Model training method, vehicle control method and device based on multi-point supervision

Abstract

The present application provides a model training method, a vehicle control method, and a device based on multi-point supervision, belonging to the field of artificial intelligence technology. The target street scene semantic features of a street scene image and the intermediate street scene semantic features of the street scene image at each intermediate supervision node are extracted through the feature extraction layer of an initial semantic segmentation model, and the corresponding intermediate street scene semantic labels are predicted based on each intermediate street scene semantic feature through the output layer of the initial semantic segmentation model. The target street scene semantic label is predicted based on the target street scene semantic features through the output layer, and the auxiliary street scene semantic label of the corresponding intermediate street scene semantic feature is predicted through the intermediate supervision node. According to each intermediate street scene semantic label, each auxiliary street scene semantic label, the image semantic label, and the target street scene semantic label, the target loss value is calculated, and the model parameters of the initial semantic segmentation model are updated according to the target loss value to obtain the target semantic segmentation model, which can improve the accuracy of street scene semantic understanding.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular, to a model training method, a vehicle control method, and a device based on multi-point supervision. Background Art

[0002] In the field of autonomous driving, in order to achieve accurate perception of the driving environment, a deep semantic segmentation model is usually used to capture hierarchical features in the driving environment to identify road objects, lanes, drivable areas, etc. However, as the network depth of the semantic segmentation model increases, problems such as gradient disappearance and insufficient optimization of intermediate layers will be faced during the model training process. These limitations reduce the potential of deep network architectures and affect the performance of model semantic segmentation. Summary of the Invention

[0003] The main purpose of the embodiments of this application is to propose a model training method, a vehicle control method, and a device based on multi-point supervision, aiming to improve the accuracy of street scene semantic understanding.

[0004] To achieve the above objective, in the first aspect of the embodiments of this application, a model training method based on multi-point supervision is proposed. The method includes:

[0005] Obtain a street scene image, the image semantic label of the street scene image, and an initial semantic segmentation model. The initial semantic segmentation model includes a feature extraction layer and an output layer, and multiple intermediate supervision nodes are provided in the feature extraction layer;

[0006] Extract the target street scene semantic features of the street scene image and the intermediate street scene semantic features of the street scene image at each intermediate supervision node through the feature extraction layer, and predict the corresponding intermediate street scene semantic labels based on each intermediate street scene semantic feature through the output layer, and predict the target street scene semantic label based on the target street scene semantic feature through the output layer;

[0007] For each intermediate supervision node, predict the auxiliary street scene semantic label of the corresponding intermediate street scene semantic feature through the intermediate supervision node;

[0008] Calculate the target loss value according to each intermediate street scene semantic label, each auxiliary street scene semantic label, the image semantic label, and the target street scene semantic label;

[0009] Update the model parameters of the initial semantic segmentation model according to the target loss value to obtain a target semantic segmentation model.

[0010] In some embodiments, calculating the target loss value according to each intermediate street scene semantic label, each auxiliary street scene semantic label, the image semantic label, and the target street scene semantic label includes:

[0011] Calculate the negative entropy loss of each intermediate supervision node according to each of the intermediate street view semantic tags;

[0012] Calculate the cross-entropy loss according to the image semantic tag and the target street view semantic tag;

[0013] Calculate the mutual information loss of each intermediate supervision node according to each of the auxiliary street view semantic tags and the image semantic tag;

[0014] Perform a weighted sum of the negative entropy loss of each intermediate supervision node, the cross-entropy loss, and the mutual information loss of each intermediate supervision node to obtain the target loss value.

[0015] In some embodiments, the performing a weighted sum of the negative entropy loss of each intermediate supervision node, the cross-entropy loss, and the mutual information loss of each intermediate supervision node to obtain the target loss value includes:

[0016] Determine the first weight and the second weight of each intermediate supervision node;

[0017] Perform a weighted sum of the negative entropy loss of each intermediate supervision node and the corresponding first weight to obtain a first loss value;

[0018] Perform a weighted sum of the mutual information loss of each intermediate supervision node and the corresponding second weight to obtain a second loss value;

[0019] Perform a loss sum of the cross-entropy loss, the first loss value, and the second loss value to obtain the target loss value.

[0020] In some embodiments, each intermediate supervision node is connected with an auxiliary dimension adapter. After updating the model parameters of the initial semantic segmentation model according to the target loss value to obtain a target semantic segmentation model, the model training method based on multi-point supervision further includes:

[0021] For each auxiliary dimension adapter, calculate the parameter gradient of the target loss value with respect to the model parameters of the auxiliary dimension adapter;

[0022] Determine the parameter update step size;

[0023] Update the model parameters of the auxiliary dimension adapter according to the parameter update step size and the parameter gradient.

[0024] In some embodiments, each intermediate supervision node is connected with an auxiliary dimension adapter. For each intermediate supervision node, predicting the auxiliary street view semantic tag of the corresponding intermediate street view semantic feature through the intermediate supervision node includes:

[0025] Obtain the street view semantic categories included in the target street view semantic label and the category quantity of the street view semantic categories;

[0026] Call a target generator according to the street view semantic categories and the category quantity, and generate a synthetic street view image through the target generator based on the intermediate street view semantic features of each intermediate supervision node and the target street view semantic features;

[0027] Extract the intermediate synthetic street view semantic features of the synthetic street view image at each intermediate supervision node through the feature extraction layer;

[0028] For each intermediate supervision node, perform feature fusion on the intermediate street view semantic features and the intermediate synthetic street view semantic features of the intermediate supervision node to obtain the fused semantic features of the intermediate supervision node;

[0029] Predict the auxiliary street view semantic labels of the corresponding fused semantic features through the auxiliary dimension adapter connected by each intermediate supervision node.

[0030] In some embodiments, the generating, through the target generator, a synthetic street view image based on the intermediate street view semantic features of each intermediate supervision node and the target street view semantic features includes:

[0031] Perform feature splicing on the intermediate street view semantic features of each intermediate supervision node and the target street view semantic features to obtain a first reference semantic feature;

[0032] Perform multi-scale feature extraction on the reference semantic feature to obtain sub-semantic features of multiple scales;

[0033] Obtain random noise, and perform feature fusion on the random noise and the sub-semantic features of multiple scales to obtain a second reference semantic feature;

[0034] Generate the synthetic street view image through the target generator based on the second reference semantic feature.

[0035] To achieve the above object, a second aspect of the embodiments of the present application proposes a vehicle control method, and the method includes:

[0036] Obtain a target street view image collected by a target vehicle;

[0037] Perform street view semantic segmentation on the target street view image through a target semantic segmentation model to obtain a predicted street view semantic label; wherein, the target semantic segmentation model is trained according to the above model training method based on multi-point supervision;

[0038] Control the driving of the target vehicle according to the predicted street view semantic label.

[0039] To achieve the above object, a third aspect of the embodiments of the present application proposes a vehicle control device, the device comprising:

[0040] An image acquisition module, configured to acquire a target street view image collected by a target vehicle;

[0041] A semantic segmentation module, configured to perform street view semantic segmentation on the target street view image through a target semantic segmentation model to obtain a predicted street view semantic label; wherein, the target semantic segmentation model is trained according to the above-mentioned model training method based on multi-point supervision;

[0042] A control module, configured to control the driving of the target vehicle according to the predicted street view semantic label.

[0043] To achieve the above object, a fourth aspect of the embodiments of the present application proposes an electronic device, the electronic device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the above-mentioned model training method based on multi-point supervision in the first aspect or the vehicle control method in the second aspect when executing the computer program.

[0044] To achieve the above object, a fifth aspect of the embodiments of the present application proposes a computer-readable storage medium, the computer-readable storage medium storing a computer program, and the computer program implementing the above-mentioned model training method based on multi-point supervision in the first aspect or the vehicle control method in the second aspect when executed by a processor.

[0045] The model training method, vehicle control method, vehicle control device, electronic device, and computer-readable storage medium based on multi-point supervision in the embodiments of the present application obtain a street view image, an image semantic label of the street view image, and an initial semantic segmentation model, and train the initial semantic segmentation model based on the street view image and the image semantic label. The initial semantic segmentation model includes a feature extraction layer and an output layer. To alleviate problems such as gradient disappearance and insufficient optimization of the intermediate layer during the training of the initial semantic segmentation model, multiple intermediate supervision nodes are set in the feature extraction layer to explicitly supervise the intermediate layer of the model through the intermediate supervision nodes, enabling the model to learn intermediate features related to the street view semantic understanding task. To capture semantic features that play a key role in street view semantic understanding in the driving environment, the target street view semantic features of the street view image and the intermediate street view semantic features of the street view image at each intermediate supervision node are extracted through the feature extraction layer. The corresponding intermediate street view semantic labels are predicted based on each intermediate street view semantic feature through the output layer, and the target street view semantic label is predicted based on the target street view semantic feature through the output layer to optimize the semantic segmentation performance of the model based on the intermediate street view semantic labels and the target street view semantic label of each intermediate supervision node. For each intermediate supervision node, the auxiliary street view semantic label of the corresponding intermediate street view semantic feature is predicted through the intermediate supervision node to assist in adjusting the parameters of the intermediate layer of the model based on the auxiliary street view semantic label, thereby improving the accuracy of intermediate feature extraction. According to each intermediate street view semantic label, each auxiliary street view semantic label, the image semantic label, and the target street view semantic label, the target loss value is calculated to guide the training process of the initial semantic segmentation model based on the target loss value. The model parameters of the initial semantic segmentation model are updated according to the target loss value to obtain the target semantic segmentation model. By setting intermediate supervision nodes to perform explicit supervision on the feature learning process of the intermediate layer of the model, the parameters of the intermediate layer are optimized, and an additional gradient path is introduced through the intermediate supervision nodes to alleviate the gradient disappearance problem, thereby improving the accuracy of semantic segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 is a flowchart of the model training method based on multi-point supervision provided by the embodiments of the present application;

[0047] Figure 2 is Figure 1 a flowchart of step S130 in

[0048] Figure 3 is Figure 2 a flowchart of step S220 in

[0049] Figure 4 is Figure 1 a flowchart of step S140 in

[0050] Figure 5 isFigure 4 The flowchart of step S440 in

[0051] Figure 6 is another flowchart of the model training method based on multi-point supervision provided by an embodiment of the present application;

[0052] Figure 7 is the flowchart of the vehicle control method provided by an embodiment of the present application;

[0053] Figure 8 is the structural schematic diagram of the vehicle control device provided by an embodiment of the present application;

[0054] Figure 9 is the hardware structural schematic diagram of the electronic device provided by an embodiment of the present application. Detailed implementation manners

[0055] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application.

[0056] It should be noted that although functional module division is performed in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different module division in the device or a different order in the flowchart. Terms such as "first" and "second" in the specification, claims and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily need to describe a specific order or sequence.

[0057] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the technical field to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application, and are not intended to limit the present application.

[0058] In the field of autonomous driving, in order to achieve accurate perception of the driving environment, a deep semantic segmentation model is usually used to capture hierarchical features in the driving environment to identify road objects, lanes, drivable areas, etc. However, as the network depth of the semantic segmentation model increases, limitations such as gradient disappearance and insufficient optimization of intermediate layers will be faced during the model training process. These limitations reduce the potential of the deep network architecture and affect the performance of model semantic segmentation.

[0059] Based on this, the embodiments of the present application provide a model training method, a vehicle control method, a vehicle control device, an electronic device and a computer-readable storage medium based on multi-point supervision, aiming to improve the accuracy of street scene semantic understanding.

[0060] The model training method, vehicle control method, vehicle control device, electronic device, and computer-readable storage medium based on multi-point supervision provided by the embodiments of the present application will be specifically described through the following embodiments. First, the model training method based on multi-point supervision in the embodiments of the present application will be described.

[0061] The model training method based on multi-point supervision provided by the embodiments of the present application relates to the field of artificial intelligence technology. The model training method based on multi-point supervision provided by the embodiments of the present application can be applied to a terminal, can also be applied to a server side, or can also be software running on a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a notebook computer, a desktop computer, etc.; the server side can be configured as an independent physical server, can also be configured as a server cluster or a distributed system composed of multiple physical servers, or can also be configured as a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application implementing the model training method based on multi-point supervision, etc., but is not limited to the above forms.

[0062] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld devices or portable devices, tablet devices, multi-processor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and so on. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in a distributed computing environment, where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0063] Figure 1 is an optional flowchart of a model training method based on multi-point supervision provided by the embodiments of the present application. Figure 1 The method in may include but is not limited to steps S110 to S150.

[0064] Step S110, obtain a street view image, an image semantic label of the street view image, and an initial semantic segmentation model. The initial semantic segmentation model includes a feature extraction layer and an output layer, and multiple intermediate supervision nodes are provided in the feature extraction layer.

[0065] Step S120: Extract the target street view semantic features of the street view image and the intermediate street view semantic features at each intermediate supervision node from the feature extraction layer, and predict the corresponding intermediate street view semantic labels based on each intermediate street view semantic feature through the output layer, and predict the target street view semantic label based on the target street view semantic feature through the output layer;

[0066] Step S130: For each intermediate supervision node, predict the auxiliary street view semantic label of the corresponding intermediate street view semantic feature through the intermediate supervision node;

[0067] Step S140: Calculate the target loss value according to each intermediate street view semantic label, each auxiliary street view semantic label, the image semantic label, and the target street view semantic label;

[0068] Step S150: Update the model parameters of the initial semantic segmentation model according to the target loss value to obtain the target semantic segmentation model.

[0069] In step S110 of some embodiments, collect the street view image of the surrounding environment by the shooting device carried on the vehicle, and perform semantic annotation on the street view image to obtain the image semantic label. The image semantic label is used to identify and classify the semantic categories of each element in the street view image, such as lanes, buildings, pedestrians, drivable areas, and traffic signs, etc. The image semantic label is a pixel-level semantic label, and each pixel point in the street view image is marked as a specific semantic category. Obtain the initial semantic segmentation model, and the initial semantic segmentation model is a semantic segmentation model to be trained. Each pixel in the street view image can be assigned to a predefined semantic category through the initial semantic segmentation model. The initial semantic segmentation model can be designed according to the actual situation. The initial semantic segmentation model can be a fully convolutional neural network, a U-shaped network, etc. The initial semantic segmentation model includes a feature extraction layer and an output layer. The feature extraction layer is used to extract the image feature representation useful for semantic segmentation from the street view image, and the output layer is used to predict the semantic category of the street view image according to the image feature representation extracted by the feature extraction layer.

[0070] As the network depth increases, problems such as gradient disappearance and insufficient intermediate feature optimization will occur in the deep learning network. To improve the semantic segmentation performance of the deep learning network, multiple intermediate supervision nodes are set in the feature extraction layer. Each intermediate supervision node is connected with an auxiliary dimension adapter. The auxiliary dimension adapter is added to the network layer where the intermediate supervision node is located in the form of a pluggable module, so as to explicitly supervise the intermediate features output by the intermediate layer based on the intermediate supervision node and perform feature correction on the intermediate features, so that the deep layer of the feature extraction layer can accurately encode semantic-rich and discriminative feature representations to achieve street view semantic segmentation in different driving environments such as occlusion and weather changes.

[0071] Determine the number of nodes of the intermediate supervision nodes, where the number of nodes is less than the number of network layers in the feature extraction layer. The intermediate supervision nodes can be randomly set in the network layers of the feature extraction layer, and only one intermediate supervision node can be set in one network layer. The feature extraction layer includes multiple network layers, and there are continuous network layers between every two adjacent intermediate supervision nodes.

[0072] In step S120 of some embodiments, the street view image is subjected to image feature extraction through the feature extraction layer, so as to obtain the target street view semantic features output by the feature extraction layer in the last network layer, and the intermediate street view semantic features output by each intermediate supervision node. Each intermediate street view semantic feature is respectively input into the output layer for street view semantic segmentation, so as to obtain the intermediate street view semantic labels corresponding to each intermediate street view semantic feature. The intermediate street view semantic label is the semantic category predicted by the output layer according to the intermediate street view semantic feature. The target street view semantic feature is input into the output layer for street view semantic segmentation, so as to obtain the target street view semantic label. The target street view semantic label is the semantic category predicted by the output layer according to the target street view semantic feature.

[0073] Please refer to Figure 2 , in some embodiments, step S130 may include but is not limited to steps S210 to S250:

[0074] Step S210, obtain the street view semantic categories included in the target street view semantic label and the number of categories of the street view semantic categories;

[0075] Step S220, call the target generator according to the street view semantic categories and the number of categories, and generate a synthetic street view image through the target generator based on the intermediate street view semantic features and the target street view semantic features of each intermediate supervision node;

[0076] Step S230, extract the intermediate synthetic street view semantic features of the synthetic street view image at each intermediate supervision node through the feature extraction layer;

[0077] Step S240, for each intermediate supervision node, fuse the intermediate street view semantic feature and the intermediate synthetic street view semantic feature of the intermediate supervision node to obtain the fused semantic feature of the intermediate supervision node;

[0078] Step S250, predict the auxiliary street view semantic label of the corresponding fused semantic feature through the auxiliary dimension adapter connected by each intermediate supervision node.

[0079] In step S210 of some embodiments, obtain the street view semantic categories included in the target street view semantic label and the number of categories of the street view semantic categories. The street view semantic categories are used to indicate the semantic categories of elements in the street view image, such as lanes, traffic signs, etc. The number of categories is the number of street view semantic categories included in the target street view semantic label. For example, if the target street view semantic label includes lanes, traffic signs, and buildings, the number of categories is 3.

[0080] In step S220 of some embodiments, in order to improve the model's semantic understanding ability of complex street view environments, call the target generator to expand the sample size according to the street view semantic categories and the number of categories. If the street view semantic categories included in the target street view semantic label are preset categories, call the target generator. The preset categories can be set according to the actual situation. For example, in order to improve the accuracy of street view semantic understanding under bad weather conditions, the preset categories can be bad weather categories such as rain, fog, wind, etc. In order to improve the accuracy of street view semantic understanding under complex road conditions, the preset categories can be complex road conditions categories such as mountain roads, rural roads, etc. If the target street view semantic label does not include preset categories, then judge the number of categories of the street view semantic categories. A quantity threshold is preset. If the number of categories is greater than or equal to the quantity threshold, it means that the street view indicated by the street view image is a complex scene, such as a scene with heavy traffic flow, a scene with a complex road environment, etc., then call the target generator. If the number of categories is less than the quantity threshold, for each intermediate supervision node, input the intermediate street view semantic features output by the intermediate supervision node into the auxiliary dimensional adapter connected to the intermediate supervision node for street view semantic segmentation to obtain the auxiliary street view semantic label corresponding to the intermediate street view semantic features. The auxiliary street view semantic label is the semantic category predicted by the auxiliary dimensional adapter according to the intermediate street view semantic features.

[0081] The target generator is a pre-trained generator and can be obtained through generative adversarial learning. The target generator generates an image according to the intermediate street view semantic features and the target street view semantic features of each intermediate supervision node to obtain a synthetic street view image. The intermediate street view semantic features contain the details and local structures of the street view image, and the target street view semantic features contain the global street view semantics of the street view image. The synthetic street view image generated by combining the two can maintain consistency with the street view image in terms of details and overall structure, improving the quality of image generation.

[0082] In step S230 of some embodiments, perform image feature extraction on the synthetic street view image through the feature extraction layer to obtain the output features of the synthetic street view image at each intermediate supervision node, and use this output feature as the intermediate synthetic street view semantic feature.

[0083] In step S240 of some embodiments, for each intermediate supervision node, feature fusion is performed on the intermediate street view semantic features output by the street view image at the intermediate supervision node and the intermediate synthetic street view semantic features output by the synthetic street view image at the same intermediate supervision node, so as to accurately represent local details of the street view such as traffic signs and driving conditions at the same intermediate supervision node, obtain the fused semantic features of the intermediate supervision node, and thus improve the street view semantic segmentation performance of the auxiliary dimension adapter based on the fused semantic features.

[0084] In step S250 of some embodiments, for each intermediate supervision node, the fused semantic features output by the intermediate supervision node are input into the auxiliary dimension adapter connected to the intermediate supervision node for street view semantic segmentation, and auxiliary street view semantic labels are obtained. By using the auxiliary dimension adapter to assist in street view semantic segmentation, the number of gradient propagation paths is increased, and the problem of gradient disappearance is alleviated to a certain extent. Moreover, the auxiliary street view semantic labels output by the auxiliary dimension adapter can also assist in updating the model parameters of the initial semantic segmentation model, enabling the intermediate layer to output more accurate intermediate features, compensating for the defect of insufficient optimization of intermediate features by the deep neural network, and greatly improving the street view semantic segmentation performance of the initial semantic segmentation model.

[0085] Through the above steps S210 to S250, more accurate auxiliary street view semantic labels can be obtained, so as to prompt the model to perform more accurate feature learning on the street view image based on the auxiliary street view semantic labels.

[0086] Please refer to Figure 3 , in some embodiments, step S220 may include but is not limited to steps S310 to S340:

[0087] Step S310: Feature concatenation is performed on the intermediate street view semantic features and the target street view semantic features of each intermediate supervision node to obtain the first reference semantic features;

[0088] Step S320: Multi-scale feature extraction is performed on the reference semantic features to obtain sub-semantic features of multiple scales;

[0089] Step S330: Random noise is obtained, and feature fusion is performed on the random noise and the sub-semantic features of multiple scales to obtain the second reference semantic features;

[0090] Step S340: A synthetic street view image is generated by the target generator based on the second reference semantic features.

[0091] In step S310 of some embodiments, to enable the target generator to generate images consistent with the style of street view images, a comprehensive feature representation of street view images needs to be provided to the target generator. Therefore, according to the node positions of the intermediate supervision nodes in the feature extraction layer, the intermediate street view semantic features output by each intermediate supervision node are sequentially concatenated, and the concatenated street view semantic features are concatenated with the target street view semantic features again to obtain the first reference semantic features.

[0092] In step S320 of some embodiments, the first reference semantic features include features of different scales such as building roads, roads, pedestrians, vehicles, etc. To enable the target generator to capture these semantic elements of different scales, multi-scale feature extraction is performed on the reference semantic features through a feature pyramid network to obtain sub-semantic features of multiple scales, where the scale refers to the resolution of the sub-semantic features. The feature pyramid network includes a bottom-up path and a top-down path. The bottom-up path includes N first network layers connected in series, and the top-down path includes N second network layers connected in series, where N is an integer greater than 1. The first reference semantic features are downsampled by a factor of two through the bottom-up path to obtain the first feature maps output by each first network layer.

[0093] If the first feature map output by the i-th first network layer is denoted as , then the resolution of the first feature map is of the first reference semantic features. The first feature map output by the last first network layer is upsampled by a factor of two through the first second network layer of the top-down path to obtain the second feature map output by the first second network layer. After the second feature map is upsampled by a factor of two through the second second network layer, it is fused with the first feature map output by the (N - 1)-th first network layer, that is, feature addition is performed, to obtain the second feature map output by the second network layer. After the second feature map output by the previous network layer is upsampled by a factor of two through the i-th second network layer, it is fused with the first feature map output by the (N - i + 1)-th first network layer to obtain the second feature map output by the i-th second network layer, where i is an integer greater than or equal to 2 and less than or equal to N. Repeat the above steps until multiple second feature maps are obtained, and these multiple second feature maps are used as sub-semantic features of multiple scales.

[0094] In step S330 of some embodiments, random noise is obtained to generate diverse street view images. The random noise may follow a normal distribution, a uniform distribution, or the like. To enable the target generator to accurately identify semantic elements at different scales, feature fusion is performed on the random noise and the sub-semantic features at each scale to obtain a second reference semantic feature.

[0095] In step S340 of some embodiments, the target generator may adopt different network architectures, such as a fully connected network, a convolutional network, etc. The second reference semantic feature is input to the target generator so that the target generator can capture the street scene semantics implicit in the second reference semantic feature and generate a synthetic street scene image.

[0096] Through the above steps S310 to S340, a synthetic street view image can be obtained, so as to expand the sample size based on the synthetic street view image and achieve accurate expression of the intermediate street view semantic features output by the intermediate supervisory node.

[0097] See also Figure 4 In some embodiments, step S140 may include but is not limited to steps S410 to S440:

[0098] Step S410, calculating the negative entropy loss of each intermediate supervisory node according to each intermediate street scene semantic label;

[0099] Step S420, calculating a cross entropy loss based on the image semantic label and the target street scene semantic label;

[0100] Step S430, calculating the mutual information loss of each intermediate supervisory node based on each auxiliary street view semantic label and the image semantic label;

[0101] In step S440 , a weighted sum is performed on the negative entropy loss, the cross entropy loss, and the mutual information loss of each intermediate supervisory node to obtain a target loss value.

[0102] In step S410 of some embodiments, in order to prevent overfitting caused by multi-point supervision, negative entropy is used as a regularization term. By maximizing the negative entropy, the intermediate features output by the intermediate supervisory node are penalized to reduce the uncertainty of the model's prediction results, thereby improving the model's generalization ability for unknown driving environments. The intermediate street view semantic label is used to indicate the probability that each pixel point in the intermediate street view semantic feature belongs to a specific category. Negative entropy is calculated based on the intermediate street view semantic label corresponding to the mth intermediate supervisory node to obtain the negative entropy loss of the mth intermediate supervisory node. The calculation formula of the negative entropy loss is expressed as:

[0103] ,

[0104] in, For the The negative entropy loss of one intermediate supervision node; D is the sample data set; |D| is the number of street view images in the sample data set; i represents the i-th street view image in the sample data set; is the pixel set composed of pixel points; j represents the j-th pixel point; k represents the k-th semantic category; K is the number of classification categories; For the j-th pixel point of the intermediate street view semantic feature of the i-th street view image output by one intermediate supervision node; is The probability of belonging to the k-th semantic category; is the model parameter of the initial semantic segmentation model.

[0105] In step S420 of some embodiments, cross-entropy calculation is performed according to the image semantic label and the target street view semantic label, and the cross-entropy loss is calculated. The cross-entropy loss is used to measure the difference between the image semantic label and the target street view semantic label. The calculation formula of the cross-entropy loss is expressed as:

[0106] ,

[0107] where is the cross-entropy loss; D is the sample data set; |D| is the number of street view images in the sample data set; i represents the i-th street view image in the sample data set; is the pixel set composed of pixel points; j represents the j-th pixel point; K is the number of classification categories; represents the true probability that the j-th pixel point in the i-th street view image indicated by the image semantic label belongs to the k-th semantic category; represents the j-th pixel point in the i-th street view image; is The predicted probability of belonging to the k-th semantic category; is the model parameter of the initial semantic segmentation model.

[0108] In step S430 of some embodiments, in order to measure the distance between the intermediate feature output by the intermediate supervision node and the true value, mutual information calculation is performed according to each auxiliary street view semantic label and the image semantic label, and the mutual information loss of each intermediate supervision node is obtained. By maximizing the mutual information loss, the hidden feature dimension and the ground truth dimension can be aligned, so that the auxiliary street view semantic label gradually approximates the image semantic label, thereby performing explicit supervision on the intermediate feature. The calculation formula of the mutual information loss is expressed as:

[0109] ,

[0110] where is the The mutual information loss of each intermediate supervision node; D is the sample data set; |D| is the number of street view images in the sample data set; i represents the i-th street view image in the sample data set; is a pixel set composed of pixel points; j represents the j-th pixel point; K is the number of classification categories; represents the true probability that the j-th pixel point in the i-th street view image indicated by the image semantic label belongs to the k-th semantic category; is the j-th pixel point of the intermediate street view semantic feature of the i-th street view image output by the is the auxiliary street view semantic label output by the auxiliary dimension adapter connected to the intermediate supervision node, and the auxiliary street view semantic label indicates the prediction probability of belonging to the k-th semantic category; are the parameters of the auxiliary dimension adapter connected to the

[0111] In step S440 of some embodiments, the negative entropy loss, cross-entropy loss of each intermediate supervision node, and the mutual information loss of each intermediate supervision node are weighted and summed to obtain the target loss value, so as to guide the model optimization process based on the target loss value, thereby improving the street view semantic segmentation performance of the model.

[0112] Through the above steps S410 to S440, the target loss value can be obtained to explicitly supervise the intermediate features based on the target loss value, so that the intermediate layer can learn features related to the street view semantic segmentation task, and further enable the deep network layer to encode semantic-rich and discriminative feature representations to cope with diverse driving conditions.

[0113] Please refer to Figure 5 , in some embodiments, step S440 may include but is not limited to steps S510 to S540:

[0114] Step S510, determining the first weight and the second weight of each intermediate supervision node;

[0115] Step S520, performing a weighted sum of the negative entropy loss of each intermediate supervision node and the corresponding first weight to obtain the first loss value;

[0116] Step S530, performing a weighted sum of the mutual information loss of each intermediate supervision node and the corresponding second weight to obtain the second loss value;

[0117] Step S540, performing a loss sum of the cross-entropy loss, the first loss value, and the second loss value to obtain the target loss value.

[0118] In step S510 of some embodiments, the first weight and the second weight of each intermediate supervision node are obtained. The first weight is the weight for the negative entropy loss, and the second weight is the weight for the mutual information loss.

[0119] In step S520 of some embodiments, for each intermediate supervision node, the negative entropy loss of the intermediate supervision node is multiplied by the first weight of the intermediate supervision node to obtain a first sub-loss. The first sub-losses of each intermediate supervision node are summed to obtain a first loss value.

[0120] In step S530 of some embodiments, for each intermediate supervision node, the mutual information loss of the intermediate supervision node is multiplied by the second weight of the intermediate supervision node to obtain a second sub-loss. The second sub-losses of each intermediate supervision node are summed to obtain a second loss value.

[0121] In step S540 of some embodiments, the cross-entropy loss, the first loss value, and the second loss value are added together to obtain a target loss value. The calculation formula of the target loss value is expressed as:

[0122] ,

[0123] where is the target loss value; is the cross-entropy loss; is the number of intermediate supervision nodes; represents the th intermediate supervision node; and are the first weight and the second weight of the th intermediate supervision node respectively; is the negative entropy loss of the th intermediate supervision node; is the th intermediate supervision node's mutual information loss.

[0124] Through the above steps S510 to S540, the target loss value can be obtained to prompt the model to learn the features related to the street view semantic understanding task based on the target loss value, thereby improving the accuracy of street view semantic understanding.

[0125] Please refer to Figure 6 , in some embodiments, after step S140, the model training method based on multi-point supervision may further include but is not limited to steps S610 to S630:

[0126] Step S610, for each auxiliary dimension adapter, calculate the parameter gradient of the target loss value with respect to the model parameters of the auxiliary dimension adapter;

[0127] Step S620: Determine the parameter update step size;

[0128] Step S630: Update the model parameters of the auxiliary dimension adapter according to the parameter update step size and the parameter gradient.

[0129] In step S610 of some embodiments, for each auxiliary dimension adapter, calculate the first-order partial derivative of the target loss value with respect to the model parameters of the auxiliary dimension adapter to obtain the parameter gradient.

[0130] In step S620 of some embodiments, obtain the parameter update step size. The parameter update step size is the learning rate, which is used to indicate the step size of parameter update in each iteration of the model. The parameter update step size can directly affect the convergence speed of the model. To better balance the training speed and stability, it is necessary to dynamically adjust the parameter update step size. The parameter update step size can be adjusted based on the loss value. Specifically, if the difference between the target loss values of consecutive multiple training rounds is less than the preset difference threshold, then multiply the decay rate by the parameter update step size, and use the result of the multiplication as the updated parameter update step size. The decay rate is a number greater than 0 and less than 1. The parameter update step size can also be adjusted based on the training round (iteration number). The parameter update step size at the t-th training round can be expressed as:

[0131] ,

[0132] where, is the parameter update step size at the t-th training round; r is the decay rate, and the value range of the decay rate is within the interval (0, 1); t is the iteration number; e is the base of the exponential operation; represents the multiplication operation.

[0133] In step S630 of some embodiments, multiply the parameter update step size and the parameter gradient, and subtract the result of the multiplication from the current model parameters of the auxiliary dimension adapter to update the current model parameters. The update process of the model parameters of the auxiliary dimension adapter is expressed as:

[0134] ,

[0135] where, are the model parameters of the m-th auxiliary dimension adapter; is the parameter update step size; is the target loss value for the model parameters the parameter gradient.

[0136] In the above steps S610 to S630, each auxiliary dimension adapter connected to the intermediate supervision node is optimized through the target loss value, so that the auxiliary dimension adapter can cooperate with the intermediate supervision node to perform more accurate supervision on the intermediate features.

[0137] In step S150 of some embodiments, the initial semantic segmentation model and the auxiliary dimension adapter can be synchronously iteratively updated according to the same target loss value and the same parameter update step size until the preset iteration number threshold is reached. Specifically, calculate the gradient of the target loss value with respect to the model parameters of the initial semantic segmentation model, and iteratively update the model parameters of the initial semantic segmentation model according to the parameter update step size and the gradient until the preset iteration number threshold is reached, and obtain the target semantic segmentation model. The update process of the initial semantic segmentation model is expressed as:

[0138] ,

[0139] where,[[]]END]] is the model parameter of the initial semantic segmentation model; is the parameter update step size; is the target loss value with respect to the model parameter gradient.

[0140] The street scene semantic segmentation model based on the deep learning network tends to be deepened, and the traditional training paradigm that relies on single-point supervision at the network output end reduces the street scene semantic understanding performance of the model. To alleviate this problem, the embodiments of the present application improve the performance of the deep learning network by introducing intermediate multi-channel supervision (mutual information) and normalization (negative entropy). Conducting theoretical convergence analysis on the training method of the embodiments of the present application shows that the training method has a convergence rate matching that of the standard stochastic gradient descent optimization method, and multi-point supervision does not harm the asymptotic convergence of the model. If the number of iterations of the model is T, the convergence rate is expressed as . Moreover, testing the training method of the embodiments of the present application on multiple datasets, the performance of this training method is better than the traditional training method, and the average intersection over union is increased by up to 9.19%.

[0141] Figure 7 is an optional flowchart of a vehicle control method provided by the embodiments of the present application, Figure 7 The method in

[0142] Step S710, obtaining a target street scene image collected by a target vehicle;

[0143] Step S720, performing street scene semantic segmentation on the target street scene image through the target semantic segmentation model to obtain a predicted street scene semantic label; wherein, the target semantic segmentation model is trained according to the above-mentioned model training method based on multi-point supervision;

[0144] Step S730, controlling the driving of the target vehicle according to the predicted street scene semantic label.

[0145] In step S710 of some embodiments, the vehicle terminal is a control system installed on the target vehicle. The vehicle terminal controls a photographing device mounted on the target vehicle to collect images of the surrounding environment of the vehicle, and obtains a target street view image.

[0146] In step S720 of some embodiments, the vehicle terminal obtains a target semantic segmentation model. The target semantic segmentation model includes a feature extraction layer and an output layer. The feature extraction layer extracts the street view semantic features of the target street view image, and the output layer outputs predicted street view semantic labels based on the street view semantic features. The predicted street view semantic labels are pixel-level semantic labels, which are used to indicate the semantic categories of each pixel point in the target street view image.

[0147] In step S730 of some embodiments, the vehicle terminal outputs a vehicle control instruction for the target vehicle according to the predicted street view semantic labels, and controls the target vehicle to travel according to the vehicle control instruction. The vehicle control instruction may be a vehicle steering instruction, a vehicle acceleration instruction, etc.

[0148] Through the above steps S710 to S730, the target semantic segmentation model is trained based on a multi-point supervision method, so that the intermediate layer of the target semantic segmentation model can learn semantic features associated with the street view semantic segmentation task, alleviating the problems of gradient disappearance and insufficient intermediate feature optimization caused by the deep learning network, and improving the accuracy of street view semantic understanding.

[0149] The embodiment of the present application further provides a model training device based on multi-point supervision, which can implement the above-mentioned model training method based on multi-point supervision. The model training device based on multi-point supervision includes:

[0150] An acquisition module, configured to acquire a street view image, an image semantic label of the street view image, and an initial semantic segmentation model. The initial semantic segmentation model includes a feature extraction layer and an output layer, and multiple intermediate supervision nodes are arranged in the feature extraction layer;

[0151] A first street view semantic prediction module, configured to extract the target street view semantic features of the street view image and the intermediate street view semantic features of the street view image at each intermediate supervision node through the feature extraction layer, and predict the corresponding intermediate street view semantic labels based on each intermediate street view semantic feature through the output layer, and predict the target street view semantic labels based on the target street view semantic features through the output layer;

[0152] A second street view semantic prediction module, configured to, for each intermediate supervision node, predict an auxiliary street view semantic label of the corresponding intermediate street view semantic feature through the intermediate supervision node;

[0153] A calculation module, configured to calculate a target loss value according to each intermediate street view semantic label, each auxiliary street view semantic label, the image semantic label, and the target street view semantic label;

[0154] An update module, configured to update model parameters of an initial semantic segmentation model according to a target loss value to obtain a target semantic segmentation model.

[0155] In some embodiments, the calculation module is further configured to:

[0156] Calculate the negative entropy loss of each intermediate supervision node according to each intermediate street view semantic label; calculate the cross entropy loss according to the image semantic label and the target street view semantic label; calculate the mutual information loss of each intermediate supervision node according to each auxiliary street view semantic label and the image semantic label; perform weighted summation on the negative entropy loss of each intermediate supervision node, the cross entropy loss, and the mutual information loss of each intermediate supervision node to obtain a target loss value.

[0157] In some embodiments, the calculation module is further configured to:

[0158] Determine the first weight and the second weight of each intermediate supervision node; perform weighted summation on the negative entropy loss of each intermediate supervision node and the corresponding first weight to obtain a first loss value; perform weighted summation on the mutual information loss of each intermediate supervision node and the corresponding second weight to obtain a second loss value; perform loss summation on the cross entropy loss, the first loss value, and the second loss value to obtain a target loss value.

[0159] In some embodiments, the update module is further configured to:

[0160] For each auxiliary dimension adapter, calculate the parameter gradient of the target loss value with respect to the model parameters of the auxiliary dimension adapter; determine a parameter update step; update the model parameters of the auxiliary dimension adapter according to the parameter update step and the parameter gradient.

[0161] In some embodiments, the second street view semantic prediction module is further configured to:

[0162] Obtain the street view semantic categories included in the target street view semantic label and the number of categories of the street view semantic categories; call a target generator according to the street view semantic categories and the number of categories, and generate a synthetic street view image through the target generator based on the intermediate street view semantic features and the target street view semantic features of each intermediate supervision node; extract the intermediate synthetic street view semantic features of the synthetic street view image at each intermediate supervision node through a feature extraction layer; for each intermediate supervision node, perform feature fusion on the intermediate street view semantic feature and the intermediate synthetic street view semantic feature of the intermediate supervision node to obtain the fused semantic feature of the intermediate supervision node; predict the auxiliary street view semantic label of the corresponding fused semantic feature through the auxiliary dimension adapter connected to each intermediate supervision node.

[0163] In some embodiments, the second street view semantic prediction module is further configured to:

[0164] Feature splicing is performed on the intermediate street view semantic features and the target street view semantic features of each intermediate supervision node to obtain the first reference semantic feature; multi-scale feature extraction is performed on the reference semantic feature to obtain sub-semantic features of multiple scales; random noise is obtained, and feature fusion is performed on the random noise and the sub-semantic features of multiple scales to obtain the second reference semantic feature; a synthetic street view image is generated by a target generator based on the second reference semantic feature.

[0165] Please refer to Figure 8 , the embodiment of the present application further provides a vehicle control device, which can implement the above vehicle control method. The vehicle control device includes:

[0166] An image acquisition module 810, configured to acquire a target street view image collected by a target vehicle;

[0167] A semantic segmentation module 820, configured to perform street view semantic segmentation on the target street view image through a target semantic segmentation model to obtain a predicted street view semantic label; wherein, the target semantic segmentation model is trained according to the above model training method based on multi-point supervision;

[0168] A control module 830, configured to control the driving of the target vehicle according to the predicted street view semantic label.

[0169] The specific implementation manner of this vehicle control device is basically the same as the specific embodiment of the above vehicle control method, and will not be elaborated here.

[0170] The embodiment of the present application further provides an electronic device. The electronic device includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements the above model training method based on multi-point supervision or the vehicle control method. This electronic device can be any intelligent terminal including a tablet computer, an in-vehicle computer, etc.

[0171] Please refer to Figure 9 , Figure 9 illustrates the hardware structure of an electronic device in another embodiment. The electronic device includes:

[0172] A processor 910, which can be implemented by using a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, etc., and is configured to execute relevant programs to implement the technical solutions provided by the embodiments of the present application;

[0173] The memory 920 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM), etc. The memory 920 can store an operating system and other application programs. When implementing the technical solutions provided in the embodiments of this specification through software or firmware, the relevant program codes are stored in the memory 920, and the processor 910 is used to call and execute the model training method or vehicle control method based on multi-point supervision in the embodiments of this application;

[0174] The input / output interface 930 is used to implement information input and output;

[0175] The communication interface 940 is used to implement communication interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.);

[0176] The bus 950 transmits information between various components of the device (such as the processor 910, the memory 920, the input / output interface 930, and the communication interface 940);

[0177] Among them, the processor 910, the memory 920, the input / output interface 930, and the communication interface 940 are communicatively connected to each other inside the device through the bus 950.

[0178] The embodiments of this application also provide a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it implements the above-mentioned model training method or vehicle control method based on multi-point supervision.

[0179] As a non-transitory computer-readable storage medium, the memory can be used to store non-transitory software programs and non-transitory computer-executable programs. In addition, the memory can include high-speed random access memory, and can also include non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state storage devices. In some embodiments, the memory can optionally include a memory remotely set relative to the processor, and these remote memories can be connected to the processor through a network. Examples of the above networks include, but are not limited to, the Internet, an enterprise intranet, a local area network, a mobile communication network, and combinations thereof.

[0180] The embodiments described in the embodiments of this application are to more clearly illustrate the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided in the embodiments of this application. Those skilled in the art know that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.

[0181] Those skilled in the art can understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than those shown in the figures, or combine certain steps, or different steps.

[0182] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, that is, they may be located in one place, or may be distributed to multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0183] Those of ordinary skill in the art can understand that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, and their appropriate combinations.

[0184] In the description of this application and the above-mentioned drawings, the terms "first", "second", "third", "fourth", etc. (if any) are used to distinguish similar objects, and do not necessarily have to be used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present application described here can be implemented in an order other than those illustrated or described here. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products, or devices.

[0185] It should be understood that in this application, "at least one (item)" means one or more, and "multiple" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that there can be three relationships. For example, "A and / or B" can mean: only A exists, only B exists, and both A and B exist at the same time. Among them, A and B can be singular or plural. The character " / " generally means that the associated objects before and after are in an "or" relationship. "At least one (one) of the following" or its similar expression refers to any combination of these items, including any combination of single item (one) or plural items (ones). For example, at least one (one) of a, b, or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0186] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the above division of units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. The displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces. The indirect couplings or communication connections of devices or units can be in electrical, mechanical or other forms.

[0187] The units described above as separate components may or may not be physically separated. The components shown as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0188] In addition, in each embodiment of the present application, the functional units can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above integrated units can be implemented in the form of hardware or in the form of software functional units.

[0189] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods in each embodiment of the present application. The foregoing storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROM), random access memories (RAM), magnetic disks or optical discs that can store programs.

[0190] The preferred embodiments of the embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the rights of the embodiments of the present application. Any modifications, equivalent replacements, and improvements made by those skilled in the art without departing from the scope and essence of the embodiments of the present application shall be within the scope of the rights of the embodiments of the present application.

Claims

1. A model training method based on multi-point supervision, characterized in that: The method comprises: Acquire a street view image, an image semantic label of the street view image, and an initial semantic segmentation model, wherein the initial semantic segmentation model includes a feature extraction layer and an output layer, and the feature extraction layer is provided with a plurality of intermediate supervisory nodes; Extracting target street view semantic features of the street view image and intermediate street view semantic features of the street view image at each intermediate supervisory node through the feature extraction layer, predicting corresponding intermediate street view semantic labels based on each intermediate street view semantic feature through the output layer, and predicting target street view semantic labels based on the target street view semantic features through the output layer; For each intermediate supervisory node, predicting an auxiliary street view semantic label of the corresponding intermediate street view semantic feature through the intermediate supervisory node; Calculating a target loss value according to each of the intermediate street view semantic labels, each of the auxiliary street view semantic labels, the image semantic label, and the target street view semantic label; Updating the model parameters of the initial semantic segmentation model according to the target loss value to obtain a target semantic segmentation model; The calculating a target loss value according to each of the intermediate street view semantic labels, each of the auxiliary street view semantic labels, the image semantic label, and the target street view semantic label includes: Calculating the negative entropy loss of each intermediate supervisory node according to each of the intermediate street view semantic labels; Calculating a cross entropy loss based on the image semantic label and the target street scene semantic label; Calculating the mutual information loss of each intermediate supervisory node according to each of the auxiliary street view semantic labels and the image semantic label; Perform weighted summation of the negative entropy loss, cross entropy loss, and mutual information loss of each intermediate supervisory node to obtain the target loss value; Each intermediate supervisory node is connected to an auxiliary dimension adapter. For each intermediate supervisory node, predicting the auxiliary street view semantic label of the corresponding intermediate street view semantic feature through the intermediate supervisory node includes: Obtaining the street view semantic category included in the target street view semantic tag and the number of the street view semantic categories; Invoking a target generator according to the street view semantic category and the number of categories, and generating a synthetic street view image by the target generator based on the intermediate street view semantic features of each intermediate supervisory node and the target street view semantic features; Extracting intermediate synthetic street view semantic features of the synthetic street view image at each intermediate supervisory node through the feature extraction layer; For each intermediate supervisory node, performing feature fusion on the intermediate street view semantic feature and the intermediate synthetic street view semantic feature of the intermediate supervisory node to obtain a fused semantic feature of the intermediate supervisory node; The auxiliary dimension adapter connected by each intermediate supervisory node predicts the auxiliary street view semantic label of the corresponding fused semantic feature.

2. The method according to claim 1, characterized in that The weighted summation of the negative entropy loss, the cross entropy loss, and the mutual information loss of each intermediate supervisory node to obtain the target loss value includes: Determining a first weight and a second weight of each intermediate supervisory node; Performing a weighted summation on the negative entropy loss of each intermediate supervisory node and the corresponding first weight to obtain a first loss value; Perform a weighted summation of the mutual information loss of each intermediate supervisory node and the corresponding second weight to obtain a second loss value; The cross entropy loss, the first loss value, and the second loss value are summed to obtain the target loss value.

3. The method according to claim 1, characterized in that Each intermediate supervision node is connected to an auxiliary dimension adapter. After updating the model parameters of the initial semantic segmentation model according to the target loss value to obtain the target semantic segmentation model, the model training method based on multi-point supervision further includes: For each auxiliary dimension adapter, calculating the parameter gradient of the target loss value with respect to the model parameters of the auxiliary dimension adapter; Determine the parameter update step size; The model parameters of the auxiliary dimension adapter are updated according to the parameter update step size and the parameter gradient.

4. The method according to claim 1, wherein The method of generating a synthetic street view image by the target generator based on the intermediate street view semantic features of each intermediate supervisory node and the target street view semantic features includes: Performing feature splicing on the intermediate street view semantic feature of each intermediate supervisory node and the target street view semantic feature to obtain a first reference semantic feature; Performing multi-scale feature extraction on the reference semantic features to obtain sub-semantic features of multiple scales; Obtaining random noise, and performing feature fusion on the random noise and sub-semantic features at multiple scales to obtain a second reference semantic feature; The synthetic street view image is generated by the target generator based on the second reference semantic feature.

5. A vehicle control method, characterized in that: The method comprises: Acquire a target street view image collected by a target vehicle; Performing street view semantic segmentation on the target street view image using a target semantic segmentation model to obtain a predicted street view semantic label; wherein the target semantic segmentation model is trained according to the multi-point supervision-based model training method according to any one of claims 1 to 4; The target vehicle is controlled to travel according to the predicted street view semantic label.

6. A vehicle control device, characterized in that: The device comprises: An image acquisition module, used to acquire a target street view image collected by a target vehicle; a semantic segmentation module, configured to perform street view semantic segmentation on the target street view image using a target semantic segmentation model to obtain predicted street view semantic labels; wherein the target semantic segmentation model is trained using the multi-point supervision-based model training method according to any one of claims 1 to 4; A control module is used to control the target vehicle to travel according to the predicted street view semantic label.

7. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, it implements the model training method based on multi-point supervision as described in any one of claims 1 to 4 or the vehicle control method as described in claim 5.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the model training method based on multi-point supervision according to any one of claims 1 to 4 or the vehicle control method according to claim 5 is implemented.

Images