Model training method, vehicle control method, device, equipment and medium

Abstract

The embodiment of the application provides a model training method, a vehicle control method, device, equipment and medium, and belongs to the technical field of automatic driving. The method comprises the following steps: acquiring a sample street view image, the sample street view image having a semantic segmentation label; extracting a sample latent feature of the sample street view image through a visual large model; respectively extracting a sample intermediate feature of the sample latent feature through each original expert model, and outputting a sample semantic segmentation image of the sample street view image according to the corresponding sample intermediate feature; aggregating the sample semantic segmentation image output by each original expert model to obtain a target semantic segmentation image; calculating a target loss according to the semantic segmentation label, the sample latent feature, the target semantic segmentation image and the original expert model; and updating the model parameters of each original expert model according to the target loss to obtain a target expert model. The embodiment of the application can improve the accuracy of semantic segmentation.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of autonomous driving, and in particular to a model training method, a vehicle control method, device, equipment and medium. BACKGROUND

[0002] Autonomous driving (AD) requires models to be able to process and respond to extremely diverse real-world scenarios in real time. These scenarios exhibit extremely high complexity and diversity, including but not limited to unpredictable dense urban traffic, variable lighting conditions on highways, irregular terrain in rural roads, and sensor input blurring in adverse weather such as heavy rain, fog or snow. In related technologies, a semantic segmentation model is used to perform semantic segmentation on a scene image so that a vehicle can understand its surroundings. However, the semantic segmentation model often overfits to the dominant patterns in the training data, making it difficult to generalize to edge or unknown scenarios, which results in low accuracy of semantic segmentation when the model faces diverse and complex environments. SUMMARY

[0003] The main purpose of the embodiments of the present application is to propose a model training method, a vehicle control method, device, equipment and medium, which aims to improve the accuracy of semantic segmentation.

[0004] To achieve the above-mentioned purpose, the first aspect of the embodiments of the present application proposes a model training method, which comprises:

[0005] Obtaining a sample street view image; wherein the sample street view image has a semantic segmentation label;

[0006] Extracting a sample latent feature of the sample street view image by a pre-trained visual large model;

[0007] Extracting a sample intermediate feature of the sample latent feature by each original expert model, and outputting a sample semantic segmentation map of the sample street view image according to the corresponding sample intermediate feature;

[0008] Aggregating the sample semantic segmentation map output by each original expert model to obtain a target semantic segmentation map;

[0009] According to the semantic segmentation label, the sample latent feature, the target semantic segmentation map and the original expert model, a target loss is calculated;

[0010] According to the target loss, the model parameters of each original expert model are updated respectively to obtain a target expert model; wherein the target expert model is used for semantic segmentation on a target street view image obtained.

[0011] In some embodiments, the aggregating the sample semantic segmentation maps output by each of the original expert models to obtain a target semantic segmentation map comprises:

[0012] calculating a routing probability of the original expert model;

[0013] routing the original expert model according to the routing probability to obtain a reference expert model;

[0014] aggregating the sample semantic segmentation maps output by each of the reference expert models to obtain the target semantic segmentation map.

[0015] In some embodiments, each of the original expert models has a feature retrieval library comprising a prototype feature vector, and the calculating the routing probability of the original expert model comprises:

[0016] calculating an attention weight between the sample latent feature and the prototype feature vector;

[0017] weighting and fusing the prototype feature vector according to the attention weight to obtain a retrieval prototype feature;

[0018] calculating a divergence between the sample latent feature and the retrieval prototype feature to obtain a first divergence;

[0019] calculating the routing probability according to the first divergence.

[0020] In some embodiments, the aggregating the sample semantic segmentation maps output by each of the reference expert models to obtain the target semantic segmentation map comprises:

[0021] respectively calculating a divergence between the sample intermediate feature extracted by each of the reference expert models and the sample latent feature to obtain a second divergence corresponding to the reference expert model;

[0022] calculating an aggregation weight of the reference expert model according to the second divergence;

[0023] aggregating the sample semantic segmentation map corresponding to each of the aggregation weights to obtain the target semantic segmentation map.

[0024] In some embodiments, each of the original expert models has a feature retrieval library comprising a prototype feature vector and a vector weight of the prototype feature vector, and the calculating a target loss according to the semantic segmentation label, the sample latent feature, the target semantic segmentation map and the original expert model comprises:

[0025] calculating a first loss according to the semantic segmentation label and the target semantic segmentation map;

[0026] compute an attention weight between the sample latent feature and the prototype feature vector, compute a routing probability of the original expert model according to the attention weight, the sample latent feature and the prototype feature vector, and compute a second loss according to the routing probability and a number of the original expert models;

[0027] compute a third loss according to the prototype feature vector, the vector weight and the attention weight;

[0028] weight the first loss, the second loss and the third loss to obtain the target loss.

[0029] In some embodiments, each of the original expert models has a feature retrieval library including a prototype feature vector and a vector weight of the prototype feature vector, and after the target loss is computed according to the semantic segmentation label, the sample latent feature, the target semantic segmentation map and the original expert model, the method further includes:

[0030] for each of the original expert models, map the sample intermediate feature to a prototype space to obtain a mapped prototype feature;

[0031] determine an attention weight;

[0032] update the prototype feature vector according to the attention weight and the mapped prototype feature;

[0033] update the vector weight according to the attention weight.

[0034] To achieve the above object, a second aspect of the embodiment of the present application proposes a vehicle control method, which comprises:

[0035] obtain a target street view image collected by a target vehicle and a plurality of target expert models; wherein the target expert models are trained according to the model training method of the first aspect;

[0036] extract a target latent feature of the target street view image through a pre-trained visual large model;

[0037] extract an intermediate feature of the target latent feature through each of the target expert models, and output a semantic segmentation map of the target street view image according to the corresponding intermediate feature;

[0038] route the target expert models to obtain candidate expert models;

[0039] aggregate the semantic segmentation maps output by each of the candidate expert models to obtain a target segmentation map;

[0040] Control the target vehicle to travel according to the target segmentation map.

[0041] To achieve the above object, a third aspect of the embodiments of the present application provides a vehicle control device, the device comprising:

[0042] An acquisition module is configured to acquire a target street view image collected by a target vehicle and a plurality of target expert models, wherein the target expert models are obtained by training according to the model training method of the first aspect;

[0043] A first feature extraction module is configured to extract target latent features of the target street view image by a pre-trained visual large model;

[0044] A second feature extraction module is configured to extract intermediate features of the target latent features by each of the target expert models, and output a semantic segmentation map of the target street view image according to the corresponding intermediate features;

[0045] A routing module is configured to route the target expert models to obtain candidate expert models;

[0046] An aggregation module is configured to aggregate the semantic segmentation maps output by each of the candidate expert models to obtain a target segmentation map;

[0047] A control module is configured to control the target vehicle to travel according to the target segmentation map.

[0048] To achieve the above object, a fourth aspect of the embodiments of the present application provides an electronic device, the electronic device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the model training method of the first aspect or the vehicle control method of the second aspect when executing the computer program.

[0049] To achieve the above object, a fifth aspect of the embodiments of the present application provides a computer readable storage medium, the computer readable storage medium storing a computer program, and the computer program implementing the model training method of the first aspect or the vehicle control method of the second aspect when executed by a processor.

[0050] The model training method, vehicle control method, vehicle control device, electronic device and computer readable storage medium provided by the embodiments of the present application can obtain a sample street view image, the sample street view image has a semantic segmentation label, and the training of the semantic segmentation model is performed based on the sample street view image carrying the semantic segmentation label. The style of the sample street view image is various, in order to comprehensively understand the street view image, the powerful feature extraction capability of the visual large model is used, the sample latent feature of the sample street view image is extracted through the visual large model, and the sample latent feature is taken as the input of the downstream semantic segmentation task. In order to enhance the generalization ability of the semantic segmentation model, the expert mixing paradigm based on the large model is adopted, and the semantic segmentation task is jointly performed by multiple parallel expert models. The sample intermediate feature of the sample latent feature is extracted by each original expert model respectively, the diversified features are extracted based on the expert knowledge of different expert models, so as to cope with the complex and various driving environment, and the sample semantic segmentation image of the sample street view image is output according to the corresponding sample intermediate feature. In order to improve the performance of semantic segmentation, the sample semantic segmentation image output by each original expert model is aggregated to improve the overall segmentation accuracy, and the target semantic segmentation image is obtained. According to the semantic segmentation label, the sample latent feature, the target semantic segmentation image and the original expert model, the target loss is calculated, and the training process of the expert model is guided based on the target loss. According to the target loss, the model parameters of each original expert model are updated respectively to obtain the target expert model, so as to perform semantic segmentation on the obtained target street view image through the target expert model, thereby improving the accuracy of semantic segmentation and the safety of autonomous driving. BRIEF DESCRIPTION OF DRAWINGS

[0051] Figure 1 is a flowchart of the model training method provided by the embodiments of the present application;

[0052] Figure 2 is a flowchart of step S140 in Figure 1

[0053] Figure 3 is a flowchart of step S210 in Figure 2

[0054] Figure 4 is a flowchart of step S230 in Figure 2

[0055] Figure 5 is a flowchart of step S150 in Figure 1

[0056] Figure 6 is another flowchart of the model training method provided by the embodiments of the present application;

[0057] Figure 7 is an effect diagram of the model training method provided by the embodiments of the present application;​​​​

[0058] Figure 8 is a flowchart of a vehicle control method provided by an embodiment of the present application;

[0059] Figure 9 is a structural schematic diagram of a vehicle control device provided by an embodiment of the present application;

[0060] Figure 10 is a hardware structural schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0061] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and should not be used to limit the present application.

[0062] It should be noted that although the functional modules are divided in the device schematic diagram, and the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a manner different from the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification and claims and the above-described drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence.

[0063] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0064] Autonomous driving requires models to be able to process and respond to extremely diverse real-world scenarios in real time. These scenarios exhibit extremely high complexity and diversity, including but not limited to unpredictable dense urban traffic, variable lighting conditions on highways, irregular terrain in rural roads, and sensor input ambiguity in adverse weather such as heavy rain, fog, or snow. In the related art, a semantic segmentation model is used to perform semantic segmentation on a scene image so that a vehicle can understand its surrounding environment. However, the semantic segmentation model often overfits to the dominant patterns in the training data, making it difficult to generalize to edge or unknown scenarios, which results in low accuracy of semantic segmentation when the model faces diverse and complex environments.

[0065] Based on this, the embodiments of the present application provide a model training method, a vehicle control method, a vehicle control device, an electronic device and a computer readable storage medium, which aim to improve the accuracy of semantic segmentation.

[0066] The model training method, vehicle control method, vehicle control device, electronic device, and computer readable storage medium provided in the embodiments of the present application are specifically described through the following embodiments. First, the model training method in the embodiments of the present application is described.

[0067] The model training method provided in the embodiments of the present application relates to the technical field of autonomous driving. The model training method provided in the embodiments of the present application can be applied to a terminal, can be applied to a server end, and can also be software running in the terminal or the server end. In some embodiments, the terminal can be a smartphone, a tablet computer, a notebook computer, a desktop computer, or the like; the server end can be configured as a standalone physical server, can be configured as a server cluster or a distributed system formed by multiple physical servers, can also be configured as a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDNs, and big data and artificial intelligence platforms; and the software can be an application that implements the model training method, but is not limited to the above forms.

[0068] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld devices or portable devices, tablet devices, multi-processor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. The present application can also be practiced in a distributed computing environment in which tasks are performed by remote processing devices connected by a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0069] Figure 1 The method in the model training method provided in the embodiments of the present application can include but is not limited to steps S110 to S160. Figure 1

[0070] In step S110, a sample street view image is acquired; wherein the sample street view image has a semantic segmentation label.

[0071] In step S120, a sample latent feature of the sample street view image is extracted through a pre-trained visual large model.

[0072] ​Step S130: Extract the intermediate features of the latent features of the samples through each original expert model, and output the sample semantic segmentation map of the sample street scene image according to the corresponding intermediate features.

[0073] Step S140: Aggregate the sample semantic segmentation maps output by each original expert model to obtain the target semantic segmentation map;

[0074] Step S150: Calculate the target loss based on the semantic segmentation label, sample latent features, target semantic segmentation map, and original expert model;

[0075] Step S160: Update the model parameters of each original expert model according to the target loss to obtain the target expert model; wherein, the target expert model is used to perform semantic segmentation on the acquired target street view image.

[0076] In step S110 of some embodiments, sample street view images can be obtained from a dataset or captured by a camera. The sample street view images are color RGB street view images used for model training. The sample street view images have semantic segmentation labels, which are used to indicate the semantic category of each pixel in the sample street view image, such as roads, vehicles, pedestrians, buildings, etc.

[0077] In step S120 of some embodiments, street view images exhibit diverse styles, encompassing various scenes and elements, resulting in complex semantic information. To understand this complex street view semantic information, this application introduces a pre-trained large vision model as the backbone network. Leveraging the powerful semantic understanding and feature extraction capabilities of the large vision model, latent features of the sample street view images are extracted, yielding sample latent features. These sample latent features can serve as input to downstream task models. The large vision model is a model trained on massive amounts of images or videos, possessing billions or even trillions of parameters, and can be used to process visual data such as images and videos. The large vision model can be a Vision Transformer (ViT).

[0078] The first in dataset D Zhang's sample street view images are represented as follows The model parameters of the large vision model are expressed as follows: Then the first The latent features of the sample street view images are represented as follows:

[0079] ,

[0080] in, Indicates the first Sample latent features of Zhang's street view images.

[0081] In step S130 of some embodiments, in order to improve the semantic segmentation ability of the model on unknown street view images or edge street view images, a plurality of parallel original expert models are deployed to extract diversified street view features according to the expert knowledge possessed by the original expert models, and the adaptability of the semantic segmentation model to complex and diversified street views is improved. The number of original expert models can be set according to actual conditions, which is not specifically limited here. For example, the number of models can be 10. Specifically, the sample latent features are input into each original expert model, and for each original expert model, the original expert model extracts features from the input sample latent features to obtain sample intermediate features, and classifies each pixel in the sample street view image into a predefined semantic category according to the sample intermediate features, and outputs a sample semantic segmentation map of the sample street view image. The original expert model can use a convolutional neural network (CNN). The model architecture of each original expert model can be the same or different.

[0082] The original expert model includes an encoder, and the sample intermediate features are extracted through the encoder. The feature extraction process is represented as:

[0083] ,

[0084] wherein, represents the sample intermediate features obtained by the jth original expert model for the i th sample street view image; represents the sample latent features of the i th sample street view image; represents the encoder of the j th original expert model; represents the model parameters of the encoder in the j th original expert model.

[0085] The semantic segmentation process is represented as:

[0086] ,

[0087] wherein, represents the sample semantic segmentation map output by the j th original expert model; represents the j th original expert model; represents the sample latent features of the i th sample street view image; represents the model parameters of the j th original expert model, .

[0088] Each original expert model has a feature retrieval library (FRL) for storing a prototype feature pattern for indicating the professional field and professional knowledge of the expert model. The feature retrieval library includes a prototype feature set including a prototype feature vector, i.e., a prototype feature pattern, and a vector weight of the prototype feature vector for indicating the importance of the prototype feature vector in the original expert model. The vector weight is a non-negative number, and the greater the value, the greater the contribution of the prototype feature vector to the semantic segmentation result and the more important the prototype feature vector in the original expert model. The number of prototype feature vectors included in the prototype feature set can be set according to actual conditions, such as 16.

[0089] For the feature retrieval library of the jthoriginal expert model, the prototype feature set is represented as The prototype feature set includes prototype feature vectors and a vector weight of each prototype feature vector. The kthprototype feature vector is represented as , , represents a real number space, represents the dimension of the prototype feature vector, and the vector weight thereof is represented as . The prototype feature set can be represented as .

[0090] Different original expert models perform differently under different data distributions. For example, one original expert model performs better in semantic segmentation of urban street images, and another original expert model performs better in semantic segmentation of rural road images. In order to improve the accuracy of semantic segmentation and improve the robustness of the model to different data distributions and reduce the safety risk of autonomous driving, the embodiments of the present application aggregate the sample semantic segmentation maps output by each original expert model to obtain a target semantic segmentation map.

[0091] Referring to Figure 2 In some embodiments, step S140 can include, but is not limited to, steps S210 to S230:

[0092] Step S210, calculating a routing probability of the original expert model;

[0093] Step S220, routing the original expert model according to the routing probability to obtain a reference expert model;

[0094] Step S230, aggregating the sample semantic segmentation maps output by each reference expert model to obtain a target semantic segmentation map.

[0095] In step S210 of some embodiments, among the plurality of expert models, a routing mechanism is one of the core components, which dynamically selects the most relevant expert model according to the input data. The routing probability of the original expert model is calculated, which is the probability of the original expert model being selected to process the input data.

[0096] In step S220 of some embodiments, each expert model performs better in different scenes (such as urban streets, rural roads, highways, etc.) or different targets (such as vehicles, pedestrians, buildings, etc.). In order to obtain an expert model that is good at processing the current street scene or target, the original expert models are sorted in descending order of routing probability, and the top K original expert models are selected as reference expert models. K < N, N is the total number of original expert models, and K and N are integers. Exemplarily, if N is 10, K can be 5.

[0097] In step S230 of some embodiments, the routing mechanism and the aggregation mechanism are decoupled, and the sample semantic segmentation maps output by the K reference expert models are aggregated to obtain a target semantic segmentation map.

[0098] The above steps S210 to S230 can improve the overall accuracy of semantic segmentation by aggregating the sample semantic segmentation maps output by the expert models that are good at processing the current street scene or target.

[0099] Please refer to Figure 3 In some embodiments, step S210 can include but is not limited to steps S310 to S340:

[0100] Step S310, calculate the attention weight between the sample latent feature and the prototype feature vector;

[0101] Step S320, weight fusion is performed on the prototype feature vector according to the attention weight to obtain a retrieval prototype feature;

[0102] Step S330, calculate the divergence between the sample latent feature and the retrieval prototype feature to obtain a first divergence;

[0103] Step S340, calculate the routing probability according to the first divergence.

[0104] In step S310 of some embodiments, the routing mechanism in the related art usually relies on gating functions, such as softmax-based functions, which are very sensitive to feature mismatch. In a dynamic autonomous driving environment, these routing functions tend to be difficult to accurately map the input to the optimal expert model, resulting in a decrease in prediction accuracy. In order to improve the accuracy of semantic segmentation, the embodiments of the present application propose a routing mechanism enhanced based on statistical retrieval, which calculates the routing probability by combining the statistical information retrieved in the feature retrieval library of each expert model, i.e., the prototype feature vector, and the feature statistical information extracted by the large visual model, i.e., the sample latent feature, to determine the expert model most relevant to the current input data based on the routing probability.

[0105] Specifically, the similarity between the sample latent feature and the prototype feature vector is calculated based on a similarity function, and the similarity is normalized to obtain an attention weight. Exemplarily, the similarity function can be cosine similarity. The calculation formula of the attention weight is represented as:

[0106] ,

[0107] wherein the superscript i represents the i-th sample street view image; the subscript j represents the j-th original expert model feature retrieval library; the subscripts k and are the identifiers of the prototype feature vectors, and represent the k-th prototype feature vector and the -th prototype feature vector in the j-th feature retrieval library, respectively; sim represents the similarity function; represents the sample latent feature; represents the number of prototype feature vectors in the j-th feature retrieval library; represents the attention weight between the sample latent feature of the i-th sample street view image and the k-th prototype feature vector in the j-th feature retrieval library.

[0108] In step S320 of some embodiments, for each prototype feature vector in the original expert model feature retrieval library, the attention weight is multiplied by the corresponding prototype feature vector to obtain an intermediate vector. Each intermediate vector is added to obtain the retrieval prototype feature of the original expert model. The retrieval prototype feature is represented as:

[0109] ,

[0110] wherein, represents the retrieval prototype feature of the j-th original expert model.

[0111] In step S330 of some embodiments, the sample latent feature is normalized from the feature to the distribution to obtain a first distribution. The retrieval prototype feature is normalized from the feature to the distribution to obtain a second distribution. A Jensen-Shannon (JS) divergence between the first distribution and the second distribution is calculated to obtain a first divergence, so as to align the sample latent feature with the expert knowledge in the feature retrieval library according to the first divergence, and realize accurate scene perception expert selection. For example, in a dense fog driving scene, an expert model focusing on low-visibility driving scene prediction is retrieved, so as to ensure adaptive and accurate prediction. The calculation formula of the first divergence is represented as:

[0112] ,

[0113] ,

[0114] ,

[0115] ,

[0116] wherein Norm represents a normalization function from the feature to the distribution, such as a softmax function; represents the first distribution; represents the second distribution; represents the JS divergence; represents a Kullback-Leibler divergence, i.e., a KL divergence; represents an average distribution.

[0117] Referring to the above formula, the average distribution is obtained by averaging the first distribution and the second distribution; the first KL divergence is obtained by calculating the KL divergence between the first distribution and the average distribution; the second KL divergence is obtained by calculating the KL divergence between the second distribution and the average distribution; and the first divergence is obtained by averaging the first KL divergence and the second KL divergence.

[0118] In step S340 of some embodiments, a routing score is calculated according to the first divergence, and the routing score is inversely proportional to the first divergence. The routing score is normalized to obtain a routing probability.

[0119] Specifically, a numerical stability parameter and a temperature parameter are obtained, the numerical stability parameter is a very small positive number, and the temperature parameter is a positive number used to adjust the sharpness of the output probability distribution. The reciprocal of the sum of the numerical stability parameter and the first divergence is taken as the routing score. The routing score is normalized according to the temperature parameter to obtain the routing probability. The calculation formula of the routing score is represented as:

[0120] ,

[0121] wherein, denotes a routing score of the jthoriginal expert model for the ithsample street view image; denotes a numerical stability parameter, denotes a first divergence.

[0122] The calculation formula of the routing probability is represented as:

[0123]

[0124] wherein, denotes a routing probability of the jthoriginal expert model for the ithsample street view image; denotes a temperature parameter, N is the number of original expert models.

[0125] The steps S310 to S340 above enhance the routing process by integrating the retrieval mechanism, calculate the routing probability, and dynamically select the expert model most relevant to the current scene input data based on the routing probability, thereby improving the accuracy of semantic segmentation.

[0126] Referring to Figure 4 In some embodiments, the step S230 can include but is not limited to steps S410 to S430:

[0127] Step S410, respectively calculate the divergence between the sample intermediate features extracted by each reference expert model and the sample latent features, to obtain the second divergence corresponding to the reference expert model;

[0128] Step S420, calculate the aggregation weight of the reference expert model according to the second divergence;

[0129] Step S430, aggregate the corresponding sample semantic segmentation map according to each aggregation weight to obtain the target semantic segmentation map.

[0130] In step S410 of some embodiments, traditional methods such as simple weighted average fail to fully capture the statistical dependence between expert models, resulting in underestimation of the fused activation values, and this inefficient aggregation further weakens the prediction performance of the model, which is particularly significant in non-independent and identically distributed autonomous driving scenarios. To overcome the shortcomings of the expert fusion strategy, the embodiments of the present application propose an expert aggregation mechanism based on statistical similarity, which adopts a dynamic statistical reweighting method and considers aligning the statistical distribution of expert output to aggregate expert output.

[0131] ​​Based on the Jensen-Shannon divergence, the statistical distance between the instant features extracted by each reference expert model, i.e., the sample intermediate features, and the sample latent features extracted by the ViT is calculated, to obtain the second divergence of each reference expert model. Specifically, the sample latent features are normalized from features to a distribution to obtain a first distribution. The sample intermediate features are normalized from features to a distribution to obtain a third distribution. The JS divergence between the first distribution and the third distribution is calculated to obtain the second divergence. The calculation formula of the second divergence is represented as:

[0132] ,

[0133] ,

[0134] wherein, denotes the third distribution; Norm denotes a normalization function from features to a distribution; denotes the sample intermediate features extracted by the jth reference expert model for the ith sample street view image; denotes the second divergence; denotes a set of reference expert models.

[0135] In step S420 of some embodiments, for each reference expert model, the reciprocal of the sum of the numerical stability parameter and the second divergence is calculated, and the proportion of the reciprocal in the total sum of all reference expert model reciprocals is calculated, to assign an aggregation weight to the reference expert model. The distribution alignment between the intermediate features extracted by the expert model and the latent features extracted by the visual large model can be determined through the second divergence, so as to dynamically reweight the expert model in the fusion process. The calculation formula of the aggregation weight is represented as:

[0136] ,

[0137] ,

[0138] wherein, denotes the aggregation weight of the jth reference expert model; denotes a numerical stability parameter, which is a very small positive number.

[0139] In step S430 of some embodiments, the output of the expert model aggregated by the above aggregation weight is adopted, for each reference expert model, the aggregation weight of the reference expert model is multiplied by the sample semantic segmentation map output by the reference expert model to obtain an intermediate segmentation map. Each intermediate segmentation map is added to obtain a target semantic segmentation map. The calculation formula of the aggregation is represented as:

[0140] ,

[0141] wherein, a target semantic segmentation map representing the i-th sample street view image; a sample semantic segmentation map output by the j-th reference expert model for the i-th sample street view image.

[0142] The aggregation based on statistical similarity in steps S410-S430 enables focusing on expert models that output sample latent feature distributions consistent with the target semantic segmentation map, achieving a fusion with strong adaptability and good robustness in a diversified and complex autonomous driving environment. By synergistically integrating the routing mechanism and the aggregation mechanism, the high demand for prediction accuracy and adaptability in autonomous driving can be met.

[0143] Referring to Figure 5 In some embodiments, step S150 can include but is not limited to steps S510-S540:

[0144] Step S510, calculating a first loss according to the semantic segmentation label and the target semantic segmentation map;

[0145] Step S520, calculating an attention weight between the sample latent feature and the prototype feature vector, calculating a routing probability of the original expert model according to the attention weight, the sample latent feature and the prototype feature vector, and calculating a second loss according to the routing probability and the number of original expert models;

[0146] Step S530, calculating a third loss according to the prototype feature vector, the vector weight and the attention weight;

[0147] Step S540, calculating a target loss by weighting the first loss, the second loss and the third loss.

[0148] In step S510 of some embodiments, a first loss is calculated based on a cross-entropy loss function according to the semantic segmentation label and the target semantic segmentation map. The first loss is used to measure the difference between the class probability distribution of the pixels in the target semantic segmentation map and the true class of the pixels in the semantic segmentation label.

[0149] In step S520 of some embodiments, the attention weight between the sample latent feature and the prototype feature vector is calculated with reference to step S310, and the prototype feature vectors are weighted and fused according to the attention weight to obtain a retrieval prototype feature with reference to steps S320 to S340. The divergence between the sample latent feature and the retrieval prototype feature is calculated to obtain a first divergence. The routing probability of the original expert model is calculated according to the first divergence. The routing probability of the original expert model for each sample street view image in the training batch is averaged to obtain a probability mean. The second loss is calculated according to the probability mean of each original expert model and the number of original expert models. The probability mean is used to promote the uniform routing of each original expert model, avoiding the collapse of the expert model in the training process. The second loss is an auxiliary load balancing term, which is used to control the regularization strength and avoid the collapse of the expert model by promoting the uniform routing of the expert model. The calculation formula of the second loss is represented as:

[0150] ,

[0151] ,

[0152] wherein, represents the second loss; represents the batch size of the training batch; represents the routing probability of the jthoriginal expert model for the ithsample street view image; represents the probability mean corresponding to the jthoriginal expert model; and N represents the number of original expert models.

[0153] In step S530 of some embodiments, for each prototype feature vector of each original expert model, a weight decay value is calculated according to the prototype feature vector and the vector weight. The attention weights between the sample latent features extracted based on each sample street view image in the training batch and the prototype feature vectors are summed to obtain a sparse attention value. The third loss is obtained by summing the weight decay values and the sparse attention values. The weight decay value prevents unlimited growth of the prototype feature vector and its vector weight, and the sparse attention value is used to select the relevant prototype feature vector instead of the average prototype feature vector. The calculation formula of the third loss is represented as:

[0154] ,

[0155] wherein, represents the third loss; and N represents the number of original expert models. represents the number of prototype feature vectors in the feature retrieval library of the jthoriginal expert model; represents the kthprototype feature vector in the feature retrieval library of the jthoriginal expert model; denotes the vector weight of the k-th prototype feature vector in the j-th original expert model feature retrieval library; B is the batch size of the training batch; denotes the attention weight between the sample latent feature extracted based on the i-th sample street view image and the prototype feature vector; denotes the weight decay value; denotes the sparse attention value.

[0156] In step S540 of some embodiments, the first loss, the second loss and the third loss are weighted and summed to obtain a target loss. The calculation formula of the target loss is represented as:

[0157] ,

[0158] wherein, denotes the target loss; denotes the first loss; denotes the second loss; denotes the weight of the second loss; denotes the third loss; denotes the weight of the third loss.

[0159] The weight of the second loss and the weight of the third loss can be set according to actual conditions. Exemplarily, the weight of the second loss is 0.01, and the weight of the third loss is 1e-4.

[0160] Through the above steps S510 to S540, the target loss can be obtained to guide the training process of the expert model based on the target loss, thereby improving the accuracy of semantic segmentation.

[0161] The feature retrieval library is used to store the professional knowledge and experience of each expert model, and can be updated in a reading-updating manner based on an attention mechanism. The updating is parallel to the forward propagation, and can be performed after reading the prototype feature set of the feature retrieval library to adaptively refine the prototype feature set.

[0162] Please refer to Figure 6 In some embodiments, after step 150, the model training method can further include, but is not limited to, steps S610 to S640:

[0163] Step S610, for each original expert model, mapping the sample intermediate feature to the prototype space to obtain a mapped prototype feature;

[0164] Step S620, determining an attention weight;

[0165] Step S630, updating the prototype feature vector according to the attention weight and the mapped prototype feature;

[0166] Step S640, updating the vector weight according to the attention weight.

[0167] In step S610 of some embodiments, each original expert model has a mapping function for mapping features to a prototype space. For each original expert model, the intermediate features in the corresponding sample are mapped to the prototype space by the mapping function to obtain mapped prototype features.

[0168] In step S620 of some embodiments, referring to step S310, the attention weight between the sample latent feature and the prototype feature vector is calculated.

[0169] In step S630 of some embodiments, the prototype feature vector is updated according to the attention weight, the mapped prototype feature, and a preset learning rate, where the learning rate can be set to 3e-4. The update formula is represented as:

[0170] ,

[0171] wherein, denotes the learning rate; denotes the attention weight; denotes the prototype feature vector; denotes the mapping function; denotes the intermediate features of the sample.

[0172] In step S640 of some embodiments, the vector weight is updated according to the attention weight and a preset learning rate, and the update formula is represented as:

[0173] ,

[0174] wherein, denotes the learning rate; denotes the attention weight; denotes the vector weight of the prototype feature vector.

[0175] Through the above steps S610 to S640, a compact prototype feature with scene awareness can be maintained, so that the expert model has the ability to evolve with the scene.

[0176] In step S160 of some embodiments, the target loss is minimized, the model parameters of each original expert model are updated, and a target expert model is obtained to perform semantic segmentation on the obtained target street view image through the target expert model.

[0177] The semantic segmentation performance of the target expert model trained according to the model training method of the embodiments of the present application and other models is tested on an autonomous driving dataset, the other models including BiSeNetV2, SegNet, SegFormer, AttaNet, HRDA, TopFormer, SeaFormer, ViT+ASSP, LinearMoE, NonlinearMoE, SoftMoE, and the semantic segmentation performance is measured by mIoU (mean Intersection over Union), and the test results are as shown in Figure 7 The Epochs represent the training rounds. Figure 7 It can be seen that the model trained according to the embodiments of the present application can achieve better semantic segmentation performance.

[0178] Figure 8 is an optional flowchart of the vehicle control method provided by the embodiments of the present application, Figure 8 The method in the flowchart can include but is not limited to steps S810 to S860.

[0179] Step S810, obtaining a target street view image collected by a target vehicle and a plurality of target expert models;

[0180] Step S820, extracting a target latent feature of the target street view image by a pre-trained visual large model;

[0181] Step S830, extracting an intermediate feature of the target latent feature by each target expert model respectively, and outputting a semantic segmentation map of the target street view image according to the corresponding intermediate feature;

[0182] Step S840, routing the target expert models to obtain candidate expert models;

[0183] Step S850, aggregating the semantic segmentation maps output by each candidate expert model to obtain a target segmentation map;

[0184] Step S860, controlling the target vehicle to travel according to the target segmentation map.

[0185] In step S810 of some embodiments, a target street view image collected by a target vehicle and a plurality of target expert models trained according to the above model training method are obtained.

[0186] In step S820 of some embodiments, referring to step S120, the target street view image is input to the pre-trained visual large model for feature extraction to obtain a target latent feature.

[0187] In step S830 of some embodiments, the target latent features are respectively input into each target expert model, intermediate features of the target latent features are extracted by each target expert model, and a semantic segmentation map of the target street view image is output according to the intermediate features.

[0188] In step S840 of some embodiments, referring to steps S210 to S230, the routing probability of each target expert model is calculated, the first K target expert models are selected in descending order of the routing probability, and the candidate expert models are obtained.

[0189] In step S850 of some embodiments, referring to steps S410 to S430, the semantic segmentation maps output by the candidate expert models are aggregated to obtain a target segmentation map.

[0190] In step S860 of some embodiments, a control instruction is generated according to the target segmentation map, and the target vehicle is controlled to travel according to the control instruction. The control instruction can be a speed control instruction, a steering control instruction, etc.

[0191] The steps S810 to S860 described above can realize accurate street view semantic segmentation based on the routing mechanism and the aggregation mechanism, thereby improving the safety of vehicle travel in an autonomous driving scenario.

[0192] Please refer to Figure 9 The embodiments of the present application also provide a vehicle control device, which can implement the vehicle control method described above. The vehicle control device comprises:

[0193] The acquisition module 910 is configured to acquire a target street view image collected by a target vehicle and a plurality of target expert models. The target expert models are obtained by training according to the model training method described above.

[0194] The first feature extraction module 920 is configured to extract target latent features of the target street view image by using a pre-trained visual large model.

[0195] The second feature extraction module 930 is configured to extract intermediate features of the target latent features by using each target expert model respectively, and output a semantic segmentation map of the target street view image according to the corresponding intermediate features.

[0196] The routing module 940 is configured to route the target expert models to obtain candidate expert models.

[0197] The aggregation module 950 is configured to aggregate the semantic segmentation maps output by each candidate expert model to obtain a target segmentation map.

[0198] The control module 960 is configured to control the target vehicle to travel according to the target segmentation map.

[0199] The specific embodiment of the vehicle control device is basically the same as the specific embodiment of the above vehicle control method, and will not be described here.

[0200] The embodiment of the present application also provides an electronic device, which comprises a memory and a processor. The memory stores a computer program, and the processor implements the above model training method or vehicle control method when executing the computer program. The electronic device can be any intelligent terminal, such as a tablet computer or a vehicle-mounted computer.

[0201] Please refer to Figure 10 , Figure 10 The hardware structure of the electronic device of another embodiment is illustrated, which comprises:

[0202] The processor 1010 can be implemented in the form of a general central processing unit (CPU), a microprocessor, an application specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute related programs to implement the technical solutions provided by the embodiments of the present application.

[0203] The memory 1020 can be implemented in the form of a read only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1020 can store an operating system and other application programs. When the technical solutions provided by the embodiments of the present application are implemented by software or firmware, the related program codes are stored in the memory 1020 and are called and executed by the processor 1010 to implement the model training method of the embodiments of the present application.

[0204] The input / output interface 1030 is used to realize information input and output.

[0205] The communication interface 1040 is used to realize the communication interaction between the device and other devices. The communication can be realized by wired means (such as USB, network cable, etc.), or by wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0206] The bus 1050 transmits information between various components (such as the processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040) of the device.

[0207] The processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040 are connected to each other through the bus 1050 for internal communication in the device.

[0208] The embodiment of the present application further provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the model training method or the vehicle control method.

[0209] The memory, as a non-transitory computer readable storage medium, can be used to store non-transitory software programs and non-transitory computer executable programs. In addition, the memory can include a high-speed random access memory, and can also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory can optionally include a memory disposed remotely relative to the processor, and these remote memories can be connected to the processor through a network. Examples of the network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0210] The embodiments described in the embodiments of the present application are used to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art can know that, with the evolution of technology and the appearance of new application scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0211] Those skilled in the art can understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and can include more or fewer steps than those shown in the figures, or combine certain steps, or different steps.

[0212] The device embodiments described above are only schematic, and the units described as separate components can or can not be physically separate, that is, can be located in one place, or can be distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiments of the present application.

[0213] Those skilled in the art can understand that all or some steps in the above disclosed method, the functions of the modules / units in the system and the device can be implemented as software, firmware, hardware and their appropriate combinations.

[0214] The terms "first", "second", "third", "fourth", and the like in the description and in the claims of this application, if any, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the use of the terms so termed is interchangeable under appropriate circumstances such that the embodiments of the application described herein are, for example, capable of orderly or chronological mundane operation, reverse order operation, based on circuitry availability, based on stated preference or the like, and that "default" or other orderings are thus permissible. Further, the terms "comprise", "comprising", "include", "including", and the like, are specifically intended to be open-ended. That is, references to individual steps and the like do not suhstantially exclude the presence of two or more of a recited step or its integral sub-steps or additional steps whether or not readily ascertainable from the description or the like. Further, the words "a" or "an", as used herein in the disclosure and elsewhere, are used indiscriminately and are to be interpreted in the same way, i.e. as meaning "one or more".

[0215] It should be understood that, in this application, "at least one" means one or more, "multiple" means two or more. "And / or", used to describe the relationship between associated objects, means that there can be three relationships, for example, "A and / or B" can mean: only A, only B, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects. "At least one of the following" or the like means any combination of these items, including single or multiple combinations. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0216] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the above units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be omitted or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed objects can be indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0217] The units described above as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e. they can be located in one place or distributed on a plurality of network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0218] In addition, each function unit in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software function unit.

[0219] When the integrated unit is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application, essentially or in part, or all or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes multiple instructions used to cause a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods in the embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various other media that can store programs.

[0220] The preferred embodiments of the embodiments of the present application are described above with reference to the accompanying drawings, and are not intended to limit the scope of the embodiments of the present application. Any modifications, equivalent replacements and improvements made by those skilled in the art without departing from the scope and essence of the embodiments of the present application shall be within the scope of the embodiments of the present application.

Claims

1. A model training method, characterized in that, The method comprises: acquiring a sample street view image; wherein the sample street view image has a semantic segmentation label; extracting a sample latent feature of the sample street view image through a pre-trained visual large model; extracting a sample intermediate feature of the sample latent feature through each original expert model respectively, and outputting a sample semantic segmentation map of the sample street view image according to the corresponding sample intermediate feature; aggregating the sample semantic segmentation map output by each original expert model to obtain a target semantic segmentation map; calculating a target loss according to the semantic segmentation label, the sample latent feature, the target semantic segmentation map and the original expert model; updating the model parameters of each original expert model according to the target loss respectively to obtain a target expert model; wherein the target expert model is used for semantic segmentation of an acquired target street view image.

2. The method of claim 1, wherein, The aggregation of the sample semantic segmentation map output by each original expert model to obtain a target semantic segmentation map comprises: calculating a routing probability of the original expert model; routing the original expert model according to the routing probability to obtain a reference expert model; aggregating the sample semantic segmentation map output by each reference expert model to obtain the target semantic segmentation map.

3. The method of claim 2, wherein, Each original expert model has a feature retrieval library, and the feature retrieval library comprises a prototype feature vector. The calculation of the routing probability of the original expert model comprises: calculating an attention weight between the sample latent feature and the prototype feature vector; weighting and fusing the prototype feature vector according to the attention weight to obtain a retrieval prototype feature; calculating a divergence between the sample latent feature and the retrieval prototype feature to obtain a first divergence; calculating the routing probability according to the first divergence.

4. The method of claim 2, wherein, The aggregation of the sample semantic segmentation map output by each reference expert model to obtain the target semantic segmentation map comprises: calculating a divergence between the sample intermediate feature extracted by each reference expert model and the sample latent feature to obtain a second divergence corresponding to the reference expert model; wherein the reference expert model is an original expert model selected through a routing operation; calculating an aggregation weight of the reference expert model according to the second divergence; aggregating the corresponding sample semantic segmentation map according to each aggregation weight to obtain the target semantic segmentation map.

5. The method of claim 1, wherein, Each original expert model has a feature retrieval library, and the feature retrieval library comprises a prototype feature vector and a vector weight of the prototype feature vector. The calculation of the target loss according to the semantic segmentation label, the sample latent feature, the target semantic segmentation map and the original expert model comprises: calculating a first loss according to the semantic segmentation label and the target semantic segmentation map; calculating a second loss according to the sample latent feature and the target semantic segmentation map; and calculating the target loss according to the first loss and the second loss. computing an attention weight between the sample latent feature and the prototype feature vector, computing a routing probability of the original expert model according to the attention weight, the sample latent feature and the prototype feature vector, and computing a second loss according to the routing probability and a model number of the original expert model; computing a third loss according to the prototype feature vector, the vector weight and the attention weight; performing weighted computation on the first loss, the second loss and the third loss to obtain the target loss.

6. The method according to any one of claims 1 to 5, characterized in that, Each of the original expert models has a feature retrieval library including a prototype feature vector and a vector weight of the prototype feature vector, and after the target loss is computed according to the semantic segmentation label, the sample latent feature, the target semantic segmentation map and the original expert model, the method further includes: for each of the original expert models, mapping the sample intermediate feature to a prototype space to obtain a mapped prototype feature; determining an attention weight; updating the prototype feature vector according to the attention weight and the mapped prototype feature; updating the vector weight according to the attention weight.

7. A vehicle control method characterized by The method includes: obtaining a target street view image collected by a target vehicle and a plurality of target expert models, wherein the target expert models are trained according to the model training method of any one of claims 1 to 6; extracting a target latent feature of the target street view image through a pre-trained visual large model; extracting an intermediate feature of the target latent feature through each of the target expert models respectively, and outputting a semantic segmentation map of the target street view image according to the corresponding intermediate feature; routing the target expert models to obtain candidate expert models; aggregating the semantic segmentation maps output by each of the candidate expert models to obtain a target segmentation map; controlling the target vehicle to travel according to the target segmentation map.

8. A vehicle control device characterized by comprising: The device includes: an obtaining module configured to obtain a target street view image collected by a target vehicle and a plurality of target expert models, wherein the target expert models are trained according to the model training method of any one of claims 1 to 6; a first feature extraction module configured to extract a target latent feature of the target street view image through a pre-trained visual large model; a second feature extraction module configured to extract an intermediate feature of the target latent feature through each of the target expert models respectively, and output a semantic segmentation map of the target street view image according to the corresponding intermediate feature; a routing module configured to route the target expert models to obtain candidate expert models; an aggregation module configured to aggregate the semantic segmentation maps output by each of the candidate expert models to obtain a target segmentation map; a control module configured to control the target vehicle to travel according to the target segmentation map.

9. An electronic device, comprising: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the model training method of any one of claims 1 to 6 or the vehicle control method of claim 7 when executing the computer program. The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the model training method of any one of claims 1 to 6 or the vehicle control method of claim 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 9. The computer program, when executed by a processor, implements the model training method of any one of claims 1 to 6 or the vehicle control method of claim 7.

Images