Training method of street scene understanding model based on large visual model assistance

Abstract

The embodiment of the present application discloses a training method for a street view understanding model assisted by a large visual model, which belongs to the field of artificial intelligence technology, and includes: extracting street view instantaneous features for a current street view image frame, and extracting street view stable and invariant features for all historical street view image frames, respectively obtaining the difference between the current street view image and each historical street view image, and extracting street view dynamic features for all street view difference images, randomly mixing the street view instantaneous features, the street view stable and invariant features, and the street view dynamic features to obtain mixed street view features, performing street view semantic segmentation on the mixed street view features through an initial street view understanding model to obtain a predicted street view mask of the current street view image frame, generating a sample street view mask of the mixed street view features through a large visual model, updating the model parameters of the initial street view understanding model according to the predicted street view mask and the sample street view mask to obtain a target street view understanding model, thereby improving the accuracy of street view understanding.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a training method for a street scene understanding model assisted by a large visual model. Background Art

[0002] In the field of autonomous driving, vehicles collect street view images and use built-in street view understanding models to identify various elements in street view images, such as pedestrians and traffic signs, to control the driving of the vehicle. However, when designing the street view understanding model, the impact of sunny weather conditions on vehicle driving performance was mainly considered, and the impact of bad weather conditions was not fully considered. When the vehicle encounters bad weather such as rain, snow, fog, etc., the street view understanding model's recognition accuracy of street view images will decrease, which will in turn affect the driving of the vehicle. Summary of the invention

[0003] The main purpose of the embodiments of the present application is to propose a training method for a street scene understanding model assisted by a large visual model, aiming to improve the accuracy of street scene understanding.

[0004] To achieve the above objectives, a first aspect of an embodiment of the present application proposes a training method for a street scene understanding model assisted by a large visual model, the method comprising:

[0005] Acquire street view image data; the street view image data includes a current street view image frame and a plurality of continuous historical street view image frames before the current street view image frame;

[0006] Extracting instantaneous street view features for the current street view image frame, and extracting stable and invariant street view features for all the historical street view image frames;

[0007] respectively obtaining a difference between the current street view image frame and each of the historical street view image frames to obtain a plurality of street view difference images, and extracting street view dynamic features for all of the street view difference images;

[0008] Randomly mixing the instantaneous street view features, the stable and unchanging street view features, and the dynamic street view features to obtain mixed street view features;

[0009] Performing street view semantic segmentation on the mixed street view features through a preset initial street view understanding model to obtain a predicted street view mask of the current street view image frame;

[0010] Generate a true mask of the mixed street scene feature through a preset large visual model to obtain a sample street scene mask;

[0011] Based on a preset optimizer, model parameters of the initial street view understanding model are updated according to the predicted street view mask and the sample street view mask to obtain a target street view understanding model; the target street view understanding model is used to control the driving of the target vehicle according to features corresponding to street view image data around the target vehicle.

[0012] In some embodiments, the step of generating a real mask of the mixed street view feature by using a preset large visual model to obtain a sample street view mask includes:

[0013] Extracting features of the mixed street scene features by using the large visual model to obtain a potential feature map; the potential feature map includes a plurality of first pixel points;

[0014] Determine the number of cluster centers and the number of iterations, and initialize each cluster center according to the number;

[0015] For each of the first pixel points, calculate the similarity between the first pixel point and each of the cluster centers, and classify the first pixel point into the cluster center with the highest similarity;

[0016] For each cluster center, the cluster center is iteratively updated according to all first pixel points classified into the cluster center until the number of iterative updates reaches the number of iterations or the cluster center converges, thereby obtaining the sample street view mask.

[0017] In some embodiments, the optimizer includes a first loss function, a second loss function, and a third loss function, and the preset-based optimizer updates the model parameters of the initial street view understanding model according to the predicted street view mask and the sample street view mask to obtain a target street view understanding model, including:

[0018] Based on the first loss function, a first loss calculation is performed according to the predicted street view mask and the sample street view mask to obtain an image pixel loss; the predicted street view mask has a predicted category label, and the sample street view mask has a sample category label;

[0019] Based on the second loss function, performing a second loss calculation according to the predicted street view mask to obtain an inter-class contrast loss;

[0020] Based on the third loss function, performing a third loss calculation according to the predicted category label and the sample category label to obtain a category loss;

[0021] The model parameters of the initial street view understanding model are updated according to the image pixel loss, the inter-class contrast loss and the category loss to obtain the target street view understanding model.

[0022] In some embodiments, after generating the real mask of the mixed street view feature by using the preset large visual model to obtain the sample street view mask, the method further includes:

[0023] Expand the preset optimizer into multiple network layers;

[0024] Inputting the predicted street view mask and the sample street view mask into a first network layer, updating the predicted street view mask through the first network layer, and obtaining a reference street view mask output by the first network layer;

[0025] Inputting the reference street view mask and the sample street view mask output by the previous network layer into the current network layer, and updating the reference street view mask output by the previous network layer through the current network layer to obtain the reference street view mask output by the current network layer;

[0026] Until the reference street view mask output by the last network layer is obtained, the reference street view mask output by the last network layer is used as the mask of the current street view image frame.

[0027] In some embodiments, the optimizer includes a first loss function and a second loss function, and updating the predicted street view mask through the first network layer to obtain a reference street view mask output by the first network layer includes:

[0028] Determine the gradient of the first loss function according to the predicted street view mask and the sample street view mask to obtain a first gradient;

[0029] Determine the gradient of the second loss function according to the predicted street view mask to obtain a second gradient;

[0030] The predicted street view mask is updated according to the first gradient and the second gradient to obtain the reference street view mask.

[0031] In some embodiments, the predicted street view mask includes a plurality of second pixel points, and determining the gradient of the second loss function according to the predicted street view mask to obtain the second gradient includes:

[0032] Selecting one of the second pixel points of the predicted street view mask as an anchor point;

[0033] Obtain all second pixel points that belong to the same category as the anchor point to obtain a positive example set, and obtain all second pixel points that belong to different categories from the anchor point to obtain a negative example set;

[0034] The gradient of the second loss function is determined according to the anchor point, the positive example set, and the negative example set to obtain the second gradient.

[0035] To achieve the above object, a second aspect of an embodiment of the present application provides a vehicle control method, the method comprising:

[0036] Acquire a target street view image around a target vehicle within a target time period to obtain street view image data;

[0037] Performing mixed feature extraction on the street view image data to obtain mixed street view features;

[0038] Performing street view semantic segmentation on the mixed street view features through a target street view understanding model to obtain a target street view mask; the target street view understanding model is trained according to the training method of the street view understanding model based on the assistance of a large visual model as described in the first aspect;

[0039] The target vehicle is controlled to travel according to the target street view mask.

[0040] To achieve the above-mentioned purpose, a third aspect of an embodiment of the present application provides a vehicle control device, the device comprising:

[0041] An acquisition module, used to acquire a target street view image around a target vehicle within a target period of time, and obtain street view image data;

[0042] A feature extraction module, used for performing mixed feature extraction on the street view image data to obtain mixed street view features;

[0043] A street view understanding module, configured to perform street view semantic segmentation on the mixed street view features through a target street view understanding model to obtain a target street view mask; the target street view understanding model is trained according to the training method of the street view understanding model based on the assistance of a large visual model as described in the first aspect;

[0044] A control module is used to control the target vehicle to travel according to the target street view mask.

[0045] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, it implements the training method of the street scene understanding model based on the assistance of a large visual model described in the first aspect or the vehicle control method described in the second aspect.

[0046] To achieve the above-mentioned purpose, the fifth aspect of an embodiment of the present application proposes a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the training method of the street scene understanding model based on the assistance of a large visual model described in the first aspect or the vehicle control method described in the second aspect.

[0047] The training method of the street view understanding model based on the assistance of a large visual model, the vehicle control method, the vehicle control device, the electronic device and the computer-readable storage medium proposed in the present application require that autonomous driving needs to perceive the changes in the surrounding street view environment in real time, and obtain street view image data, which includes the current street view image frame and multiple continuous historical street view image frames before the current street view image frame, so that the continuous multiple image frames can provide the temporal continuity and rich contextual information of the street view surrounding environment, which helps the vehicle to accurately identify each element in the street view image. The instantaneous features of the street view for the current street view image frame are extracted to obtain the real-time street view information of the current street view image frame based on the instantaneous features of the street view. The stable and invariant features of the street view for all historical street view image frames are extracted to capture the elements that remain unchanged over time in multiple continuous image frames, so that the street view understanding model can distinguish between static elements and dynamic elements, thereby enhancing the vehicle's safe navigation capabilities. In order to capture the dynamic changes of the street scene environment, the difference between the current street scene image and each historical street scene image is obtained to obtain multiple street scene difference images, and the street scene dynamic features for all street scene difference images are extracted, so that the street scene understanding model can perceive the dynamic changes of the street scene environment based on the street scene dynamic features and understand the dynamic elements in the street scene environment. Due to the bad weather conditions, the current street scene image frame cannot provide accurate environmental information. Through the street scene instantaneous features, street scene stable features and street scene dynamic features, the street scene understanding model can perceive the time correlation features between multiple consecutive image frames and provide redundant information for the current street scene image frame to improve the accuracy of street scene understanding. In order to avoid the street scene understanding model only learning the street scene semantic information between specific image sequences and improve the generalization and robustness of the street scene understanding model, the street scene instantaneous features, street scene stable features and street scene dynamic features are randomly mixed to obtain mixed street scene features. In order to obtain a street view model that can cope with various adverse weather conditions, the preset initial street view understanding model is used to perform street view semantic segmentation on the mixed street view features to obtain the predicted street view mask of the current street view image frame, so as to train the model based on the predicted street view mask. When the difference between multiple consecutive image frames is small, it will make it difficult to label multiple consecutive image frames. In order to reduce the difficulty of labeling, a large visual model is introduced. The real mask of the mixed street view features is generated by the large visual model to obtain a sample street view mask. Based on the preset optimizer, the model parameters of the initial street view understanding model are updated according to the predicted street view mask and the sample street view mask to perform model training and obtain the target street view understanding model, so that the street view environment around the vehicle can be accurately understood based on the target street view understanding model, thereby improving the safety of vehicle driving. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 is a flow chart of a method for training a street scene understanding model assisted by a large visual model provided in an embodiment of the present application;

[0049] Figure 2 yes Figure 1 Flow chart of step S160 in FIG.

[0050] Figure 3 yes Figure 1 Flowchart of step S170 in FIG.

[0051] Figure 4 is another flow chart of a method for training a street scene understanding model assisted by a large visual model provided in an embodiment of the present application;

[0052] Figure 5 yes Figure 4 Flow chart of step S420 in FIG.

[0053] Figure 6 yes Figure 5 Flow chart of step S520 in FIG.

[0054] Figure 7 is a flow chart of a vehicle control method provided by an embodiment of the present application;

[0055] Figure 8 It is a rendering of a method for training a street scene understanding model assisted by a large visual model provided in an embodiment of the present application;

[0056] Fig. 9 This is another effect diagram of the training method of the street scene understanding model assisted by a large visual model provided in an embodiment of the present application;

[0057] Fig.10 is a schematic diagram of the structure of a vehicle control device provided in an embodiment of the present application;

[0058] Fig.11 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0060] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0061] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0062] In the field of autonomous driving, vehicles collect street view images and use built-in street view understanding models to identify various elements in street view images, such as pedestrians and traffic signs, to control the driving of the vehicle. However, when designing the street view understanding model, the impact of sunny weather conditions on vehicle driving performance was mainly considered, and the impact of bad weather conditions was not fully considered. When the vehicle encounters bad weather such as rain, snow, fog, etc., the street view understanding model's recognition accuracy of street view images will decrease, which will in turn affect the driving of the vehicle.

[0063] Based on this, the embodiments of the present application provide a training method for a street view understanding model assisted by a large visual model, a vehicle control method, a vehicle control device, an electronic device and a computer-readable storage medium, aiming to improve the accuracy of street view understanding.

[0064] The training method of the street view understanding model based on the assistance of a large visual model, the vehicle control method, the vehicle control device, the electronic device and the computer-readable storage medium provided in the embodiments of the present application are specifically illustrated by the following embodiments. First, the training method of the street view understanding model based on the assistance of a large visual model in the embodiments of the present application is described.

[0065] The training method of the street view understanding model based on the assistance of a large visual model provided in the embodiment of the present application relates to the field of artificial intelligence technology. The training method of the street view understanding model based on the assistance of a large visual model provided in the embodiment of the present application can be applied to a terminal, can also be applied to a server side, and can also be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or can be configured as a server cluster or a distributed system composed of multiple physical servers, and can also be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the training method of the street view understanding model based on the assistance of a large visual model, etc., but is not limited to the above forms.

[0066] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0067] Figure 1 is an optional flowchart of a method for training a street scene understanding model based on a large visual model assistance provided in an embodiment of the present application. Figure 1 The method may include but is not limited to steps S110 to S170.

[0068] Step S110, acquiring street view image data; the street view image data includes a current street view image frame and a plurality of continuous historical street view image frames before the current street view image frame;

[0069] Step S120, extracting instantaneous street view features for the current street view image frame, and extracting stable and invariant street view features for all historical street view image frames;

[0070] Step S130, respectively obtaining the difference between the current street view image frame and each historical street view image frame to obtain a plurality of street view difference images, and extracting street view dynamic features for all street view difference images;

[0071] Step S140, randomly mixing the instantaneous street scene features, the stable and unchanging street scene features, and the dynamic street scene features to obtain mixed street scene features;

[0072] Step S150, performing street view semantic segmentation on the mixed street view features by using a preset initial street view understanding model to obtain a predicted street view mask of the current street view image frame;

[0073] Step S160, generating a real mask of mixed street scene features through a preset large visual model to obtain a sample street scene mask;

[0074] Step S170, based on a preset optimizer, the model parameters of the initial street view understanding model are updated according to the predicted street view mask and the sample street view mask to obtain a target street view understanding model; the target street view understanding model is used to control the driving of the target vehicle according to the features corresponding to the street view image data around the target vehicle.

[0075] In step S110 of some embodiments, during the offline training phase, when the target object is driving the vehicle, it is necessary to collect image information of the environment on the way the vehicle passes, and organize and save this information. Each vehicle terminal uses the camera carried to collect image data of the environment around the vehicle to obtain street view image data. The street view image data contains visual information of the road and the environment around the vehicle, such as weather conditions, road conditions, traffic flow, obstacles, etc. The street view image data is an image sequence, which is used to train the local street view understanding model of the vehicle terminal, including multiple continuous street view image frames, and the multiple street view image frames include the current street view image frame and multiple continuous historical street view image frames before the current street view image frame. Among them, the current street view image frame is a street view image collected at the current time, and the historical street view image frame is a street view image collected before the current time. For example, the street view image data includes t+1 street view images, and the street view image collected at the time of T represents the current street view image frame, and the street view images collected at the time of T-1, T-2, ..., Tt represent multiple continuous historical street view image frames. The street scene understanding model is used to identify the street scene environment around the vehicle, so that the vehicle terminal can output control instructions. The control instructions are used to control the vehicle's steering, acceleration, braking, shifting, windows, etc., to achieve automatic driving of the vehicle.

[0076] It should be noted that cameras of various models or parameters can be used to collect street view images. Street view images can be images of various resolutions, such as 1920x1080, 960x640, etc. Different three-color spaces such as RGB, Gray, CMY, HSV, HIS, Lab, YUV, etc. can also be used to represent street view images. The image collection time interval can also be set, such as 500ms, 1s, 1.5s, etc., and street view images can be collected according to the collection time interval. Different fields of view can be set for the camera, such as 120°, etc. In addition to the above-mentioned various models and parameters, the configuration of the camera on the vehicle terminal can have various optional combinations. For example, multiple or single cameras can be set; the camera can be configured at the front of the vehicle, or one or more cameras can be configured at the front, left, and right of the vehicle; the pitch angle (pitch), heading angle (yaw), and roll angle (roll) of the camera can also be appropriately adjusted to collect various required environmental information.

[0077] It should be further explained that when collecting image data, it is necessary not only to collect a large amount of data, but also to pay attention to the richness of the collected data, that is, the collected data should include various situations. In theory, the richer the collected data, the better the performance of the autonomous driving system. If a large amount of data is collected, but the data is very single, the final performance of the autonomous driving system will be very poor.

[0078] In step S120 of some embodiments, in order to enable the street view understanding model to cope with adverse weather conditions and ensure the driving safety of the vehicle under adverse weather conditions, a domain adaptive method is used to use the street view image taken under clear weather conditions as a reference image, and to reduce the difference between the street view image taken under adverse weather conditions and the reference image. However, the street view of autonomous driving varies greatly, and the weather is various, such as cloudy, foggy, rainy, dark, etc. It is impossible to stop the vehicle at the same location to capture street view images under various weather conditions, making it difficult to obtain reference images. In addition, the domain adaptive method can often only cope with a single adverse condition such as foggy weather, and cannot adapt to unknown weather changes or a combination of multiple adverse conditions, resulting in reduced accuracy of street view understanding. In order to reduce the dependence of the street view understanding model on the reference image and improve the accuracy of street view understanding, the embodiment of the present application introduces a local sequential mechanism (Locally Sequential Mechanism, LSM) to extract local temporal correlations between consecutive street view image frames. Under severe weather conditions, the quality of street view images collected by vehicles will be reduced. Comprehensive street view features can be obtained through local temporal correlation, so that the street view understanding model can analyze the street view features under similar weather conditions, and then learn the street view changes that may occur under specific weather conditions, so that the street view understanding model can be independent of the reference image and robust under various severe weather conditions.

[0079] The local order mechanism includes three different units: instant unit (InsU), integral unit (IntU) and derivative unit (DU). Among them, the instant unit is used to detect instantaneous information in the input, the integral unit is used to capture stable and shared background information in the street view environment, and the derivative unit is used to perceive dynamic changes in the street view environment. The instant unit, integral unit and derivative unit can all use pre-trained neural network models, such as convolutional neural network, recurrent neural network, long short-term memory network, autoencoder, etc. For the integral unit and derivative unit, the depth of the local order mechanism, that is, the number of consecutive historical street view image frames, is a hyperparameter. The hyperparameters of the two units can be the same or different.

[0080] The instantaneous unit is designed to process the current image captured by the on-board camera during the vehicle's driving process. The instantaneous unit extracts features from the current street view image frame to extract real-time information from the image and obtain street view instantaneous features. Street view instantaneous features are features of the street view environment around the vehicle, such as vehicle and pedestrian flow, traffic signal status, weather changes (such as rain, snow, fog), lighting changes (such as sunrise, sunset), etc. Through street view instantaneous features, the vehicle can perceive changes in the surrounding street view environment in real time and accurately understand the surrounding environment, and then adopt a safe and suitable control strategy for driving conditions.

[0081] The integration unit extracts stable background information from the street view environment by analyzing multiple consecutive historical street view image frames before the current street view image frame. All historical street view image frames are input into the integration unit, and the integration unit extracts features from all historical street view image frames to obtain stable and invariant features of the street view. Stable and invariant features of the street view are features that remain unchanged or change very little in all historical street view image frames, such as roads, traffic signs, buildings, etc. Through the stable and invariant features of the street view, the vehicle can identify and understand elements that remain unchanged over time and establish a consistent understanding of the street view environment to distinguish between static objects and moving objects, thereby enhancing its ability to navigate safely.

[0082] In step S130 of some embodiments, the derivative unit analyzes the dynamic changes of the street view environment by comparing the current street view image frame with the historical street view image frame. The current street view image frame is spatially aligned with each historical street view image frame, and the pixel values ​​of the pixels in the current street view image frame are subtracted from the pixel values ​​of the pixels at the corresponding positions in each historical street view image frame to obtain the difference between the current street view image frame and each historical street view image frame, thereby obtaining a plurality of street view difference images. All street view difference images are input into the derivative unit, and the street view dynamic features of all street view difference images are extracted by the derivative unit. The street view dynamic features are features used to reflect the dynamic changes between the current street view image frame and the historical street view image frame, such as the movement of vehicles and pedestrians, changes in weather conditions, changes in lighting conditions, changes in traffic flow, etc. Through the street view dynamic features, the vehicle can identify the differences in the street view environment caused by dynamic elements such as moving objects (such as vehicles and pedestrians), weather changes, and changes in lighting, so that it can quickly respond to new street view changes to maintain the safety and efficiency of operation.

[0083] In step S140 of some embodiments, since LSM integrates the temporal pattern of historical continuous frames, the street view understanding model is prone to overfitting to a specific input sequence order, i.e., a specific temporal pattern. In order to alleviate the time-related overfitting problem of LSM, the embodiment of the present application introduces a global shuffle mechanism (GSM) to rearrange the fragments processed by LSM. By randomly mixing the instantaneous features of the street view, the stable and unchanging features of the street view, and the dynamic features of the street view through GSM, a mixed street view feature is obtained, so that the street view understanding model can not only learn the features of a street view image sequence in a specific order, but also learn a more general, more robust and generalizable pattern across different street view image sequences, which helps to avoid the problem of overfitting of the temporal pattern, improves the performance and generalization ability of the model, accelerates the convergence speed of the model, and enhances the model's adaptability to changes in the order of data, thereby establishing a more reliable and effective street view understanding model.

[0084] Specifically, the instantaneous features of the street scene are divided into a plurality of feature segments to obtain a plurality of first feature segments. The stable and invariant features of the street scene are divided into a plurality of feature segments to obtain a plurality of second feature segments. The dynamic features of the street scene are divided into a plurality of feature segments to obtain a plurality of third feature segments. The feature segments are randomly exchanged between the first feature segments, between the first feature segments and the second feature segments, between the second feature segments, between the second feature segments and the third feature segments, and / or between the first feature segments and the third feature segments to obtain a mixed street scene feature. The mixed street scene feature includes the instantaneous features of the street scene after the random mixing operation, the stable and invariant features of the street scene, and the dynamic features of the street scene.

[0085] In step S150 of some embodiments, LSM has two fusion methods, early fusion and late fusion, wherein early fusion and late fusion are two strategies for fusing instantaneous features of street scenes, stable and unchanging features of street scenes, and dynamic features of street scenes. In the early fusion method, the features from the instantaneous unit, the integral unit, and the derivative unit are first merged, and then the merged features are processed by the shared model. Early fusion enables the model to utilize the correlation between these different types of features from the beginning, thereby enhancing the model's ability to understand the street scene. In the late fusion method, three independent models are used to process the features from the instantaneous unit, the integral unit, and the derivative unit respectively, and after the processing is completed, these different types of features are integrated. Through late fusion, each type of feature can be individually analyzed in detail, which helps to retain the unique characteristics of each type of information.

[0086] If the early fusion strategy is adopted, the initial street view understanding model uses a convolutional neural network to input the mixed street view features into the shared initial street view understanding model for street view semantic segmentation, so that the initial street view understanding model can fully learn the association between different types of features, enhance the model's street view semantic segmentation ability, and obtain the predicted street view mask of the current street view image frame. The predicted street view mask is a mask generated by the initial street view understanding model, and the predicted street view mask carries object category information.

[0087] If the late fusion strategy is adopted, the initial street view understanding model adopts a convolutional neural network + a semantic segmentation head, and the street view instantaneous features, the street view stable and unchanging features, and the street view dynamic features in the mixed street view features are respectively input into three independent convolutional neural networks for feature extraction, so as to obtain a first feature map corresponding to the street view instantaneous features, a second feature map corresponding to the street view stable and unchanging features, and a third feature map corresponding to the street view dynamic features. The first feature map, the second feature map, and the third feature map are merged, and the merged features are semantically segmented by the semantic segmentation head to obtain the predicted street view mask of the current street view image frame.

[0088] In the related art, supervised learning is used to train street view understanding models, which requires obtaining label information of a large amount of street view data. However, in the field of autonomous driving, the difference between consecutive street view image frames is small, which makes it more difficult to annotate consecutive image frames. In order to reduce the difficulty of annotating consecutive image frames, the embodiment of the present application introduces a large visual model, which is used to generate a real mask of mixed street view features, and obtain a sample street view mask, so that the model can be trained by unsupervised learning.

[0089] The process of generating labels for large visual models involves multiple complex steps to ensure that the generated labels are accurate and applicable to a variety of application scenarios. First, the image data needs to be preprocessed, including scaling to a uniform size, normalizing pixel values, and increasing data diversity through data augmentation techniques such as random cropping, rotation, and flipping. These preprocessing steps help improve the robustness of the model and the efficiency of training.

[0090] Next, the preprocessed image is fed into a large vision model. These models are usually composed of multiple convolutional layers, pooling layers, and other structures to extract high-level features of the image. These features capture key information in the image, such as edges, textures, and shapes, allowing the model to better understand the image content.

[0091] After feature extraction, the model converts the extracted features into feature vectors of specific dimensions through fully connected layers or other dimensionality reduction techniques. These feature vectors are the core of the model's understanding and representation of image content. Based on this, the model generates labels for specific tasks. For image classification tasks, the feature vectors are input to a classifier (such as a Softmax layer), which outputs the probability of each class based on the value of the feature vector and selects the class with the highest probability as the final label. For other tasks, such as object detection and semantic segmentation, the label generation process is different. In object detection, the model needs to generate not only the class label but also the bounding box coordinates of each object; in semantic segmentation, the model needs to generate a class label for each pixel. In order to improve the accuracy of these labels, post-processing steps are usually performed. For example, in object detection, non-maximum suppression (NMS) is used to remove redundant detection boxes and only retain the best detection results; smoothing the labels can reduce noise.

[0092] Finally, the labels generated through these steps are used as the output of the model and can be applied to various scenarios, such as target detection in autonomous driving, face recognition in security monitoring, lesion detection in medical imaging, etc. This process ensures that the model can provide accurate and reliable labels in different environments and conditions, thereby supporting complex visual tasks and applications.

[0093] See also Figure 2 In some embodiments, step S160 may include but is not limited to steps S210 to S240:

[0094] Step S210, extracting the mixed street scene features through a large visual model to obtain a potential feature map; the potential feature map includes a plurality of first pixel points;

[0095] Step S220, determining the number of cluster centers and the number of iterations, and initializing each cluster center according to the number;

[0096] Step S230, for each first pixel point, calculating the similarity between the first pixel point and each cluster center, and classifying the first pixel point to the cluster center with the highest similarity;

[0097] Step S240, for each cluster center, iteratively update the cluster center according to all first pixel points classified into the cluster center, until the number of iterative updates reaches the number of iterations, or the cluster center converges, to obtain a sample street view mask.

[0098] In step S210 of some embodiments, the big vision model adopts a semantic segmentation model (SemanticAttention Module, SAM), and uses the powerful zero-sample reasoning ability of the big vision model to generate masks for mixed street scene features, uses the generated mask as a potential feature map, and uses the pixel point in the potential feature map as the first pixel point.

[0099] In step S220 of some embodiments, although the semantic segmentation model can generate a mask, it cannot generate object category information of objects such as pedestrians and vehicles. The embodiment of the present application uses a clustering algorithm to generate a mask carrying object category information. Initialize the number of cluster centers and the maximum number of iterations, and randomly select the number of first pixels from the potential feature map as each cluster center, and assign an index to each cluster center, which is used to characterize the object category information.

[0100] In step S230 of some embodiments, for each first pixel, the inner product calculation is performed on the coordinates of the first pixel and the coordinates of each cluster center to obtain the similarity between the first pixel and each cluster center, the first pixel is classified to the cluster center with the highest similarity, and the index of the cluster center is assigned to the first pixel. The index of the first pixel classified to the cluster center is the same as the index of the cluster center. If the coordinates of the first pixel are represented as X and the cluster center is represented as centroid, the similarity between the first pixel and the cluster center is represented as , T represents transpose.

[0101] In step S240 of some embodiments, for each cluster center, the cluster center is iteratively updated according to all first pixel points classified into the cluster center. The cluster center update method is expressed as:

[0102] ,

[0103] Wherein, j represents the index; represents the set of first pixel points classified into cluster center j; Represents the pixel number of the first pixel in the set; Represents the coordinates of the i-th first pixel in the set.

[0104] Repeat steps S230 to S240 until the number of iterative updates reaches the maximum number of iterations, or the cluster centers converge, that is, the coordinates of the cluster centers remain unchanged or change very little, to obtain a sample street view mask. In the sample street view mask, the first pixel points classified to the same cluster center belong to the same category and have the same index (sample category label); conversely, the first pixel points classified to different cluster centers belong to different categories and have different indexes.

[0105] Through the above steps S210 to S240, a mask carrying object category information can be obtained, so that the street view understanding model can distinguish objects of different categories in the street view image, improve the model's street view semantic understanding ability, and further enhance the vehicle's automatic navigation ability.

[0106] The predicted street view mask and the sample street view mask are input into the optimizer for loss calculation, and the model parameters of the initial street view understanding model are updated by minimizing the loss to obtain the target street view understanding model, which is used to control the target vehicle to drive according to the features corresponding to the street view image data around the target vehicle. The optimizer is an algorithm program used to adjust the model parameters to minimize the loss function, including the Adam optimizer, the RMSprop optimizer, etc. The optimizer includes the first loss function, the second loss function, and the third loss function. The first loss function is an image-level regularizer, the second loss function is an inter-class contrast regularizer, and the third loss function is a pixel-level cross entropy loss. The complexity of the model is penalized by two regularizers to improve the generalization ability of the model under different severe weather conditions.

[0107] See also Figure 3 In some embodiments, step S170 may include but is not limited to steps S310 to S340:

[0108] Step S310, based on the first loss function, performing a first loss calculation according to the predicted street view mask and the sample street view mask to obtain an image pixel loss; the predicted street view mask has a predicted category label, and the sample street view mask has a sample category label;

[0109] Step S320, based on the second loss function, performing a second loss calculation according to the predicted street view mask to obtain an inter-class contrast loss;

[0110] Step S330, based on the third loss function, performing a third loss calculation according to the predicted category label and the sample category label to obtain a category loss;

[0111] Step S340, updating the model parameters of the initial street scene understanding model according to the image pixel loss, the inter-class contrast loss and the category loss to obtain the target street scene understanding model.

[0112] In step S310 of some embodiments, the first loss function is expressed as:

[0113] ,

[0114] Wherein, X represents the current street view image frame; Represents the probability distribution of pixel x in the predicted street view mask; Represents the probability distribution of pixel x in the sample street view mask.

[0115] Substitute the predicted street view mask and the sample street view mask into the first loss function, calculate the Bhattacharyya distance between the predicted street view mask and the sample street view mask, and obtain the image pixel loss. The predicted street view mask has a predicted category label, and the sample street view mask has a sample category label. The predicted category label is the object category of each object in the current street view image frame predicted by the street view understanding model, and the sample category label is the actual object category of each object in the current street view image frame.

[0116] In step S320 of some embodiments, the second loss function is expressed as:

[0117] ,

[0118] Where |u| represents the total number of pixels in the positive example set, and the category of each pixel in the positive example set is the same as the category of pixel x; Represents the similarity measure between pixel x and the mth pixel in the positive example set, such as cosine similarity; represents the temperature parameter; |v| represents the total number of pixels in the negative example set, and the category of each pixel in the negative example set is different from the category of pixel x; Represents the similarity measure between pixel x and the kth second pixel in the negative example set, such as cosine similarity; Represents a multiplication operation.

[0119] A pixel point is randomly selected from the predicted street view mask, and other pixels in the predicted street view mask belonging to the same category as the pixel point are obtained to obtain a positive example set, and pixels in the predicted street view mask belonging to different categories from the pixel point are obtained to obtain a negative example set. The pixel value of the pixel point, the pixel value of the pixel point in the positive example set, and the pixel value of the pixel point in the negative example set are substituted into the second loss function to calculate the inter-class contrast loss, and the inter-class contrast loss is obtained.

[0120] In step S330 of some embodiments, the third loss function is expressed as:

[0121] ,

[0122] Wherein, i represents the i-th pixel in the current street view image frame; Represents the sample category label of the i-th pixel; Represents the predicted category label of the i-th pixel.

[0123] Substitute the predicted category label of each pixel in the predicted street view mask and the sample category label of the pixel at the corresponding position in the sample street view mask into the third loss function to perform the third loss calculation to obtain the category loss.

[0124] In step S340 of some embodiments, the image pixel loss, the inter-class contrast loss, and the class loss are weighted to obtain target loss data, and the target loss data is minimized by an optimizer to update the model parameters of the initial street view understanding model to obtain a target street view understanding model. The calculation formula of the target loss data is defined as:

[0125] ,

[0126] in, and are all weight parameters.

[0127] The above steps S310 to S340, by introducing an image-level regularizer and an inter-class contrast regularizer, can enhance the generalization of the model to cope with different weather conditions, and obtain a target street scene understanding model that can accurately understand the semantics of street scenes.

[0128] See also Figure 4 In some embodiments, after step S160, the training method of the street view understanding model assisted by the large visual model may also include but is not limited to steps S410 to S440:

[0129] Step S410, expanding the preset optimizer into multiple network layers;

[0130] Step S420, inputting the predicted street view mask and the sample street view mask into the first network layer, updating the predicted street view mask through the first network layer, and obtaining a reference street view mask output by the first network layer;

[0131] Step S430, inputting the reference street view mask and the sample street view mask output by the previous network layer into the current network layer, and updating the reference street view mask output by the previous network layer through the current network layer to obtain the reference street view mask output by the current network layer;

[0132] Step S440, until the reference street view mask output by the last network layer is obtained, the reference street view mask output by the last network layer is used as the mask of the current street view image frame.

[0133] In step S410 of some embodiments, in order to avoid instability and suboptimal performance of the model during training, the optimizer is expanded into multiple network layers through the deep expansion technology, that is, the image-level regularizer and the inter-class contrast regularizer are expanded into layers, and the iterative training process of the optimizer on the model is converted into an update process of the predicted street view mask by multiple network layers, thereby avoiding heuristic or exhaustive search of the weights of the two regularizers, thereby preventing instability and suboptimal performance. The number of network layers is equal to the number of iterative training times. If the number of iterative training times is L, the number of network layers is L.

[0134] In step S420 of some embodiments, the predicted street view mask and the sample street view mask are input into the first network layer, and the predicted street view mask is updated by the first network layer to obtain a reference street view mask output by the first network layer. The reference street view mask is the updated predicted street view mask.

[0135] In step S430 of some embodiments, the reference street view mask and the sample street view mask output by the previous network layer are input to the current network layer, and the gradient of the first loss function for the pixel points of the reference street view mask and the gradient of the second loss function for the pixel points of the reference street view mask are calculated by the current network layer. According to the two gradients, the weight parameter for the first loss function, the weight parameter for the second loss function, and the update step size, the pixel value of the pixel point (anchor point) of the reference street view mask output by the previous network layer is updated until all the pixel points in the reference street view mask output by the previous network layer are updated, and the reference street view mask output by the current network layer is obtained. The reference street view mask is updated as follows:

[0136] ,

[0137] in, l Indicates l Network layers; Indicates l The pixel value of the reference street view mask output by the network layer; Represents the pixel point of the first loss function for the reference street view mask The gradient of Represents the pixel point of the second loss function for the reference street view mask The gradient of Indicates l The weight parameters of the network layer for the first loss function; Indicates l The weight parameters of the network layer for the second loss function; Indicates l The update step size of each network layer.

[0138] In step S440 of some embodiments, step S430 is repeated until the reference street view mask output by the last network layer is obtained, and the reference street view mask output by the last network layer is used as the mask of the current street view image frame. The driving of the target vehicle is controlled according to the mask of the current street view image frame.

[0139] Through the above steps S410 to S440, the mask of the current street view image frame can be directly obtained, which speeds up the semantic understanding of the street view without spending a lot of time on model training, thereby ensuring the real-time control of the vehicle under severe weather conditions.

[0140] See also Figure 5 In some embodiments, step S420 may also include but is not limited to steps S510 to S530:

[0141] Step S510, determining the gradient of a first loss function according to the predicted street view mask and the sample street view mask to obtain a first gradient;

[0142] Step S520, determining the gradient of the second loss function according to the predicted street view mask to obtain a second gradient;

[0143] Step S530: updating the predicted street view mask according to the first gradient and the second gradient to obtain a reference street view mask.

[0144] In step S510 of some embodiments, the predicted street view mask includes a plurality of pixels, and the pixels of the predicted street view mask are used as second pixels. According to the pixel values ​​of each second pixel in the predicted street view mask and the pixel values ​​of the pixels at the corresponding positions in the sample street view mask, the gradient of the first loss function with respect to the pixel is determined to obtain a first gradient. The calculation method of the first gradient is as follows:

[0145] ,

[0146] Among them, x represents the pixel point; represents the first loss function; represents the gradient of the first loss function to the pixel point; X represents the current street view image frame; Represents the probability distribution of pixel x in the predicted street view mask; Represents the probability distribution of pixel x in the sample street view mask.

[0147] In step S520 of some embodiments, a second pixel point is selected from multiple second pixel points of the predicted street view mask as an anchor point, and the gradient of the second loss function with respect to the anchor point is determined according to the predicted street view mask to obtain a second gradient.

[0148] In step S530 of some embodiments, the anchor point of the predicted street view mask is updated according to the first gradient and the second gradient, and each second pixel point of the predicted street view mask is used as an anchor point in turn until all anchor points are updated to obtain a reference street view mask output by the first network layer. The anchor point update method is as follows:

[0149] ,

[0150] in, represents the anchor point of the predicted street view mask; , and represents the learnable parameters of the first network layer, and represents the weight parameter, represents the update step size; represents the gradient of the first loss function with respect to the anchor point, that is, the gradient of the first loss function with respect to the pixel point x; Represents the gradient of the second loss function with respect to the anchor point; represents the updated anchor point output by the first network layer, that is, the pixel point corresponding to the anchor point in the reference street view mask.

[0151] Through the above steps S510 to S530, the predicted street view mask can be optimized, thereby improving the accuracy of street view semantic understanding.

[0152] See also Figure 6 In some embodiments, step S520 may include but is not limited to steps S610 to S630:

[0153] Step S610, selecting a second pixel point from a plurality of second pixel points of the predicted street view mask as an anchor point;

[0154] Step S620, obtaining all second pixel points that belong to the same category as the anchor point to obtain a positive example set, and obtaining all second pixel points that belong to different categories from the anchor point to obtain a negative example set;

[0155] Step S630, determining the gradient of the second loss function according to the anchor point, the positive example set, and the negative example set to obtain a second gradient.

[0156] In step S610 of some embodiments, a second pixel point is randomly selected from a plurality of second pixel points of the predicted street view mask as an anchor point. Each second pixel point in the predicted street view mask can be used as an anchor point.

[0157] In step S620 of some embodiments, all second pixel points in the predicted street view mask that belong to the same category as the anchor point are included in a set to obtain a positive example set. The positive example set does not include the anchor point. All second pixel points in the predicted street view mask that belong to different categories from the anchor point are included in a set to obtain a negative example set.

[0158] In step S630 of some embodiments, the gradient of the second loss function with respect to the anchor point is determined according to the anchor point, the positive example set, and the negative example set to obtain a second gradient. The second gradient is calculated as follows:

[0159] ,

[0160] Where x represents the anchor point, represents the second loss function; Represents the gradient of the second loss function with respect to the anchor point; A and B are both intermediate parameters; and Represent the gradients of A and B with respect to the anchor point x respectively.

[0161] A, B, and The calculation method is as follows:

[0162] ,

[0163] ,

[0164] ,

[0165] ,

[0166] Where |u| represents the number of second pixels in the positive example set; Represents the similarity measure between the anchor point and the mth second pixel in the positive example set, such as cosine similarity; represents the temperature parameter; |v| represents the number of second pixels in the negative example set; Represents the similarity measure between the anchor point and the k-th second pixel in the negative example set, such as cosine similarity; Represents a multiplication operation.

[0167] Through the above steps S610 to S630, a second gradient can be obtained, so as to optimize the predicted street view mask based on the second gradient, thereby improving the accuracy of street view understanding.

[0168] Figure 7 is an optional flow chart of the vehicle control method provided in the embodiment of the present application. Figure 7 The method may include but is not limited to steps S710 to S740:

[0169] Step S710, acquiring a target street view image around a target vehicle within a target time period to obtain street view image data;

[0170] Step S720, performing mixed feature extraction on the street view image data to obtain mixed street view features;

[0171] Step S730, performing street view semantic segmentation on the mixed street view features through a target street view understanding model to obtain a target street view mask; the target street view understanding model is trained according to a training method for a street view understanding model assisted by a large visual model;

[0172] Step S740: Control the target vehicle to travel according to the target street view mask.

[0173] In step S710 of some embodiments, for the online reasoning stage, the target street view understanding model obtained in the offline training stage is deployed on the target vehicle, and the same system hardware and software configuration as that used to collect street view image data is used. During the movement of the target vehicle, the target street view image around the target vehicle in the target time period is acquired through the camera to obtain street view image data. The target vehicle is a vehicle to be controlled for driving, and the target time period is a time period when the target vehicle needs automatic driving or autonomous decision-making, such as a time period when bad weather occurs, a peak period of traffic congestion, etc. The target time period covers the current time and multiple continuous historical times between the current time. The target street view image is a street view image of the vehicle surrounding the target vehicle collected during the target time period. The street view image data includes the target street view image at the current time and multiple continuous target street view images before the current time.

[0174] In step S720 of some embodiments, referring to steps S120 to S140, the instantaneous unit extracts features from the target street view image at the current time to obtain street view instantaneous features, the integral unit extracts features from multiple consecutive target street view images before the current time to obtain street view stable and invariant features, and the difference between the target street view image at the current time and each target street view image before the current time is obtained to obtain multiple street view difference images. The derivative unit extracts features from all street view difference images to obtain street view dynamic features. According to the GSM random mixing of street view instantaneous features, street view stable and invariant features, and street view dynamic features, a mixed street view feature is obtained.

[0175] In step S730 of some embodiments, the mixed street view features are input into the target street view understanding model to perform street view semantic segmentation, and a mask image of the target street view image at the current time is obtained, and the mask image is used as the target street view mask.

[0176] In step S740 of some embodiments, a driving control instruction is output according to the target street view mask, and the target vehicle is controlled to drive according to the driving control instruction. The driving control instruction is an instruction or signal for controlling the driving state of the vehicle, including instructions such as acceleration, deceleration, steering, and shifting.

[0177] In the above steps S710 to S740, the target street view understanding model can accurately understand the street view semantics under various weather conditions including severe weather conditions, and obtain the target street view mask, so that the vehicle can distinguish different objects and areas in the street view image according to the target street view mask, such as distinguishing roads, buildings, pedestrians, vehicles, etc., and then control the vehicle driving to ensure the safety of the vehicle driving.

[0178] The training method of the street scene understanding model based on the assistance of a large visual model in the embodiment of the present application takes into account local temporal correlation to enhance model performance, while taking into account global randomness to prevent overfitting of specific temporal patterns. In addition to global randomness, the embodiment of the present application also introduces regularization technology to enhance the generalization ability of the model under different weather conditions through image-level regularizers and inter-class contrast regularizers to prevent over-reliance on specific temporal patterns of adjacent frames. The model performance of the target street scene understanding model was tested on the Apolloscapes dataset and the carla simulation dataset, respectively, and the four indicators of mean intersection over Union (mIoU), mean precision (mPre), mean recall (mRecall, mRec) and mean F1 score (mF1) were used to compare the effects of the target street scene understanding model and the DeepLabv3+ model on these two datasets. The effects of the target street scene understanding model and the DeepLabv3+ model on the Apolloscapes dataset are shown in the figure. Figure 8 As shown in Figure 2, the effects of the target street scene understanding model and the DeepLabv3+ model on the carla simulation dataset are as follows: Fig. 9 As shown. Figure 8 and Fig. 9 In the figure, the target street scene understanding model is denoted as AdvImmu, the left bar graph of the double bar graph is the performance of the DeepLabv3+ model, and the right bar graph is the performance of the AdvImmu model. Figure 8 and Fig. 9 It can be seen that the average intersection-over-union ratio, average precision, average recall rate and average F1 score of AdvImmu are all higher than those of the DeepLabv3+ model. It can effectively identify various objects in street scenes under different weather conditions, has good generalization performance under various severe weather conditions, and ensures the accuracy of street scene semantic understanding.

[0179] See also Fig.10 The embodiment of the present application further provides a vehicle control device, which is applied to a terminal device and can implement the above-mentioned vehicle control method. The device includes:

[0180] An acquisition module 1010 is used to acquire a target street view image around a target vehicle within a target time period to obtain street view image data;

[0181] A feature extraction module 1020 is used to extract mixed features from the street view image data to obtain mixed street view features;

[0182] The street view understanding module 1030 is used to perform street view semantic segmentation on the mixed street view features through a target street view understanding model to obtain a target street view mask; the target street view understanding model is trained according to a training method of a street view understanding model assisted by a large visual model;

[0183] The control module 1040 is used to control the target vehicle to travel according to the target street view mask.

[0184] In some embodiments, the control module 1040 is further configured to:

[0185] Acquire street view image data; the street view image data includes a current street view image frame and a plurality of continuous historical street view image frames before the current street view image frame;

[0186] Extracting instantaneous street view features for a current street view image frame, and extracting stable and invariant street view features for all historical street view image frames;

[0187] respectively obtaining the difference between the current street view image frame and each historical street view image frame to obtain a plurality of street view difference images, and extracting street view dynamic features for all street view difference images;

[0188] The instantaneous features of the street scene, the stable and unchanging features of the street scene and the dynamic features of the street scene are randomly mixed to obtain mixed street scene features;

[0189] Performing street view semantic segmentation on the mixed street view features through a preset initial street view understanding model to obtain a predicted street view mask of the current street view image frame;

[0190] Generate a real mask of mixed street scene features through a preset large visual model to obtain a sample street scene mask;

[0191] Based on a preset optimizer, model parameters of the initial street view understanding model are updated according to the predicted street view mask and the sample street view mask to obtain a target street view understanding model; the target street view understanding model is used to control the driving of the target vehicle according to features corresponding to street view image data around the target vehicle.

[0192] In some embodiments, the control module 1040 is further configured to:

[0193] Extracting features of the mixed street scene features through a large visual model to obtain a potential feature map; the potential feature map includes a plurality of first pixel points;

[0194] Determine the number of cluster centers and the number of iterations, and initialize each cluster center according to the number;

[0195] For each first pixel point, calculate the similarity between the first pixel point and each cluster center, and classify the first pixel point into the cluster center with the highest similarity;

[0196] For each cluster center, the cluster center is iteratively updated according to all first pixel points classified into the cluster center until the number of iterative updates reaches the number of iterations, or the cluster center converges, and a sample street view mask is obtained.

[0197] In some embodiments, the control module 1040 is further configured to:

[0198] Based on the first loss function, a first loss calculation is performed according to the predicted street view mask and the sample street view mask to obtain an image pixel loss; the predicted street view mask has a predicted category label, and the sample street view mask has a sample category label;

[0199] Based on the second loss function, the second loss is calculated according to the predicted street view mask to obtain the inter-class contrast loss;

[0200] Based on the third loss function, the third loss is calculated according to the predicted category label and the sample category label to obtain the category loss;

[0201] The model parameters of the initial street scene understanding model are updated according to the image pixel loss, inter-class contrast loss and category loss to obtain the target street scene understanding model.

[0202] In some embodiments, the control module 1040 is further configured to:

[0203] Expand the preset optimizer into multiple network layers;

[0204] Input the predicted street view mask and the sample street view mask into the first network layer, update the predicted street view mask through the first network layer, and obtain the reference street view mask output by the first network layer;

[0205] The reference street view mask and the sample street view mask output by the previous network layer are input into the current network layer, and the reference street view mask output by the previous network layer is updated through the current network layer to obtain the reference street view mask output by the current network layer;

[0206] Until the reference street view mask output by the last network layer is obtained, the reference street view mask output by the last network layer is used as the mask of the current street view image frame.

[0207] In some embodiments, the control module 1040 is further configured to:

[0208] Determine a gradient of a first loss function according to the predicted street view mask and the sample street view mask to obtain a first gradient;

[0209] Determine the gradient of the second loss function according to the predicted street view mask to obtain a second gradient;

[0210] The predicted street view mask is updated according to the first gradient and the second gradient to obtain a reference street view mask.

[0211] In some embodiments, the control module 1040 is further configured to:

[0212] Selecting a second pixel point from a plurality of second pixel points of the predicted street view mask as an anchor point;

[0213] Get all the second pixel points that belong to the same category as the anchor point to get a positive example set, and get all the second pixel points that belong to different categories from the anchor point to get a negative example set;

[0214] The gradient of the second loss function is determined according to the anchor point, the positive example set, and the negative example set to obtain a second gradient.

[0215] The specific implementation of the vehicle control device is basically the same as the specific implementation of the above-mentioned vehicle control method, and will not be repeated here.

[0216] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned street scene understanding model training method or vehicle control method based on the assistance of a large visual model when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a vehicle-mounted computer, etc.

[0217] See also Fig.11 , Fig.11 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0218] The processor 1110 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0219] The memory 1120 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1120 can store an operating system and other applications. When the technical solution provided in the embodiments of this specification is implemented by software or firmware, the relevant program code is stored in the memory 1120, and the processor 1110 calls and executes the training method of the street view understanding model assisted by a large visual model or the vehicle control method of the embodiment of the present application;

[0220] Input / output interface 1130, used to implement information input and output;

[0221] The communication interface 1140 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WIFI, Bluetooth, etc.);

[0222] The bus 1150 transmits information between the various components of the device (e.g., the processor 1110, the memory 1120, the input / output interface 1130, and the communication interface 1140);

[0223] The processor 1110 , the memory 1120 , the input / output interface 1130 , and the communication interface 1140 are connected to each other in communication within the device via the bus 1150 .

[0224] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the above-mentioned street scene understanding model training method or vehicle control method based on the assistance of a large visual model.

[0225] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0226] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0227] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0228] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0229] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0230] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0231] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0232] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0233] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0234] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0235] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0236] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A method for training a street scene understanding model assisted by a large visual model, characterized in that: The method comprises: Acquire street view image data; the street view image data includes a current street view image frame and a plurality of continuous historical street view image frames before the current street view image frame; Extracting instantaneous street view features for the current street view image frame, and extracting stable and invariant street view features for all the historical street view image frames; respectively obtaining a difference between the current street view image frame and each of the historical street view image frames to obtain a plurality of street view difference images, and extracting street view dynamic features for all of the street view difference images; Randomly mixing the instantaneous street view features, the stable and unchanging street view features, and the dynamic street view features to obtain mixed street view features; Performing street view semantic segmentation on the mixed street view features through a preset initial street view understanding model to obtain a predicted street view mask of the current street view image frame; Generate a true mask of the mixed street scene feature through a preset large visual model to obtain a sample street scene mask; Based on a preset optimizer, model parameters of the initial street view understanding model are updated according to the predicted street view mask and the sample street view mask to obtain a target street view understanding model; the target street view understanding model is used to control the driving of the target vehicle according to features corresponding to street view image data around the target vehicle; The method of generating a real mask of the mixed street view feature by using a preset large visual model to obtain a sample street view mask includes: Extracting features of the mixed street scene features by using the large visual model to obtain a potential feature map; the potential feature map includes a plurality of first pixel points; Determine the number of cluster centers and the number of iterations, and initialize each cluster center according to the number; For each of the first pixel points, calculate the similarity between the first pixel point and each of the cluster centers, and classify the first pixel point into the cluster center with the highest similarity; For each cluster center, iteratively update the cluster center according to all first pixel points classified into the cluster center until the number of iterative updates reaches the number of iterations, or the cluster center converges, to obtain the sample street view mask; The optimizer includes a first loss function, a second loss function and a third loss function. The preset optimizer updates the model parameters of the initial street view understanding model according to the predicted street view mask and the sample street view mask to obtain a target street view understanding model, including: Based on the first loss function, a first loss calculation is performed according to the predicted street view mask and the sample street view mask to obtain an image pixel loss; the predicted street view mask has a predicted category label, and the sample street view mask has a sample category label; Based on the second loss function, performing a second loss calculation according to the predicted street view mask to obtain an inter-class contrast loss; Based on the third loss function, performing a third loss calculation according to the predicted category label and the sample category label to obtain a category loss; The model parameters of the initial street view understanding model are updated according to the image pixel loss, the inter-class contrast loss and the category loss to obtain the target street view understanding model.

2. The method according to claim 1, characterized in that After the real mask of the mixed street view feature is generated by the preset large visual model to obtain a sample street view mask, the method further includes: Expand the preset optimizer into multiple network layers; Inputting the predicted street view mask and the sample street view mask into a first network layer, updating the predicted street view mask through the first network layer, and obtaining a reference street view mask output by the first network layer; Inputting the reference street view mask and the sample street view mask output by the previous network layer into the current network layer, and updating the reference street view mask output by the previous network layer through the current network layer to obtain the reference street view mask output by the current network layer; Until the reference street view mask output by the last network layer is obtained, the reference street view mask output by the last network layer is used as the mask of the current street view image frame.

3. The method according to claim 2, characterized in that The optimizer includes a first loss function and a second loss function, and the predicted street view mask is updated through the first network layer to obtain a reference street view mask output by the first network layer, including: Determine the gradient of the first loss function according to the predicted street view mask and the sample street view mask to obtain a first gradient; Determine the gradient of the second loss function according to the predicted street view mask to obtain a second gradient; The predicted street view mask is updated according to the first gradient and the second gradient to obtain the reference street view mask.

4. The method according to claim 3, characterized in that The predicted street view mask includes a plurality of second pixel points, and determining the gradient of the second loss function according to the predicted street view mask to obtain a second gradient includes: Selecting one of the second pixel points of the predicted street view mask as an anchor point; Obtain all second pixel points that belong to the same category as the anchor point to obtain a positive example set, and obtain all second pixel points that belong to different categories from the anchor point to obtain a negative example set; The gradient of the second loss function is determined according to the anchor point, the positive example set, and the negative example set to obtain the second gradient.

5. A vehicle control method, characterized in that: The method comprises: Acquire a target street view image around a target vehicle within a target time period to obtain street view image data; Performing mixed feature extraction on the street view image data to obtain mixed street view features; Performing street view semantic segmentation on the mixed street view features through a target street view understanding model to obtain a target street view mask; the target street view understanding model is trained according to the training method of a street view understanding model based on the assistance of a large visual model according to any one of claims 1 to 4; The target vehicle is controlled to travel according to the target street view mask.

6. A vehicle control device, characterized in that: The device comprises: An acquisition module is used to acquire a target street view image around a target vehicle within a target period of time to obtain street view image data; A feature extraction module, used for performing mixed feature extraction on the street view image data to obtain mixed street view features; A street view understanding module, configured to perform street view semantic segmentation on the mixed street view features through a target street view understanding model to obtain a target street view mask; the target street view understanding model is trained according to the training method for a street view understanding model based on a large visual model as described in any one of claims 1 to 4; A control module is used to control the target vehicle to travel according to the target street view mask.

7. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the training method of the street scene understanding model based on the assistance of a large visual model as described in any one of claims 1 to 4 or the vehicle control method as described in claim 5 when executing the computer program.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, it implements the training method of a street scene understanding model based on the assistance of a large visual model as described in any one of claims 1 to 4 or the vehicle control method as described in claim 5.

Images