Large model application method, device, equipment and medium based on autonomous driving scenario

Abstract

The embodiments of the present application provide a large model application method, device, equipment and medium based on an autonomous driving scenario. The method includes: obtaining an image to be recognized from the current autonomous driving data; extracting features of the image to be recognized based on the visual backbone network of the pre-trained target visual model, and inputting the extracted features into the perception head of the target visual model to obtain a first feature to be recognized; performing random gradient descent processing on the first feature to be recognized through the posterior optimization trajectory generator in the target visual model to obtain a second feature to be recognized for indicating the subsequent reasoning direction of the target visual model; performing semantic recognition on the target feature to be recognized obtained by fusing the first feature to be recognized and the second feature to be recognized through the target visual model to obtain an image recognition result of the image to be recognized. In this way, the accuracy of image recognition by the model can be improved while improving the recognition efficiency of image recognition.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a large model application method, device, equipment and medium based on an autonomous driving scenario. Background Art

[0002] Autonomous driving refers to the use of mechatronics, artificial intelligence and multi-agent systems to achieve assisted or fully automatic control of vehicles such as cars, trucks, airplanes or water vehicles. Street scene semantic understanding is an important part of autonomous driving, which aims to accurately identify and segment pedestrians, obstacles, roads and landmarks, and provide necessary road information for autonomous driving technology.

[0003] In the related art, visual models are generally used to recognize street scene images. However, due to the power consumption and space limitations of the vehicle computing platform, the model may increase the delay in the actual process of street scene image recognition, thereby reducing the model's recognition efficiency for street scene images; on the other hand, due to the lack of adaptive capabilities of general visual models, the model's recognition accuracy for new environments is reduced when facing new environments. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to propose a large model application method, device, equipment and medium based on an autonomous driving scenario, which can improve the recognition efficiency of image recognition while improving the accuracy of the model in image recognition.

[0005] To achieve the above-mentioned purpose, a first aspect of an embodiment of the present application proposes a large model application method based on an autonomous driving scenario, the method comprising:

[0006] Obtain an image to be recognized from current autonomous driving data;

[0007] Extracting features of the image to be recognized according to the visual backbone network of the pre-trained target visual model, and inputting the extracted features into the perception head of the target visual model to obtain a first feature to be recognized;

[0008] Performing a random gradient descent process on the first feature to be identified by a posterior optimization trajectory generator in the target visual model to obtain a second feature to be identified for indicating a subsequent reasoning direction of the target visual model;

[0009] The target visual model is used to perform semantic recognition on the target feature to be recognized obtained by fusing the first feature to be recognized and the second feature to be recognized, so as to obtain an image recognition result of the image to be recognized.

[0010] Accordingly, a second aspect of an embodiment of the present application proposes a large model application device based on an autonomous driving scenario, the device comprising:

[0011] An acquisition module, used to acquire the image to be recognized from the current autonomous driving data;

[0012] An input module, used to extract features of the image to be identified according to the visual backbone network of the target visual model obtained by pre-training, and input the extracted features into the perception head of the target visual model to obtain a first feature to be identified;

[0013] A processing module, configured to perform a stochastic gradient descent process on the first feature to be identified through a posterior optimization trajectory generator in the target visual model, so as to obtain a second feature to be identified for indicating a subsequent reasoning direction of the target visual model;

[0014] The recognition module is used to perform semantic recognition on the target feature to be recognized obtained by fusing the first feature to be recognized and the second feature to be recognized through the target visual model to obtain an image recognition result of the image to be recognized.

[0015] In some embodiments, the large model application device based on the autonomous driving scenario further includes a fusion module for:

[0016] Acquire first weight information corresponding to the first feature to be identified, and second weight information corresponding to the second feature to be identified;

[0017] Determine a first product according to the product of the first feature to be identified and the first weight information;

[0018] Determine a second product according to the product of the second feature to be identified and the second weight information;

[0019] The target feature to be identified is obtained according to the sum of the first product and the second product.

[0020] In some embodiments, the processing module is further used to:

[0021] Obtaining a first predicted label corresponding to the first feature to be identified and a first estimated label corresponding to the image to be identified;

[0022] Determining, by a posterior optimization trajectory generator of the target visual model, a first loss of the perception head when predicting the first feature to be identified based on the first estimated label and the first predicted label;

[0023] Determine, based on the first loss and the first feature to be identified, a first gradient for performing stochastic gradient descent processing on the first feature to be identified;

[0024] Updating the first gradient of the first feature to be identified by using a gradient descent algorithm to obtain an updated first feature to be identified;

[0025] Repeatedly obtain the first predicted label corresponding to the updated first feature to be identified and the first estimated label corresponding to the image to be identified; determine the first loss of the perception head when predicting the updated first feature to be identified based on the first estimated label and the first predicted label through the posterior optimization trajectory generator of the target visual model; determine the first gradient of the updated first feature to be identified for random gradient descent processing based on the first loss and the updated first feature to be identified; update the first gradient of the first feature to be identified through the gradient descent algorithm to obtain the updated first feature to be identified, until the number of updates reaches the preset first gradient number, and obtain the second feature to be identified for indicating the subsequent reasoning direction of the target visual model according to the updated first feature to be identified.

[0026] In some embodiments, the large model application device based on the autonomous driving scenario further includes a training module for:

[0027] Obtain sample images to be identified from historical autonomous driving data;

[0028] Inputting the sample image to be recognized into a visual backbone network in a first visual model corresponding to the autonomous driving scene for feature extraction, and inputting the extracted sample features into a perception head to obtain first sample features to be recognized;

[0029] Performing a random gradient descent process on the first sample feature to be identified by the posterior optimization trajectory generator in the first visual model to obtain a second sample feature to be identified for indicating a subsequent optimization direction of the first visual model;

[0030] Performing semantic recognition on the target sample features to be recognized obtained by fusing the first sample features to be recognized and the second sample features to be recognized by using the first visual model to obtain a sample prediction result of the sample image to be recognized;

[0031] Obtaining a true sample label of the sample to be identified image, and determining a target loss based on a gap between the true sample label and the sample prediction result;

[0032] The parameters of the first visual model are adjusted based on the target loss until a preset convergence condition is reached to obtain an updated target visual model.

[0033] In some embodiments, the large model application device based on the autonomous driving scenario further includes an updating module for:

[0034] Obtaining the update time of the last update of the first visual model, and determining a sample image to be recognized after the update time from historical autonomous driving data;

[0035] Acquire historical images before the update time from the historical autonomous driving data, and calculate the scene difference between the sample image to be identified and the historical images;

[0036] A scene difference threshold is obtained, and when the scene difference is less than the scene difference threshold, an update operation on the first visual model is triggered.

[0037] In some embodiments, the training module is further used to:

[0038] Acquire first sample weight information corresponding to the feature to be identified of the first sample, and second sample weight information corresponding to the feature to be identified of the second sample;

[0039] Determine a first sample product according to the product of the first sample feature to be identified and the first sample weight information;

[0040] Determine a second sample product according to the product of the second sample feature to be identified and the second sample weight information;

[0041] The to-be-identified feature of the target sample is obtained according to the sum of the first sample product and the second sample product.

[0042] In some embodiments, the training module is further used to:

[0043] Obtaining a first sample prediction label corresponding to the first sample feature to be identified and a first sample estimated label corresponding to the image to be identified;

[0044] Determining, by means of a posterior optimization trajectory generator of the target visual model, a first sample loss of the perception head when predicting a feature to be identified of the first sample based on the first sample estimated label and the first sample predicted label;

[0045] Determine, based on the first sample loss and the first sample feature to be identified, a first sample gradient for performing stochastic gradient descent processing on the first sample feature to be identified;

[0046] Updating the first sample gradient for the first sample feature to be identified by using a gradient descent algorithm to obtain an updated first sample feature to be identified;

[0047] Repeatedly obtain the first sample prediction label corresponding to the updated first sample feature to be identified and the first sample estimated label corresponding to the image to be identified; determine the first sample loss of the perception head when predicting the updated first sample feature to be identified based on the first sample estimated label and the first sample prediction label through the posterior optimization trajectory generator of the target visual model; determine the first sample gradient for performing random gradient descent processing on the updated first sample feature to be identified based on the first sample loss and the updated first sample feature to be identified; update the first sample gradient of the first sample feature to be identified through the gradient descent algorithm to obtain the updated first sample feature to be identified, until the number of updates reaches the preset first sample gradient number, and obtain the second sample feature to be identified for indicating the subsequent optimization direction of the target visual model according to the updated first sample feature to be identified.

[0048] Correspondingly, the third aspect of the embodiments of the present application proposes a computer device, which includes a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, it implements the large model application method based on the autonomous driving scenario as described in any one of the embodiments of the first aspect of the present application.

[0049] Correspondingly, the fourth aspect of the embodiments of the present application proposes a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the large model application method based on the autonomous driving scenario as described in any one of the embodiments of the first aspect of the present application.

[0050] The embodiment of the present application obtains an image to be recognized from the current autonomous driving data; extracts features from the image to be recognized according to the visual backbone network of the pre-trained target visual model, and inputs the extracted features into the perception head of the target visual model to obtain a first feature to be recognized; the first feature to be recognized is subjected to random gradient descent processing by the posterior optimization trajectory generator in the target visual model to obtain a second feature to be recognized for indicating the subsequent reasoning direction of the target visual model; the target feature to be recognized obtained by fusion of the first feature to be recognized and the second feature to be recognized by the target visual model performs semantic recognition to obtain the image recognition result of the image to be recognized. In this way, the second feature to be recognized representing the predicted reasoning direction of the target visual model in the future direction can be generated by random gradient descent to accelerate the decision-making speed in the reasoning stage. Moreover, by fusing the first feature to be recognized and the second feature to be recognized to obtain the enhanced target feature to be recognized, the target visual model can be helped to output the prediction result more quickly and accurately during scene understanding and decision-making. In summary, the present application can improve the accuracy of image recognition by the target visual model while improving the recognition efficiency of image recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 It is a schematic diagram of the architecture of a large model application system based on an autonomous driving scenario provided in an embodiment of the present application;

[0052] Figure 2 is a flow chart of a large model application method based on an autonomous driving scenario provided in an embodiment of the present application;

[0053] Figure 3 It is a general architecture diagram of the target visual model provided in the embodiment of the present application;

[0054] Figure 4 It is a schematic diagram of the gradient descent process of the posterior optimization trajectory generator provided in an embodiment of the present application;

[0055] Figure 5 It is a feature fusion graph provided in an embodiment of the present application;

[0056] Figure 6 It is a functional module diagram of a large model application device based on an autonomous driving scenario provided in an embodiment of the present application;

[0057] Figure 7 It is a schematic diagram of the hardware structure of the computer device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0059] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0060] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0061] Autonomous driving refers to the use of mechatronics, artificial intelligence and multi-agent systems to achieve assisted or fully automatic control of vehicles such as cars, trucks, airplanes or water vehicles. Street scene semantic understanding is an important part of autonomous driving, which aims to accurately identify and segment pedestrians, obstacles, roads and landmarks, and provide necessary road information for autonomous driving technology.

[0062] In the related art, visual models are generally used to recognize street scene images. However, due to the power consumption and space limitations of the vehicle computing platform, the model may increase the delay in the actual process of street scene image recognition, thereby reducing the model's recognition efficiency for street scene images; on the other hand, due to the lack of adaptive capabilities of general visual models, the model's recognition accuracy for new environments is reduced when facing new environments.

[0063] Moreover, in autonomous driving, semantic understanding of street scene images is an extremely critical and complex task. For example, when an autonomous vehicle enters an unfamiliar environment, its performance may drop significantly compared to operating in a known environment. In order to improve the generalization ability of the model, the vehicle needs to train the model over time using continuously collected data. Over time, the amount of data adapted to the autonomous driving model continues to expand, which helps to achieve a continuous and significant improvement in the generalization ability of the autonomous driving model. However, this increasing amount of data is a double-edged sword for the autonomous driving model. Specifically, as the amount of dynamically collected data grows beyond the fitting ability of the autonomous driving model, the risk of underfitting the autonomous driving model increases, resulting in a decrease in performance accuracy.

[0064] Based on this, the embodiments of the present application provide a large model application method, device, equipment and medium based on an autonomous driving scenario, which can improve the recognition efficiency of image recognition while improving the accuracy of the model in image recognition.

[0065] The large model application method, device, equipment and medium based on the autonomous driving scenario provided in the embodiments of the present application are specifically illustrated through the following embodiments. First, the large model application system based on the autonomous driving scenario in the embodiments of the present application is described.

[0066] Please refer to Figure 1 In some implementations, an embodiment of the present application provides a large model application system based on an autonomous driving scenario, including a terminal 11 and a server 12.

[0067] Specifically, the terminal 11 may be a device directly used for data collection, processing and interaction with the server 12, such as an onboard computer, a smart phone, a camera, etc. of an autonomous driving vehicle. The terminal 11 may collect image or video data in real time through a camera or other device, and perform preprocessing on the collected image, such as adjusting image resolution, eliminating noise, and compressing data.

[0068] Furthermore, the device of the terminal 11 (such as an onboard computer or embedded device equipped with powerful computing power) can also perform preliminary image processing and reasoning tasks to reduce the load of the back-end server. The terminal 11 can transmit the collected data or the processed results to the server 12, and the server 12 further processes and analyzes the data.

[0069] Exemplarily, the server 12 may be a device for centralized processing, storage, and management of data, which is usually deployed in the cloud or in a data center. For example, the server 12 may be a high-performance graphics processing unit (GPU) server, a tensor processing unit (TPU) server, a central processing unit (CPU) server, a storage server, etc. The server 12 may receive data transmitted by the terminal 11, perform complex image processing and recognition tasks on the data, and return the results to the terminal 11 after processing.

[0070] Furthermore, the server 12 may update the model regularly and send the latest model parameters to the terminal 11, and the terminal 11 may call the updated model to perform reasoning tasks.

[0071] The large model application method based on the autonomous driving scenario in the embodiment of the present application can be illustrated by the following embodiment.

[0072] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first. Moreover, the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or separate consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or separate consent, the necessary user-related data for enabling the normal operation of the embodiment of the present application will be obtained.

[0073] In the embodiment of the present application, the large model application device based on the autonomous driving scenario will be described from the perspective of the large model application device based on the autonomous driving scenario, which can be specifically integrated into a computer device. Figure 2 , Figure 2 This is a flowchart of the steps of the large model application method based on the autonomous driving scenario provided in the embodiment of the present application. The embodiment of the present application takes the large model application device based on the autonomous driving scenario as an example, which is specifically integrated on a terminal or a server. When the processor on the terminal or the server executes the program instructions corresponding to the large model application method based on the autonomous driving scenario, the specific process is as follows:

[0074] Step 101, obtaining an image to be recognized from current autonomous driving data.

[0075] In some embodiments, in order to ensure that the target visual model can understand and respond to the environment around the vehicle in real time and accurately, the image to be recognized can be obtained from the current autonomous driving data so that the vehicle can make timely decisions.

[0076] The autonomous driving data may be sensor data collected in real time during the autonomous driving process, including but not limited to camera images, lidar point clouds, radar data, ultrasonic sensor data, etc. These data are used to perceive the environment around the vehicle, including roads, pedestrians, obstacles, traffic signs, etc.

[0077] The image to be identified can be an image extracted from the current autonomous driving data, usually an image from a vehicle-mounted camera. The image to be identified can be used for further visual processing and semantic understanding to identify and segment key elements such as pedestrians, obstacles, roads, and landmarks. It is understandable that the image to be identified can also be obtained from historical autonomous driving data. As long as the image needs to be semantically identified, it can be used as the image to be identified.

[0078] For example, assume that an autonomous vehicle is driving and is equipped with the following sensors: front-view camera: resolution 1920×1080, frame rate 30 FPS; side-view camera: resolution 960×640, frame rate 30 FPS; lidar: used to obtain 3D point cloud data of the surrounding environment; radar: used to detect the distance and speed of obstacles.

[0079] Exemplarily, the front-view camera can collect 30 frames of images per second, and the side-view camera can collect 30 frames of images per second, and the timestamps of the front-view camera and the side-view camera can be synchronized to ensure that the image data of both are collected at the same time point. Furthermore, the latest frame of image collected by the front-view camera (for example, the timestamp is 16:00:00.123) can be converted into RGB format, and the latest frame of image (timestamp is 16:00:00.123) can be selected from the front-view camera as the image to be identified, and the selected image to be identified (front-view camera image) can be temporarily stored in the memory of the on-board computing platform to obtain the image to be identified. Furthermore, if side-view information is required, the image with the corresponding timestamp (timestamp is 16:00:00.123) can also be selected from the side-view camera.

[0080] In some embodiments, the frame rate of the images captured by the front-view camera and the side-view camera can be adjusted according to different specific scenes.

[0081] Furthermore, the autonomous driving vehicle can be equipped with cameras at multiple angles, such as one or more cameras at the front, left, and right of the vehicle, and the pitch angle, heading angle, and roll angle of each camera can be adjusted according to actual conditions. It is understood that the timestamps of cameras at different angles should be equal.

[0082] By obtaining the images to be recognized from the current autonomous driving data, it can be ensured that the target visual model can understand and respond to the environment around the vehicle in real time and accurately, so that the subsequent target visual model can perform rapid recognition and improve user satisfaction.

[0083] Step 102, based on the visual backbone network of the target visual model obtained by pre-training, feature extraction is performed on the image to be identified, and the extracted features are input into the perception head of the target visual model to obtain the first feature to be identified.

[0084] In some embodiments, in order to accurately identify an image to be identified, the image to be identified can be input into a pre-trained target visual model, and rich common features can be extracted through the visual backbone network. The extracted common features can be further converted into feature representations specific to the image recognition task through the perception head, so as to better capture the detailed information related to the task, thereby improving the performance of the model.

[0085] The target visual model can be a complete neural network model for completing image recognition tasks. The target visual model can be composed of a visual backbone network and a perception head, and a delayed optimization trajectory generator is set on the perception head. The target visual model is designed to solve specific visual recognition tasks, such as object detection, semantic segmentation or image classification.

[0086] The visual backbone network, also known as the Large Visual Model (LVM), can be a pre-trained deep neural network in the target visual model, which is responsible for extracting high-level feature representations from the input image to capture complex patterns and semantic information in the image to be recognized. Exemplarily, the visual backbone network can be ResNet, VGG, Inception, EfficientNet, and pre-trained image generation pre-trained transformer (ImageGPT, iGPT), etc.

[0087] Among them, the perception head can be the part of the target visual model located after the visual backbone network. The perception head can receive features from the visual backbone network and further process them to obtain the first feature to be identified.

[0088] Among them, the first feature to be identified can be the feature vector output by the perception head after the general feature extracted by the visual backbone network is input into the perception head. The first feature to be identified represents the preliminary understanding and prediction of the image to be identified.

[0089] Exemplarily, if the visual backbone network is iGPT, the image to be identified can be input into iGPT. iGPT can capture the complex patterns and semantic information in the image to be identified through a multi-layer attention mechanism and generate a high-dimensional feature vector. The high-dimensional feature vector contains information such as the position, shape, and color of the object in the image to be identified.

[0090] Furthermore, the high-dimensional feature vector can be input into the perception head for further processing to obtain a specific feature representation related to the current image recognition task, that is, the first feature to be recognized, so as to better meet the task requirements of image recognition. The first feature to be recognized contains more specific task-related information, such as the confidence score of each category.

[0091] Please refer to Figure 3 , Figure 3 The overall architecture diagram of the target visual model provided for the embodiment of the present application. Specifically, the target visual model includes a visual backbone network (LVM), a perception head (Head) and a posterior optimized trajectory generator (POTGen), wherein the posterior optimized trajectory generator can be regarded as a part of the perception head. Specifically, the visual backbone network can extract high-level features from the image to be identified collected by the vehicle, and pass the high-level feature input to the perception head. After receiving the high-level features transmitted by the visual backbone network, the perception head performs specific processing for the image recognition task to obtain the first feature to be identified.

[0092] Through the above method, an accurate first feature to be identified can be obtained for the image recognition task, so that the subsequent target model can process the first feature to be identified to obtain the second feature to be identified.

[0093] Step 103, performing stochastic gradient descent processing on the first feature to be identified through the posterior optimization trajectory generator in the target visual model to obtain a second feature to be identified for indicating the subsequent reasoning direction of the target visual model.

[0094] In some embodiments, in order to improve the efficiency and accuracy of the target visual model prediction, the first feature to be identified can be subjected to stochastic gradient descent processing through a posterior optimization trajectory generator to generate features of the optimization direction in advance, so that the target visual model can find the optimal solution more quickly in the subsequent reasoning process.

[0095] Among them, the posterior optimization trajectory generator (POTGen) can be a component in the target visual model, located after the perception head. The posterior optimization trajectory generator can generate a feature vector representing the future reasoning direction in advance by performing stochastic gradient descent processing on the first feature to be identified, which is the second feature to be identified.

[0096] The second feature to be identified can be a feature vector obtained by performing random gradient descent processing on the first feature to be identified by the posterior optimization trajectory generator, which is an optimized representation of the first feature to be identified and is used to indicate the path of the target visual model from the current state to a better state. The second feature to be identified contains information about the future reasoning direction, which is used to guide the subsequent reasoning direction of the target visual model.

[0097] Please refer to Figure 3 For example, the first feature to be identified can be input into the posterior optimization trajectory generator, and the posterior optimization trajectory generator can first make a preliminary prediction based on the first feature to be identified to obtain a first predicted label. Further, the first estimated label of the image to be identified can be obtained, and the first loss of the perception head in predicting the first feature to be identified can be determined based on the gap between the first predicted label and the first estimated label.

[0098] Furthermore, the first gradient order of the stochastic gradient descent can be determined based on the first loss and the first feature to be identified, and the first gradient order can be pre-set or dynamically adjusted according to the first loss. Furthermore, the first feature to be identified can be updated to the first gradient order by using a gradient descent algorithm through a posterior optimization trajectory generator, and the first feature to be identified can be adjusted during each update, so that the first feature to be identified is closer to the ideal reasoning direction and optimization direction, and finally the second feature to be identified is obtained.

[0099] By generating the second feature to be recognized, the subsequent reasoning direction of the target visual model can be effectively guided, thereby improving the efficiency and accuracy of the target visual model in predicting the image to be recognized.

[0100] In some embodiments, in order to make the target visual model have unique advantages in autonomous driving and other scene understanding tasks that require high efficiency and high performance, the first feature to be identified can be optimized by introducing a posteriori optimization trajectory generator to obtain a second feature to be identified for indicating a path from the current state to a more advanced state. For example, step 103 may include:

[0101] (103.1) Obtaining a first predicted label corresponding to a first feature to be identified and a first estimated label corresponding to an image to be identified;

[0102] (103.2) determining, by a posterior optimized trajectory generator of the target visual model, a first loss of the perception head in predicting the first feature to be identified based on the first estimated label and the first predicted label;

[0103] (103.3) determining a first gradient for performing stochastic gradient descent processing on the first feature to be identified based on the first loss and the first feature to be identified;

[0104] (103.4) updating the first gradient of the first feature to be identified by using a gradient descent algorithm to obtain an updated first feature to be identified;

[0105] (103.5) Repeatedly obtain the first predicted label corresponding to the updated first feature to be identified and the first estimated label corresponding to the image to be identified; determine the first loss of the perception head when predicting the updated first feature to be identified based on the first estimated label and the first predicted label through the posterior optimization trajectory generator of the target visual model; determine the first gradient of the updated first feature to be identified for random gradient descent processing based on the first loss and the updated first feature to be identified; update the first gradient of the first feature to be identified through the gradient descent algorithm to obtain the updated first feature to be identified, until the number of updates reaches the preset first gradient number, and obtain the second feature to be identified for indicating the subsequent reasoning direction of the target visual model according to the updated first feature to be identified.

[0106] Among them, the first predicted label can be a label obtained by the perception head through preliminary prediction based on the first feature to be identified. The first predicted label is an initial classification result of the target visual model for the current first feature to be identified. The first predicted label reflects the current prediction ability of the target visual model.

[0107] The first estimated label can be a label estimated based on the training data of the first visual model. During the training of the first visual model, the first visual model can learn the relationship between features and labels, so that after the target visual model is trained, a label associated with the first feature to be identified can be determined, thereby estimating the first estimated label. The first estimated label reflects the knowledge learned by the target visual model during the training process.

[0108] The first loss may be a loss value calculated based on the gap between the first estimated label and the first predicted label. The loss function for calculating the first loss may be a cross-entropy loss, a mean squared error loss, and the like.

[0109] The first gradient may refer to the gradient calculated based on the current first feature to be identified and the first loss during the stochastic gradient descent process. The first gradient indicates the adjustment direction of the model parameters in the current state to minimize the loss function, and is used to update the first feature to be identified so that the updated first feature to be identified is closer to the ideal optimization direction.

[0110] The first gradient number may be the number of iterations for updating the first feature to be identified by the gradient descent algorithm. The first gradient number may be pre-set or dynamically adjusted according to the first loss.

[0111] For example, the first feature to be identified corresponding to the image A to be identified can be obtained by the sensing head Prediction is performed to obtain a preliminary classification result output by the perception head, that is, a first predicted label, for example, the first predicted label can be a vehicle. Further, based on the feature-label relationship in the training data, it can be estimated that the first estimated label of the image A to be identified in the training data is "pedestrian".

[0112] Furthermore, the first gradient order and learning rate can be set according to actual conditions, for example, the preset first gradient order is 10, 30, etc. The first loss can be calculated using a cross entropy loss function based on the gap between the first predicted label and the first estimated label. Specifically, the first loss can be calculated based on the first estimated label and probability distribution and the probability distribution of the first predicted label. The calculation formula for the first loss can be as follows:

[0113]

[0114] Where N is the number of samples, C is the number of categories, is the probability distribution of the first estimated label of the jth gradient descent batch, is the first predicted label probability distribution of the jth gradient descent batch.

[0115] Furthermore, the first gradient of stochastic gradient descent can be determined according to the first loss value. Assume that the number of iterations is set to 10. The specific formula is as follows:

[0116]

[0117] in, represents the first gradient, represents the first loss, Indicates the first feature to be identified to be updated.

[0118] Furthermore, the gradient descent formula can be used to update the first feature to be identified , get the updated first feature to be identified :

[0119]

[0120] in, Yes and The learnable variables of each layer are used to adjust the learning rate, represents the first feature to be identified before updating, It represents the first feature to be identified after the update, j represents the number in a gradient descent batch, k represents the number of the layer in the posterior optimization trajectory generator POT, POT contains a total of k layers, and the number of layers k can be adjusted.

[0121] The final posterior optimization generator can output the second feature to be identified for:

[0122] =

[0123] in, represents the initial first feature to be identified, represents the second feature to be identified after performing K gradient descent updates on the first feature to be identified, and K represents the number of layers in the posterior optimization trajectory generator POT.

[0124] Please refer to Figure 4 Furthermore, when the number of updates of the stochastic gradient descent does not reach the first gradient number (that is, the level of the posterior optimization trajectory generator), the first feature to be identified can be repeatedly obtained according to the update Re-acquire the first predicted label corresponding to the updated first feature to be identified, and obtain the corresponding first estimated label, and determine the first loss of the perception head when predicting the updated first feature to be identified based on the first estimated label and the first predicted label through the posterior optimization trajectory generator of the target visual model; determine the first gradient of the updated first feature to be identified by stochastic gradient descent based on the first loss and the updated first feature to be identified; update the first gradient of the first feature to be identified by the gradient descent algorithm to obtain the updated first feature to be identified, until the number of updates reaches the preset first gradient number (update all levels), that is, the number of layers of the posterior optimization trajectory generator is reached, that is, the first feature to be identified after the random gradient descent update can be used as the second feature to be identified for indicating the subsequent reasoning direction of the target visual model. The repeated update process is not described in detail here.

[0125] Through the above method, the first feature to be identified can be effectively adjusted so that the adjusted second feature to be identified is closer to the feature corresponding to the optimal solution, which helps to accelerate the reasoning of the target visual model and improves the efficiency and accuracy of the target visual model in identifying the image to be identified.

[0126] Step 104 , semantic recognition is performed on the target feature to be recognized obtained by fusing the first feature to be recognized and the second feature to be recognized through the target visual model to obtain an image recognition result of the image to be recognized.

[0127] In some embodiments, in order to improve the accuracy and robustness of semantic recognition, the first feature to be identified and the second feature to be identified can be fused to obtain an enhanced target feature to be identified. In this way, the preliminary predicted information and the optimized information can be combined to improve the accuracy of semantic recognition.

[0128] The target feature to be identified may be an enhanced feature vector obtained by fusing the first feature to be identified and the second feature to be identified.

[0129] The image recognition result can be the classification, positioning or other related attributes of the object or scene in the image to be recognized obtained after the target visual model processes the target features to be recognized. This result can be a category label, bounding box, key point, etc.

[0130] It should be noted that the first feature to be identified is a high-level feature directly extracted by the target visual model based on the image to be identified, which reflects the target visual model's preliminary understanding of the image to be identified. The second feature to be identified is a feature obtained after multiple rounds of gradient descent optimization based on the first feature to be identified, which contains more contextual information and optimization direction, and can better reflect the information of the real label. Therefore, the first feature to be identified and the second feature to be identified can be fused to obtain an enhanced target feature to be identified. Based on the target feature to be identified, the target visual model can reduce the impact of noise while retaining useful information, thus avoiding the problem of insufficient robustness and generalization ability of a single feature.

[0131] Exemplarily, the first feature to be identified and the second feature to be identified may be directly added to obtain the target feature to be identified, or different weights may be set for the first feature to be identified and the second feature to be identified and then fused to obtain the target feature to be identified.

[0132] Furthermore, the fusion weights of the first feature to be identified and the second feature to be identified can be dynamically adjusted according to the first loss. For example, when the first loss corresponding to the initial first feature to be identified (when gradient descent calculation is not performed) is high, a larger weight can be given to the second feature to be identified to utilize its optimization information; conversely, if the initial loss is low, a larger weight can be given to the first feature to be identified. Alternatively, for different areas in the image to be identified, a spatial attention mechanism can be used to assign different weights to different areas of the first feature to be identified and the second feature to be identified, and then fuse them to obtain the target feature to be identified. Alternatively, the weight of the fusion of the first feature to be identified and the second feature to be identified can be flexibly adjusted by itself, for example, the corresponding weight of the first feature to be identified is set to 0.6, and the corresponding weight of the second feature to be identified is set to 0.4, etc. The embodiment of the present application does not impose specific restrictions on this.

[0133] By setting different weights for the first feature to be identified and the second feature to be identified and fusing them, the target feature to be identified is obtained. This not only retains the useful information of the first feature to be identified, but also obtains the reasoning feature representing the future reasoning direction, so as to accelerate the reasoning speed of the target visual model.

[0134] In some embodiments, in order to more effectively combine the initially extracted features and the optimized features, different weights may be set for the first feature to be identified and the second feature to be identified and weighted fusion may be performed to combine the advantages of both and provide a more comprehensive and accurate feature representation. Exemplarily, the target feature to be identified of the first feature to be identified and the second feature to be identified may be obtained by fusion in the following manner:

[0135] (A.1) Obtaining first weight information corresponding to a first feature to be identified, and second weight information corresponding to a second feature to be identified;

[0136] (A.2) determining a first product according to the product of the first feature to be identified and the first weight information;

[0137] (A.3) determining a second product according to the product of the second feature to be identified and the second weight information;

[0138] (A.4) Based on the sum of the first product and the second product, the target feature to be identified is obtained.

[0139] The first weight information may be a weight coefficient for the first feature to be identified.

[0140] The second weight information may be a weight coefficient for the second feature to be identified.

[0141] The first product may be an element-by-element product of the first feature to be identified and the first weight information.

[0142] The second product may be an element-by-element product of the second feature to be identified and the second weight information.

[0143] Exemplarily, the formula for fusing the first feature to be identified and the second feature to be identified is as follows:

[0144] 0.1]

[0145] in, represents the target feature to be identified, σ represents the first weight, represents the first feature to be identified, (1-σ) represents the second weight, Indicates the second feature to be identified.

[0146] Specifically, the first feature to be identified and the first weight information may be multiplied to obtain a first product, which is the first feature to be identified after the weight is adjusted, and the second feature to be identified and the second weight information may be multiplied to obtain a second product, which is the second feature to be identified after the weight is adjusted. The first product and the second product are added together, that is, the first feature to be identified after the weight is adjusted and the second feature to be identified after the weight is adjusted are fused to obtain the target feature to be identified. By fusion of the features, the optimized features can be quickly reached.

[0147] Furthermore, the first weight information and the second weight information can be obtained according to a pre-set setting, for example, the first weight information can be set to 0.6, the second weight information can be set to 0.4, and so on. Alternatively, the first weight information and the second weight information can be adjusted based on the initial first loss. For example, when the first loss is small, the first weight information can be appropriately increased and the second weight information can be reduced; when the first loss is large, the first weight information can be appropriately reduced and the second weight information can be increased. This application does not limit the specific settings of the first weight information and the second weight information.

[0148] The embodiment of the present application obtains an image to be recognized from the current autonomous driving data; extracts features from the image to be recognized according to the visual backbone network of the pre-trained target visual model, and inputs the extracted features into the perception head of the target visual model to obtain a first feature to be recognized; the first feature to be recognized is subjected to random gradient descent processing by the posterior optimization trajectory generator in the target visual model to obtain a second feature to be recognized for indicating the subsequent reasoning direction of the target visual model; the target feature to be recognized obtained by fusion of the first feature to be recognized and the second feature to be recognized by the target visual model performs semantic recognition to obtain the image recognition result of the image to be recognized. In this way, the second feature to be recognized representing the predicted reasoning direction of the target visual model in the future direction can be generated by random gradient descent to accelerate the decision-making speed in the reasoning stage. Moreover, by fusing the first feature to be recognized and the second feature to be recognized to obtain the enhanced target feature to be recognized, the target visual model can be helped to output the prediction result more quickly and accurately during scene understanding and decision-making. In summary, the present application can improve the accuracy of image recognition by the target visual model while improving the recognition efficiency of image recognition.

[0149] In some embodiments, as autonomous vehicles continue to collect new data, the data set becomes increasingly large and diverse. This data heterogeneity leads to insufficient generalization of the model and extended training cycles when facing new environments, which in turn makes it difficult for the model to fully capture the deep and complex structure of the data, resulting in underfitting, while also taking up a lot of computing resources. Therefore, this application proposes a new model architecture that can not only extract the features of the data in depth, but also accelerate the convergence process of the model, so that the visual model can still run efficiently on a resource-constrained vehicle platform, meeting the real-time and computing efficiency requirements of the autonomous driving system. Exemplarily, the target visual model is trained in the following way:

[0150] (B.1) Obtain sample images to be identified from historical autonomous driving data;

[0151] (B.2) Inputting the sample image to be recognized into the visual backbone network in the first visual model corresponding to the autonomous driving scene for feature extraction, and inputting the extracted sample features into the perception head to obtain the first sample feature to be recognized;

[0152] (B.3) performing a stochastic gradient descent process on the first sample feature to be identified by using the posterior optimization trajectory generator in the first visual model to obtain a second sample feature to be identified for indicating a subsequent optimization direction of the first visual model;

[0153] (B.4) performing semantic recognition on the target sample features to be identified by fusing the first sample features to be identified and the second sample features to be identified using the first visual model to obtain a sample prediction result of the sample image to be identified;

[0154] (B.5) Obtain the true sample label of the sample image to be identified, and determine the target loss based on the gap between the true sample label and the sample prediction result;

[0155] (B.6) Adjusting the parameters of the first visual model based on the target loss until a preset convergence condition is reached to obtain an updated target visual model.

[0156] The historical autonomous driving data may be sensor data collected during past autonomous driving processes, including but not limited to camera images, lidar point clouds, radar data, ultrasonic sensor data, etc. The historical autonomous driving data is autonomous driving data to be trained that is collected after the previous model training is completed, and is used to train subsequent visual models to update the visual models.

[0157] The sample image to be identified may be an image extracted from historical autonomous driving data for training the first visual model, and is usually an image from a vehicle-mounted camera.

[0158] Among them, the first sample feature to be identified can be the sample feature vector output by the perception head after the sample feature extracted by the visual backbone network of the first visual model is input into the perception head. The first feature to be identified represents the preliminary understanding and prediction of the first visual model for the sample image to be identified.

[0159] The second sample feature to be identified can be a sample feature vector obtained by performing random gradient descent processing on the first feature to be identified through the posterior optimization trajectory generator of the first visual model, which is an optimized representation of the first sample feature to be identified and is used to indicate the optimization path of the first visual model from the current state to a better state. The second sample feature to be identified contains information about the future optimization direction, which is used to guide the subsequent optimization direction of the first visual model.

[0160] The first visual model may be an untrained visual model or a visual model that needs to be updated. The first visual model includes a visual backbone network, a perception head, and a posterior optimization trajectory generator. The visual backbone network is pre-trained, and only the parameters of the perception head and the posterior optimization trajectory generator need to be updated and adjusted.

[0161] Among them, the visual backbone network is also a large visual model (LVM), which can be a pre-trained deep neural network in the first visual model, responsible for extracting high-level feature representations from the input image to capture complex patterns and semantic information in the image to be recognized. The visual backbone network has multiple layers, each layer has many neurons, which can extract the hierarchical structure of data from simple to complex and capture more complex patterns of data; at the same time, the visual backbone network has a large number of learnable parameters, so that it can effectively capture and learn the underlying distribution of data, especially for the complex task of semantic understanding in the context of autonomous driving; in addition, the visual backbone network can use the attention mechanism to enable the first visual model to focus on different parts of the input sample image to be recognized when generating each part of the output. Therefore, by setting the visual backbone network in the first visual model, the first visual model can generate more context-aware first sample features to be recognized, and can also effectively solve the problem of underfitting. Exemplarily, the visual backbone network can be ResNet, VGG, Inception, EfficientNet, and pre-trained image generation pre-trained transformer (ImageGPT, iGPT), etc.

[0162] Among them, the posterior optimization trajectory generator can be a mechanism for generating the future optimization direction of the first visual model (or target visual model) in advance. Based on the first sample features to be identified, the second sample features to be identified corresponding to the future (posterior) optimization direction can be generated in advance to guide the current optimization iteration, thereby reducing the computing resources and time required to reach convergence, while also improving the model performance and the model's generalization ability for new data.

[0163] The target sample feature to be identified may be an enhanced sample feature vector obtained by fusing the first sample feature to be identified and the second sample feature to be identified.

[0164] The sample prediction result may be a result of classification, positioning or other related attributes of an object or scene in the sample image to be identified, obtained after the first visual model processes the target sample features to be identified. The sample prediction result may be a category label, a bounding box, a key point, etc.

[0165] The real sample label can be the correct label of the sample image to be identified, which can be obtained through manual annotation or from a reliable data source. The real sample label contains information such as the actual category and location of the object in the sample image to be identified.

[0166] The target loss may be a loss value calculated based on the gap between the true sample label and the sample prediction result. The target loss may be a cross entropy loss, a mean square error loss, etc.

[0167] The convergence condition may be a preset condition for stopping training. When the convergence condition is met, it is considered that the first visual model has achieved sufficient performance, and the training of the first visual model may be stopped to obtain the target visual model. For example, the convergence condition may be that the number of training times for the first visual model has reached a preset number of training times, such as 100 times, 200 times, etc.; or the number of times the target loss is lower than the loss threshold has reached a preset number of times, such as the number of times the target loss a exceeds the first loss threshold b has reached 5 times. The convergence condition may be set according to actual conditions.

[0168] Exemplarily, sample images to be identified can be obtained from historical driving data, and the sample images to be identified should be selected from images that have not been used as training data recently. For example, the sample images to be identified can be selected from historical driving data from the last time the first visual model was trained to the current time. The sample images to be identified can include different scenes such as city streets, highways, and rural roads, as well as scenes under different lighting conditions such as sunny days, rainy days, foggy days, and nights.

[0169] Furthermore, when the pre-trained visual backbone network is iGPT, the iGPT can be used to extract features of the sample image to be identified, and then the extracted sample features are input into the perception head to obtain a preliminary first sample feature to be identified, which reflects the first visual model's preliminary understanding of the sample image to be identified. It is understandable that the weight of the perception head can be dynamically adjusted according to different driving environments to adapt to different scene requirements.

[0170] Exemplarily, the first sample feature to be identified is input into the posterior optimization trajectory generator, and through multiple random gradient descent processes, a second sample feature to be identified is generated to represent the future optimization direction, and the first sample feature to be identified and the second sample feature to be identified are fused to obtain the target sample feature to be identified, and semantic recognition is performed on the target sample feature to be identified to obtain a sample prediction result.

[0171] Furthermore, the real sample label can be used as the category label, and the cross entropy loss function can be used to calculate the gap between the sample prediction result and the real label to obtain the target loss. Exemplarily, multiple loss functions can also be combined, such as constructing a comprehensive loss function by combining the cross entropy loss function, L1 loss, L2 loss, etc., and the target loss can be calculated by the comprehensive loss function.

[0172] Furthermore, the parameters of the perception head and the posterior optimization trajectory generator of the first visual model can be adjusted based on the target loss until the preset convergence condition is reached. For example, when the number of training times reaches 200, the training of the first visual model can be stopped to obtain the target visual model.

[0173] By generating the future optimization direction (the features to be identified in the second sample) in advance through the posterior optimization trajectory generator to guide the current optimization iteration, the convergence speed of the first visual model can be accelerated, and the computing resources and time required to reach convergence can be reduced, so that the model can run efficiently on a resource-constrained vehicle platform.

[0174] In some embodiments, since the autonomous driving vehicle will encounter various driving environments during actual operation, in order to ensure that the autonomous driving system of the autonomous vehicle can adapt to the ever-changing driving environment, the autonomous driving system can collect image information of the environment in which the autonomous driving vehicle passes, and regularly detect the scene differences between new sample images to be identified and historical images (images that have been used for previous visual model training) to timely detect changes in the environment. When the changes are large, the visual model is updated to improve the generalization ability of the visual model for new data, so that the autonomous driving vehicle can adapt to new environmental conditions. For example, before (B.1), it can also include:

[0175] (C.1) Obtaining the last update time of the first visual model, and determining a sample image to be recognized after the update time from historical autonomous driving data;

[0176] (C.2) Obtain historical images before the update time from the historical autonomous driving data, and calculate the scene difference between the sample image to be identified and the historical image;

[0177] (C.3) Obtaining a scene difference threshold, and when the scene difference is less than the scene difference threshold, triggering an update operation on the first visual model.

[0178] The update time may be the time point when the first visual model was last updated or trained.

[0179] The scene difference can be used to measure the difference between the sample image to be identified and the historical image in terms of visual features, environmental conditions, etc. The scene difference can be quantified by structural similarity index, mean square error, histogram comparison, etc.

[0180] The scene difference threshold may be a preset value used to determine whether the scene difference is large enough. When the scene difference is less than the scene difference threshold, it indicates that the driving environment has changed significantly and needs to trigger an update operation of the first visual model. The scene difference threshold may be set according to specific application requirements and experience.

[0181] Exemplarily, assuming that the last update time of the first visual model is September 1, 2024, the sample images to be recognized correspond to September 1, 2024 to the current moment, and the sample images to be recognized contain environmental information that has not been trained on the first visual model; before September 1, 2024, the images that have been used to update the first visual model, and a group of representative images are extracted from the images before September 1, 2024 as historical images, and the historical images reflect the typical environmental conditions of the historical update period.

[0182] Furthermore, the multiple structural similarity indexes corresponding to the multiple sample images to be identified and the corresponding historical images can be averaged to obtain the scene difference between the sample images to be identified and the corresponding historical images. The range of the structural similarity index is between 0 and 1, and the closer the value is to 1, the more similar the two images are. For example, assuming that 3 sample images to be identified and 3 historical images are selected, the scene difference between sample image 1 to be identified and the corresponding historical image 1 is 0.25, the structural similarity index between sample image 2 to be identified and the corresponding historical image 2 is 0.65, and the structural similarity index between sample image 3 to be identified and the corresponding historical image 3 is 0.33. Then the scene difference between the sample image to be identified and the corresponding historical image is (0.25+0.65+0.34) / 3=0.41.

[0183] It should be noted that the present application does not limit the number of historical images and sample images to be identified for calculating scene differences. The number of historical images and sample images to be identified can be flexibly set without departing from the concept of the present application.

[0184] Exemplarily, if the scene difference threshold is 0.6, and the calculated scene difference is less than 0.6, then an update operation on the first visual model needs to be triggered.

[0185] In some implementations, in addition to triggering an update operation on the first visual model based on the scene difference being less than a scene difference threshold, the first visual model may also be updated at a scheduled time, or the update may be manually triggered by an operator, and so on.

[0186] It is understandable that, during the process of autonomous driving, when collecting sample images to be recognized for training the first visual model, the collected images should be as rich as possible and as many as possible so that the first visual model can be fully trained.

[0187] Through the above methods, it can be ensured that the first visual model can adapt to new environmental conditions, improve its robustness and generalization ability, and thus ensure the safety and reliability of autonomous driving.

[0188] In some embodiments, in order to more effectively combine the initially extracted features and the optimized features, different weights may be set for the first feature to be identified and the second feature to be identified and weighted fusion may be performed to combine the advantages of both and provide a more comprehensive and accurate feature representation. Exemplarily, the target sample feature to be identified of the first sample feature to be identified and the second sample feature to be identified may be obtained by fusion in the following manner:

[0189] (D.1) Obtaining first sample weight information corresponding to the feature to be identified of the first sample, and second sample weight information corresponding to the feature to be identified of the second sample;

[0190] (D.2) determining a first sample product according to the product of the first sample feature to be identified and the first sample weight information;

[0191] (D.3) determining a second sample product according to the product of the second sample feature to be identified and the second sample weight information;

[0192] (D.4) Obtain the to-be-identified features of the target sample based on the sum of the first sample product and the second sample product.

[0193] The first sample weight information may be a weight coefficient for the feature to be identified in the first sample.

[0194] The second sample weight information may be a weight coefficient for the feature to be identified in the second sample.

[0195] The first sample product may be an element-by-element product of the first sample feature to be identified and the first sample weight information.

[0196] The second sample product may be an element-by-element product of the second sample feature to be identified and the second sample weight information.

[0197] Exemplarily, the formula for fusing the first sample feature to be identified and the second sample feature to be identified is as follows:

[0198] 0.1]

[0199] in, represents the target sample feature to be identified, σ represents the first sample weight, represents the feature to be identified in the first sample, (1-σ) represents the weight of the second sample, Indicates the features to be identified of the second sample.

[0200] Specifically, the first sample product can be obtained by multiplying the first sample feature to be identified and the first sample weight information, and the second sample product can be obtained by multiplying the second sample feature to be identified and the second sample weight information, and the second sample product can be obtained, and the second sample product can be obtained by adding the first sample product and the second sample product, that is, fusing the first sample feature to be identified after the weight is adjusted and the second sample feature to be identified after the weight is adjusted, to obtain the target sample feature to be identified, and the optimized feature can be quickly reached through the fused feature.

[0201] Please refer to Figure 5 , Figure 5 In the example, the posterior optimization trajectory generator can generate a second sample feature to be identified that represents the future optimization trajectory, and the first feature to be identified and the second feature to be identified are fused to obtain the optimization direction guided by the posterior optimization trajectory generator. In this way, the first visual model can converge quickly according to the optimization direction.

[0202] Furthermore, the first sample weight information and the second sample weight information can be obtained according to a pre-set setting, for example, the first sample weight information can be set to 0.6, the second sample weight information can be set to 0.4, and so on. Alternatively, the first sample weight information and the second sample weight information can be adjusted based on the initial first sample loss. For example, when the first sample loss is small, the first sample weight information can be appropriately increased and the second sample weight information can be reduced; when the first sample loss is large, the first sample weight information can be appropriately reduced and the second sample weight information can be increased. This application does not limit the specific settings of the first sample weight information and the second sample weight information.

[0203] In some embodiments, in order to speed up the training efficiency of the first visual model, the first sample to be identified features can be subjected to stochastic gradient descent processing by a posterior optimization trajectory generator to generate features of the optimization direction in advance, so that the target visual model can find the optimal solution more quickly in the subsequent optimization process. Exemplarily, (B.3) may include:

[0204] (B.3.1) Obtaining a first sample predicted label corresponding to the first sample feature to be identified and a first sample estimated label corresponding to the image to be identified;

[0205] (B.3.2) determining, by means of a posterior optimized trajectory generator of the target visual model, a first sample loss of the perception head in predicting the first sample feature to be identified based on the first sample estimated label and the first sample predicted label;

[0206] (B.3.3) determining a first sample gradient for performing stochastic gradient descent processing on the first sample feature to be identified based on the first sample loss and the first sample feature to be identified;

[0207] (B.3.4) updating the first sample gradient for the first sample feature to be identified by using a gradient descent algorithm to obtain an updated first sample feature to be identified;

[0208] (B.3.5) Repeatedly obtain the first sample prediction label corresponding to the updated first sample feature to be identified, and the first sample estimated label corresponding to the image to be identified; determine the first sample loss of the perception head when predicting the updated first sample feature to be identified based on the first sample estimated label and the first sample prediction label through the posterior optimization trajectory generator of the target visual model; determine the first sample gradient for performing stochastic gradient descent processing on the updated first sample feature to be identified based on the first sample loss and the updated first sample feature to be identified; update the first sample gradient of the first sample feature to be identified through the gradient descent algorithm to obtain the updated first sample feature to be identified, until the number of updates reaches the preset first sample gradient number, and obtain the second sample feature to be identified for indicating the subsequent optimization direction of the target visual model according to the updated first sample feature to be identified.

[0209] The first sample prediction label may be a prediction result obtained after the perception head makes a preliminary prediction based on the first sample feature to be identified. The first sample prediction label may be a category label, a bounding box, or other related attributes. The first sample prediction label reflects the current prediction capability of the first visual model.

[0210] The first sample estimated label may be a label estimated by the first visual model (or a previous model of the first visual model) based on the relationship between features and labels learned during the historical training process. The first sample estimated label reflects the knowledge learned by the first visual model during the training process.

[0211] The first sample loss may be a loss value calculated based on the gap between the first sample estimated label and the first sample predicted label. The first sample loss may be a cross entropy loss, a mean square error loss, or the like.

[0212] The first sample gradient may be a gradient calculated based on the current first sample feature to be identified and the first sample loss during the stochastic gradient descent process, and represents a partial derivative of the first sample loss with respect to the first sample feature to be identified.

[0213] The first sample gradient number may be the number of iterations of performing random gradient descent processing on the first sample feature to be identified.

[0214] For example, the first sample feature to be identified corresponding to the sample image A to be identified can be sensed by the sensing head (Head) Prediction is performed to obtain the preliminary classification result output by the perception head, that is, the first sample prediction label, for example, the first sample prediction label can be a vehicle. Further, based on the feature-label relationship in the training data, it can be estimated that the first sample estimated label of the sample image A to be identified in the training data is "pedestrian".

[0215] Furthermore, the first sample gradient times and learning rate can be set according to actual conditions, for example, the preset first sample gradient times are 10, 30, etc. The first sample loss can be calculated using a cross entropy loss function based on the gap between the first sample predicted label and the first sample estimated label. Specifically, the first sample loss can be calculated based on the first sample estimated label and probability distribution and the probability distribution of the first sample predicted label. The calculation formula for the first sample loss can be as follows:

[0216]

[0217] Where N is the number of samples, C is the number of categories, is the probability distribution of the estimated label of the first sample in the jth gradient descent batch, is the predicted label probability distribution of the first sample in the jth gradient descent batch.

[0218] Furthermore, the first sample gradient of stochastic gradient descent can be determined according to the first sample loss value. Assume that the number of iterations is set to 10. The specific formula is as follows:

[0219]

[0220] in, represents the first sample gradient, represents the first sample loss, Indicates the first sample feature to be updated.

[0221] Furthermore, the gradient descent formula can be used to update the first sample feature to be identified , get the updated first sample feature to be identified :

[0222]

[0223] in, Yes and The learnable variables of each layer are used to adjust the learning rate, represents the first sample feature to be identified before updating, It represents the updated first sample feature to be identified, j represents the number in a gradient descent batch, k represents the number of the layer in the posterior optimization trajectory generator POT, POT contains a total of k layers, and the number of layers k can be adjusted.

[0224] Finally, the posterior optimization generator can output the second sample features to be identified for:

[0225] =

[0226] in, represents the initial first sample feature to be identified, represents the feature of the second sample to be identified after the feature of the first sample to be identified is updated by K gradient descents, and K represents the number of layers in the posterior optimization trajectory generator POT.

[0227] Please refer to Figure 3 and Figure 4 Furthermore, when the number of updates of the stochastic gradient descent does not reach the number of the first sample gradient (that is, the level of the posterior optimization trajectory generator), the first sample feature to be identified can be repeatedly obtained according to the update Re-acquire the first sample prediction label corresponding to the updated first sample feature to be identified, and obtain the corresponding first sample estimated label, and determine the first sample loss of the perception head when predicting the updated first sample feature to be identified based on the first sample estimated label and the first sample prediction label through the posterior optimization trajectory generator of the target visual model; determine the first sample gradient for random gradient descent processing of the updated first sample feature to be identified based on the first sample loss and the updated first sample feature to be identified; update the first sample gradient of the first sample feature to be identified through the gradient descent algorithm to obtain the updated first sample feature to be identified, until the number of updates reaches the preset first sample gradient number, that is, the number of layers of the posterior optimization trajectory generator is reached, that is, the first sample feature to be identified after the random gradient descent update can be used as the second sample feature to be identified for indicating the subsequent reasoning direction of the target visual model. The repeated update process is not described in detail here.

[0228] Through the above method, the features to be identified in the first sample can be effectively adjusted, and the posterior (future) optimization direction can be generated in advance to guide the current optimization iteration. This method can enable the first visual model to more accurately capture the complex patterns in the data. At the same time, it can enable the first visual model to converge quickly in a very short period of time, which is conducive to reducing the computational burden of the first visual model. This is of great significance for autonomous driving vehicles with limited computing resources.

[0229] Experiments show that, compared with other models, the method proposed in this application improves model performance by 66.48% and converges more than 6 times faster.

[0230] See also Figure 6 The embodiment of the present application further provides a large model application device based on an autonomous driving scenario, which can implement the large model application method based on an autonomous driving scenario. The large model application device based on an autonomous driving scenario includes:

[0231] An acquisition module 61 is used to acquire an image to be recognized from current autonomous driving data;

[0232] An input module 62 is used to extract features of the image to be identified based on the visual backbone network of the target visual model obtained by pre-training, and input the extracted features into the perception head of the target visual model to obtain a first feature to be identified;

[0233] A processing module 63 is used to perform a stochastic gradient descent process on the first feature to be identified through a posterior optimization trajectory generator in the target visual model to obtain a second feature to be identified for indicating a subsequent reasoning direction of the target visual model;

[0234] The recognition module 64 is used to perform semantic recognition on the target feature to be recognized obtained by fusing the first feature to be recognized and the second feature to be recognized through the target visual model to obtain an image recognition result of the image to be recognized.

[0235] The specific implementation of the large model application device based on the autonomous driving scenario is basically the same as the specific implementation of the large model application method based on the autonomous driving scenario, and will not be repeated here. On the premise of meeting the requirements of the embodiment of this application, the large model application device based on the autonomous driving scenario can also be provided with other functional modules to implement the large model application method based on the autonomous driving scenario in the above embodiment.

[0236] The embodiment of the present application also provides a computer device, the computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned large model application method based on the autonomous driving scenario when executing the computer program. The computer device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0237] See also Figure 7 , Figure 7 The hardware structure of a computer device according to another embodiment is shown, and the computer device includes:

[0238] The processor 71 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0239] The memory 72 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 72 can store an operating system and other applications. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 72, and the processor 71 is called to execute the large model application method based on the autonomous driving scenario in the embodiment of this application;

[0240] Input / output interface 73, used to implement information input and output;

[0241] Communication interface 74, used to realize communication interaction between the device and other devices, which can be realized by wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.);

[0242] A bus 75 that transmits information between the various components of the device (e.g., the processor 71, the memory 72, the input / output interface 73, and the communication interface 74);

[0243] The processor 71 , the memory 72 , the input / output interface 73 and the communication interface 74 are connected to each other in communication within the device via a bus 75 .

[0244] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the above-mentioned large model application method based on the autonomous driving scenario.

[0245] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0246] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0247] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0248] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0249] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0250] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0251] It should be understood that in the present application, "at least one (item)" and "several" refer to one or more, and "plurality" refers to two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and subsequent associated objects are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0252] In the several embodiments provided in the present application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0253] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0254] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0255] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0256] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A large model application method based on an autonomous driving scenario, characterized in that: The method comprises: Obtain an image to be recognized from current autonomous driving data; Extracting features of the image to be recognized according to the visual backbone network of the pre-trained target visual model, and inputting the extracted features into the perception head of the target visual model to obtain a first feature to be recognized; Obtain a first predicted label corresponding to the first feature to be identified, and a first estimated label corresponding to the image to be identified; determine a first loss of the perception head when predicting the first feature to be identified based on the first estimated label and the first predicted label through the posterior optimization trajectory generator of the target visual model; determine a first gradient for performing stochastic gradient descent processing on the first feature to be identified based on the first loss and the first feature to be identified; update the first gradient of the first feature to be identified through a gradient descent algorithm to obtain an updated first feature to be identified; repeatedly obtain a first predicted label corresponding to the updated first feature to be identified, and a first estimated label corresponding to the image to be identified. an estimated label; determining, through the posterior optimization trajectory generator of the target visual model, a first loss of the perception head when predicting the updated first feature to be identified based on the first estimated label and the first predicted label; determining, based on the first loss and the updated first feature to be identified, a first gradient for performing stochastic gradient descent processing on the updated first feature to be identified; updating the first gradient of the first feature to be identified through a gradient descent algorithm to obtain an updated first feature to be identified, until the number of updates reaches a preset first gradient number, and obtaining a second feature to be identified for indicating a subsequent reasoning direction of the target visual model according to the updated first feature to be identified; The target visual model is used to perform semantic recognition on the target feature to be recognized obtained by fusing the first feature to be recognized and the second feature to be recognized, so as to obtain an image recognition result of the image to be recognized.

2. The large model application method based on the autonomous driving scenario according to claim 1 is characterized in that: The target features to be identified are obtained by fusion in the following way: Acquire first weight information corresponding to the first feature to be identified, and second weight information corresponding to the second feature to be identified; Determine a first product according to the product of the first feature to be identified and the first weight information; Determine a second product according to the product of the second feature to be identified and the second weight information; The target feature to be identified is obtained according to the sum of the first product and the second product.

3. The large model application method based on the autonomous driving scenario according to claim 1 is characterized in that: The target visual model is trained in the following way: Obtain sample images to be identified from historical autonomous driving data; Inputting the sample image to be recognized into a visual backbone network in a first visual model corresponding to the autonomous driving scene for feature extraction, and inputting the extracted sample features into a perception head to obtain first sample features to be recognized; Performing a random gradient descent process on the first sample feature to be identified by the posterior optimization trajectory generator in the first visual model to obtain a second sample feature to be identified for indicating a subsequent optimization direction of the first visual model; Performing semantic recognition on the target sample features to be recognized obtained by fusing the first sample features to be recognized and the second sample features to be recognized by using the first visual model to obtain a sample prediction result of the sample image to be recognized; Obtaining a true sample label of the sample to be identified image, and determining a target loss based on a gap between the true sample label and the sample prediction result; The parameters of the first visual model are adjusted based on the target loss until a preset convergence condition is reached to obtain an updated target visual model.

4. The large model application method based on the autonomous driving scenario according to claim 3 is characterized in that: Before obtaining the sample image to be recognized from the historical autonomous driving data, the method further includes: Obtaining the update time of the last update of the first visual model, and determining a sample image to be recognized after the update time from historical autonomous driving data; Acquire historical images before the update time from the historical autonomous driving data, and calculate the scene difference between the sample image to be identified and the historical images; A scene difference threshold is obtained, and when the scene difference is less than the scene difference threshold, an update operation on the first visual model is triggered.

5. The large model application method based on the autonomous driving scenario according to claim 3 is characterized in that: The target sample feature to be identified of the first sample feature to be identified and the target sample feature to be identified of the second sample feature to be identified are obtained by fusing the feature in the following manner: Acquire first sample weight information corresponding to the feature to be identified of the first sample, and second sample weight information corresponding to the feature to be identified of the second sample; Determine a first sample product according to the product of the first sample feature to be identified and the first sample weight information; Determine a second sample product according to the product of the second sample feature to be identified and the second sample weight information; The to-be-identified feature of the target sample is obtained according to the sum of the first sample product and the second sample product.

6. The large model application method based on the autonomous driving scenario according to claim 3 is characterized in that: The step of performing random gradient descent processing on the first sample feature to be identified by the posterior optimization trajectory generator in the first visual model to obtain a second sample feature to be identified for indicating a subsequent optimization direction of the first visual model includes: Obtaining a first sample prediction label corresponding to the first sample feature to be identified and a first sample estimated label corresponding to the image to be identified; Determining, by means of a posterior optimization trajectory generator of the target visual model, a first sample loss of the perception head when predicting a feature to be identified of the first sample based on the first sample estimated label and the first sample predicted label; Determine, based on the first sample loss and the first sample feature to be identified, a first sample gradient for performing stochastic gradient descent processing on the first sample feature to be identified; Updating the first sample gradient for the first sample feature to be identified by using a gradient descent algorithm to obtain an updated first sample feature to be identified; Repeatedly obtain the first sample prediction label corresponding to the updated first sample feature to be identified and the first sample estimated label corresponding to the image to be identified; determine the first sample loss of the perception head when predicting the updated first sample feature to be identified based on the first sample estimated label and the first sample prediction label through the posterior optimization trajectory generator of the target visual model; determine the first sample gradient for performing random gradient descent processing on the updated first sample feature to be identified based on the first sample loss and the updated first sample feature to be identified; update the first sample gradient of the first sample feature to be identified through the gradient descent algorithm to obtain the updated first sample feature to be identified, until the number of updates reaches the preset first sample gradient number, and obtain the second sample feature to be identified for indicating the subsequent optimization direction of the target visual model according to the updated first sample feature to be identified.

7. A large model application device based on an autonomous driving scenario, characterized in that: The device comprises: An acquisition module is used to acquire the image to be recognized from the current autonomous driving data; An input module, used to extract features of the image to be identified according to the visual backbone network of the target visual model obtained by pre-training, and input the extracted features into the perception head of the target visual model to obtain a first feature to be identified; A processing module, used to obtain a first predicted label corresponding to the first feature to be identified, and a first estimated label corresponding to the image to be identified; determine a first loss of the perception head when predicting the first feature to be identified based on the first estimated label and the first predicted label through the posterior optimization trajectory generator of the target visual model; determine a first gradient of the first feature to be identified by stochastic gradient descent processing based on the first loss and the first feature to be identified; update the first gradient of the first feature to be identified by a gradient descent algorithm to obtain an updated first feature to be identified; repeatedly obtain the first predicted label corresponding to the updated first feature to be identified, and the image to be identified. a first estimated label corresponding to the first target visual model; determining, through the posterior optimization trajectory generator of the target visual model, the first loss of the perception head when predicting the updated first feature to be identified based on the first estimated label and the first predicted label; determining, based on the first loss and the updated first feature to be identified, the first gradient of the updated first feature to be identified by stochastic gradient descent processing; updating the first gradient of the first feature to be identified by the gradient descent algorithm to obtain the updated first feature to be identified, until the number of updates reaches the preset first gradient number, and obtaining the second feature to be identified for indicating the subsequent reasoning direction of the target visual model according to the updated first feature to be identified; The recognition module is used to perform semantic recognition on the target feature to be recognized obtained by fusing the first feature to be recognized and the second feature to be recognized through the target visual model to obtain an image recognition result of the image to be recognized.

8. A computer device, characterized in that: The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the large model application method based on the autonomous driving scenario as described in any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the large model application method based on the autonomous driving scenario as described in any one of claims 1 to 6 is implemented.

Images