Vehicle automatic driving control method and device

Abstract

The invention discloses a vehicle automatic driving control method and device. The method comprises the following steps: acquiring real-time pose information of a vehicle, and acquiring a real-time surrounding environment image of the vehicle; the surrounding environment image is analyzed through a target image recognition model, information of obstacles around the vehicle is determined, and the target image recognition model is obtained by replacing a self-attention mechanism in a standard ViT model with a Performer attention mechanism and conducting weight pruning on the adjusted model; and adjusting the driving direction of the vehicle according to the pose information and the obstacle information. According to the method and the device, the technical problems that more resources are occupied and the processing efficiency is low when a ViT model is used for image recognition in a related automatic driving scheme are solved.

Description

Technical Field

[0001] The present application relates to the field of autonomous driving technology, and more specifically, to a method and device for controlling an autonomous driving vehicle. Background Art

[0002] In recent years, the rapid development of autonomous driving technology is closely related to the progress of artificial intelligence, computer vision and deep learning. Traditional autonomous driving systems generally use CNN (Convolutional Neural Networks) or ResNet (Residual Network) for image processing and recognition to achieve understanding of the surrounding environment and decision-making. However, with the development of technology, the ViT (Vision Transformer) model has begun to attract more and more researchers and engineers due to its superior performance in processing image sequence data. However, the structure of the ViT model is relatively complex. For vehicle-mounted systems that need to process a large amount of data in real time but have limited computing resources, the use of the ViT model may lead to slow data processing speed and high energy consumption. In addition, during the autonomous driving process, the vehicle may enter areas with poor satellite navigation signals such as underground garages and culverts, resulting in the inability to accurately obtain the vehicle's positioning information, thereby affecting the vehicle's driving safety.

[0003] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0004] The embodiments of the present application provide a vehicle automatic driving control method and device to at least solve the technical problem that when the ViT model is used for image recognition in related automatic driving solutions, more resources are occupied and the processing efficiency is slow.

[0005] According to one aspect of an embodiment of the present application, a vehicle automatic driving control method is provided, including: obtaining real-time posture information of the vehicle, and obtaining real-time surrounding environment images of the vehicle; analyzing the surrounding environment images using a target image recognition model to determine obstacle information around the vehicle, wherein the target image recognition model is a model obtained by replacing the self-attention mechanism in the standard ViT model with the Performer attention mechanism and performing weight pruning on the adjusted model; adjusting the vehicle's driving direction according to the posture information and obstacle information.

[0006] Optionally, obtaining the real-time position and posture information of the vehicle includes: when no satellite navigation signal is detected, obtaining measurement parameters in the inertial navigation system on the vehicle-mounted terminal, and determining the real-time position and posture information of the vehicle based on the measurement parameters; when a satellite navigation signal is detected, determining the real-time position and posture information of the vehicle based on the satellite navigation signal, and adjusting the accumulated error of the measurement parameters in the inertial navigation system based on the satellite navigation signal.

[0007] Optionally, obtaining measurement parameters in an inertial navigation system on a vehicle-mounted terminal, and determining real-time posture information of the vehicle based on the measurement parameters, includes: obtaining target parameters measured by the inertial navigation system on the vehicle-mounted terminal, and determining dead reckoning information based on the target parameters, wherein the target parameters include at least one of the following: initial position, initial heading, driving speed, driving time, acceleration, and angular velocity of the vehicle; obtaining installation angle information, non-integrity constraint bracket information, and magnetic heading correction information of the vehicle-mounted terminal; and using an adaptive extended Kalman filter algorithm based on correlation entropy to fuse the dead reckoning information, installation angle information, non-integrity constraint bracket information, and magnetic heading correction information to obtain real-time posture information of the vehicle.

[0008] Optionally, the training process of the target image recognition model includes: obtaining a standard ViT model and replacing the self-attention mechanism in the ViT model with a Performer attention mechanism to obtain a first model; pruning the weights in the first model to obtain a second model; obtaining multiple environmental images collected during vehicle driving, performing image enhancement processing on each environmental image to obtain multiple training samples, and marking the obstacles in each training sample as sample labels; dividing the multiple training samples into a training set and a verification set; iteratively training the second model using the training set, and verifying the training results of the second model using the verification set to obtain the target image recognition model.

[0009] Optionally, the weights in the first model are pruned to obtain the second model, including: determining the L1 norm of the weights of each neuron connection in the first model; determining the weights whose L1 norm is less than a preset threshold as the target weights, or sorting the weights of each neuron connection from large to small according to the L1 norm, and determining a preset proportion of weights in the latter sort as the target weights; pruning the target weights to adjust the value of the target weights to 0; and adjusting the model hyperparameters based on the pruning results to obtain the second model.

[0010] Optionally, image enhancement processing is performed on each environmental image, including: for each environmental image, randomly generating at least one shadow area of ​​a first preset shape, a first target size, and a first shadow intensity to cover the environmental image, wherein the first target size is randomly taken within the first preset size range, and the first shadow intensity is randomly taken within the preset shadow intensity range; and / or, randomly selecting at least one target area of ​​a second preset shape and a second target size in the environmental image, and filling the target area with black, wherein the second target size is randomly taken within the second preset size range; and / or, for each pixel in the environmental image, adjusting the brightness of the pixel using a target adjustment coefficient, wherein the target adjustment coefficient is randomly taken within the preset adjustment coefficient range.

[0011] Optionally, the second model is iteratively trained using the training set, and the training results of the second model are verified using the verification set to obtain a target image recognition model, including: in each training batch, the training samples corresponding to the training batch are input into the second model in sequence to obtain recognition results of the second model; the target loss function is determined based on the recognition results and sample labels, and the model parameters of the second model are adjusted by minimizing the loss function; the trained second model is verified using the verification set to obtain verification results; when the iterative training reaches a preset number of times, or the verification result meets a preset convergence condition, the model training is stopped, and the obtained model is used as the target image recognition model.

[0012] According to another aspect of an embodiment of the present application, a vehicle automatic driving control device is also provided, including: an acquisition module, used to obtain real-time posture information of the vehicle, and obtain real-time surrounding environment images of the vehicle; an analysis module, used to analyze the surrounding environment images using a target image recognition model to determine obstacle information around the vehicle, wherein the target image recognition model is a model obtained by replacing the self-attention mechanism in the standard ViT model with the Performer attention mechanism, and performing L1 norm pruning on the weights in the adjusted model; a control module, used to adjust the vehicle's driving direction according to the posture information and obstacle information.

[0013] According to another aspect of an embodiment of the present application, a computer program product is also provided, which includes: a computer program, wherein when the computer program is executed by a processor, the above-mentioned vehicle automatic driving control method is implemented.

[0014] According to another aspect of an embodiment of the present application, an electronic device is also provided, which includes: a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to execute the above-mentioned vehicle automatic driving control method through the computer program.

[0015] In the embodiments of the present application, the existing ViT model is improved. By replacing the self-attention mechanism in the standard ViT model with the Performer attention mechanism, the complexity of attention calculation can be reduced. Then, the weight pruning of the adjusted model can reduce the number of parameters in the model calculation, further improving the calculation efficiency. In the actual vehicle automatic driving control process, the real-time posture information of the vehicle can be obtained, and the real-time surrounding environment image of the vehicle can be obtained. Then, the surrounding environment image is analyzed using the trained target image recognition model to determine the obstacle information around the vehicle. Then, the driving direction of the vehicle is adjusted according to the posture information and obstacle information, which can ensure the safe driving of the vehicle. The present application scheme effectively solves the technical problems of occupying more resources and having lower processing efficiency when using the ViT model for image recognition in related automatic driving schemes. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0017] Figure 1 is a flow chart of an optional vehicle automatic driving control method according to an embodiment of the present application;

[0018] Figure 2 is a schematic diagram of an optional model training process according to an embodiment of the present application;

[0019] Figure 3 is a schematic diagram of an optional vehicle posture determination process according to an embodiment of the present application;

[0020] Figure 4 is a schematic structural diagram of an optional vehicle automatic driving control device according to an embodiment of the present application;

[0021] Figure 5 It is a schematic diagram of the structure of an optional electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0022] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0023] It should be noted that the terms "first", "second", etc. in the specification, claims and drawings of the present application are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0024] Example 1

[0025] According to an embodiment of the present application, a vehicle automatic driving control method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0026] Figure 1 is a flow chart of a vehicle automatic driving control method provided according to an embodiment of the present application, such as Figure 1 As shown, the method comprises the following steps:

[0027] Step S102, obtaining real-time position information of the vehicle and obtaining real-time surrounding environment images of the vehicle;

[0028] Step S104, using a target image recognition model to analyze the surrounding environment image to determine obstacle information around the vehicle, wherein the target image recognition model is a model obtained by replacing the self-attention mechanism in the standard ViT model with the Performer attention mechanism and performing weight pruning on the adjusted model;

[0029] Step S106, adjusting the driving direction of the vehicle according to the position information and obstacle information.

[0030] The following describes each step of the vehicle automatic driving control method in conjunction with a specific implementation process.

[0031] As an optional implementation, the target image recognition model can be first trained in the following manner: obtain a standard ViT model, and replace the self-attention mechanism in the ViT model with the Performer attention mechanism to obtain a first model; prune the weights in the first model to obtain a second model; obtain multiple environmental images collected during vehicle driving, perform image enhancement processing on each environmental image to obtain multiple training samples, and mark the obstacles in each training sample as sample labels; divide the multiple training samples into a training set and a verification set; use the training set to iteratively train the second model, and use the verification set to verify the training results of the second model to obtain the target image recognition model.

[0032] Among them, Performer is a variant of the self-attention mechanism. Its core idea is to decompose the softmax attention function into the inner product of a set of orthogonal random features. This method is called FAVOR+ (Fast Attention Via positive Orthogonal Random features). Performer allows long sequences to be processed with linear time complexity, which can reduce the computational complexity of attention from the traditional O(N2) to O(N), where N represents the sequence length. Specifically, the Performer attention mechanism usually includes the following key components:

[0033] 1) Linear transformation component, using the weight matrix W of the same dimension Q , W K , W V Perform a linear matrix transformation on the input hidden state X to obtain the query Q (Query), key K (Key), and value V (Value);

[0034] 2) Scaling factor component, which scales Q and K by the scaling factor to control the range of internal values. The specific formula is:

[0035] 3) A random feature mapping component that maps Q and K to a random feature space using a mapping function to approximate the exponential function in softmax attention, where the mapping function is implemented by matrix multiplication with a random matrix;

[0036] 4) Random feature generation component, which generates random feature representations Q of Q and K by multiplying them with a random matrix features and K features ;

[0037] 5) Attention weight calculation component, which uses random feature representation to approximate attention weight. The specific calculation formula is: KV = einsum(′…nd,…ne->...de′,Kfeatures ,V), QKV=einsum(′…nd,…de->…ne′,Q features ,KV). The einsum operation allows efficient tensor multiplication and summation to calculate the weighted value.

[0038] Assume that the input hidden state X∈R^(n×d), where n is the sequence length, d is the hidden dimension size, and the number of random features is m, then Q,K∈R^(n×d) is scaled and multiplied by the random matrix R∈R^(d×m) to obtain Q features ,K features ∈R^(n×m); by calculating Q features and K features The outer product of is then multiplied by V to get the final output QKV∈R^(n×d).

[0039] By introducing random feature maps, Performer can approximate the attention weights at a low cost, making it feasible to process long sequences. This method significantly reduces the time and space complexity of the self-attention mechanism, making its application on large-scale datasets and long sequences more efficient.

[0040] Generally, in the ViT model, the importance of weights can be measured by the size of their absolute values. The smaller the absolute value of the weight, the smaller its impact on the model, and it can be considered unimportant. On this basis, the embodiment of the present application proposes an L1 norm pruning method to reduce the number of effective parameters of the model without changing the network architecture by removing relatively unimportant weights.

[0041] Optionally, when pruning the weights in the first model, the pruning can be performed in the following manner: determining the L1 norm of the weights of each neuron connection in the first model, wherein the L1 norm is also called Manhattan distance or sparse norm, which is the sum of the absolute values ​​of each element in the vector. For the weight vector w of the neural network, its L1 norm is defined as: In the formula, w i represents the i-th element in the weight vector, and n represents the dimension of the vector; the weight whose L1 norm is less than the preset threshold is determined as the target weight, or the weights of each neuron connection are sorted from large to small according to the L1 norm, and the weights of a preset proportion of the sorted weights are determined as the target weights, wherein the preset threshold or the preset proportion can be adjusted as needed and is not specifically limited here; the target weight is pruned to adjust the value of the target weight to 0; considering that pruning may affect the model performance, the model hyperparameters need to be adjusted according to the pruning results to obtain the second model.

[0042] As an optional implementation, when obtaining training samples, you can first obtain multiple environmental images collected during the vehicle's driving process, for example, control the unmanned vehicle to drive in an industrial park, and use three cameras set on the front side of the unmanned vehicle to collect surrounding environmental images, and then pre-process the collected images, and then perform image enhancement processing on each environmental image in turn to obtain multiple training samples, wherein the image enhancement processing method includes at least one of the following: random shadow processing, random erasure processing, and random brightness processing.

[0043] Optionally, when performing random shadow processing, a shadow area with at least one first preset shape, first target size, and first shadow intensity can be randomly generated and covered on the environmental image, wherein the first target size is randomly taken within a first preset size range, and the first shadow intensity is randomly taken within a preset shadow intensity range.

[0044] Assuming that the width of the environment image is W, the height is H, and the shape of the shadow area is a quadrilateral, the coordinates (x, y) of the four vertices of the shadow area can be randomly generated in the following way:

[0045] x1=rand()×W,y1=rand()×H

[0046] x2=rand()×W,y2=rand()×H

[0047] x3=x1+rand()×(W-x1), y3=y1+rand()×(H-y1)

[0048] x4=x2+rand()×(W-x2), y4=y2+rand()×(H-y2)

[0049] Where rand() is a function that generates random numbers between [0,1).

[0050] When using the shadow area to cover the environment image, the weight of the shadow area can be determined by the following formula: mask weight =uniform(w low ,w high ), the weight of the environment image is: mask origin =1-mask weight , where w low and w high is the minimum and maximum value of the preset shadow intensity, uniform(w low ,w high ) generates a random number in this range; you can use OpenCV's cv.addWeighted function to mix the shadow area and the environment image according to the weight to generate an image with a shadow.

[0051] Optionally, when performing random erasing processing, at least one target area of ​​a second preset shape and a second target size in the environmental image may be randomly selected and filled with black, wherein the second target size is randomly selected within a second preset size range.

[0052] For example, the transforms.RandomErasing method can be used to erase the target area at a random position in the environment image. The following key parameters need to be set: p: represents the probability that the environment image is randomly erased; scale = (s l ,s h ): represents the ratio of the target area to the environment image area, s l and h are the minimum and maximum values ​​of the ratio respectively; ratio = (r1, r2): indicates the aspect ratio range of the target area, r1 and r2 are the minimum and maximum values ​​of the range respectively; value = 0: indicates that the target area is filled with 0, that is, filled with black. The erasing operation here is achieved by randomly transforming the image, so it is necessary to first convert the image into a tensor, apply random erasing, and then convert it back to image format.

[0053] Optionally, when performing random brightness processing, for each pixel in the environment image, the brightness of the pixel may be adjusted using a target adjustment coefficient, wherein the target adjustment coefficient randomly takes a value within a preset adjustment coefficient range.

[0054] For example, for each pixel, its brightness can be adjusted using the following formula: new =α×I old +β, where I old Represents the original pixel value, I new represents the adjusted pixel value, α is the brightness adjustment factor, that is, the target adjustment coefficient, which can be randomly selected by the np.random.uniform() function, and β is a constant added to each pixel, representing the degree of linear shift. The above formula and OpenCV's cv.convertScaleAbs function can be used to adjust the image pixel brightness.

[0055] After obtaining multiple training samples, they can be divided into a training set and a validation set according to a preset ratio, such as training set: validation set = 4:1; then the training set is used to iteratively train the second model, and the validation set is used to verify the training results of the second model until the target image recognition model is obtained.

[0056] Optionally, during the training process, in each training batch, the training samples corresponding to the training batch can be input into the second model in turn to obtain the recognition results of the second model; the target loss function is determined based on the recognition results and sample labels, and the model parameters of the second model are adjusted by minimizing the loss function; the trained second model is then verified using the verification set to obtain the verification results; when the iterative training reaches a preset number of times, or the verification result meets the preset convergence conditions, the model training is stopped, and the obtained model is used as the target image recognition model.

[0057] Figure 2 A complete training process of a target image recognition model is shown.

[0058] In the actual vehicle driving control process, the real-time position information of the vehicle and the real-time surrounding environment image of the vehicle can be obtained first; then the surrounding environment image is analyzed using the trained target image recognition model to determine the obstacle information around the vehicle; and then the vehicle's driving direction is adjusted based on the position information and obstacle information.

[0059] During the autonomous driving process, the vehicle may enter areas with poor satellite navigation signals such as underground garages and culverts. If the vehicle's positioning information cannot be accurately obtained at this time, the vehicle's driving safety will be affected. To solve this problem, the embodiment of the present application provides a solution for assisting positioning using a terminal vehicle-mounted positioning algorithm.

[0060] Optionally, when acquiring the real-time position and posture information of the vehicle, it is possible to first detect whether a satellite navigation signal exists; if no satellite navigation signal is detected, the measurement parameters in the inertial navigation system on the vehicle terminal can be acquired, and the real-time position and posture information of the vehicle can be determined based on the measurement parameters.

[0061] Specifically, the target parameters measured by the inertial navigation system on the vehicle terminal can be obtained, and the dead reckoning information can be determined according to the target parameters, wherein the target parameters include at least one of the following: the initial position, initial heading, driving speed, driving time, acceleration, and angular velocity of the vehicle; the installation angle information, the non-integrity constraint bracket information, and the magnetic heading correction information of the vehicle terminal are obtained; and the dead reckoning information, the installation angle information, the non-integrity constraint bracket information, and the magnetic heading correction information are integrated using an adaptive extended Kalman filter algorithm based on correlation entropy to obtain the real-time posture information of the vehicle.

[0062] Among them, the formula used by the adaptive extended Kalman filter algorithm based on correlation entropy is:

[0063] x k =F k-1 x k-1 +w k-1

[0064] z k =H k x k +v k

[0065] The above two equations can be reconstructed into the form of nonlinear recursive equations, which can be expressed as:

[0066]

[0067] Further we get:

[0068]

[0069] By E[v k v k T ]=R k The error covariance matrix of the new linear equation can be obtained as:

[0070]

[0071] B k The new error covariance matrix is ​​obtained by Cholesky decomposition.

[0072] In-style Multiply both sides by left By simplification, we can get:

[0073] L k =Γ k x k +e k

[0074] Where: F k-1 is the system state transfer matrix; w k-1 、v k are process noise and measurement noise which are independent of each other and obey zero-mean Gaussian distribution; x k is the vehicle state vector at time k; z k is the direction measured at time k; H k is the Jacobian matrix of h(·); I is the identity matrix; e k =L k -Γ k x k .

[0075] In order to solve the problem of the sensitivity of the minimum mean square error criterion to faults, the maximum correlation entropy criterion is introduced to reduce the adverse effects of pseudorange measurement failures. The reconstructed cost function is:

[0076]

[0077] The optimal solution of the state quantity can be expressed as:

[0078] x k = argmaxJ MCC (x k )

[0079] By taking the first-order derivative of the above formula, we get:

[0080]

[0081] The weight matrix is ​​defined as:

[0082]

[0083] C P,k =diag(G σ (e 1,k ),…,G σ (e m,k ))

[0084] C R,k =diag(G σ (e m+1,k ),…,G σ (e n,k ))

[0085] Where: m and n are the dimensions of the covariance error matrix and the measurement noise matrix respectively; C k is the weight matrix; C P,k is the weight matrix of the error covariance; C R,k is the weight matrix of the measurement noise. MCC (x k ) is maximum, the error between the measured value and the predicted value is minimum, and the optimal estimate of the system state can be obtained. k The new error covariance matrix α is obtained by weighting k for:

[0086]

[0087] The modified prior error covariance matrix and measurement noise covariance matrix are:

[0088]

[0089] Further calculation of the filter gain matrix And update the state and error covariance to get:

[0090]

[0091] The above process derives the correlation entropy extended Kalman filter based on fixed kernel width, and selects Gaussian kernel function instead of least mean square estimation, which can obtain higher-order measurement statistical information and reduce the weight of faulty measurement information, thereby ensuring the robustness of the filter estimation performance and the accuracy of the output position, velocity and attitude.

[0092] Optionally, when a satellite navigation signal is detected, the real-time position information of the vehicle can be determined based on the satellite navigation signal, and the accumulated error of the measurement parameters in the inertial navigation system can be adjusted based on the satellite navigation signal.

[0093] Figure 3 A complete vehicle posture determination process is shown. First, it is determined whether the satellite navigation signal exists. If so, the vehicle posture information is directly determined. If not, the relevant parameters measured by the inertial navigation system on the vehicle terminal are obtained, such as the measurement parameters of the gyroscope and accelerometer in the inertial measurement unit, and these parameters are mechanically arranged by inertial navigation to obtain the arrangement result. At the same time, the non-integrity constraint information is determined, and the magnetic heading correction information of the magnetometer is obtained. Then, the adaptive extended Kalman filter algorithm based on correlation entropy is used to fuse the above information to obtain the vehicle posture information.

[0094] Specifically, the target parameters measured by the inertial navigation system on the vehicle terminal can be obtained, and the dead reckoning information can be determined according to the target parameters, wherein the target parameters include at least one of the following: the initial position, initial heading, driving speed, driving time, acceleration, and angular velocity of the vehicle; the installation angle information, the non-integrity constraint bracket information, and the magnetic heading correction information of the vehicle terminal are obtained; and the dead reckoning information, the installation angle information, the non-integrity constraint bracket information, and the magnetic heading correction information are integrated using an adaptive extended Kalman filter algorithm based on correlation entropy to obtain the real-time posture information of the vehicle.

[0095] In the embodiment of the present application, by replacing the self-attention mechanism in the standard ViT model with the Performer attention mechanism, the complexity of attention calculation can be reduced, and then the weight pruning of the adjusted model can reduce the number of parameters of model calculation, further improving the calculation efficiency; in the actual vehicle automatic driving control process, the real-time posture information of the vehicle can be obtained, and the real-time surrounding environment image of the vehicle can be obtained, and then the surrounding environment image can be analyzed using the trained target image recognition model to determine the obstacle information around the vehicle, and then the driving direction of the vehicle is adjusted according to the posture information and obstacle information, which can ensure the safe driving of the vehicle. The present application scheme effectively solves the technical problems of occupying more resources and slow processing efficiency when using the ViT model for image recognition in related automatic driving schemes.

[0096] Example 2

[0097] According to an embodiment of the present application, a vehicle automatic driving control device for implementing the vehicle automatic driving control method in embodiment 1 is also provided. Figure 4 As shown, the vehicle automatic driving control device at least includes: an acquisition module 41, an analysis module 42 and a control module 43, wherein:

[0098] The acquisition module 41 is used to acquire the real-time position information of the vehicle and the real-time surrounding environment image of the vehicle;

[0099] An analysis module 42 is used to analyze the surrounding environment image using a target image recognition model to determine obstacle information around the vehicle, wherein the target image recognition model is a model obtained by replacing the self-attention mechanism in the standard ViT model with the Performer attention mechanism and performing L1 norm pruning on the weights in the adjusted model;

[0100] The control module 43 is used to adjust the driving direction of the vehicle according to the position information and obstacle information.

[0101] The functions of each module of the vehicle automatic driving control device are explained below in conjunction with the specific implementation process.

[0102] As an optional implementation, the training process of the target image recognition model includes: obtaining a standard ViT model and replacing the self-attention mechanism in the ViT model with a Performer attention mechanism to obtain a first model; pruning the weights in the first model to obtain a second model; obtaining multiple environmental images collected during vehicle driving, performing image enhancement processing on each environmental image to obtain multiple training samples, and marking the obstacles in each training sample as sample labels; dividing the multiple training samples into a training set and a verification set; iteratively training the second model using the training set, and verifying the training results of the second model using the verification set to obtain a target image recognition model.

[0103] Among them, when pruning the weights in the first model, it can be done in the following way: determine the L1 norm of the weights of each neuron connection in the first model; determine the weight whose L1 norm is less than a preset threshold as the target weight, or, sort the weights of each neuron connection from large to small according to the L1 norm, and determine the weights of a preset proportion of the sorted weights as the target weights; prune the target weights and adjust the value of the target weights to 0; adjust the model hyperparameters according to the pruning results to obtain the second model.

[0104] When obtaining training samples, you can first obtain multiple environmental images collected during the vehicle's driving process, pre-process the collected images, and then perform image enhancement processing on each environmental image in turn to obtain multiple training samples, wherein the image enhancement processing method includes at least one of the following: random shadow processing, random erasure processing, and random brightness processing.

[0105] Optionally, when performing random shadow processing, a shadow area with at least one first preset shape, first target size, and first shadow intensity can be randomly generated and covered on the environmental image, wherein the first target size is randomly taken within a first preset size range, and the first shadow intensity is randomly taken within a preset shadow intensity range.

[0106] Optionally, when performing random erasing processing, at least one target area of ​​a second preset shape and a second target size in the environmental image may be randomly selected and filled with black, wherein the second target size is randomly selected within a second preset size range.

[0107] Optionally, when performing random brightness processing, for each pixel in the environment image, the brightness of the pixel may be adjusted using a target adjustment coefficient, wherein the target adjustment coefficient randomly takes a value within a preset adjustment coefficient range.

[0108] After obtaining multiple training samples, they can be divided into a training set and a validation set according to a preset ratio, such as training set: validation set = 4:1; then the training set is used to iteratively train the second model, and the validation set is used to verify the training results of the second model until the target image recognition model is obtained.

[0109] In which, during the training process, in each training batch, the training samples corresponding to the training batch can be input into the second model in turn to obtain the recognition results of the second model; the target loss function is determined based on the recognition results and sample labels, and the model parameters of the second model are adjusted by minimizing the loss function; the trained second model is then verified using the verification set to obtain the verification results; when the iterative training reaches a preset number of times, or the verification result meets the preset convergence conditions, the model training is stopped, and the obtained model is used as the target image recognition model.

[0110] During the actual vehicle driving control process, the acquisition module first obtains the real-time posture information of the vehicle and the real-time surrounding environment image of the vehicle; the analysis module uses the trained target image recognition model to analyze the surrounding environment image and determine the obstacle information around the vehicle; the control module adjusts the vehicle's driving direction based on the posture information and obstacle information.

[0111] During the autonomous driving process, the vehicle may enter areas with poor satellite navigation signals such as underground garages and culverts. If the vehicle's positioning information cannot be accurately obtained at this time, the vehicle's driving safety will be affected. To solve this problem, the embodiment of the present application provides a solution for assisting positioning using a terminal vehicle-mounted positioning algorithm.

[0112] Optionally, when acquiring the real-time position and posture information of the vehicle, the acquisition module can first detect whether there is a satellite navigation signal; if no satellite navigation signal is detected, the measurement parameters in the inertial navigation system on the vehicle-mounted terminal can be acquired, and the real-time position and posture information of the vehicle can be determined based on the measurement parameters; if a satellite navigation signal is detected, the real-time position and posture information of the vehicle can be determined based on the satellite navigation signal, and the cumulative error of the measurement parameters in the inertial navigation system can be adjusted based on the satellite navigation signal.

[0113] Among them, when no satellite navigation signal is detected, the target parameters measured by the inertial navigation system on the vehicle terminal can be obtained first, and the dead reckoning information can be determined based on the target parameters, and the target parameters include at least one of the following: the initial position, initial heading, driving speed, driving time, acceleration, and angular velocity of the vehicle; then the installation angle information, non-integrity constraint bracket information and magnetic heading correction information of the vehicle terminal are obtained; the dead reckoning information, installation angle information, non-integrity constraint bracket information and magnetic heading correction information are integrated using an adaptive extended Kalman filter algorithm based on correlation entropy to obtain real-time posture information of the vehicle.

[0114] It should be noted that each module in the vehicle automatic driving control device in the embodiment of the present application corresponds one by one to each implementation step of the vehicle automatic driving control method in Example 1. Since a detailed description has been given in Example 1, some details not reflected in this embodiment can be referred to Example 1 and will not be repeated here.

[0115] Example 3

[0116] According to an embodiment of the present application, a computer program product is also provided, which includes a computer program, wherein when the computer program is executed by a processor, the vehicle automatic driving control method in Example 1 is implemented.

[0117] According to an embodiment of the present application, a non-volatile storage medium is also provided, which includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the vehicle automatic driving control method in Example 1 by running the computer program.

[0118] According to an embodiment of the present application, a processor is also provided, which is used to run a computer program, wherein the vehicle automatic driving control method in Example 1 is executed when the computer program is running.

[0119] According to an embodiment of the present application, an electronic device is also provided, which includes: a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to execute the vehicle automatic driving control method in Example 1 through the computer program.

[0120] Specifically, the following steps are executed when the computer program is running: obtaining the real-time posture information of the vehicle and the real-time surrounding environment image of the vehicle; analyzing the surrounding environment image using a target image recognition model to determine obstacle information around the vehicle, wherein the target image recognition model is a model obtained by replacing the self-attention mechanism in the standard ViT model with the Performer attention mechanism and performing weight pruning on the adjusted model; adjusting the driving direction of the vehicle according to the posture information and obstacle information.

[0121] As an optional implementation, the electronic device may be in the form of a mobile terminal, a computer terminal or a similar computing device. Figure 5 FIG. 1 shows a hardware structure block diagram of an electronic device for implementing a vehicle automatic driving control method. Figure 5 As shown, the electronic device 50 may include one or more (502a, 502b, ..., 502n are used to illustrate) processors 502 (the processor 502 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 504 for storing data, and a transmission device 506 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It can be understood by those skilled in the art that Figure 5 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 5 More or fewer components as shown, or with Figure 5 Different configurations shown.

[0122] It should be noted that the one or more processors 502 and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the electronic device 50. As described in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0123] The memory 504 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the vehicle automatic driving control method in the embodiment of the present application. The processor 502 executes various functional applications and data processing by running the software programs and modules stored in the memory 504, that is, realizing the vulnerability detection method of the above-mentioned application. The memory 504 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 504 may further include a memory remotely arranged relative to the processor 502, and these remote memories may be connected to the electronic device 50 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0124] The transmission device 506 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the electronic device 50. In one example, the transmission device 506 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 506 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0125] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the electronic device 50 .

[0126] The serial numbers of the above embodiments are only for description and do not represent the advantages or disadvantages of the embodiments.

[0127] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0128] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0129] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed over multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0130] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0131] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or all or part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of each embodiment method of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk, etc. Various media that can store program codes.

[0132] The above are only preferred implementations of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A vehicle automatic driving control method, characterized in that: include: Acquire real-time position information of the vehicle and obtain real-time surrounding environment images of the vehicle; Analyzing the surrounding environment image using a target image recognition model to determine obstacle information around the vehicle, wherein the target image recognition model is a model obtained by replacing the self-attention mechanism in a standard visual transformer ViT model with a Performer attention mechanism and performing weight pruning on the adjusted model; The driving direction of the vehicle is adjusted according to the position information and the obstacle information.

2. The method according to claim 1, characterized in that Get the real-time position information of the vehicle, including: In the case where no satellite navigation signal is detected, obtaining measurement parameters in an inertial navigation system on a vehicle-mounted terminal, and determining real-time position and posture information of the vehicle based on the measurement parameters; When a satellite navigation signal is detected, the real-time position information of the vehicle is determined based on the satellite navigation signal, and the accumulated error of the measurement parameters in the inertial navigation system is adjusted based on the satellite navigation signal.

3. The method according to claim 2, characterized in that Acquiring measurement parameters in an inertial navigation system on a vehicle-mounted terminal, and determining real-time position information of the vehicle based on the measurement parameters, including: Acquire target parameters measured by the inertial navigation system on the vehicle-mounted terminal, and determine dead reckoning information according to the target parameters, wherein the target parameters include at least one of the following: initial position, initial heading, driving speed, driving time, acceleration, and angular velocity of the vehicle; Obtaining installation angle information, non-integrity constraint bracket information and magnetic heading correction information of the vehicle-mounted terminal; The dead reckoning information, the installation angle information, the non-integrity constraint bracket information and the magnetic heading correction information are fused using an adaptive extended Kalman filter algorithm based on correlation entropy to obtain real-time posture information of the vehicle.

4. The method according to claim 1, characterized in that The training process of the target image recognition model includes: Obtain a standard ViT model, and replace the self-attention mechanism in the ViT model with the Performer attention mechanism to obtain a first model; Pruning the weights in the first model to obtain a second model; Acquire multiple environmental images collected during vehicle driving, perform image enhancement processing on each of the environmental images to obtain multiple training samples, and mark obstacles in each of the training samples as sample labels; Dividing the multiple training samples into a training set and a validation set; The second model is iteratively trained using the training set, and the training result of the second model is verified using the verification set to obtain the target image recognition model.

5. The method according to claim 4, characterized in that Pruning the weights in the first model to obtain a second model includes: Determining the L1 norm of the weights of each neuron connection in the first model; Determine the weight whose L1 norm is less than a preset threshold as the target weight, or sort the weights of the neuron connections from large to small according to the L1 norm, and determine the weights of a preset proportion in the latter sort as the target weight; Pruning the target weight to adjust the value of the target weight to 0; The model hyperparameters are adjusted according to the pruning results to obtain the second model.

6. The method according to claim 4, characterized in that Performing image enhancement processing on each of the environment images, including: For each of the environment images, randomly generate at least one shadow area of ​​a first preset shape, a first target size, and a first shadow intensity and cover it on the environment image, wherein the first target size is randomly selected within a first preset size range, and the first shadow intensity is randomly selected within a preset shadow intensity range; and / or, Randomly selecting at least one target area of ​​a second preset shape and a second target size in the environment image, and filling the target area with black, wherein the second target size is randomly selected within a second preset size range; and / or, For each pixel in the environment image, the brightness of the pixel is adjusted using a target adjustment coefficient, wherein the target adjustment coefficient is randomly selected within a preset adjustment coefficient range.

7. The method according to claim 4, characterized in that Iteratively training the second model using the training set, and verifying the training result of the second model using the verification set to obtain the target image recognition model, including: In each training batch, the training samples corresponding to the training batch are sequentially input into the second model to obtain the recognition result of the second model; Determining a target loss function according to the recognition result and the sample label, and adjusting model parameters of the second model by minimizing the loss function; Using the verification set to verify the trained second model to obtain a verification result; When the iterative training reaches a preset number of times, or the verification result meets the preset convergence condition, the model training is stopped, and the obtained model is used as the target image recognition model.

8. A vehicle automatic driving control device, characterized in that: include: An acquisition module is used to acquire real-time position information of the vehicle and acquire real-time surrounding environment images of the vehicle; An analysis module, used to analyze the surrounding environment image using a target image recognition model to determine obstacle information around the vehicle, wherein the target image recognition model is a model obtained by replacing the self-attention mechanism in the standard ViT model with the Performer attention mechanism and performing L1 norm pruning on the weights in the adjusted model; A control module is used to adjust the driving direction of the vehicle according to the posture information and the obstacle information.

9. A computer program product, characterized in that include: A computer program, wherein when the computer program is executed by a processor, it implements the vehicle automatic driving control method described in any one of claims 1 to 7.

10. An electronic device, characterized in that: include: A memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the vehicle automatic driving control method described in any one of claims 1 to 7 through the computer program.

Images