Vehicle trajectory processing method and device, and storage medium

Abstract

The application discloses a driving track processing method and device of a vehicle and a storage medium. The method comprises the following steps: in the driving process of the vehicle, a plurality of driving images are collected by a camera carried on the vehicle, wherein the collection time points of the plurality of driving images are continuous; a first image and a second image in the plurality of driving images are stacked to obtain a plurality of stacked images, wherein the first image and the second image are images collected at adjacent time points; and the plurality of stacked images are identified respectively to obtain the driving track of the vehicle. The application solves the technical problem that the accuracy of predicting the driving track of the vehicle is low in the related art.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of vehicles, in particular to a driving track processing method and device of a vehicle and a storage medium. BACKGROUND

[0002] At present, there are a large number of visual mileage calculation methods in the academic and industrial circles. The traditional visual mileage calculation method can be divided into a feature-based method and a direct method. The feature-based method usually includes steps of camera calibration, feature detection, feature matching, outlier rejection, motion estimation, scale estimation and optimization. However, how to detect appropriate features to restore specific motion is still a very challenging problem, and the accuracy of the driving track prediction of the vehicle through the driving images obtained by the camera is low.

[0003] At present, no effective solution has been proposed for the above problems. SUMMARY

[0004] The embodiments of the present application provide a driving track processing method, device and storage medium of a vehicle, to at least solve the technical problem of low accuracy of predicting the driving track of the vehicle in the related art.

[0005] According to an aspect of the embodiments of the present application, a driving track processing method of a vehicle is provided, comprising: in the driving process of the vehicle, collecting a plurality of driving images through a camera mounted on the vehicle, wherein the collection time points of the plurality of driving images are continuous; stacking a first image and a second image in the plurality of driving images to obtain a plurality of stacked images, wherein the first image and the second image are images collected at adjacent time points; respectively identifying the plurality of stacked images to obtain the driving track of the vehicle.

[0006] According to an aspect of the embodiments of the present application, a driving track processing device of a vehicle is provided, comprising: a collection module, configured to collect a plurality of driving images through a camera mounted on the vehicle in the driving process of the vehicle, wherein the collection time points of the plurality of driving images are continuous; a stacking module, configured to stack a first image and a second image in the plurality of driving images to obtain a plurality of stacked images, wherein the first image and the second image are images collected at adjacent time points; an identification module, configured to respectively identify the plurality of stacked images to obtain the driving track of the vehicle.

[0007] According to an aspect of the embodiments of the present application, a non-volatile storage medium is provided, characterized in that the non-volatile storage medium comprises a stored program, wherein when the program is running, the processor of the device controls the execution of the driving track processing method of the vehicle in any embodiment.

[0008] In the embodiment of the present application, during the driving of the vehicle, a plurality of driving images are collected by the camera mounted on the vehicle, wherein the collection time points of the plurality of driving images are continuous; a first image and a second image in the plurality of driving images are stacked to obtain a plurality of stacked images, wherein the first image and the second image are images collected at adjacent time points; the plurality of stacked images are identified respectively to obtain the driving trajectory of the vehicle, which realizes the driving trajectory of the vehicle with high accuracy, and through stacking the collected images, the experience of multiple images can be combined in the identification process to improve the identification accuracy, thereby solving the technical problem of low accuracy in predicting the driving trajectory of the vehicle in the related art. BRIEF DESCRIPTION OF DRAWINGS

[0009] The drawings described herein are used to provide further understanding of the present application, and form a part of the present application. The illustrative embodiments of the present application and their descriptions serve to explain the present application, and do not constitute improper limitations on the present application. In the drawings:

[0010] Figure 1 is a flowchart of a driving trajectory processing method of a vehicle according to an embodiment of the present application;

[0011] Figure 2 is a schematic diagram of a driving trajectory processing structure of a vehicle according to an embodiment of the present application;

[0012] Figure 3 is a schematic diagram of a neural network structure according to an embodiment of the present application;

[0013] Figure 4 is a schematic diagram of a driving trajectory processing device of a vehicle according to an embodiment 2 of the present application. DETAILED DESCRIPTION

[0014] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should fall within the scope of protection of the present application.

[0015] It is to be understood that the terms "first", "second", and the like, used in the description and the claims of the application as well as the above description of the drawings merely serve to distinguish the different singular objects, but do not limit the order or the precedence of the objects mentioned. It is to be understood that the data so distinguished can be interchanged under appropriate circumstances such that one embodiment of the application described herein can be practiced with other than the orders or sequences illustrated or described. Furthermore, the terms "comprise" and "include", and their conjugates, do not exclude the presence of other elements or steps than those listed. They are used to specify the presence of the stated features, steps or components, but they do not preclude the presence or addition of one or more other features, steps or components, nor do they preclude equivalents.

[0016] Visual odometry (VO) is an important research problem in the field of autonomous vehicles and mobile service robots. Among various visual odometry methods developed for different sensor modalities (e.g., monocular camera, stereo camera, omnidirectional camera, laser, lidar, etc.), monocular visual odometry is one of the most challenging ones. Most of the current visual odometry methods are developed based on a standard framework, including modules of feature extraction, feature matching, motion estimation, local optimization, etc. Although some of these methods have demonstrated superior performance, they usually need to be carefully designed and fine-tuned to adapt to different task and environmental requirements. Moreover, monocular visual odometry lacks scale information and often needs some prior knowledge to recover the absolute scale estimation. Recently, deep learning based methods have provided new solutions for odometry, relocalization, and semantic mapping. However, the feature representation of existing deep learning based algorithms is not accurate enough, especially for the important visual features.

[0017] The application aims to solve the problems of key feature capture and expression in the related art. By adding a self-attention mechanism module in the network framework, different weights are given to different channels and pixels at different positions of the video frame, thereby improving the accuracy and robustness of visual odometry positioning. Therefore, the application proposes a novel end-to-end monocular VO framework. Since it is trained and configured in an end-to-end manner, it can directly calculate the pose from a series of original RGB images (Red, Green, Blue) or videos without using any modules in traditional VO frameworks. The model is based on a Visual Transformer (ViT) network and an Attention Bidirectional Long Short-Term Memory (Att-Bi-LSTM) network, which has two dimensions of advantages: on the one hand, it can automatically learn the effective feature representation of the VO problem through a Convolutional Neural Network (CNN) and a Multi-Head Self-Attention mechanism; on the other hand, it can model the time sequence model (motion model) and data association model (image sequence) through an Attention Long Short-Term Memory network. The model can fully utilize the information of the sequence frames to achieve accurate and robust visual odometry positioning. The main problem of visual odometry is how to estimate the camera motion from several adjacent images.

[0018] There is an overlap between the two consecutive frames of the camera during the movement, that is, some scene and feature points in the three-dimensional world will be observed at the same time. These scene feature points will be projected onto the 2D picture, and through the alignment of the picture or the matching of the features, the corresponding relationship of the feature points on the front and back pictures can be found. By using the imaging geometry model (including camera parameters) and constraints, the motion information (rotation matrix R and translation t) between the two frames can be solved. In this way, a series of camera relative change matrices can be obtained, and thus the camera pose information can be derived.

[0019] At present, there are a large number of visual odometry calculation methods in the academic and industrial circles. Traditional visual odometry calculation methods can be divided into feature-based methods and direct methods. The feature-based method usually includes camera calibration, feature detection, feature matching, outlier rejection, motion estimation, scale estimation and optimization. However, how to detect appropriate features to recover specific motion is still a highly challenging problem. Unlike the feature-based method, the direct method tracks the motion of pixels and predicts the vehicle pose by minimizing the photometric error. The principle of this type of algorithm makes it extremely susceptible to changes in light.

[0020] An important property of odometry is that it only cares about motion over a local time, most often between two time instants. When time is sampled at some interval, the motion of the moving object within each time interval can be estimated. Since this estimate is affected by noise, the estimation error at previous time instances will accumulate to the motion at later times, which is called drift. With the continuous development of deep learning in computer vision, its application in visual odometry has also attracted widespread attention in the industry. Deep learning is a data-driven method that can automatically extract deep features from input videos (or image sequences) without the need for manually designed feature extraction algorithms. Such algorithms encode high-level priors to predict poses. Compared with traditional visual odometry, deep learning-based algorithms have the advantages of low computational cost and do not need to rely on camera internal parameters. Researchers have proposed some methods based on deep learning for camera motion recovery. Although they have achieved satisfactory performance, they do not consider the different responses of visual cues and the influence of pixel movement in different directions in input images on camera motion, so they may output trajectories with large errors.

[0021] For visual odometry based on neural network design, existing work relies on two modules of feature extraction and sequential learning to compose a deep learning-based network model, because learning only the representation information of a single image will severely limit the generalization ability of VO in new scenes. For video frames at different time instants, not only spatial information but also temporal information needs to be learned. Therefore, convolutional neural networks and recurrent neural networks need to be used in series to meet this requirement.

[0022] The present application designs an end-to-end visual odometry based on a self-attention mechanism. Through deep learning technology, the results of the feature extractor based on the ViT network structure and the Att-Bi-LSTM capable of mining and utilizing the correlation between images taken on a long track are combined for sequential modeling, realizing the training and deployment of end-to-end monocular visual odometry, thereby enhancing the positioning accuracy and robustness of visual odometry. It can solve the problem of the current CNN-based image feature extraction model relying on the local receptive field of convolution operation and only being able to focus on local features without being able to focus on global features, i.e., the limitation of long-range dependence of images. It can also solve the problem of the current deep learning-based method being unable to completely represent the information of the entire sequence in the image sequence or video.

[0023] Embodiment 1

[0024] According to the embodiment of the present application, an embodiment of a driving track processing method of a vehicle is provided. It should be noted that the steps shown in the flowchart of the drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described herein can be executed in an order different from that shown herein.

[0025] Figure 1 is a flowchart of a driving track processing method of a vehicle according to an embodiment of the present application, as shown in the figure, the method comprises the following steps: Figure 1

[0026] Step S102, during the driving process of the vehicle, a plurality of driving images are collected by the camera mounted on the vehicle.

[0027] Among them, the collection time points of the plurality of driving images are continuous.

[0028] The vehicle described above can be a common vehicle, and can also be an automatic driving vehicle, which is not limited here.

[0029] The camera described above can be a driving recorder installed on the vehicle, and can also be a camera separately provided on the vehicle, which is not limited here.

[0030] The plurality of driving images described above can be any orientation of the environment image collected during the driving process of the vehicle, for example, it can be the environment image in front of the vehicle during the driving process, and can also be the environment image behind the vehicle during the driving process, which is only an example here and is not limited.

[0031] The collection time interval of the plurality of driving images collected by the camera described above can be set according to the actual needs, which is not limited here, and the collection time interval can also be a fixed time interval.

[0032] In an optional embodiment, during the driving process of the vehicle, a plurality of driving images can be continuously collected by the camera mounted on the vehicle.

[0033] The plurality of driving images described above are continuously collected driving images, so there is a certain correlation.

[0034] Step S104, stacking the first image and the second image in the plurality of driving images to obtain a plurality of stacked images.

[0035] Among them, the first image and the second image are images collected at adjacent time points.

[0036] The number of the plurality of driving images described above can be greater than a preset number, wherein the preset number can be 3.

[0037] ​In an optional embodiment, two adjacent images in the plurality of driving images can be stacked to obtain a plurality of stacked images, wherein the first image and the second image are the two adjacent images.

[0038] In another optional embodiment, the plurality of driving images can include a first driving image collected at a first time point, a second driving image collected at a second time point, and a third driving image collected at a third time point; in the process of stacking the first driving image and the second driving image, the first driving image can be taken as the first image, the second driving image can be taken as the second image, and the first image and the second image can be stacked to obtain a stacked image of the first time point and the second time point; in the process of stacking the second driving image and the third driving image, the second driving image can be taken as the first image, the third driving image can be taken as the second image, and the first image and the second image can be stacked to obtain a stacked image of the second time point and the third time point. Finally, two stacked images can be obtained.

[0039] When the plurality of driving images include driving images collected at more time points, the plurality of driving images can be stacked in the above-mentioned manner to obtain a plurality of stacked images.

[0040] In the scenario of the above-mentioned stacking manner, the number of stacked images is less than the number of driving images, and specifically, the number of stacked images can be n, and the number of driving images can be n-1.

[0041] In yet another optional embodiment, the number of first images can be greater than 1, and the number of second images can also be greater than 1, that is, two or more driving images can be stacked to obtain a stacked image.

[0042] In step S106, the plurality of stacked images are respectively recognized to obtain the driving trajectory of the vehicle.

[0043] The driving trajectory of the vehicle mentioned above can be obtained by concatenating a plurality of pose information, wherein the plurality of pose information can be obtained by recognizing the plurality of stacked images.

[0044] In an optional embodiment, since each stacked image includes a plurality of driving images, the recognition of each stacked image actually combines the recognition results of the plurality of driving images, so that the changes between images can be recognized, the key places in the driving images can be determined, and the key features can be extracted, so as to focus on the recognition of the key features, and thus the accuracy of recognizing the plurality of stacked images can be improved, and thus the driving trajectory of the vehicle with higher accuracy can be obtained.

[0045] By the above steps, during the driving of the vehicle, a plurality of driving images are collected by the camera carried on the vehicle, wherein the collection time points of the plurality of driving images are continuous; a first image and a second image in the plurality of driving images are stacked to obtain a plurality of stacked images, wherein the first image and the second image are images collected at adjacent time points; the plurality of stacked images are respectively identified to obtain the driving trajectory of the vehicle, which realizes obtaining the driving trajectory of the vehicle with high accuracy. By stacking the collected images, the experience of multiple images can be combined in the identification process to improve the accuracy of identification, thereby solving the technical problem of low accuracy in predicting the driving trajectory of the vehicle in the related art.

[0046] In the above embodiments of the present application, respectively identifying the plurality of stacked images to obtain the driving trajectory of the vehicle includes: respectively identifying the plurality of stacked images to obtain a plurality of target pose information of the camera; and determining the driving trajectory of the vehicle based on the plurality of target pose information.

[0047] The above-mentioned pose information can be six degrees of freedom, wherein the six degrees of freedom can be three degrees of displacement and three degrees of spatial rotation.

[0048] In an optional embodiment, each stacked image can be identified, and the pose information of each stacked image can be used to determine the driving trajectory of the vehicle.

[0049] In the above embodiments of the present application, stacking the first image and the second image in the plurality of driving images to obtain the plurality of stacked images includes: obtaining a first image channel of the first image and a second image channel of the second image; stacking the first image channel and the second image channel to obtain a target image channel; and generating the plurality of stacked images based on the target image channel.

[0050] The first image channel of the first image can be an RGB channel of the first image.

[0051] In an optional embodiment, the first image channel of the first image and the second image channel of the second image can be obtained, the second image channel can be stacked on the first image channel to obtain a target image channel, the stacked image corresponding to the target image channel can be obtained, the image channels corresponding to any continuous images in the plurality of driving images can be superimposed to obtain a plurality of stacked target image channels, and the stacked images can be generated according to each stacked target image channel.

[0052] In the above embodiments of the present application, the plurality of stacked images are respectively identified to obtain the plurality of target pose information of the camera, including: identifying the plurality of stacked images by using a first neural network to obtain initial feature vectors of the plurality of stacked images, wherein the first neural network is used to extract the initial feature vectors; and processing the initial feature vectors of the plurality of stacked images by using a second neural network to obtain the plurality of target pose information of the camera, wherein the second neural network is used to adjust the weight proportion of the initial feature vectors.

[0053] The first neural network described above can be a convolutional neural network (CNN).

[0054] The initial feature vector described above can be a feature vector of a key region in the stacked image, wherein the key region can be determined according to a specific scene to which the first neural network is applied. If the vehicle scene is extracted, the key region can be a road region in the vehicle scene, a prompt sign lamp, which is not limited here. It should be noted that the first neural network can also be applied to other scenes.

[0055] In an optional embodiment, the first neural network can be used to identify the plurality of stacked images to extract an initial feature vector of a key region in each stacked image. The initial feature vector is used to represent the pose information in the stacked image.

[0056] In another optional embodiment, a cropped monocular RGB image can be directly used as a sequence input. For a given image x∈R H×W×C with a spatial resolution of H×W and a channel number of C, the plane of the input image x is cut into a plurality of image blocks with a size of P×P, and N represents the number of image blocks (i.e., the length of the input sequence), which is specifically represented as follows:

[0057]

[0058] The model can perform a linear projection x p of the image block vector on a D-dimensional hidden state space. For the spatial information and the position information of the image, the model uses a specific encoding embedding to preserve the position information while learning the spatial information, wherein x pos represents an image block embedding vector, and E N×D represents an image block position embedding, which is specifically represented as follows:

[0059]

[0060] The deep neural network architecture of the present application has achieved breakthrough success in computer vision tasks. The method adopted by the present application learns knowledge from appearance information and image context information during the training process. However, the visual odometry calculation method is rooted in geometric features rather than appearance features. The present application aims at the spatial and temporal features of visual odometry, takes the obtained image tensor as the input of the feature extraction network to generate effective geometric features of monocular visual odometry, and then derives the relationship between consecutive image frames through a bidirectional long short-term memory network with an attention mechanism and performs sequential learning.

[0061] In the above embodiment of the present application, the initial feature vectors of the plurality of stacked images are processed by the second neural network to obtain a plurality of target pose information of the camera, including: obtaining a first initial feature vector of a first stacked image, wherein the first initial feature vector is obtained by merging an initial feature vector of the first stacked image and a second initial feature vector of a second stacked image, and the first stacked image and the second stacked image are stacked images of adjacent time points in the plurality of stacked images; identifying the first initial feature vector by using the second neural network to obtain a first target feature vector of the first stacked image; and generating a plurality of target pose information based on the first target feature vector and a second target feature vector of the second stacked image.

[0062] The second stacked image described above can be obtained by stacking two driving images collected at time points earlier, and the first stacked image can be obtained by stacking driving images collected at time points later than the driving image of the first stacked image.

[0063] The adjacent time points in the plurality of stacked images can be obtained according to the driving images corresponding to the stacked images, for example, the stacked image is obtained by stacking a driving image collected at a first time point and a driving image collected at a second time point, and the time point of the stacked image can be a time point obtained by averaging the first time point and the second time point, or a middle time point of the first time point and the second time point.

[0064] For example, there are a first driving image, a second driving image, and a third driving image collected in sequence, the first driving image and the second driving image are stacked to obtain a second stacked image, and the second driving image and the third driving image are stacked to obtain a first stacked image.

[0065] In an optional embodiment, a first initial feature vector of the first stacked image can be obtained, wherein the first initial feature vector is obtained by merging an initial feature vector of the first stacked image and a second initial feature vector of the second stacked image, so that more information is contained in the first initial feature vector, and the expression capability of the feature vector is enhanced. At this time, the first initial feature vector is identified by the second neural network, so that the identification result of the stacked image is improved, and the first target feature vector of the first stacked image is obtained with higher accuracy. After obtaining the first target feature vector, the second target feature vector output by the stacked image of the adjacent time point can be combined to obtain target pose information with higher accuracy. A plurality of stacked images are processed in turn, and a plurality of target pose information can be generated.

[0066] In the process of identification by the second neural network, the expression capability of the feature vector can be enhanced by combining the feature vectors of a plurality of images, so that the final output result is more accurate.

[0067] In the above embodiments of the present application, the first initial feature vector is identified by the second neural network to obtain the first target feature vector of the first stacked image, which comprises adjusting the weight of the first initial feature vector by the second neural network to obtain the first target feature of the first stacked image.

[0068] In an optional embodiment, the second neural network can assign different weights to the feature vectors, and the processing of the previous stacked image can be combined to determine the feature vectors with strong correlation. The weight of the first initial feature vector with strong correlation is increased to improve the expression capability of the first initial feature vector with strong correlation, so that the first target feature of the first stacked image with higher accuracy is obtained.

[0069] In the above embodiments of the present application, the method further comprises: obtaining a training set of the vehicle scene task, wherein the training set comprises: a sample image and label information of a target region in the sample image, and the target region is a region associated with the vehicle scene task in the sample image; and training the initial neural network based on the training set to obtain the first neural network.

[0070] The vehicle scene task described above can be obtaining the driving track of the vehicle, and can also be obtaining the current pose of the vehicle.

[0071] The sample image described above can be a driving image collected during the driving of the vehicle, and the label information described above can be label information of a key region in the driving image. The label information can be a label frame, a label text, etc., which is not limited here.

[0072] The target region described above can be a region that needs to be focused on in the vehicle scene, for example, other vehicles in the vehicle scene, road conditions, etc.

[0073] In an optional embodiment, a training set of the vehicle scene task can be acquired, the initial neural network is trained by using the training set, and the first neural network capable of identifying the key region in the driving image is obtained.

[0074] In the above embodiments of the present application, the driving trajectory of the vehicle is determined based on the plurality of target pose information, which comprises: acquiring the acquisition time points of the plurality of driving images; determining the time sequence of the plurality of stacked images based on the acquisition time points of the plurality of driving images; and concatenating the plurality of target pose information based on the time sequence to obtain the driving trajectory of the vehicle.

[0075] The acquisition time points of the plurality of driving images can be acquired through the acquisition log of the camera.

[0076] The time sequence is used to represent the acquisition sequence of the plurality of driving images.

[0077] In an optional embodiment, the acquisition time points of the plurality of driving images can be acquired, the time sequence of the plurality of stacked images can be determined according to the acquisition time points of the plurality of driving images, the plurality of target pose information can be concatenated in the time sequence, the driving trajectory of the vehicle is obtained, and the driving trajectory of the vehicle is displayed.

[0078] Further, the driving trajectory of the vehicle can be displayed in a preset manner.

[0079] Figure 2 is a schematic diagram of a driving trajectory processing structure of a vehicle according to an embodiment of the present application, as shown in Figure 2 The driving images acquired at time t and the driving images acquired at time t+1 and time t+2 are stacked to obtain two groups of stacked images as shown in Figure 2 The two groups of stacked images can be input into the processing module, the output result of the processing module is processed by using the first neural network to obtain the processing result, the subsequent multiple processing modules and the classification module are processed to obtain the initial feature vector, the initial feature vector and the second initial feature vector of the adjacent stacked image are merged to obtain the first initial feature vector, the first initial feature vector is identified by using the second neural network to obtain the first target feature vector of the first stacked image, and the first target feature vector and the second target feature vector of the second stacked image are used to generate the plurality of target pose information, as shown in Figure 2 The first neural network includes a normalization layer (Layer Norm), a multi-head self-attention layer (Multi Head Self Attention), and a multilayer perceptron (Multilayer Perceptron).

[0080] Figure 3 is a schematic diagram of a neural network structure according to an embodiment of the present application, x1, x2, x3, x4 represent the feature vector output by the neural network, so that the network can capture more rich feature information, on the basis of self-attention, a multi-head self-attention mechanism is derived, and is applied to a machine vision task, wherein the model parameters (Query, Key, Value) are first subjected to linear transformation, input into the scaling dot product attention layer (Attention) and run h times, to obtain Figure 3 z1, z2, z3, z4 in the formula, the parameter matrix for linear transformation of Q, K, V is different each time. Then the results of the h times of scaling dot product attention layer are spliced, and the spliced z1, z2, z3, z4 are input into the full connection layer for linear transformation, to obtain the final multi-head self-attention mechanism result. The conversion process can be represented by the following formula:

[0081]

[0082]

[0083] wherein Q is a query vector sequence, K is a key vector sequence, V is a value vector sequence, and d k represents the dimension of the vector. Adding position coding information in word embedding can make up for the characteristics of the model ignoring word order. The advantage of multi-head self-attention mechanism is that it can filter out a small amount of important information from a large amount of information, and ignore a large amount of unimportant information. The greater the weight, the more focused on the corresponding Value value, which can optimize the parallel poor and long-distance dependence problems of the traditional network structure.

[0084] The present application combines convolutional neural network with feature extraction module (transformer), first uses CNN for original image feature extraction to generate feature mapping for input. Then embed the extracted hidden vector into the multi-head self-attention module, and finally output the extracted image feature information.

[0085] In the encoder-decoder (Encoder-Decoder) framework, all information is compressed into a vector, and this approach becomes increasingly unreasonable as the length of the Encoder side becomes longer. In view of this phenomenon, the related technology introduces an attention mechanism on the basis of the original model, directly retains all the outputs of the units to avoid the problem of information loss. It can be represented by the following formula:

[0086] S t =f(S t-1 ,Y t-1 ,C t );

[0087]

[0088]

[0089] where h is the input hidden vector, f is the activation function; Ct is the attention distribution, which can be interpreted as the degree of attention of the t-th information when given the relevant query q; alpha tj The attention obtained for each input. After introducing the attention mechanism, the representation model can better observe the global, where the scoring function is generally composed of additive models, dot product models, scaled dot product models or bilinear models, as shown below:

[0090] s(x i , q) = v T tan h(W x i +Uq);

[0091]

[0092]

[0093]

[0094] where W, U, v T are learnable network parameters, and d is the dimension of the input information. The complexity of the additive model and the dot product model is approximately the same, but the dot product model can use matrix multiplication, which is more efficient in calculation. When the dimension d of the input information is high, the variance of the dot product model is also larger, which will make the gradient of the softmax function smaller, so it is more reasonable to use scaled dot product at this time. The bilinear model introduces asymmetry when calculating similarity, and is the most generalized dot product model.

[0095] Because the convolutional neural network is difficult to learn the image sequence information at different times, a serial RNN recurrent network is needed to learn the connection and information between video frames at different times. In a flexible attention mechanism, the information at left and right times is connected together, which can prevent the problem of gradient disappearance and weakening of image time sequence long-term dependence.

[0096] where W, U, vT are learnable network parameters, and d is the dimension of the input information. The complexity of the additive model and the dot product model is approximately the same, but the dot product model can use matrix multiplication, which is more efficient in calculation. When the dimension d of the input information is high, the variance of the dot product model is also larger, which will make the gradient of the softmax function smaller, so it is more reasonable to use scaled dot product at this time. The bilinear model introduces asymmetry when calculating similarity, and is the most generalized dot product model.

[0097] Because the convolutional neural network is difficult to learn the image sequence information at different times, a serial recurrent network is needed to learn the connection and information between image frames at different times, to connect the information at left and right times in a flexible attention mechanism, to prevent the problem of gradient vanishing and weakening of image time sequence long-term dependence.

[0098] It should be noted that, for the foregoing method embodiments, in order to simply describe, they are all expressed as a series of action combinations, but those skilled in the art should know that the present application is not limited to the action sequence described, because according to the present application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the present application.

[0099] From the above description of the embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be realized by means of software and the necessary general hardware platform, of course, it can also be realized by hardware, but in many cases the former is a better embodiment. Based on such understanding, the technical solutions of the present application or the parts that contribute to the related art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes a plurality of instructions for making a terminal device (which can be a mobile phone, computer, server, or network device, etc.) execute the method of each embodiment of the present application.

[0100] Embodiment 2

[0101] According to the embodiments of the present application, a vehicle driving track processing device for implementing the vehicle driving track processing method is also provided, Figure 4 is a schematic diagram of a vehicle driving track processing device according to Embodiment 2 of the present application, as Figure 4 shown, the device 400 includes a collection module 402, a stacking module 404, and an identification module 406.

[0102] The collection module is configured to collect a plurality of driving images through a camera mounted on the vehicle during the driving process of the vehicle, wherein the collection time points of the plurality of driving images are continuous; the stacking module is configured to stack a first image and a second image in the plurality of driving images to obtain a plurality of stacked images, wherein the first image and the second image are images collected at adjacent time points; and the identification module is configured to identify the plurality of stacked images respectively to obtain the driving track of the vehicle.

[0103] It should be noted that the above collection module 402, stacking module 404, and identification module 406 correspond to steps S202-S206 of Embodiment 1, and the three modules have the same instances and application scenarios as the corresponding steps, but are not limited to the content disclosed in Embodiment 1.

[0104] In the above embodiments of the present application, the identification module is further configured to identify the plurality of stacked images respectively to obtain a plurality of target pose information of the camera, and determine the driving trajectory of the vehicle based on the plurality of target pose information.

[0105] In the above embodiments of the present application, the stacking module is further configured to obtain a first image channel of the first image and a second image channel of the second image, stack the first image channel and the second image channel to obtain a target image channel, and generate a plurality of stacked images based on the target image channel.

[0106] In the above embodiments of the present application, the identification module is further configured to identify the plurality of stacked images using a first neural network to obtain an initial feature vector of the plurality of stacked images, wherein the first neural network is configured to extract the initial feature vector, and process the initial feature vector of the plurality of stacked images using a second neural network to obtain a plurality of target pose information of the camera, wherein the second neural network is configured to adjust a weight ratio of the initial feature vector.

[0107] In the above embodiments of the present application, the identification module is further configured to obtain a first initial feature vector of a first stacked image, wherein the first initial feature vector is obtained by merging an initial feature vector of the first stacked image and a second initial feature vector of a second stacked image, the first stacked image and the second stacked image are adjacent stacked images at consecutive time points in the plurality of stacked images, identify the first initial feature vector using the second neural network to obtain a first target feature vector of the first stacked image, and generate the plurality of target pose information based on the first target feature vector and a second target feature vector of the second stacked image.

[0108] In the above embodiments of the present application, the identification module is further configured to adjust a weight of the first initial feature vector using the second neural network to obtain a first target feature of the first stacked image.

[0109] In the above embodiments of the present application, the device is further configured to obtain a training set of a vehicle scene task, wherein the training set includes sample images and label information of a target region in the sample images, and the target region is a region associated with the vehicle scene task in the sample images; and train the initial neural network based on the training set to obtain the first neural network.

[0110] In the above embodiments of the present application, the identification module is further configured to obtain collection time points of the plurality of driving images; in the above embodiments of the present application, the identification module is further configured to determine a time sequence of the plurality of stacked images based on the collection time points of the plurality of driving images; in the above embodiments of the present application, the identification module is further configured to concatenate the plurality of target pose information based on the time sequence to obtain the driving trajectory of the vehicle.

[0111] Embodiment 3

[0112] According to the embodiments of the present application, a non-volatile storage medium is provided, which comprises a stored program, wherein when the program is running, the processor of the device is controlled to execute the driving trajectory processing method of the vehicle in the above embodiments.

[0113] It should be noted that the preferred embodiments involved in the above embodiments of the present application are the same as the schemes and application scenarios and implementation processes provided in Embodiment 1, but are not limited to the schemes provided in Embodiment 1.

[0114] The above-mentioned embodiment numbers are only for description, and do not represent the advantages and disadvantages of the embodiments.

[0115] In the above embodiments of the present application, the description of each embodiment has its own focus, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments.

[0116] In the several embodiments provided by the present application, it should be understood that the disclosed technology can be implemented in other ways. Of course, the unit embodiment described above is only illustrative, and for example, the division of units can be a logical function division, and there can be another division manner in actual implementation, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interface, unit or module, and can be electrical or other forms.

[0117] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on multiple units. Part or all of the units can be selected according to actual needs to achieve the purpose of the present embodiment scheme.

[0118] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The above integrated unit can be realized in the form of hardware or in the form of software functional unit.

[0119] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application, essentially or in other words, the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a number of instructions to make a computer device (which can be a personal computer, a server or a network device, etc.) execute all or part of the steps of the methods described in various embodiments of the present application. The aforementioned storage medium includes: a U disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a mobile hard disk, a magnetic disk or an optical disk, and various media that can store program codes.

[0120] The above is only the preferred embodiment of the present application, it should be pointed out that, for those skilled in the art, without departing from the principles of the present application, can make a number of improvements and refinements, these improvements and refinements should also be considered as the protection scope of the present application.

Claims

1. A travel trajectory processing method of a vehicle, characterized by, The method comprises the following steps: During the driving of a vehicle, a plurality of driving images are collected by a camera mounted on the vehicle, wherein the collection time points of the plurality of driving images are continuous; A first image and a second image in the plurality of driving images are stacked to obtain a plurality of stacked images, wherein the first image and the second image are images collected at adjacent time points; The plurality of stacked images are respectively identified to obtain a driving track of the vehicle; Wherein, respectively identifying the plurality of stacked images to obtain the driving track of the vehicle comprises: respectively identifying the plurality of stacked images to obtain a plurality of target pose information of the camera; determining the driving track of the vehicle based on the plurality of target pose information; Respectively identifying the plurality of stacked images to obtain the plurality of target pose information of the camera comprises: identifying the plurality of stacked images by using a first neural network to obtain an initial feature vector of the plurality of stacked images, wherein the first neural network is used to extract the initial feature vector; processing the initial feature vector of the plurality of stacked images by using a second neural network to obtain the plurality of target pose information of the camera, wherein the second neural network is used to adjust the weight proportion of the initial feature vector; Processing the initial feature vector of the plurality of stacked images by using the second neural network to obtain the plurality of target pose information of the camera comprises: obtaining a first initial feature vector of a first stacked image, wherein the first initial feature vector is obtained by merging an initial feature vector of the first stacked image and a second initial feature vector of a second stacked image, the first stacked image and the second stacked image being adjacent time point stacked images in the plurality of stacked images; identifying the first initial feature vector by using the second neural network to obtain a first target feature vector of the first stacked image; generating a plurality of target pose information based on the first target feature vector and a second target feature vector of the second stacked image.

2. The method of claim 1, wherein, Stacking the first image and the second image in the plurality of driving images to obtain a plurality of stacked images comprises: Obtaining a first image channel of the first image and a second image channel of the second image; Stacking the first image channel and the second image channel to obtain a target image channel; Generating the plurality of stacked images based on the target image channel.

3. The method of claim 1, wherein, Identifying the first initial feature vector by using the second neural network to obtain the first target feature vector of the first stacked image comprises: Adjusting the weight of the first initial feature vector by using the second neural network to obtain the first target feature of the first stacked image.

4. The method of claim 1, wherein, The method further comprises: Obtaining a training set of a vehicle scene task, wherein the training set comprises: a sample image and label information of a target region in the sample image, the target region being a region associated with the vehicle scene task in the sample image; Training an initial neural network based on the training set to obtain the first neural network.

5. The method of claim 1, wherein, determining a driving track of the vehicle based on the plurality of target pose information, comprising: acquiring collection time points of the plurality of driving images; determining a time sequence of the plurality of stacked images based on the collection time points of the plurality of driving images; concatenating the plurality of target pose information based on the time sequence to obtain the driving track of the vehicle.

6. A travel trajectory processing device of a vehicle, characterized by, comprising: a collection module, configured to collect a plurality of driving images through a camera mounted on a vehicle during driving of the vehicle, wherein collection time points of the plurality of driving images are continuous; a stacking module, configured to stack a first image and a second image in the plurality of driving images to obtain a plurality of stacked images, wherein the first image and the second image are images collected at adjacent time points; an identification module, configured to identify the plurality of stacked images respectively to obtain a driving track of the vehicle; wherein the identification module is further configured to identify the plurality of stacked images respectively to obtain a plurality of target pose information of the camera; and determine the driving track of the vehicle based on the plurality of target pose information; the identification module is further configured to identify the plurality of stacked images using a first neural network to obtain initial feature vectors of the plurality of stacked images, wherein the first neural network is configured to extract the initial feature vectors; and process the initial feature vectors of the plurality of stacked images using a second neural network to obtain the plurality of target pose information of the camera, wherein the second neural network is configured to adjust weight proportions of the initial feature vectors; the identification module is further configured to acquire a first initial feature vector of a first stacked image, wherein the first initial feature vector is obtained by merging an initial feature vector of the first stacked image and a second initial feature vector of a second stacked image, the first stacked image and the second stacked image being adjacent stacked images at continuous time points in the plurality of stacked images; identify the first initial feature vector using the second neural network to obtain a first target feature vector of the first stacked image; and generate a plurality of target pose information based on the first target feature vector and a second target feature vector of the second stacked image.

7. A non-volatile storage medium, characterized by The non-volatile storage medium comprises a stored program, wherein when the program is running, the processor of the device is controlled to execute the driving track processing method of the vehicle of any one of claims 1-5.

Images