A mobile target trajectory prediction method, device, equipment and medium

Abstract

The application discloses a mobile target trajectory prediction method and device, equipment and medium, and relates to the technical field of automatic driving. The method comprises the following steps: acquiring image data including a mobile target collected by each vehicle camera at a current time during vehicle driving; performing image data splicing processing on the image data of different vehicle cameras according to a preset image splicing method to obtain a current splicing image; performing feature extraction on the current splicing image and a historical splicing image obtained at a previous time by using a preset feature extraction algorithm to obtain corresponding current features and historical features; connecting the current features and the historical features by using a preset connection function to obtain dynamic flow features; and predicting the motion trajectory of the mobile target according to the mobile trend information in the dynamic flow features based on position information and a moving speed corresponding to the mobile target. The image splicing processing can save computer resources, and connecting the features of adjacent time points can capture the mobile trend information, thereby realizing accurate trajectory prediction.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and in particular to a method, device, equipment and medium for predicting the trajectory of a moving target. Background Art

[0002] Decision-making biases and errors in autonomous driving can endanger human life. As vehicle speed increases, incorrect decisions tend to become more serious. For safety reasons, the perception model for autonomous driving needs to perform fast reasoning within low latency. Figure 1 As shown in the figure, while driving, autonomous vehicles typically capture continuous images through cameras and radar data through radar. These continuous images and radar data are then fed into a target detection algorithm to determine the position and velocity of each target. Based on the position and velocity of each target, the target detection algorithm's evaluation results are then calculated. Finally, the autonomous driving decision module makes a behavioral decision. However, in real-world online scenarios, no matter how fast the algorithm is, the state of the world around the vehicle changes once it processes the latest observation data. This problem is exacerbated as speed increases. The greater the speed, the greater the offset. The discrepancy between the changed state of an object and the perceived result can lead to unsafe autonomous driving decisions. Furthermore, in the Hermes on-board computing platform, the vehicle is equipped with four cameras. The images captured by different cameras simultaneously have overlapping portions. Processing these images separately results in repeated processing of the overlapping data, wasting computing resources.

[0003] In summary, how to accurately predict the motion trajectory of moving targets around the vehicle during autonomous driving and reduce the waste of computer resources is a problem that needs to be solved. Summary of the Invention

[0004] In view of this, the purpose of the present invention is to provide a moving target trajectory prediction method, device, equipment, and medium that can accurately predict the motion trajectory of moving targets around the vehicle during autonomous driving and reduce the waste of computer resources. The specific solution is as follows:

[0005] In a first aspect, the present application discloses a method for predicting a moving target trajectory, comprising:

[0006] Acquire image data including moving objects captured by each vehicle-mounted camera at a current moment during vehicle travel, and stitch the image data from different vehicle-mounted cameras according to a preset image stitching method to obtain a current stitched image;

[0007] Using a preset feature extraction algorithm, feature extraction is performed on the current spliced ​​image and the historical spliced ​​image obtained at a previous moment to obtain corresponding current features and historical features, and the current features and the historical features are connected by a preset connection function to obtain dynamic flow features;

[0008] Based on the position information and moving speed corresponding to the moving target and according to the moving trend information in the dynamic flow feature, the motion trajectory of the moving target is predicted.

[0009] Optionally, before predicting the motion trajectory of the moving target based on the position information and moving speed corresponding to the moving target and according to the movement trend information in the dynamic flow feature, the method further includes:

[0010] Acquire continuous image data and radar data including moving targets collected by each vehicle-mounted camera and radar during vehicle driving;

[0011] The continuous image data and the radar data are detected using a target detection algorithm to obtain the position information and moving speed of the moving target.

[0012] Optionally, stitching the image data from different vehicle-mounted cameras according to a preset image stitching method to obtain a current stitched image includes:

[0013] forming adjacent image groups based on image data from adjacent vehicle-mounted cameras, and calculating a relative displacement between two images in each adjacent image group;

[0014] A directed graph is constructed based on the relative displacement, and an absolute displacement is calculated based on the directed graph, so as to synthesize a current stitched image using the absolute displacement.

[0015] Optionally, calculating the relative displacement between two images in each adjacent image group includes:

[0016] Performing Fourier transform on the two images in each of the adjacent image groups to obtain a Fourier transform image group, and calculating a normalized correlation coefficient of the Fourier transform image group;

[0017] Performing an inverse Fourier transform on the normalized correlation coefficient based on a preset maximum value, and mapping an index value corresponding to the preset maximum value to an image coordinate to obtain a relative displacement corresponding to each of the adjacent image groups.

[0018] Optionally, the using a preset feature extraction algorithm to perform feature extraction on the current stitched image and the historical stitched image obtained at a previous moment to obtain corresponding current features and historical features, and connecting the current features and the historical features through a preset connection function to obtain dynamic flow features, includes:

[0019] Using the CSPDarknet-53 algorithm, feature extraction is performed on the current stitched image and the historical stitched image obtained at the previous moment to obtain current FPN features and historical FPN features;

[0020] The number of channels of the current FPN feature and the historical FPN feature is reduced by a batch normalization layer and a SiLU activation function, and then the current FPN feature and the historical FPN feature are connected using a concat function to obtain a dynamic flow feature.

[0021] Optionally, the using a preset feature extraction algorithm to perform feature extraction on the current stitched image and the historical stitched image obtained at a previous moment to obtain corresponding current features and historical features includes:

[0022] A preset feature extraction algorithm is used to perform feature extraction on the current spliced ​​image to obtain current features, and historical features corresponding to the historical spliced ​​image at a previous moment are obtained from a preset feature buffer.

[0023] Optionally, the predicting the motion trajectory of the moving target based on the position information and the moving speed corresponding to the moving target and according to the movement trend information in the dynamic flow feature includes:

[0024] Adaptive weights are generated for the moving targets of different moving speeds based on the movement trend information in the dynamic flow features, and the adaptive weights are dynamically allocated to predict the motion trajectory of the moving target based on the position information corresponding to the moving target.

[0025] In a second aspect, the present application discloses a moving target trajectory prediction device, comprising:

[0026] An image stitching module is used to obtain image data including moving objects collected by each vehicle-mounted camera at the current moment during the vehicle's driving process, and stitch the image data of different vehicle-mounted cameras according to a preset image stitching method to obtain a current stitched image;

[0027] a dynamic flow feature acquisition module, configured to extract features from the current spliced ​​image and the historical spliced ​​image obtained at a previous moment using a preset feature extraction algorithm to obtain corresponding current features and historical features, and to connect the current features and the historical features using a preset connection function to obtain dynamic flow features;

[0028] The trajectory prediction module is used to predict the motion trajectory of the moving target based on the position information and moving speed corresponding to the moving target and the moving trend information in the dynamic flow feature.

[0029] In a third aspect, the present application discloses an electronic device, comprising:

[0030] Memory, used to store computer programs;

[0031] A processor is used to execute the computer program to implement the steps of the aforementioned mobile target trajectory prediction method.

[0032] In a fourth aspect, the present application discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, the steps of the aforementioned mobile target trajectory prediction method are implemented.

[0033] It can be seen that the present application obtains image data including moving targets collected by each vehicle-mounted camera at the current moment during vehicle driving, and stitches the image data of different vehicle-mounted cameras according to a preset image stitching method to obtain a current stitched image; uses a preset feature extraction algorithm to perform feature extraction on the current stitched image and the historical stitched image obtained at the previous moment to obtain corresponding current features and historical features, and connects the current features and the historical features through a preset connection function to obtain dynamic flow features; based on the position information and moving speed corresponding to the moving target, and according to the movement trend information in the dynamic flow features, the motion trajectory of the moving target is predicted. It can be seen that the present application can avoid the problem of wasting computer resources caused by repeated processing of overlapping data by stitching the image data collected by different vehicle-mounted cameras at the current moment, and extract features from the current stitched image obtained at the current moment and the historical stitched image of the previous moment adjacent to it through a preset feature extraction algorithm to obtain corresponding features, and then connect the extracted features to obtain dynamic flow features, and then capture the movement trend information of the moving target in the dynamic flow features to predict the motion trajectory of the moving target, thereby improving the accuracy of the motion trajectory prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without any creative work.

[0035] Figure 1 This is a schematic diagram of an existing autonomous driving behavior decision-making process disclosed in this application;

[0036] Figure 2 This is a flow chart of a moving target trajectory prediction method disclosed in this application;

[0037] Figure 3 This is a schematic diagram of an improved autonomous driving behavior decision-making process disclosed in this application;

[0038] Figure 4 This is a flow chart of a specific moving target trajectory prediction method disclosed in this application;

[0039] Figure 5 A specific relative displacement calculation flow chart disclosed in this application;

[0040] Figure 6 This is a schematic diagram of a specific dual-flow sensing module disclosed in this application;

[0041] Figure 7 This is a schematic structural diagram of a moving target trajectory prediction device disclosed in this application;

[0042] Figure 8 This is a structural diagram of an electronic device disclosed in this application. DETAILED DESCRIPTION

[0043] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0044] Currently, in real online scenarios, no matter how fast the algorithm is, once the algorithm processes the latest observation data, the state of the world around the vehicle will change; and as the speed increases, the problem will be further exacerbated. The faster the speed, the greater the offset. The deviation between the changed state of the object and the perception result may lead to unsafe autonomous driving decisions. At the same time, in the Hermes on-board computing platform, the vehicle is equipped with four on-board cameras. The images obtained by different cameras at the same time have overlapping parts. Processing these images separately will cause the overlapping data to be processed repeatedly, thereby wasting computer resources. To this end, the embodiments of the present application disclose a moving target trajectory prediction method, device, equipment and medium, which can accurately predict the motion trajectory of moving targets around the vehicle during autonomous driving and reduce the waste of computer resources.

[0045] See also Figure 2 and Figure 3 As shown, the embodiment of the present application discloses a method for predicting a moving target trajectory, the method comprising:

[0046] Step S11: acquiring image data including moving targets collected by each vehicle-mounted camera at a current moment during vehicle driving, and stitching the image data from different vehicle-mounted cameras according to a preset image stitching method to obtain a current stitched image.

[0047] In this embodiment, during the automatic driving process of the vehicle, image data including moving targets captured by each on-board camera at the current moment is obtained. It is understandable that in the Hermes on-board computing platform, the vehicle is usually equipped with four on-board cameras. The images obtained by different cameras at the same time will have overlapping parts, and processing these images separately will cause the overlapping data to be repeatedly processed, thereby wasting computer resources. Therefore, after obtaining the image data captured by different on-board cameras at the same time, the present application splices these image data according to a preset image stitching method to obtain the current stitched image. In this way, the problem of repeated processing of overlapping data can be avoided through image stitching processing, thereby saving a large amount of computer resources.

[0048] Step S12: using a preset feature extraction algorithm to perform feature extraction on the current stitched image and the historical stitched image obtained at the previous moment to obtain corresponding current features and historical features, and connecting the current features and the historical features through a preset connection function to obtain dynamic flow features.

[0049] In this embodiment, a preset feature extraction algorithm is used to extract features from the current spliced ​​image and the historical spliced ​​image obtained at the previous moment to obtain corresponding current features and historical features, and then the current features and historical features are connected through a preset connection function to generate dynamic flow features. A cascade fusion operation is usually used to achieve optimal efficiency and performance. In addition, this embodiment also needs to obtain static flow features. For static flow features, this embodiment reasonably adds the original features of the current frame through a residual connection, which not only provides basic information for detection, but also improves the prediction accuracy at different driving speeds. That is, the present application discloses a dual-stream sensing module, which is specifically composed of a dynamic stream and a static stream in parallel, which capture movement trends and basic detection features respectively, and track one or more objects at the same time to provide data for predicting target actions; in terms of model architecture, the dynamic stream focuses on capturing the movement trend of the object, while the static stream provides basic information and detection features through a residual connection.

[0050] Furthermore, the aforementioned method of extracting features from the current stitched image and the historical stitched image obtained at the previous moment using a preset feature extraction algorithm to obtain corresponding current and historical features includes: extracting features from the current stitched image using the preset feature extraction algorithm to obtain current features, and obtaining historical features corresponding to the historical stitched image at the previous moment from a preset feature buffer. It will be appreciated that the preset feature buffer can be used to store the historical features of the latest frame. Therefore, only the current features of the current stitched image need to be extracted, and then the historical features stored at the last moment can be directly aggregated, thereby saving the time of reprocessing the last frame.

[0051] Step S13: predicting the motion trajectory of the moving target based on the position information and moving speed corresponding to the moving target and the moving trend information in the dynamic flow feature.

[0052] In this embodiment, the motion trajectory of the moving target is predicted based on the position information and moving speed corresponding to the moving target and the movement trend information captured in the dynamic flow features. It should be noted that before the above-mentioned prediction of the motion trajectory of the moving target based on the position information and moving speed corresponding to the moving target and the movement trend information in the dynamic flow features, it also includes: obtaining continuous image data and radar data including the moving target collected by each on-board camera and radar during the vehicle's driving process; using a target detection algorithm to detect the continuous image data and the radar data to obtain the position information and movement speed of the moving target. That is, during the automatic driving process of the vehicle, continuous images including the moving target collected by the on-board camera and radar data including the moving target collected by the radar are obtained, and then target detection is performed on the continuous image data and radar data using a target detection algorithm to obtain the position information and movement speed of the moving targets around the vehicle.

[0053] It can be seen that the present application obtains image data including moving targets collected by each vehicle-mounted camera at the current moment during vehicle driving, and stitches the image data of different vehicle-mounted cameras according to a preset image stitching method to obtain a current stitched image; uses a preset feature extraction algorithm to perform feature extraction on the current stitched image and the historical stitched image obtained at the previous moment to obtain corresponding current features and historical features, and connects the current features and the historical features through a preset connection function to obtain dynamic flow features; based on the position information and moving speed corresponding to the moving target, and according to the movement trend information in the dynamic flow features, the motion trajectory of the moving target is predicted. It can be seen that the present application can avoid the problem of wasting computer resources caused by repeated processing of overlapping data by stitching the image data collected by different vehicle-mounted cameras at the current moment, and extract features from the current stitched image obtained at the current moment and the historical stitched image of the previous moment adjacent to it through a preset feature extraction algorithm to obtain corresponding features, and then connect the extracted features to obtain dynamic flow features, and then capture the movement trend information of the moving target in the dynamic flow features to predict the motion trajectory of the moving target, thereby improving the accuracy of the motion trajectory prediction.

[0054] See also Figure 4 As shown, the embodiment of the present application discloses a specific method for predicting the trajectory of a moving target. Compared with the previous embodiment, this embodiment further illustrates and optimizes the technical solution. Specifically, it includes:

[0055] Step S21: acquiring image data including moving targets captured by each vehicle-mounted camera at the current moment while the vehicle is traveling, forming adjacent image groups based on the image data of adjacent vehicle-mounted cameras, and calculating the relative displacement between two images in each adjacent image group.

[0056] In this embodiment, after acquiring the image data captured by each vehicle-mounted camera at the current moment, adjacent image groups are formed based on the image data from adjacent vehicle-mounted cameras, and then the relative displacement between two images in the adjacent image groups is calculated. Specifically, calculating the relative displacement between two images in each adjacent image group includes: performing a Fourier transform on the two images in each adjacent image group to obtain a Fourier transform image group, and calculating a normalized correlation coefficient for the Fourier transform image group; performing an inverse Fourier transform on the normalized correlation coefficient based on a preset maximum value, and mapping the index value corresponding to the preset maximum value to image coordinates to obtain the relative displacement corresponding to each adjacent image group. That is, first read the two image files Fi and Fj in the adjacent image group into objects Ii and Ij respectively, then perform a two-dimensional Fourier transform (FFT) on Ii and Ij to obtain the Fourier transform image group, which is recorded as FFTi and FFTj, and then calculate the normalized cross correlation coefficient (NCC) of FFTi and FFTj to obtain the corresponding NCC ij , which is the element-wise normalized conjugate multiplication of two complex vectors; then the two-dimensional inverse Fourier transform (IFFT) of the normalized correlation coefficient is calculated to obtain NCC -ij 1 , and reduce the inverse transform to a preset maximum value (i.e. max ij ), and mapping the index value corresponding to the preset maximum value back to the image coordinate (x, y). It can be understood that the Fourier transform is periodic in nature, so the overlap distances x and y are ambiguous and can be interpreted as x or (wx) and y or (hy). Finally, the cross-correlation factors CCF1ij..4 are calculated; where each cross-correlation factor corresponds to one of the four overlap patterns, namely (x or wx), (y or hy), (x or y), and (w–x or hy), and CCFmaxij is found and mapped back to (x, y) ij Finally, Fi and Fj can be synthesized and spliced ​​to obtain Fij. The above steps can be found in Figure 5 As shown, the technical solution adopted for other adjacent image groups such as Fm and Fn is consistent with the processing method of Fi and Fj, and Fmn is obtained. Finally, the obtained Fij and Fmn are processed according to the above processing method to obtain the final stitched image.

[0057] Step S22: constructing a directed graph based on the relative displacement, and calculating the absolute displacement based on the directed graph, so as to synthesize the current stitched image using the absolute displacement.

[0058] In this embodiment, after the relative displacements are calculated, these displacements form an overconstrained system, which can be specifically represented as a directed graph, where vertices represent images and edges are associated with adjacent images. It is understandable that the overconstraint in the system is due to the equivalence between the absolute displacement of the image and the path sum in the graph, and the path sum must be path-invariant. Therefore, it is necessary to resolve the overconstraint in the system and calculate the absolute displacement. Specifically, a subset of the relative displacements can be selected or adjusted to the path-invariant state in the graph using a global optimization method, and then the absolute displacements are used to synthesize the current stitched image.

[0059] It should be noted that the aforementioned image stitching process, synthesizing a large number of images, is a computationally and data-intensive task. Specifically, a scalable hybrid CPU-GPU image stitching implementation was employed. This method uses a Fourier-based image stitching method to process large image sets at near interactive rates. Image stitching can be roughly divided into three stages: the first stage computes the relative displacements between adjacent image pairs; the second stage resolves overconstraints in the system and computes the absolute displacements; and the third stage uses the absolute displacements to synthesize the stitched image. The algorithm focuses on the first stage, the relative displacement calculation, as this is the most computationally intensive stage; the second stage is less computationally intensive, while the third stage can be performed as needed as part of the stitched image visualization. It can be understood that the aforementioned implementation exploits coarse-grained parallelism, organizing computations into a pipeline architecture that spans CPU (central processing unit) and GPU (graphics processing unit) resources and overlaps computation with data movement. The hybrid CPU-GPU implementation utilizes coarse-grained parallelism, with each stage consisting of one or more CPU threads, some of which interact with the GPU. The pipeline overlaps various computations performed on the CPU or GPU kernels with data transfers between disk, main memory, and graphics memory. Each execution pipeline processes an image as follows:

[0060] (1) A CPU thread reads the image;

[0061] (2) A CPU thread copies data from the CPU to the GPU memory;

[0062] (3) A CPU thread starts FFT calculation on the GPU;

[0063] (4) A CPU thread manages the computation state, advancing adjacent tile pairs that are ready (i.e., their FFT is available) to the next stage;

[0064] (5) A CPU thread calls the relative displacement calculation (NCC, FFT-1, and maximum reduction). This stage copies the index of the maximum value (a single scalar) from the GPU to the CPU memory. It also adds an entry to the queue between stages 3 and 4 to handle memory management;

[0065] (6) Based on the number of available CPU cores, multiple CPU threads perform the CCF calculation. Each thread maps the index of the maximum value found in the previous stage to image coordinates and calculates four CCFij1..4 values. This produces the final x and y relative displacement of the image group.

[0066] The system has special memory management measures. As part of initialization, it allocates a memory pool on the GPU for each pipeline. The system allocates GPU memory only once to avoid any further allocations, which would force global synchronization of all kernels and memory transfers. This memory pool consists of a fixed number of buffers, one buffer per transform (forward or backward), and the size of the memory pool effectively limits the number of images in flight. In addition, each tile has a reference count that is decremented when the tile is used to calculate relative displacements; when the reference count reaches zero, the system reclaims the associated GPU buffer; this ensures that the system does not run out of memory.

[0067] Step S23: Using the CSPDarknet-53 algorithm, feature extraction is performed on the current stitched image and the historical stitched image obtained at the previous moment to obtain current FPN features and historical FPN features.

[0068] In this example, the current stitched image and the previous stitched image are used as input, and the shared-weight CSPDarknet-53 algorithm is used to extract features from each of the current and previous stitched images to obtain the corresponding current and historical FPN features. FPN, which stands for Feature Pyramid Networks, is used to extract features from the backbone CNN for subsequent object detection or semantic segmentation.

[0069] Step S24: reducing the number of channels of the current FPN feature and the historical FPN feature through a batch normalization layer and a SiLU activation function, and then using a concat function to connect the current FPN feature and the historical FPN feature to obtain a dynamic flow feature.

[0070] In this embodiment, a shared weight 1×1 convolution layer is used, followed by a batch normalization layer (BN) and SiLU activation function to reduce the number of channels of the two FPN features to half, and then the Backbone and FPN features of adjacent frames are connected using the concat function to obtain dynamic flow features, which are used to capture the motion trend of moving targets. Static flow measurement captures basic information and detection features through residual connections. For details of the above process, see Figure 6 shown.

[0071] Step S25: generating adaptive weights for the moving targets of different moving speeds based on the movement trend information in the dynamic flow features, and dynamically allocating the adaptive weights so as to predict the motion trajectory of the moving target based on the position information corresponding to the moving target.

[0072] In this embodiment, it is understood that the movement speed of each moving object within the same frame is completely different. Different movement trends arise from various factors: different sizes and movement states, occlusion, or different topological distances. Based on the movement trend information in the dynamic flow features, adaptive weights can be generated for moving objects of different movement speeds. Different adaptive weights are then dynamically assigned to predict the movement trajectory of the moving object based on the position information corresponding to the moving object.

[0073] It should be noted that the above-mentioned adaptive weighting is applied to each object based on its movement trend. The main process is as follows: sensor data is used as input to generate multimodal features of regions of interest and region proposals; a parallel detection and correlation network uses the region of interest and region proposal features to generate detection results; further refinement is achieved through the motion prediction and matching score ranking modules; and a mixed integer prediction module performs trajectory prediction based on this data. Generally speaking, this embodiment focuses on fast-moving objects because their future state is more difficult to predict. To quantitatively measure movement speed, a trend factor is introduced for each moving object. The Intersection over Union (IoU) matrix between the Fourier transform + 1 and the Fourier transform ground truth box is then calculated. The maximum value of the Fourier transform dimension is then performed to obtain the matching IoU of the corresponding objects between the two frames. A smaller value of this matching IoU indicates a faster moving object, and vice versa. If a new moving object appears in the form of Ft+1, there is no matching box for it, and its matching IoU is much smaller than usual. Loss weights are adaptively generated based on the movement speed of each moving object, dynamically assigning different weights to localize and predict each object.

[0074] It can be seen that in the embodiment of the present application, after the four vehicle-mounted cameras acquire real-time image data, in order to improve performance, the overlapping parts of the images are processed by using the CPU+GPU heterogeneous computing method, and these images with overlapping parts are stitched together; in order to solve the problem of unsafe autonomous driving decisions, a dual-stream sensing module and a trend factor are introduced to capture motion trends and basic detection features respectively, record and track one or more moving targets, and adaptively generate loss weights according to the trend information of each moving target, so as to dynamically assign different weights to locate and predict the motion trajectory of each object, thereby providing strong protection for autonomous driving decisions.

[0075] See also Figure 7 As shown, an embodiment of the present application discloses a moving target trajectory prediction device, which includes:

[0076] The image stitching module 11 is used to obtain image data including moving objects collected by each vehicle-mounted camera at the current moment during the vehicle's driving process, and stitch the image data of different vehicle-mounted cameras according to a preset image stitching method to obtain a current stitched image;

[0077] A dynamic flow feature acquisition module 12 is configured to extract features from the current spliced ​​image and the historical spliced ​​image obtained at a previous moment using a preset feature extraction algorithm to obtain corresponding current features and historical features, and to connect the current features and the historical features using a preset connection function to obtain dynamic flow features;

[0078] The trajectory prediction module 13 is configured to predict the motion trajectory of the moving target based on the position information and moving speed corresponding to the moving target and the moving trend information in the dynamic flow feature.

[0079] It can be seen that the present application obtains image data including moving targets collected by each vehicle-mounted camera at the current moment during vehicle driving, and stitches the image data of different vehicle-mounted cameras according to a preset image stitching method to obtain a current stitched image; uses a preset feature extraction algorithm to perform feature extraction on the current stitched image and the historical stitched image obtained at the previous moment to obtain corresponding current features and historical features, and connects the current features and the historical features through a preset connection function to obtain dynamic flow features; based on the position information and moving speed corresponding to the moving target, and according to the movement trend information in the dynamic flow features, the motion trajectory of the moving target is predicted. It can be seen that the present application can avoid the problem of wasting computer resources caused by repeated processing of overlapping data by stitching the image data collected by different vehicle-mounted cameras at the current moment, and extract features from the current stitched image obtained at the current moment and the historical stitched image of the previous moment adjacent to it through a preset feature extraction algorithm to obtain corresponding features, and then connect the extracted features to obtain dynamic flow features, and then capture the movement trend information of the moving target in the dynamic flow features to predict the motion trajectory of the moving target, thereby improving the accuracy of the motion trajectory prediction.

[0080] In some specific embodiments, before the trajectory prediction module 13, the following steps may also be included:

[0081] A data acquisition unit, configured to acquire continuous image data and radar data including moving targets collected by each vehicle-mounted camera and radar during vehicle travel;

[0082] The target detection unit is used to detect the continuous image data and the radar data using a target detection algorithm to obtain the position information and moving speed of the moving target.

[0083] In some specific embodiments, the image stitching module 11 may specifically include:

[0084] a relative displacement calculation submodule, configured to form adjacent image groups based on image data from adjacent vehicle-mounted cameras, and calculate the relative displacement between two images in each adjacent image group;

[0085] The absolute displacement calculation unit is used to construct a directed graph based on the relative displacement and calculate the absolute displacement based on the directed graph, so as to synthesize the current spliced ​​image using the absolute displacement.

[0086] In some specific embodiments, the relative displacement calculation submodule may specifically include:

[0087] a Fourier transform unit, configured to perform Fourier transform on the two images in each of the adjacent image groups to obtain a Fourier transform image group, and calculate a normalized correlation coefficient of the Fourier transform image group;

[0088] An image coordinate mapping unit is used to perform inverse Fourier transform on the normalized correlation coefficient based on a preset maximum value, and map the index value corresponding to the preset maximum value to image coordinates to obtain a relative displacement corresponding to each of the adjacent image groups.

[0089] In some specific embodiments, the dynamic flow feature acquisition module 12 may specifically include:

[0090] A feature extraction unit is used to perform feature extraction on the current spliced ​​image and the historical spliced ​​image obtained at a previous moment using a CSPDarknet-53 algorithm to obtain current FPN features and historical FPN features;

[0091] A feature connection unit is used to reduce the number of channels of the current FPN feature and the historical FPN feature through a batch normalization layer and a SiLU activation function, and then use a concat function to connect the current FPN feature and the historical FPN feature to obtain a dynamic flow feature.

[0092] In some specific embodiments, the dynamic flow feature acquisition module 12 may specifically include:

[0093] The feature acquisition unit is used to extract features from the current spliced ​​image using a preset feature extraction algorithm to obtain current features, and to acquire historical features corresponding to the historical spliced ​​image at a previous moment from a preset feature buffer.

[0094] In some specific embodiments, the trajectory prediction module 13 may specifically include:

[0095] A weight allocation unit is used to generate adaptive weights for the moving targets of different moving speeds based on the movement trend information in the dynamic flow characteristics, and dynamically allocate the adaptive weights so as to predict the motion trajectory of the moving target based on the position information corresponding to the moving target.

[0096] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Specifically, the device may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 is used to store a computer program, which is loaded and executed by the processor 21 to implement the relevant steps of the mobile target trajectory prediction method performed by the electronic device as disclosed in any of the aforementioned embodiments.

[0097] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and the external device. The communication protocol it follows is any communication protocol that can be applied to the technical solution of this application and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world. Its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0098] Among them, the processor 21 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 21 can be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 21 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 21 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 21 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0099] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or CD, etc. The resources stored thereon include an operating system 221, a computer program 222 and data 223, etc. The storage method can be temporary storage or permanent storage.

[0100] The operating system 221 is used to manage and control the hardware devices and computer programs 222 on the electronic device 20, enabling the processor 21 to calculate and process the massive amount of data 223 in the memory 22. It can be Windows, Unix, Linux, etc. In addition to including computer programs capable of implementing the mobile target trajectory prediction method performed by the electronic device 20 as disclosed in any of the aforementioned embodiments, the computer programs 222 may further include computer programs capable of performing other specific tasks. In addition to data received by the electronic device from external devices, the data 223 may also include data collected by its own input and output interface 25.

[0101] Furthermore, an embodiment of the present application also discloses a computer-readable storage medium, in which a computer program is stored. When the computer program is loaded and executed by a processor, the method steps performed in the mobile target trajectory prediction process disclosed in any of the aforementioned embodiments are implemented.

[0102] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from the other embodiments. Reference can be made to the descriptions of the identical or similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple, and the relevant parts can be referred to the descriptions of the methods.

[0103] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0104] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0105] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

[0106] The above is a detailed introduction to a mobile target trajectory prediction method, device, equipment and storage medium provided by the present invention. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core ideas. At the same time, for those skilled in the art, according to the ideas of the present invention, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A method for predicting a moving target trajectory, characterized in that: include: Acquire image data including moving objects captured by each onboard camera at a current moment during vehicle travel, and stitch the image data from different onboard cameras according to a preset image stitching method to obtain a current stitched image; wherein the number of the onboard cameras is four; Using a preset feature extraction algorithm, feature extraction is performed on the current spliced ​​image and the historical spliced ​​image obtained at a previous moment to obtain corresponding current features and historical features, and the current features and the historical features are connected by a preset connection function to obtain dynamic flow features; Predicting a motion trajectory of the moving target based on the position information and moving speed corresponding to the moving target and according to the moving trend information in the dynamic flow feature; The step of stitching the image data of different vehicle-mounted cameras according to a preset image stitching method to obtain a current stitched image includes: forming adjacent image groups based on image data from adjacent vehicle-mounted cameras, and calculating a relative displacement between two images in each adjacent image group; A directed graph is constructed based on the relative displacement, and an absolute displacement is calculated based on the directed graph, so as to synthesize a current stitched image using the absolute displacement.

2. The method for predicting a moving target trajectory according to claim 1, wherein: Before predicting the motion trajectory of the moving target based on the position information and the moving speed corresponding to the moving target and the moving trend information in the dynamic flow feature, the method further includes: Acquire continuous image data and radar data including moving targets collected by each vehicle-mounted camera and radar during vehicle driving; The continuous image data and the radar data are detected using a target detection algorithm to obtain the position information and moving speed of the moving target.

3. The method for predicting a moving target trajectory according to claim 1, wherein: The calculating the relative displacement between two images in each adjacent image group includes: Performing Fourier transform on the two images in each of the adjacent image groups to obtain a Fourier transform image group, and calculating a normalized correlation coefficient of the Fourier transform image group; Performing an inverse Fourier transform on the normalized correlation coefficient based on a preset maximum value, and mapping an index value corresponding to the preset maximum value to an image coordinate to obtain a relative displacement corresponding to each of the adjacent image groups.

4. The method for predicting a moving target trajectory according to claim 1, wherein: The method of extracting features from the current spliced ​​image and the historical spliced ​​image obtained at a previous moment using a preset feature extraction algorithm to obtain corresponding current features and historical features, and connecting the current features and the historical features using a preset connection function to obtain dynamic flow features includes: Using the CSPDarknet-53 algorithm, feature extraction is performed on the current stitched image and the historical stitched image obtained at the previous moment to obtain current FPN features and historical FPN features; The number of channels of the current FPN feature and the historical FPN feature is reduced by a batch normalization layer and a SiLU activation function, and then the current FPN feature and the historical FPN feature are connected using a concat function to obtain a dynamic flow feature.

5. The method for predicting a moving target trajectory according to claim 1, wherein: The method of extracting features from the current spliced ​​image and the historical spliced ​​image obtained at a previous moment using a preset feature extraction algorithm to obtain corresponding current features and historical features includes: A preset feature extraction algorithm is used to perform feature extraction on the current spliced ​​image to obtain current features, and historical features corresponding to the historical spliced ​​image at a previous moment are obtained from a preset feature buffer.

6. The method for predicting a moving target trajectory according to any one of claims 1 to 5, wherein: The predicting of the motion trajectory of the mobile target based on the position information and the moving speed corresponding to the mobile target and the moving trend information in the dynamic flow feature includes: Adaptive weights are generated for the moving targets of different moving speeds based on the movement trend information in the dynamic flow features, and the adaptive weights are dynamically allocated to predict the motion trajectory of the moving target based on the position information corresponding to the moving target.

7. A moving target trajectory prediction device, characterized in that: include: An image stitching module, configured to obtain image data including moving objects captured by each vehicle-mounted camera at a current moment while the vehicle is traveling, and stitch the image data from different vehicle-mounted cameras according to a preset image stitching method to obtain a current stitched image; wherein the number of vehicle-mounted cameras is four; a dynamic flow feature acquisition module, configured to extract features from the current spliced ​​image and the historical spliced ​​image obtained at a previous moment using a preset feature extraction algorithm to obtain corresponding current features and historical features, and to connect the current features and the historical features using a preset connection function to obtain dynamic flow features; a trajectory prediction module, configured to predict the motion trajectory of the moving target based on the position information and moving speed corresponding to the moving target and the movement trend information in the dynamic flow feature; The image stitching module specifically includes: a relative displacement calculation submodule, configured to form adjacent image groups based on image data from adjacent vehicle-mounted cameras, and calculate the relative displacement between two images in each adjacent image group; The absolute displacement calculation unit is used to construct a directed graph based on the relative displacement and calculate the absolute displacement based on the directed graph, so as to synthesize the current spliced ​​image using the absolute displacement.

8. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the steps of the moving target trajectory prediction method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that Used to store a computer program; wherein, when the computer program is executed by a processor, the steps of the mobile target trajectory prediction method according to any one of claims 1 to 6 are implemented.

Images