Moving target trajectory prediction method and system based on deep learning

Abstract

A moving target trajectory prediction method based on deep learning optimizes input image data through image equalization and normalization processing, and uses a YOLO network for training to extract image features and classify the image features. And after the training is completed, inputting the data set test set image into the YOLO network for detection, and evaluating whether the trained YOLO network meets the detection requirement or not. And matching the detection target of the current frame with the prediction target at the previous moment through a KM matching algorithm. And then, predicting the position of each moving target at the next moment by adopting a self-adaptive unscented Kalman filtering algorithm. And the system constructs a trajectory prediction model of the moving target through the dynamic observation equation, continuously optimizes the filtering process, and finally outputs a trajectory prediction result of the moving target. According to the method, under the requirements of real-time performance and high precision, recognition and prediction of moving objects such as balls and athletes are achieved, and improvement of training scientificity, prevention of injuries and diseases, optimization of competition strategies, promotion of technical innovation and the like are facilitated.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing and deep learning, and in particular to a method and system for predicting the trajectory of a moving target based on deep learning. Background Art

[0002] As an important branch of artificial intelligence, deep learning does not require complex rules to be set in advance. Through continuous learning and optimization, deep learning models can demonstrate capabilities beyond human capabilities in many specific tasks. With the rapid development of the sports industry, athletes' training and development plans are also facing higher requirements. In order to improve the competitive level and competition results, more scientific means are needed to guide athletes' training and preparation. As a cutting-edge artificial intelligence technology, deep learning has shown significant advantages in improving the scientific nature of training, preventing injuries, optimizing competition strategies, and promoting technological innovation by predicting the next moment position of sports targets such as balls and athletes. Deep learning technology can help coaching teams develop more personalized and accurate training plans, optimize resource allocation, and thus improve training efficiency. With the help of deep learning technology, the sports industry can better discover and cultivate talents and promote the continuous progress and development of sports.

[0003] Current motion trajectory prediction research focuses on building kinematic and dynamic models. However, since the motion trajectories of most targets present nonlinear characteristics, complex curved trajectories often appear, which makes model-based trajectory prediction algorithms face the challenge of low accuracy when dealing with these problems. In recent years, some scholars have turned to studying data-driven trajectory prediction algorithms, using classification and regression to deal with trajectory prediction problems. For example, the prior art discloses a "trajectory prediction method for multi-dimensional spatiotemporal feature fusion for autonomous driving". This method uses the Transformer encoder combined with the long short-term memory network (LSTM) for time modeling to achieve the effect of predicting motion trajectories. However, this method faces challenges in real-time and accuracy in the target detection network, and is prone to loss of targets and data. At the same time, trajectory prediction also requires a large amount of continuous data and a large amount of calculation. Therefore, there is an urgent need for a motion target trajectory prediction method and system based on deep learning to solve the problems of the prior art. Summary of the invention

[0004] In view of the above problems, the present invention is proposed to provide a method and system for predicting the trajectory of a moving target based on deep learning, which overcomes the above problems or at least partially solves the above problems.

[0005] In order to solve the above technical problems, the embodiments of the present application disclose the following technical solutions:

[0006] In a first aspect, an embodiment of the present invention discloses a moving target trajectory prediction method based on deep learning, comprising:

[0007] S100. Read the training set images in the training set, perform brightness equalization and normalization processing on the training set images, and train the YOLO network model using the images after brightness equalization and normalization processing to obtain the YOLO network model after training;

[0008] S200. Input the test set image in the data set into the trained YOLO network model for detection, and evaluate the trained YOLO network model by calculating the difference between the observed vector and the expected measurement value;

[0009] S300. If the YOLO network model is evaluated to be passed after the training is completed, the moving target image acquired by the camera in real time is input into the YOLO network model after the training is completed, and the moving target image is preprocessed before the moving target trajectory prediction;

[0010] S400. Use an adaptive unscented Kalman filter algorithm on the preprocessed moving target image to predict the position of the moving target image at the next moment, and output a predicted value of the moving target trajectory.

[0011] Furthermore, in S100, brightness equalization processing is performed on the training set images. The specific method includes: converting the RGB image in the training set images into a YCbCr color model, performing histogram processing on the brightness component in the model, and converting the processed image YCbCr model into an RGB model output. The histogram processing formula is:

[0012]

[0013] Among them, S n Indicates the brightness component value after processing; X i Represents the number of pixels with brightness component value i; N represents the total number of pixels.

[0014] Furthermore, in S100, the YOLO network model is trained using the image after brightness equalization and normalization processing, and the specific method includes: inputting the processed image into the convolution layer of the YOLO network model, extracting image feature information, and classifying the image feature information, the convolution layer includes three types of convolution kernels: 1×1, 3×3 and 7×7, and the convolution layer uses a leakage correction function as an activation function; constructing a loss function in the YOLO network model training, the loss function is the error between the detection value and the actual situation, and optimizing the weight parameters of the YOLO network model through back propagation to perform network training. When the loss function is less than a threshold, the network model is in a detection state, otherwise it is retrained.

[0015] Furthermore, in S100, during the training of the YOLO network model, the loss of the YOLO network model is the mean square error between the predicted values ​​and the true values ​​of all cells. The loss function of the YOLO network model consists of three parts, namely, boundary loss, confidence loss, and classification loss. The expression of the loss function is:

[0016] loss = α 1 loss 1 +α 2 loss 2 +α 3 loss 3

[0017] Among them, loss represents the total loss; loss 1 Indicates the error between the predicted coordinates and the actual coordinates, i.e., the boundary loss; loss 2 Represents the confidence error of each bounding box prediction value, i.e., confidence loss; loss 3 Represents the probability value error during classification, i.e., classification loss; α 1 , α 2 , α 3 Represents the error loss factor.

[0018] Further, in S200, the observation vector includes the bounding box coordinates, category and confidence predicted by the model; the expected measurement value includes the bounding box coordinates and category of the real annotated bounding box; for each predicted bounding box, the intersection and union ratio of the predicted bounding box and the nearest real bounding box is calculated, and when the intersection and union ratio exceeds a preset threshold, the YOLO network model evaluation passes after the training is completed.

[0019] Furthermore, in S300, the moving target image is preprocessed before moving target trajectory prediction, and the specific method includes:

[0020] S301. Perform brightness equalization and normalization on the camera image, and load the trained YOLO network model;

[0021] S302. Detect the position x(k), y(k), s(k), and r(k) of the moving target in the image at the current moment according to the YOLO network model, assign an ID to the moving target created for detection, place each target object in the detection set G, and determine whether the input image is the first frame. If the input image is the first frame, directly proceed to S400; if the input image is not the first frame, directly proceed to S303;

[0022] S303, target matching, for the predicted set A of the target predicted at the previous moment, and the union T of the unsuccessfully matched set B, let T = A∪B; use the KM matching algorithm to determine whether the current frame detection target set G matches the set T, and the specific determination method includes: first determine whether the number of elements in the current detection target set G is greater than or equal to the number of elements in the set T, if the number of elements in the set G is greater than or equal to the number of elements in the set T, then place the set G on the right side of the bipartite graph to initiate matching; if the number of elements in the set G is less than the number of elements in the set T, place the set T on the right side of the bipartite graph to initiate matching; then use the intersection and union ratio of the current frame detection frame and the predicted frame at the previous moment as the weight of the bipartite graph, and use the KM matching algorithm to obtain an optimal match that maximizes the sum of the weights.

[0023] Further, in S400, an adaptive unscented Kalman filter algorithm is used on the preprocessed moving target image to predict the position of the moving target image at the next moment, and a predicted value of the moving target image trajectory is output. The specific method includes:

[0024] S401. Construct state vector Where f is a nonlinear function, h is the observation function, x is the state vector, y is the observation vector, and w is the k is the L-dimensional process noise matrix, v k is the L-dimensional measurement noise matrix;

[0025] S402. Initialize parameters and obtain the expected value through process noise variance Q and measurement noise R and the covariance matrix where x k represents the state vector of the object at time k, P x,0 is the variance of the initial state and augmented state, E is the expected value, and a is the test point;

[0026] S403. Obtain sigma test points and perform proportional sampling correction on the obtained sigma test points; the proportional correction parameter α∈[0,1] is used to construct an augmented sigma point according to the expected value and covariance matrix in S402 Where N is the dimension in the augmented state, j is the element index;

[0027] S404. Update equations through time Find the estimated value of the sampling point state Where k and k-1 represent the k-th and k-1-th moments, x and w represent the test points, and j represents the j-th element in X;

[0028] S405. Update equations through observations Find the observed estimate h represents the observation function, thus obtaining the prior measurement estimate and the prior mean square error P y,k , and introduce the noise adaptive coefficient β for optimization, the noise adaptive coefficient β∈[0,1];

[0029] S406. Update the Kalman filter according to the output sigma point, sampling point state estimate and observation estimate obtained in S403, S404 and S305;

[0030] S407. Establish state equation X k =AX k-1 +W k , the state transfer matrix T is the time interval, W k is zero-mean Gaussian noise. The current coordinate position and velocity of the moving target are (x, y, z) and (vx, vy, vz) respectively. The state vector at time t is represented by X t =[x,v x ,y,v y ,z,v z ] T ;

[0031] S408. Output the predicted value of the moving target trajectory and compare it with the true value. When the difference between the predicted value and the true value is less than a preset threshold, it is determined that the current predicted value meets the requirements, and the predicted value of the moving target trajectory is output.

[0032] In a second aspect, an embodiment of the present invention discloses a moving target trajectory prediction system based on deep learning, including: a YOLO network model training unit, a YOLO network model evaluation unit, a moving target trajectory prediction preprocessing unit and a moving target trajectory prediction value output unit; wherein:

[0033] The YOLO network model training unit is used to read the training set images in the training set, perform brightness equalization and normalization processing on the training set images, and train the YOLO network model using the images after brightness equalization and normalization processing to obtain the YOLO network model after training.

[0034] A YOLO network model evaluation unit, used for inputting the test set image in the data set into the trained YOLO network model for detection, and evaluating the trained YOLO network model by calculating the difference between the observed vector and the expected measurement value;

[0035] A moving target trajectory prediction preprocessing unit, used for inputting the moving target image acquired by the camera in real time into the trained YOLO network model, and performing preprocessing on the moving target image before moving target trajectory prediction;

[0036] The moving target trajectory prediction value output unit is used to use an adaptive unscented Kalman filter algorithm on the moving target image after preprocessing to predict the position of the moving target image at the next moment and output the prediction value of the moving target trajectory.

[0037] In a third aspect, an embodiment of the present invention discloses an electronic device, including:

[0038] one or more processors;

[0039] A memory for storing one or more programs;

[0040] When the one or more programs are executed by the one or more processors, the one or more processors implement the moving target trajectory prediction method.

[0041] The beneficial effects of the above technical solution provided by the embodiment of the present invention include at least:

[0042] The invention discloses a motion target trajectory prediction method based on deep learning. By image equalization and normalization processing, input image data is optimized, and a YOLO network is used for training to extract image features and classify them. After the training is completed, the data set test set image is input into the YOLO network for detection to evaluate whether the trained YOLO network meets the detection needs. The detection target of the current frame is matched with the predicted target of the previous moment by the KM matching algorithm. Subsequently, an adaptive unscented Kalman filter algorithm is used to predict the position of each motion target at the next moment. The system constructs a trajectory prediction model of the motion target through a dynamic observation equation, continuously optimizes the filtering process, and finally outputs the trajectory prediction result of the motion target. The invention solves the recognition and prediction of motion objects such as balls and athletes under the requirements of real-time and high precision, which is conducive to improving the scientific nature of training, preventing injuries, optimizing competition strategies, and promoting technological innovation.

[0043] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0045] Figure 1 This is a flowchart of a method for predicting moving target trajectories based on deep learning in Example 1 of the present invention;

[0046] Figure 2 This is a logic diagram of a moving target trajectory prediction method based on deep learning in Example 1 of the present invention;

[0047] Figure 3 This is a flow chart of performing brightness equalization processing on an image in Embodiment 1 of the present invention;

[0048] Figure 4 This is a structural diagram of a moving target trajectory prediction system based on deep learning in Example 2 of the present invention;

[0049] Figure 5 This is a schematic diagram of the structure of an electronic device in Example 3 of the present invention. DETAILED DESCRIPTION

[0050] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0051] In order to solve the problems existing in the prior art, an embodiment of the present invention provides a method and system for predicting the trajectory of a moving target based on deep learning.

[0052] Example 1

[0053] The present invention discloses a method for predicting the trajectory of a moving target based on deep learning. Figure 1 and 2 ,include:

[0054] S100. Read the training set images in the training set, perform brightness equalization and normalization processing on the training set images, and use the images after brightness equalization and normalization processing to train the YOLO network model to obtain the YOLO network model after training; Specifically, YOLO (You Only Look Once) is a real-time target detection algorithm, which abandons the sliding window and candidate region generation steps commonly used in traditional target detection algorithms, and directly predicts the target's bounding box and category probability through a single forward propagation. YOLO transforms the target detection problem into a single regression problem, thereby significantly improving the detection speed and accuracy.

[0055] In this embodiment, the YOLO network model is composed of CSPDarkNet53 as the backbone network, SPPF as the field of view enhancement module and PANet feature fusion module. In S100 of this embodiment, the training set image is subjected to brightness equalization processing to eliminate interference factors in the natural environment, such as the influence of occlusions, lighting changes and weather conditions on the experimental results, and enhance the image feature information. Figure 3The specific method includes: converting the RGB image in the training set image into a YCbCr color model, performing histogram processing on the brightness component in the model, and converting the processed image YCbCr model into an RGB model output. The histogram processing formula is:

[0056]

[0057] Among them, S n Indicates the brightness component value after processing; X i Represents the number of pixels with brightness component value i; N represents the total number of pixels.

[0058] In S100 of this embodiment, the image is normalized. In view of the problems of inconsistent image format, inconsistent size, irregular naming, etc. in the data set, an image normalization method is used for unified processing. The YOLO network model is trained using the image after brightness equalization and normalization. The specific method includes: inputting the processed image into the convolution layer of the YOLO network model, extracting image feature information, and classifying the image feature information, the convolution layer includes three types of convolution kernels: 1×1, 3×3 and 7×7, and the convolution layer uses a leakage correction function as an activation function; constructing a loss function in the YOLO network model training, the loss function is the error between the detection value and the actual situation, and optimizing the weight parameters of the YOLO network model through back propagation to perform network training. When the loss function is less than the threshold, the network model is in a detection state, otherwise it is retrained.

[0059] In S100 of this embodiment, during the training process of the YOLO network model, the loss of the YOLO network model is the mean square error between the predicted values ​​and the true values ​​of all cells. The loss function of the YOLO network model consists of three parts, namely, boundary loss, confidence loss and classification loss. The expression of the loss function is:

[0060] loss = α 1 loss 1 +α 2 loss 2 +α 3 loss 3

[0061] Among them, loss represents the total loss; loss 1 Indicates the error between the predicted coordinates and the actual coordinates, i.e., the boundary loss; loss 2 Represents the confidence error of each bounding box prediction value, i.e., confidence loss; loss 3 Represents the probability value error during classification, i.e., classification loss; α 1 , α 2 , α 3Represents the error loss factor.

[0062] S200. Input the test set image in the data set into the trained YOLO network model for detection, and evaluate the trained YOLO network model by calculating the difference between the observed vector and the expected measurement value;

[0063] In S200 of this embodiment, the observation vector includes the bounding box coordinates, category and confidence predicted by the model; the expected measurement value includes the bounding box coordinates and category of the real annotated bounding box; for each predicted bounding box, the intersection and union ratio of the predicted bounding box and the nearest real bounding box is calculated, and when the intersection and union ratio exceeds a preset threshold, the YOLO network model evaluation passes after the training is completed.

[0064] S300. If the YOLO network model is evaluated to be passed after the training is completed, the moving target image acquired by the camera in real time is input into the YOLO network model after the training is completed, and the moving target image is preprocessed before the moving target trajectory prediction;

[0065] In S300 of this embodiment, the moving target image is preprocessed before moving target trajectory prediction, and the specific method includes:

[0066] S301. Perform brightness equalization and normalization on the camera image, and load the trained YOLO network model;

[0067] S302. Detect the position x(k), y(k), s(k), and r(k) of the moving target in the image at the current moment according to the YOLO network model, assign an ID to the moving target created for detection, place each target object in the detection set G, and determine whether the input image is the first frame. If the input image is the first frame, directly proceed to S400; if the input image is not the first frame, directly proceed to S303;

[0068] S303, target matching, for the predicted set A of the target predicted at the previous moment, and the union T of the unsuccessfully matched set B, let T = A∪B; use the KM matching algorithm to determine whether the current frame detection target set G matches the set T, and the specific determination method includes: first determine whether the number of elements in the current detection target set G is greater than or equal to the number of elements in the set T, if the number of elements in the set G is greater than or equal to the number of elements in the set T, then place the set G on the right side of the bipartite graph to initiate matching; if the number of elements in the set G is less than the number of elements in the set T, place the set T on the right side of the bipartite graph to initiate matching; then use the intersection and union ratio of the current frame detection frame and the predicted frame at the previous moment as the weight of the bipartite graph, and use the KM matching algorithm to obtain an optimal match that maximizes the sum of the weights.

[0069] S400. Use an adaptive unscented Kalman filter algorithm on the preprocessed moving target image to predict the position of the moving target image at the next moment, and output a predicted value of the moving target trajectory.

[0070] In S400 of this embodiment, an adaptive unscented Kalman filter algorithm is used on the preprocessed moving target image to predict the position of the moving target image at the next moment, and a predicted value of the moving target image trajectory is output. The specific method includes:

[0071] S401. Construct state vector Where f is a nonlinear function, h is the observation function, x is the state vector, y is the observation vector, and w is the k is the L-dimensional process noise matrix, v k is the L-dimensional measurement noise matrix;

[0072] S402. Initialize parameters and obtain the expected value through process noise variance Q and measurement noise R and the covariance matrix where x k represents the state vector of the object at time k, P x,0 is the variance of the initial state and augmented state, E is the expected value, and a is the test point;

[0073] S403, obtain sigma test points, and perform proportional sampling correction on the obtained sigma test points; the proportional correction parameter α∈[0,1], constructs an augmented sigma point according to the expected value and covariance matrix in S402 Where N is the dimension in the augmented state, j is the element index;

[0074] S404. Update equations through time Find the estimated value of the sampling point state Where k and k-1 represent the k-th and k-1-th moments, x and w represent the test points, and j represents the j-th element in X;

[0075] S405. Update equations through observations Find the observed estimate h represents the observation function, thus obtaining the prior measurement estimate and the prior mean square error P y,k , and introduce the noise adaptive coefficient β for optimization, the noise adaptive coefficient β∈[0,1];

[0076] S406. Update the Kalman filter according to the output sigma point, sampling point state estimate and observation estimate obtained in S403, S404 and S305;

[0077] S407. Establish state equation Xk =AX k-1 +W k , the state transfer matrix T is the time interval, W k is zero-mean Gaussian noise. The current coordinate position and velocity of the moving target are (x, y, z) and (vx, vy, vz) respectively. The state vector at time t is represented by X t =[x,v x ,y,v y ,z,v z ] T ;

[0078] S408. Output the predicted value of the moving target trajectory and compare it with the true value. When the difference between the predicted value and the true value is less than a preset threshold, it is determined that the current predicted value meets the requirements, and the predicted value of the moving target trajectory is output.

[0079] The present embodiment discloses a method for predicting the trajectory of a moving target based on deep learning. By image equalization and normalization processing, the input image data is optimized, and the YOLO network is used for training to extract image features and classify them. After the training is completed, the test set image of the data set is input into the YOLO network for detection to evaluate whether the trained YOLO network meets the detection needs. The detection target of the current frame is matched with the predicted target of the previous moment by the KM matching algorithm. Subsequently, an adaptive unscented Kalman filter algorithm is used to predict the position of each moving target at the next moment. The system constructs a trajectory prediction model of the moving target through a dynamic observation equation, continuously optimizes the filtering process, and finally outputs the trajectory prediction result of the moving target. The present invention solves the recognition and prediction of moving objects such as balls and athletes under real-time and high-precision requirements, which is conducive to improving the scientific nature of training, preventing injuries, optimizing competition strategies, and promoting technological innovation.

[0080] Example 2

[0081] Based on the same inventive concept, the embodiment of the present disclosure also provides a moving target trajectory prediction system based on deep learning, such as Figure 4 , including: a YOLO network model training unit, a YOLO network model evaluation unit, a moving target trajectory prediction pre-processing unit and a moving target trajectory prediction value output unit; wherein:

[0082] The YOLO network model training unit is used to read the training set images in the training set, perform brightness equalization and normalization processing on the training set images, and train the YOLO network model using the images after brightness equalization and normalization processing to obtain the YOLO network model after training.

[0083] A YOLO network model evaluation unit, used for inputting the test set image in the data set into the trained YOLO network model for detection, and evaluating the trained YOLO network model by calculating the difference between the observed vector and the expected measurement value;

[0084] A moving target trajectory prediction preprocessing unit, used for inputting the moving target image acquired by the camera in real time into the trained YOLO network model, and performing preprocessing on the moving target image before moving target trajectory prediction;

[0085] The moving target trajectory prediction value output unit is used to use an adaptive unscented Kalman filter algorithm on the moving target image after preprocessing to predict the position of the moving target image at the next moment and output the prediction value of the moving target trajectory.

[0086] The moving target trajectory prediction value output unit is used to use an adaptive unscented Kalman filter algorithm on the moving target image after preprocessing to predict the position of the moving target image at the next moment and output the predicted value of the moving target trajectory. The specific method includes:

[0087] S401. Construct state vector Where f is a nonlinear function, h is the observation function, x is the state vector, y is the observation vector, and w is the k is the L-dimensional process noise matrix, v k is the L-dimensional measurement noise matrix;

[0088] S402. Initialize parameters and obtain the expected value through process noise variance Q and measurement noise R and the covariance matrix where x k represents the state vector of the object at time k, P x,0 is the variance of the initial state and augmented state, E is the expected value, and a is the test point;

[0089] S403. Obtain sigma test points and perform proportional sampling correction on the obtained sigma test points; the proportional correction parameter α∈[0,1] is used to construct an augmented sigma point according to the expected value and covariance matrix in S402 Where N is the dimension in the augmented state, j is the element index;

[0090] S404. Update equations through time Find the estimated value of the sampling point state Where k and k-1 represent the k-th and k-1-th moments, x and w represent the test points, and j represents the j-th element in X;

[0091] S405. Update equations through observations Find the observed estimate h represents the observation function, thus obtaining the prior measurement estimate and the prior mean square error P y,k , and introduce the noise adaptive coefficient β for optimization, the noise adaptive coefficient β∈[0,1];

[0092] S406. Update the Kalman filter according to the output sigma point, sampling point state estimate and observation estimate obtained in S403, S404 and S305;

[0093] S407. Establish state equation X k =AX k-1 +W k , the state transfer matrix T is the time interval, W k is zero-mean Gaussian noise. The current coordinate position and velocity of the moving target are (x, y, z) and (vx, vy, vz) respectively. The state vector at time t is represented by X t =[x,v x ,y,v y ,z,v z ] T ;

[0094] S408. Output the predicted value of the moving target trajectory and compare it with the true value. When the difference between the predicted value and the true value is less than a preset threshold, it is determined that the current predicted value meets the requirements, and the predicted value of the moving target trajectory is output.

[0095] In this embodiment, the specific methods of the YOLO network model training unit, the YOLO network model evaluation unit, and the moving target trajectory prediction pre-processing unit have been described in detail in Example 1, and will not be repeated in this embodiment.

[0096] Example 3

[0097] Based on the same inventive concept, the embodiment of the present disclosure also provides an electronic device. FIG. 5 is a schematic diagram of the structure of an electronic device according to the embodiment of the present disclosure. Figure 5 As shown, an embodiment of the present disclosure provides an electronic device including: one or more processors 101, a memory 102, and one or more I / O interfaces 103. The memory 102 stores one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors implement any optimization method in the above-mentioned embodiment 1; the one or more I / O interfaces 103 are connected between the processor and the memory, and are configured to implement information interaction between the processor and the memory.

[0098] Among them, the processor 101 is a device with data processing capabilities, including but not limited to a central processing unit (CPU), etc.; the memory 102 is a device with data storage capabilities, including but not limited to random access memory (RAM, more specifically SDRAM, DDR, etc.), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory (FLASH); the I / O interface (read-write interface) 103 is connected between the processor 101 and the memory 102, and can realize information interaction between the processor 101 and the memory 102, including but not limited to a data bus (Bus), etc.

[0099] In some embodiments, the processor 101 , the memory 102 , and the I / O interface 103 are connected to each other via a bus 104 , and further connected to other components of the computing device.

[0100] In some embodiments, the one or more processors 101 include a field programmable gate array.

[0101] According to an embodiment of the present disclosure, a computer-readable medium is further provided, wherein a computer program is stored on the computer-readable medium, wherein when the program is executed by a processor, the steps in any optimization method in the above-mentioned embodiment are implemented.

[0102] It should be understood that the specific order or hierarchy of steps in the disclosed process is an example of an exemplary method. Based on design preferences, it should be understood that the specific order or hierarchy of steps in the process can be rearranged without departing from the scope of protection of the present disclosure. The attached method claims present the elements of the various steps in an exemplary order and are not intended to be limited to the specific order or hierarchy described.

[0103] In the above detailed description, various features are grouped together in a single embodiment to simplify the disclosure. This method of disclosure should not be interpreted as reflecting an intention that the embodiments of the claimed subject matter require more features than are clearly stated in each claim. On the contrary, as reflected in the appended claims, the invention is in a state of having less than all the features of the disclosed individual embodiments. Therefore, the appended claims are hereby expressly incorporated into the detailed description, with each claim standing on its own as a separate preferred embodiment of the invention.

[0104] Those skilled in the art will also appreciate that the various illustrative logic blocks, modules, circuits, and algorithmic steps described in conjunction with the embodiments herein can all be implemented as electronic hardware, computer software, or a combination thereof. In order to clearly illustrate the interchangeability between hardware and software, various illustrative components, blocks, modules, circuits, and steps are generally described above around their functions. Whether such functions are implemented as hardware or software depends on specific applications and the design constraints imposed on the entire system. A skilled person can implement the described functions in an alternative manner for each specific application, but such implementation decisions should not be interpreted as departing from the scope of protection of the present disclosure.

[0105] The steps of the method or algorithm described in conjunction with the embodiments herein may be directly embodied as hardware, a software module executed by a processor, or a combination thereof. The software module may be located in a RAM memory, a flash memory, a ROM memory, an EPROM memory, an EEPROM memory, a register, a hard disk, a mobile disk, a CD-ROM, or any other form of storage medium well known in the art. An exemplary storage medium is connected to the processor so that the processor can read information from the storage medium and can write information to the storage medium. Of course, the storage medium may also be an integral part of the processor. The processor and the storage medium may be located in an ASIC. The ASIC may be located in a user terminal. Of course, the processor and the storage medium may also be present in a user terminal as discrete components.

[0106] For software implementation, the techniques described in this application can be implemented with modules (e.g., procedures, functions, etc.) that perform the functions described in this application. These software codes can be stored in a memory unit and executed by a processor. The memory unit can be implemented within the processor or outside the processor. In the latter case, it is coupled to the processor in a communication manner via various means, which are well known in the art.

[0107] The above description includes examples of one or more embodiments. Of course, it is impossible to describe all possible combinations of components or methods for the purpose of describing the above embodiments, but it should be recognized by those skilled in the art that the various embodiments may be further combined and arranged. Therefore, the embodiments described herein are intended to cover all such changes, modifications and variations that fall within the scope of protection of the appended claims. In addition, with respect to the term "comprising" used in the specification or claims, the word is covered in a manner similar to the term "including", just as "including," is explained as a transitional word in the claims. In addition, any term "or" used in the specification of the claims is intended to mean "non-exclusive or".

Claims

1. A moving target trajectory prediction method based on deep learning, characterized in that: include: S100. Read the training set images in the training set, perform brightness equalization and normalization processing on the training set images, and train the YOLO network model using the images after brightness equalization and normalization processing to obtain the YOLO network model after training; S200. Input the test set image in the data set into the trained YOLO network model for detection, and evaluate the trained YOLO network model by calculating the difference between the observed vector and the expected measurement value; S300. If the YOLO network model is evaluated to be passed after the training is completed, the moving target image acquired by the camera in real time is input into the YOLO network model after the training is completed, and the moving target image is preprocessed before the moving target trajectory prediction; S400. Use an adaptive unscented Kalman filter algorithm on the preprocessed moving target image to predict the position of the moving target image at the next moment, and output a predicted value of the moving target trajectory.

2. A method for predicting moving target trajectories based on deep learning as claimed in claim 1, characterized in that: In S100, brightness equalization processing is performed on the training set images. The specific method includes: converting the RGB image in the training set images into a YCbCr color model, performing histogram processing on the brightness component in the model, and converting the processed image YCbCr model into an RGB model output. The histogram processing formula is: Among them, S n Indicates the brightness component value after processing; X i Represents the number of pixels with brightness component value i; N represents the total number of pixels.

3. A method for predicting moving target trajectories based on deep learning as claimed in claim 1, characterized in that: In S100, the YOLO network model is trained using the image after brightness equalization and normalization processing. The specific method includes: inputting the processed image into the convolution layer of the YOLO network model, extracting image feature information, and classifying the image feature information, the convolution layer includes three types of convolution kernels: 1×1, 3×3 and 7×7, and the convolution layer uses a leakage correction function as an activation function; constructing a loss function in the YOLO network model training, the loss function is the error between the detection value and the actual situation, and optimizing the weight parameters of the YOLO network model through back propagation to perform network training. When the loss function is less than a threshold, the network model is in a detection state, otherwise it is retrained.

4. A method for predicting moving target trajectories based on deep learning as claimed in claim 1, characterized in that: In S100, during the training of the YOLO network model, the loss of the YOLO network model is the mean square error between the predicted values ​​and the true values ​​of all cells. The loss function of the YOLO network model consists of three parts, namely boundary loss, confidence loss and classification loss. The expression of the loss function is: loss=α1loss1+α2loss2+α3loss3 Among them, loss represents the total loss; loss1 represents the error between the predicted coordinates and the actual coordinates, that is, the boundary loss; loss2 represents the confidence error of the predicted value of each bounding box, that is, the confidence loss; loss3 represents the probability value error during classification, that is, the classification loss; α1, α2, and α3 represent error loss factors.

5. The method for predicting moving target trajectories based on deep learning according to claim 1, characterized in that: In S200, the observation vector includes the bounding box coordinates, category and confidence predicted by the model; the expected measurement value includes the bounding box coordinates and category of the real annotated bounding box; for each predicted bounding box, the intersection and union ratio of the predicted bounding box and the nearest real bounding box is calculated, and when the intersection and union ratio exceeds a preset threshold, the YOLO network model evaluation passes after the training is completed.

6. A method for predicting moving target trajectories based on deep learning as claimed in claim 1, characterized in that: In S300, the moving target image is preprocessed before moving target trajectory prediction, and the specific method includes: S301. Perform brightness equalization and normalization on the camera image, and load the trained YOLO network model; S302. Detect the position x(k), y(k), s(k), and r(k) of the moving target in the image at the current moment according to the YOLO network model, assign an ID to the moving target created for detection, place each target object in the detection set G, and determine whether the input image is the first frame. If the input image is the first frame, directly proceed to S400; if the input image is not the first frame, directly proceed to S303; S303, target matching, for the predicted set A of the target predicted at the previous moment, and the union T of the unsuccessfully matched set B, let T = A∪B; use the KM matching algorithm to determine whether the current frame detection target set G matches the set T, and the specific judgment method includes: first determine whether the number of elements in the current detection target set G is greater than or equal to the number of elements in the set T, if the number of elements in the set G is greater than or equal to the number of elements in the set T, then place the set G on the right side of the bipartite graph to initiate matching; if the number of elements in the set G is less than the number of elements in the set T, place the set T on the right side of the bipartite graph to initiate matching; then use the intersection and union ratio of the current frame detection frame and the predicted frame at the previous moment as the weight of the bipartite graph, and obtain an optimal match that maximizes the sum of the weights through the KM matching algorithm.

7. A method for predicting moving target trajectories based on deep learning as claimed in claim 1, characterized in that: In S400, an adaptive unscented Kalman filter algorithm is used on the preprocessed moving target image to predict the position of the moving target image at the next moment, and a predicted value of the moving target image trajectory is output. The specific method includes: S401. Construct state vector Where f is a nonlinear function, h is the observation function, x is the state vector, y is the observation vector, and w is the k is the L-dimensional process noise matrix, v k is the L-dimensional measurement noise matrix; S402. Initialize parameters and obtain the expected value through process noise variance Q and measurement noise R and the covariance matrix where x k represents the state vector of the object at time k, P x,0 is the variance of the initial state and augmented state, E is the expected value, and a is the test point; S403. Obtain sigma test points and perform proportional sampling correction on the obtained sigma test points; the proportional correction parameter α∈[0,1] is used to construct an augmented sigma point according to the expected value and covariance matrix in S402 Where N is the dimension in the augmented state, j is the element index; S404. Update equations through time Find the estimated value of the sampling point state Where k and k-1 represent the k-th and k-1-th moments, x and w represent the test points, and j represents the j-th element in X; S405. Update equations through observations Find the observed estimate h represents the observation function, thus obtaining the prior measurement estimate and the prior mean square error P y,k , and introduce the noise adaptive coefficient β for optimization, the noise adaptive coefficient β∈[0,1]; S406. Update the Kalman filter according to the output sigma point, sampling point state estimate and observation estimate obtained in S403, S404 and S305; S407. Establish state equation X k =AX k-1 +W k , the state transfer matrix T is the time interval, W k is zero-mean Gaussian noise. The current coordinate position and velocity of the moving target are (x, y, z) and (vx, vy, vz) respectively. The state vector at time t is represented by X t =[x,v x ,y,v y ,z,v z ] T ; S408. Output the predicted value of the moving target trajectory and compare it with the true value. When the difference between the predicted value and the true value is less than a preset threshold, it is determined that the current predicted value meets the requirements, and the predicted value of the moving target trajectory is output.

8. A moving target trajectory prediction system based on deep learning, characterized in that: include: YOLO network model training unit, YOLO network model evaluation unit, moving target trajectory prediction pre-processing unit and moving target trajectory prediction value output unit; wherein: The YOLO network model training unit is used to read the training set images in the training set, perform brightness equalization and normalization processing on the training set images, and train the YOLO network model using the images after brightness equalization and normalization processing to obtain the YOLO network model after training. A YOLO network model evaluation unit, used for inputting the test set image in the data set into the trained YOLO network model for detection, and evaluating the trained YOLO network model by calculating the difference between the observed vector and the expected measurement value; A moving target trajectory prediction preprocessing unit, used for inputting the moving target image acquired by the camera in real time into the trained YOLO network model, and performing preprocessing on the moving target image before moving target trajectory prediction; The moving target trajectory prediction value output unit is used to use an adaptive unscented Kalman filter algorithm on the moving target image after preprocessing to predict the position of the moving target image at the next moment and output the prediction value of the moving target trajectory.

9. A moving target trajectory prediction system based on deep learning as claimed in claim 8, characterized in that: The moving target trajectory prediction value output unit is used to use the adaptive unscented Kalman filter algorithm on the moving target image after preprocessing to predict the position of the moving target image at the next moment and output the predicted value of the moving target trajectory. The specific method includes: S401. Construct state vector Where f is a nonlinear function, h is the observation function, x is the state vector, y is the observation vector, and w is the k is the L-dimensional process noise matrix, v k is the L-dimensional measurement noise matrix; S402. Initialize parameters and obtain the expected value through process noise variance Q and measurement noise R and the covariance matrix where x k represents the state vector of the object at time k, P x,0 is the variance of the initial state and augmented state, E is the expected value, and a is the test point; S403. Obtain sigma test points and perform proportional sampling correction on the obtained sigma test points; the proportional correction parameter α∈[0,1] is used to construct an augmented sigma point according to the expected value and covariance matrix in S402 Where N is the dimension in the augmented state, j is the element index; S404. Update equations through time Find the estimated value of the sampling point state Where k and k-1 represent the k-th and k-1-th moments, x and w represent the test points, and j represents the j-th element in X; S405. Update equations through observations Find the observed estimate h represents the observation function, thus obtaining the prior measurement estimate and the prior mean square error P y,k , and introduce the noise adaptive coefficient β for optimization, the noise adaptive coefficient β∈[0,1]; S406. Update the Kalman filter according to the output sigma point, sampling point state estimate and observation estimate obtained in S403, S404 and S305; S407. Establish state equation X k =AX k-1 +W k , the state transfer matrix T is the time interval, W k is zero-mean Gaussian noise. The current coordinate position and velocity of the moving target are (x, y, z) and (vx, vy, vz) respectively. The state vector at time t is represented by X t =[x,v x ,y,v y ,z,v z ] T ; S408. Output the predicted value of the moving target trajectory and compare it with the true value. When the difference between the predicted value and the true value is less than a preset threshold, it is determined that the current predicted value meets the requirements, and the predicted value of the moving target trajectory is output.

10. An electronic device comprising: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement any moving target trajectory prediction method in claims 1-7.

Images