A video prediction method and system based on key points and Kalman filtering

Abstract

The present invention belongs to the field of prediction of video data frames in automatic driving, and provides a video prediction method and system based on key points and Kalman filtering. First, a neural network is trained to extract features from the collected data, and the extracted features are converted into key points; secondly, the key points are reconstructed, and the reconstructed features are reconstructed together with the features of the initial image frame through another neural network, and the feature extraction network and image reconstruction network are obtained by optimizing the loss function. Finally, the image key points extracted in the process are combined with the image prior information obtained by the RNN model, and the Kalman filter is used to predict the future data frame, and the predicted image is compared with the real image, and the corresponding loss function is optimized to train the RNN model.

Description

Technical Field

[0001] The present invention belongs to the field of video data frame prediction in autonomous driving, and in particular to a video prediction method and system based on key points and Kalman filtering. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] In recent years, autonomous driving, as a new application field, has attracted the attention of the industry. Autonomous driving observes the surrounding road conditions, environment, pedestrians, etc. through data acquisition devices such as cameras and videos, and analyzes and judges the road conditions based on the collected data to make decisions. Video prediction technology is a relatively cutting-edge research technology. Since the data images collected by video acquisition equipment have high dimensions and large data volumes, it brings considerable challenges to problem analysis. The technology is not very mature at present. Existing video prediction technologies include predicting the next frame based on the previous frame image and predicting the next frame based on multi-frame image information. However, the method of predicting the next frame based on the previous frame image fails to fully and effectively utilize historical information; the processing data for predicting the next frame based on multi-frame image information is too large, resulting in low computational efficiency and serious overfitting of the prediction effect. Summary of the invention

[0004] In order to solve at least one technical problem existing in the above-mentioned background technology, the present invention provides a video prediction method and system based on key points and Kalman filtering, which includes two stages of image feature extraction and image reconstruction, and data frame prediction using key points and Kalman filtering. It is necessary not only to identify the behavior of the object, but also to predict its possible next behavior, so as to better improve safety performance.

[0005] In order to achieve the above object, the present invention adopts the following technical solution:

[0006] The first aspect of the present invention provides a video prediction method based on key points and Kalman filtering, comprising the following steps:

[0007] Obtain multi-dimensional data during the original autonomous driving process;

[0008] Extract image features based on multi-dimensional data and the trained image feature extraction model, and obtain multi-dimensional key point information of the image based on the extracted features;

[0009] Based on the multi-dimensional key point information and the image prior information obtained by the prediction model, the Kalman filter is used to predict the current data frame to obtain the predicted image of the next data frame, and the next behavior of the identified object is predicted based on the predicted image.

[0010] A second aspect of the present invention provides a video prediction system based on key points and Kalman filtering, comprising: a data acquisition module configured to: acquire multi-dimensional data in an original autonomous driving process;

[0011] The key point information extraction module is configured to: extract image features according to the multi-dimensional data and the trained image feature extraction model, and obtain multi-dimensional key point information of the image according to the extracted features;

[0012] The video image prediction module is configured as follows: based on multi-dimensional key point information and image prior information estimated by the prediction model, the Kalman filter is used to predict the current data frame, and the predicted image of the next data frame is obtained, and the next behavior of the identified object is predicted based on the predicted image.

[0013] A third aspect of the present invention provides a computer-readable storage medium.

[0014] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps in the above-mentioned video prediction method based on key points and Kalman filtering.

[0015] A fourth aspect of the present invention provides a computer device.

[0016] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps in the video prediction method based on key points and Kalman filtering as described above are implemented.

[0017] Compared with the prior art, the present invention has the following beneficial effects:

[0018] First, the key point extraction model and the prediction model are trained separately, which reduces the overall complexity of the model and improves the prediction efficiency. Secondly, the feature extraction technology of key points is used to reduce the dimension of data in the dynamic prediction process, reduce the computational complexity, and effectively alleviate the overfitting phenomenon. Finally, the Kalman filtering method is used to make full use of historical prior information, making the image prediction effect smoother to better improve safety performance. The two stages of data frame prediction using key points and filtering methods are adopted. The image key points and the prior information of the image can not only identify the behavior of the object, but also predict its possible next behavior based on the prior information to better improve safety performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0020] Figure 1 It is an overall schematic diagram of the feature extraction and image reconstruction process of the first embodiment of the present invention;

[0021] Figure 2 is a schematic diagram of a feature extraction process in Embodiment 1 of the present invention;

[0022] Figure 3 It is a convolutional neural network module according to the first embodiment of the present invention;

[0023] Figure 4 is a schematic diagram of an image reconstruction model according to a first embodiment of the present invention;

[0024] Figure 5 This is a flow chart of data frame prediction based on Kalman filtering according to Embodiment 1 of the present invention. DETAILED DESCRIPTION

[0025] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0026] It should be noted that the following detailed descriptions are all illustrative and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0027] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0028] Embodiment 1

[0029] like Figure 1 As shown, this embodiment provides a video prediction method based on key points and Kalman filtering, comprising the following steps:

[0030] S1: Obtaining the image and video data collected during the original autonomous driving process;

[0031] S2: extracting image features based on the collected image and video data and the trained image feature extraction model, and obtaining multi-dimensional key point information of the image based on the extracted features;

[0032] S3: Rendering the multi-dimensional key points of the image into low-dimensional data, and reconstructing the obtained low-dimensional data through an image reconstruction model in combination with the original data;

[0033] S4: extract the multi-dimensional key point information of the image, combine it with the image prior information obtained by the RNN model, use Kalman filtering to predict the next data frame to obtain a predicted image, and predict the next behavior of the identified object based on the predicted image;

[0034] S5: Compare the predicted image with the real image and optimize the corresponding loss function;

[0035] Among them, in S1, the image and video data collected during the original autonomous driving process are collected through cameras, videos and other data acquisition devices to collect surrounding road conditions, environment, pedestrians, etc.

[0036] like Figure 2 As shown, after obtaining the original data, the data is preprocessed and the data frame is randomly shuffled to maintain the independent and identical distribution of the data;

[0037] The feature extraction model includes multiple convolutional neural network modules, each of which includes: batch normalization, convolution layer, pooling layer, non-linear activation layer, etc.

[0038] like Figure 3 As shown, in S2, converting the extracted features into image key points includes:

[0039] Assume that by feature extraction φ det After that, k features are obtained, denoted as S u (v,k);

[0040] Then the obtained k feature key points are:

[0041] like Figure 4 As shown in S3, the image reconstruction model φ rec Use neural network models, such as ResNet network, DenseNet network, etc.

[0042] like Figure 1 As shown, in the key point extraction stage, the loss function used is:

[0043] λ is a hyperparameter.

[0044] In S4, the key points of the image are extracted, and the image prior information obtained by the RNN model is combined to predict the data frame using the Kalman filter method to obtain the predicted image;

[0045] After extracting the multi-dimensional key point information, the image key points are restored. The method of restoring the image key points is: through Gaussian mapping, the key points are Renders as:

[0046] Among them, Su (v,k) represents the features extracted from image v using the kth convolution kernel. represents the features reconstructed from the key points obtained by the kth convolution kernel of image v, k represents the number of features, and u represents a Gaussian distribution random variable. v represents the corresponding image, Represents the key points of image v using the kth convolution kernel.

[0047] in With S u The (v,k) dimensions have the same size.

[0048] like Figure 5 As shown, prior information is very important for estimation. In this model, the prior information is estimated as follows:

[0049] First, before observing the current frame image, the historical information is integrated into the RNN model φ prior From here, get the prior information φ of the current frame prior (h t-1 )=p(z t |x <t,z<t);

[0050] Secondly, combined with the current frame image information x t With prior information Using Bayesian estimation Obtain the posterior estimate of the prediction model

[0051] Again, by the posterior estimate And the current frame key point information x t , using Kalman filtering Predict the key point information of the next frame

[0052] Finally, the hidden state is updated through RNN

[0053] like Figure 5 As shown, the loss function of the future image prediction stage is:

[0054]

[0055] Among them, λ1, λ2, λ3 are hyper parameters, v t+1 is the t+1 frame image, is the predicted image of frame t+1, Loss_kl is the relative entropy loss, L sep is the separation loss, L sparse is the sparse loss; L sparse =∑ k |μk |， <x>Represents the mean of the key points x.

[0056] Embodiment 2

[0057] This embodiment provides a video prediction system based on key points and Kalman filtering, including:

[0058] The data acquisition module is configured to: acquire multi-dimensional data in the original autonomous driving process;

[0059] The key point information extraction module is configured to: extract image features according to the multi-dimensional data and the trained image feature extraction model, and obtain multi-dimensional key point information of the image according to the extracted features;

[0060] The video image prediction module is configured to: predict the current data frame using Kalman filtering based on multi-dimensional key point information and image prior information obtained by the prediction model estimation to obtain the predicted image of the next data frame.

[0061] Embodiment 3

[0062] This embodiment provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps in the video prediction method based on key points and Kalman filtering as described above are implemented.

[0063] Embodiment 4

[0064] This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps in the video prediction method based on key points and Kalman filtering as described above are implemented.

[0065] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) containing computer-usable program code.

[0066] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0067] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0068] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0069] A person skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0070] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.< / x>

Claims

1. A video prediction method based on key points and Kalman filtering, characterized in that: The steps include: Obtaining the image data and video data collected during the original autonomous driving process; Extract image features based on the multidimensional data and the trained image feature extraction model, obtain multidimensional key point information of the image based on the extracted features, render the multidimensional key point information into low-dimensional data, and reconstruct the image through the image reconstruction model by combining the low-dimensional data obtained by combining the first frame of the image; Extract the multi-dimensional key point information of the reconstructed image, combine it with the image prior information estimated by the RNN model, use the Kalman filter to predict the current data frame, obtain the predicted image of the next data frame, and predict the next behavior of the identified object based on the predicted image; The process of obtaining the image prior information includes: Before observing the current frame image, historical information is integrated into the RNN model to obtain the prior information of the current frame; Combining the current frame image information with the prior information, using Bayesian estimation, the posterior estimation of the prediction model is obtained; Based on the posterior estimation and the key point information of the current frame, the Kalman filter is used to predict the key point information of the next frame; The loss function of the prediction model is: ， in is a super parameter, is the t+1 frame image, is the predicted image for frame t+1, is the relative entropy loss, For separation loss, is the sparse loss; , , , , Represents the mean of the key points x.

2. A video prediction method based on key points and Kalman filtering as claimed in claim 1, characterized in that: The feature extraction model includes multiple convolutional neural network modules, each of which includes: batch normalization, a convolution layer, a pooling layer, and a nonlinear activation layer.

3. A video prediction method based on key points and Kalman filtering as claimed in claim 1, characterized in that: The multi-dimensional key point information of the reconstructed image is extracted to restore the image key points. The method for restoring the image key points is: rendering the key points through Gaussian mapping.

4. A video prediction method based on key points and Kalman filtering as claimed in claim 1, characterized in that: After obtaining the original data, the data is preprocessed and randomly shuffled.

5. A video prediction system based on key points and Kalman filtering, characterized in that: include: The data acquisition module is configured to: acquire image data and video data collected during the original autonomous driving process; The key point information extraction module is configured to: extract image features according to the multidimensional data and the trained image feature extraction model, obtain multidimensional key point information of the image according to the extracted features, render the multidimensional key point information into low-dimensional data, and reconstruct the image through the image reconstruction model in combination with the low-dimensional data obtained from the first frame of the image; The video image prediction module is configured to: extract the multi-dimensional key point information of the reconstructed image, combine the image prior information obtained by RNN model estimation, use Kalman filtering to predict the current data frame, obtain the predicted image of the next data frame, and predict the next behavior of the identified object based on the predicted image; The process of obtaining the image prior information includes: Before observing the current frame image, historical information is integrated into the RNN model to obtain the prior information of the current frame; Combining the current frame image information with the prior information, using Bayesian estimation, the posterior estimation of the prediction model is obtained; Based on the posterior estimation and the key point information of the current frame, the Kalman filter is used to predict the key point information of the next frame; The loss function of the prediction model is: ， in is a super parameter, is the t+1 frame image, is the predicted image for frame t+1, is the relative entropy loss, For separation loss, is the sparse loss; , , , , Represents the mean of the key points x.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps in a video prediction method based on key points and Kalman filtering as described in any one of claims 1 to 4 are implemented.

7. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps in the video prediction method based on key points and Kalman filtering as described in any one of claims 1 to 4 are implemented.

Images