A panoramic video two-frame pedestrian trajectory prediction method and device and a storage medium

Abstract

The present application relates to a kind of panoramic video two frame pedestrian trajectory prediction method, device and storage medium, wherein method includes: obtaining panoramic video, extracting the position information, speed information, action information and optical flow information of pedestrian in panoramic video, pedestrian is divided into free moving pedestrian and social moving pedestrian;Calculate free pedestrian dataset and all pedestrian dataset;Pedestrian trajectory prediction is carried out using training completed network, wherein free pedestrian prediction network is trained based on free pedestrian dataset, future social pool network and social pedestrian prediction network are trained based on all pedestrian dataset.The present application provides a kind of panoramic video pedestrian trajectory prediction method, constructs the pedestrian trajectory prediction network model with pedestrian historical position information, historical action information and historical optical flow information as input, to achieve the goal of higher precision pedestrian trajectory prediction.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of artificial intelligence, and particularly relates to a panoramic video two-frame pedestrian trajectory prediction method, device and storage medium. BACKGROUND

[0002] Under the background of the intensifying phenomenon of population aging and the increasing cost of human resources, mobile robots are increasingly applied in the fields of medical treatment, catering, culture and entertainment, and are commonly used in places such as hospitals, restaurants, amusement parks and museums, and can assist staff in carrying goods or patrol along a designated route. The working nature of the mobile robot determines that it usually works in a human-machine-object shared space, and needs to dynamically change the trajectory for obstacle avoidance when moving in the interference of pedestrians and goods. The trajectory of the pedestrian has great uncertainty, and accurate prediction of the future trajectory of the pedestrian helps the mobile robot to plan a reasonable trajectory to avoid collision. Compared with the traditional Kalman filter trajectory prediction, the prediction method based on recurrent neural network has the characteristics of long prediction time and high accuracy, and is widely used in pedestrian trajectory prediction. The current mature recurrent neural network includes Long-Short Time Memory (LSTM), Gated Recurrent Unit networks (GRU) and the like. Most of the current first-view pedestrian trajectory prediction algorithms based on vision use ordinary cameras or wide-angle cameras, and the viewing angle is limited, and the complete environment cannot be recorded synchronously. In reality, the trajectory of the pedestrian walking in the opposite direction in front of the robot is not only affected by the environment in front of the robot, but also affected by the environment behind the robot. The information recorded by the ordinary camera or the wide-angle camera in the first-view trajectory prediction scene is largely missing. The panoramic camera is a kind of camera that can record the front and rear of the robot at the same time, and has a small panoramic blind area. Applying the panoramic video to the pedestrian trajectory prediction method is beneficial to improve the accuracy of prediction. SUMMARY

[0003] The present application provides a panoramic video two-frame pedestrian trajectory prediction method, device and storage medium, aiming to at least solve one of the technical problems existing in the prior art. The scheme of the present application provides a panoramic video pedestrian trajectory prediction method by aiming at the larger visual blind area of the existing first-view pedestrian trajectory prediction method, which constructs a pedestrian trajectory prediction network model taking the historical position information, historical action information and historical optical flow information of the pedestrian as input, so as to achieve the goal of higher precision pedestrian trajectory prediction.

[0004] The technical scheme of the present application relates to a panoramic video two-frame pedestrian trajectory prediction method, which comprises the following steps:

[0005] S100, acquire and splice videos in different directions to obtain a panoramic video, extract position information, speed information, action information and optical flow information of pedestrians in the panoramic video, and divide the pedestrians into free-moving pedestrians and socially-moving pedestrians;

[0006] S200, calculate a free pedestrian dataset and a whole pedestrian dataset, the free pedestrian dataset including information of all free-moving pedestrians, and the whole pedestrian dataset including information of all pedestrians, divide the free pedestrian dataset into a free pedestrian training set, a free pedestrian verification set and a free pedestrian test set, and divide the whole pedestrian dataset into a whole pedestrian training set, a whole pedestrian verification set and a whole pedestrian test set;

[0007] S300, use a trained free pedestrian prediction network, a future social pooling network and a social pedestrian prediction network to perform pedestrian trajectory prediction, wherein the free pedestrian prediction network is trained based on the free pedestrian dataset, the future social pooling network and the social pedestrian prediction network are trained based on the whole pedestrian dataset, and input data of the future social pooling network is the trained free pedestrian prediction network.

[0008] The application also relates to a panoramic video two-frame pedestrian trajectory prediction device for realizing the panoramic video two-frame pedestrian trajectory prediction method.

[0009] A video acquisition device is used for acquiring and splicing videos.

[0010] A deep learning control board is used for classifying and learning pedestrians in the videos, and analyzing and predicting walking trajectories of the pedestrians in the videos.

[0011] A general control system is used for controlling the video acquisition device and the deep learning control board, and displaying result information.

[0012] The application also relates to a computer readable storage medium, which stores program instructions, and the program instructions are executed by a processor to implement the method.

[0013] Compared with the prior art, the application has the following characteristics.

[0014] The method uses panoramic videos to simultaneously obtain pedestrian information in front and back directions, significantly reduces a visual blind area during prediction, and improves the accuracy of trajectory prediction; the method uses two frames to perform prediction, and reduces the requirement for hardware during prediction; the method includes free-moving pedestrian prediction trajectories into a social pooling module, reduces the difficulty of network learning of pedestrian interaction modes, and improves the accuracy of trajectory prediction. BRIEF DESCRIPTION OF DRAWINGS

[0015] Figure 1A flowchart of a panoramic video two-frame pedestrian trajectory prediction method;

[0016] Figure 2 A panoramic video processing schematic diagram in the panoramic video two-frame pedestrian trajectory prediction method;

[0017] Figure 3 A schematic diagram of future social pooling matrix stacking in the panoramic video two-frame pedestrian trajectory prediction method;

[0018] Figure 4 A schematic diagram of using a trained free pedestrian prediction network, a future social pooling network, and a social pedestrian prediction network to perform pedestrian trajectory prediction in the panoramic video two-frame pedestrian trajectory prediction method;

[0019] Figure 5 A structural schematic diagram of a free pedestrian prediction network in the panoramic video two-frame pedestrian trajectory prediction method;

[0020] Figure 6 A structural schematic diagram of a future social pooling network in the panoramic video two-frame pedestrian trajectory prediction method;

[0021] Figure 7 A structural schematic diagram of a social pedestrian prediction network in the panoramic video two-frame pedestrian trajectory prediction method. DETAILED DESCRIPTION

[0022] To make the objects, technical solutions, and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0023] The concept, specific structure, and generated technical effects of the present application will be described clearly and completely below with reference to the embodiments and drawings, so as to fully understand the objects, solutions, and effects of the present application.

[0024] It should be noted that, unless otherwise specified, when a certain feature is called "fixed" or "connected" to another feature, it can be directly fixed or connected to the other feature, or indirectly fixed or connected to the other feature. The singular forms "a," "said," and "the" used herein are also intended to include the plural forms, unless the context clearly indicates otherwise. In addition, unless otherwise defined, all technical and scientific terms used herein have the same meaning as understood by those skilled in the art. The terms used in the specification herein are only for the purpose of describing specific embodiments and are not intended to limit the present application. The term "and / or" used herein includes any combination of one or more related listed items.

[0025] It should be understood that, although the terms first, second, third, etc. can be employed in this disclosure to describe various elements, these elements should not be limited to these terms. These terms are only used to distinguish one type of element from another type of element. For example, a first element can also be referred to as a second element, and similarly, a second element can also be referred to as a first element, without departing from the scope of the present disclosure. The use of any and all examples or exemplary language herein (e.g., "such as," "for example," etc.) is intended merely to better illustrate the embodiments of the present application and does not impose a limitation on the scope of the present application unless otherwise claimed. In addition, the industry term "pose" used herein refers to the position and attitude of an element relative to a spatial coordinate system.

[0026] Referring to Figures 1 to 7 The embodiments of the present application provide a two-frame pedestrian trajectory prediction method based on panoramic video, which specifically includes the following steps:

[0027] S100, acquiring and splicing videos in different directions to obtain a panoramic video, extracting position information, speed information, action information and optical flow information of pedestrians in the panoramic video, and dividing the pedestrians into free-moving pedestrians and social-moving pedestrians;

[0028] S200, calculating a free pedestrian dataset and a total pedestrian dataset, the free pedestrian dataset including information of all free-moving pedestrians, and the total pedestrian dataset including information of all pedestrians, dividing the free pedestrian dataset into a free pedestrian training set, a free pedestrian verification set and a free pedestrian test set, and dividing the total pedestrian dataset into a total pedestrian training set, a total pedestrian verification set and a total pedestrian test set;

[0029] S300, performing pedestrian trajectory prediction using the trained free pedestrian prediction network, the future social pooling network, and the social pedestrian prediction network, wherein the free pedestrian prediction network is trained based on a free pedestrian dataset, the future social pooling network and the social pedestrian prediction network are trained based on the full pedestrian dataset, and input data of the future social pooling network is the trained free pedestrian prediction network.

[0030] Further, step S100 comprises:

[0031] S110, acquiring two videos of relative directions, and performing horizontal splicing on the two videos to obtain a panoramic video;

[0032] S120, calculating historical position information of the pedestrian based on the panoramic video Pedes i The position information at time t is

[0033]

[0034] wherein i represents the i-th pedestrian, and are respectively the horizontal and vertical coordinates of the pedestrian Pedes i in the pixel coordinate system, and are respectively the length and width of the detection box of the pedestrian Pedes i in the pixel coordinate system;

[0035] S130, calculating historical speed information of the pedestrian based on the panoramic video Pedes i The speed information at time t is

[0036]

[0037] wherein T obs is the time interval between the two frames of the panoramic video;

[0038] S140, calculating historical motion information of the pedestrian based on the panoramic video Pedes i The motion information at time t is

[0039]

[0040] wherein are the horizontal and vertical coordinates of the 18 joints of the pedestrian after standardization;

[0041] S150, calculating historical optical flow information of the pedestrian based on the panoramic video The optical flow information of the pedestrian Pedesi at time t includes the horizontal axis component and the vertical axis component of the optical flow of the pixels in the pedestrian detection frame. The size of the pedestrian detection frame is adjusted to 10 pixels*10 pixels, and is represented as,

[0042]

[0043] S160, marking different moving types of pedestrians, including free moving pedestrians and social moving pedestrians, the free moving pedestrians being pedestrians whose movements are not affected by others throughout the whole process, and the social moving pedestrians being pedestrians whose movements are affected by others in a certain period of time.

[0044] In some embodiments, the step S100 specifically comprises: processing the panoramic video into two-view videos and horizontally splicing, the video frame rate being 30 frames, the resolution after splicing being 3840*1080, the horizontal angles of the two views being 0° and 180° respectively, the vertical angle and the roll angle being 0°, the field of view being 80°, and the distortion control coefficient being 0.65. The pedestrians are manually labeled, the pedestrian action information is extracted using the openpose algorithm, the optical flow information is extracted using the Farneback optical flow method, and the pedestrians are manually divided into two categories of free moving and social moving.

[0045] Further, the step S200 further comprises:

[0046] S210, making free moving pedestrian samples, sampling the trajectory of each free moving pedestrian at uniform time intervals, for a certain sampled free moving pedestrian numbered as Pedes1, the time sequence of the observation frame being:

[0047] T obslis t=[t0-T obs ,t0],

[0048] Wherein, t0 is the prediction start time, T obs is the time interval of the two observed frames,

[0049] The time sequence of the label frame is:

[0050] T labellist =[t0+T gap ,t0+2*T gap ,…,t0+T pred ],

[0051] Wherein, T pred is the maximum prediction time, and T gap is the time interval of each prediction,

[0052] Each free moving pedestrian sample is obtained, and the input of each free moving pedestrian sample includes the historical position information of the free moving pedestrian Free-moving pedestrian historical speed information Free-moving pedestrian historical motion information And free-moving pedestrian historical optical flow information The label of each free-moving pedestrian sample includes free-moving pedestrian label position information Free-moving pedestrian label speed information Free-moving pedestrian label motion information And free-moving pedestrian label optical flow information

[0053] In some embodiments, the step S210 specifically includes: setting the observed two-frame time interval T obs = 0.5, the maximum prediction time T pred = 2, and the time interval for each prediction T gap = 0.5, all in seconds.

[0054] S220, collect all free-moving pedestrian samples, make a free pedestrian dataset, and divide the free-moving pedestrian samples into mutually different free-moving pedestrian training set, free-moving pedestrian test set and free-moving pedestrian validation set according to a preset proportion;

[0055] S230, make a whole pedestrian sample, sample the trajectories of all pedestrians at uniform time intervals, and for a sampled pedestrian numbered as Pedes1, all pedestrians are expressed as:

[0056] Pedes 1:m = [Pedes1, Pedes2,..., Pedes m ],

[0057] Wherein, m is the number of pedestrians in the panoramic video at this moment,

[0058] The input of the whole pedestrian sample includes whole pedestrian historical position information Whole pedestrian historical speed information Whole pedestrian historical motion information And whole pedestrian historical optical flow information The label of the whole pedestrian sample includes whole pedestrian label position information Whole pedestrian label speed information Whole pedestrian label motion information And whole pedestrian label optical flow information

[0059] S240, collect all whole-moving pedestrian samples, make a whole pedestrian dataset, and divide the whole-moving pedestrian samples into mutually different whole-moving pedestrian training set, whole-moving pedestrian test set and whole-moving pedestrian validation set according to a preset proportion.

[0060] In some embodiments, step S240 specifically includes dividing the training set, test set, and verification set of all moving pedestrians into a ratio of 8:1:1.

[0061] Furthermore, the pedestrian prediction network described in step S300 is trained based on a pedestrian dataset, including:

[0062] S301, respectively, the freely moving pedestrian Pedes i Location information at time t Speed ​​information and action information Embedded into the first fully connected layer φ ex Second fully connected layer φ ev and the third fully connected layer φ ep Free-moving pedestrians Pedes i Optical flow information at time t Embedded into the first multilayer perceptron ψ ef The embedded vectors are obtained respectively. and And embed the vector and Use the first GRU network GRU in sequence ex Second GRU network GRU ev Third GRU network GRU ep and the fourth GRU network GRU ef Encode the freely moving pedestrians Pedes. i The state vector of the GRU network used for encoding at time t, containing position, velocity, motion, and optical flow information. and

[0063]

[0064]

[0065]

[0066]

[0067]

[0068]

[0069]

[0070] Where, φex φ ev These are the first and second fully connected layers using the PRelu activation function, respectively. Both the first and second fully connected layers have 4 input units and es output units, φ. ep This is a third fully connected layer using the PRelu activation function, with 36 input units and es output units, ψ. ef This is a two-layer first multilayer perceptron using the PRelu activation function. The first multilayer perceptron has 200 input layer units, 2 * e^s hidden layer units, and e^s output layer units. The first GRU network... ex The second GRU network GRU ev The third GRU network GRU ep and the fourth GRU network GRU ef The number of input layer units is es, the number of state layer units is rs, and W x W v W p W f W ex W ev W ep W ef These are trainable network parameters;

[0071] In some embodiments, step S100 specifically includes: setting es = 32 and rs = 128.

[0072] S302, the state vector containing position information, velocity information, motion information, and optical flow information. and Serialized and passed through a second multilayer perceptron ψ gc Output the state vector used for decoding.

[0073]

[0074] Where, ψ gc For a second multilayer perceptron using the PRelu activation function, the second multilayer perceptron ψ gc It is a two-layer multilayer perceptron (MLP) network, where the second multilayer perceptron ψ gc The number of input layer units and hidden layer units are both 4*rs, and the number of output layer units is 3*rs. W gc These are trainable network parameters;

[0075] S303, The state vector used for decoding Input the fifth GRU network GRU de and the third multilayer perceptron ψ gl, the free movement trajectory of the pedestrian is predicted, and the time sequence of the prediction frame is:

[0076] T predlist = [t0, t0+T gap , t0+2*T gap , …, t0+T pred ],

[0077] wherein T pred is the maximum time of prediction, and T gap is the time interval of each prediction step,

[0078] The decoding process is represented as:

[0079]

[0080]

[0081]

[0082]

[0083]

[0084] wherein, represents the state vector of the fifth GRU network used for decoding of the pedestrian Pedesi at time t, the input layer unit number of the fifth GRU network GRU de is 2*es, the state layer unit number is 3*rs, the fifth GRU network is a decoder network, and ψ gl is a third multi-layer perception MLP network with Prelu activation function using Dropout mechanism, the input layer unit number of the third multi-layer perception is 3*rs, the hidden layer unit number is 3*rs and rs respectively, and the output layer unit number is 4, represents the predicted free movement position change, and respectively represent the predicted next step free movement position and velocity, W de , W gl , W x , W v are trainable network parameters,

[0085] The free movement trajectory prediction sequence of the current sample pedestrian is obtained as

[0086]

[0087] wherein t0 is the prediction start time, T obs is the observed two-frame time interval, T pred is the maximum time of prediction, and Tgap This represents the time interval for each prediction step.

[0088] Furthermore, the pedestrian prediction network described in step S300, trained based on a pedestrian dataset, also includes:

[0089] S304. Train the loss function L of the free pedestrian prediction network. CloU Update the parameters of the free pedestrian prediction network.

[0090] L CloU =L ACIoU +L FCIoU +L Reg

[0091] Among them, L ACIoU For the average CIoU error term, L FCIoU For the final CIoU error term, L Reg The L2 regularization term for predicting network weights for free pedestrians.

[0092]

[0093]

[0094] Among them, CIoU(X) pred X gt ) represents X pred With X gt The ratio of complete intersection and union, length(T) labellist ) represents T labellist The number of elements in, k ACIoU and k FCIoU A constant that is set;

[0095] In some embodiments, step S304 specifically includes: setting k ACIoU =0.5, k FCIoU =0.5.

[0096] S305. Select the parameters of the free pedestrian prediction network, use the free pedestrian validation set to validate the network parameters of each generation during training, and select the loss function L of the free pedestrian prediction network. FCIoU The network parameter with the smallest value is used as the final parameter of the free pedestrian prediction network.

[0097] Furthermore, the future social pooling network and the social pedestrian prediction network described in step S300 are trained based on the entire pedestrian dataset, including:

[0098] S306. Using the trained free pedestrian prediction network, predict the free movement trajectories of all pedestrians in the entire pedestrian training set, resulting in the free movement trajectory prediction sequence of all pedestrians in the entire pedestrian training set, as shown below.

[0099]

[0100] Where t0 is the prediction start time, T obs T is the time interval between two observed frames. gap T is the time interval for each prediction step. pred The maximum predicted time is given by m, where m represents the number of people in all pedestrian samples in the entire pedestrian training set, and T represents the time series of all pedestrian samples in the entire pedestrian training set. list for,

[0101] T list =[t0-T obs ,t0,t0+T gap t0+2*T gap , ..., t0+T pred ];

[0102] S307, For the time series T of all pedestrian samples in the entire pedestrian training set. list Each step in the process will determine the total number of pedestrians Pedes at time t. i Location prediction information With speed prediction information Embedded in the fourth fully connected layer respectively and the fifth fully connected layer Get pedestrian Pedes i The vector after embedding the position prediction information and velocity prediction information at time t and Bundle and After being connected in series, it passes through the sixth fully connected layer. Get pedestrian Pedes i At time t, the position and velocity information are obtained using the vector embedded by the sixth fully connected layer, which is the vector of all pedestrian states in the sample.

[0103]

[0104] in, These are the fourth and fifth fully connected layers with PRelu activation functions, respectively. The fourth and fifth fully connected layers have 4 input units and es output units. The sixth fully connected layer... The number of input layer units is 2*es, and the number of output units is es. are trainable network parameters;

[0105] S308, embedding the relative positions of the pdm pedestrians closest to the pedestrian Pedes1 in the pixel coordinate system into a seventh fully connected layer obtaining the relative positions of the pedestrian Pedes1 and the pedestrian Pedes i the vector of the relative position information at time t after embedding using a fully connected layer for predicting the trajectory of the pedestrian Pedes1, the process is represented as:

[0106]

[0107] wherein the seventh fully connected layer is a fully connected layer using a PRelu activation function, the input layer unit number of the seventh fully connected layer is 2, and the output unit number is es, are trainable network parameters, and pdm is the number of other pedestrians closest to the pedestrian Pedes1 in the pixel coordinate system;

[0108] In some embodiments, the step S308 specifically comprises: setting pdm = 8.

[0109] S309, training the time sequence T of all pedestrian samples in the entire pedestrian training set list at each step, stacking the social pooling matrix SP of the pedestrian Pedes1 at time t the social pooling matrix SP of the pedestrian Pedes1 at time t t after the convolution network δ sp extracting information to obtain SPC t , inputting SPC t into an eighth fully connected layer φ es obtaining the social pooling vector ES of the pedestrian Pedes1 at time t by extracting information through the eighth fully connected layer t , the process is represented as:

[0110]

[0111] SPC t = δ sp (SP t ; W sp )(t = T list ),

[0112] ES t = φ es (SPC t ; W es )

[0113] wherein the convolution network δ spIt is a three-layer convolutional network with PRelu activation function, wherein the convolutional network δ sp The convolutional kernel size is 3, and the number of output channels are 5, 7, and 9 respectively. The eighth fully connected layer φ es For a fully connected layer with 6*es output units, W sp W es These are trainable network parameters;

[0114] S310, using the sixth GRU network GRU es Encoding the future social pooling vector sequence yields the state vector of pedestrian Pedes1 at time t, representing the social information of the sixth GRU network used for encoding. The process is represented as follows:

[0115]

[0116] Among them, the sixth GRU network GRU es It is a GRU recurrent neural network with 6 * es input layer units and 2 * rs state layer units. W ges These are the trainable network parameters.

[0117] Furthermore, the future social pooling network and the social pedestrian prediction network described in step S300 are trained based on the entire pedestrian dataset, and also include:

[0118] S311, respectively, pedestrian Pedes i Location information at time t Speed ​​information and action information Embedded into the ninth fully connected layer 10th Fully Connected Layer and the eleventh fully connected layer Pedestrian Pedes i Optical flow information at time t Embedded into the fourth multilayer perceptron The embedded vectors were obtained respectively. and And embed the vector and Use the seventh GRU network in sequence Eighth GRU Network Ninth GRU Network and the 10th GRU network Encode the pedestrians to obtain the Pedes. i The state vector of the GRU network used for encoding at time t, containing position, velocity, motion, and optical flow information. and The process is represented as follows:

[0119]

[0120]

[0121]

[0122]

[0123]

[0124]

[0125]

[0126] wherein the ninth fully connected layer and the tenth fully connected layer use a PRelu activation function, the input layer unit number of the ninth fully connected layer and the tenth fully connected layer is 4, the output unit number is es, the eleventh fully connected layer uses a PRelu activation function, the input unit number of the eleventh fully connected layer is 36, the output unit number is es, the fourth multi-layer perception is a two-layer multi-layer perception MLP using a PRelu activation function, the fourth multi-layer perception has an input layer unit number of 200, both of the hidden layer unit numbers are 2*es, and the output layer unit number is es, the seventh GRU network the eighth GRU network the ninth GRU network and the tenth GRU network are all GRU recurrent neural networks, the input layer unit number of the seventh GRU network the eighth GRU network the ninth GRU network and the tenth GRU network is es, the state layer unit number is rs, is a trainable network parameter;

[0127] S312, the state vector of the position information, the speed information, the action information, the optical flow information and the social information of the GRU network used for encoding at t time i and are connected in series and then pass through a fifth multi-layer perception to obtain a final historical information encoding vector The process is represented as: ​

[0128]

[0129] The fifth multi-layer perception machine The fifth multi-layer perception machine is a two-layer multi-layer perception machine MLP network using a PRelu activation function The number of input layer units is 6*rs, the number of hidden layer units is 4*rs, and the number of output layer units is 3*rs.

[0130] S313, encode the final historical information vector Input the eleventh GRU network And the sixth multi-layer perception machine Predict the social movement trajectory of all pedestrians, and obtain the predicted social movement position change The next step of the predicted social movement position And the speed The process is represented as:

[0131]

[0132]

[0133]

[0134]

[0135]

[0136] Wherein, The state vector of the GRU network used for decoding at t time for the pedestrian Pedesi, the eleventh GRU network The number of input layer units is 2*es, the number of state layer units is 3*rs, and the sixth multi-layer perception machine The sixth multi-layer perception machine is a three-layer multi-layer perception machine MLP network using a PRelu activation function with a Dropout mechanism The number of input layer units is 3*rs, the number of hidden layer units is 3*rs and rs respectively, and the number of output layer units is 4, The trainable network parameters are

[0137] Obtain the trajectory prediction sequence of the social pedestrian Pedes1,

[0138]

[0139] S314, as step S304, training the loss function of the future social pooling network and the social pedestrian prediction network, updating the network parameters of the future social pooling network and the social pedestrian prediction network, as step S305, verifying the network parameters of each generation in the training process using the full-body pedestrian verification set, selecting the network parameters with the minimum loss function term of the future social pooling network and the social pedestrian prediction network as the final parameters of the future social pooling network and the social pedestrian prediction network.

[0140] Further, step S300 uses the trained free pedestrian prediction network, future social pooling network, and social pedestrian prediction network to perform pedestrian trajectory prediction, including:

[0141] For two frames of panoramic video data with a time interval T obs , obtain the historical position information of all pedestrians in the panoramic video historical speed information historical action information and historical optical flow information

[0142] Use the free pedestrian prediction network to predict the free movement trajectory of all pedestrians in the graph, and then use the future social pooling network and the social pedestrian prediction network to sequentially predict the social movement trajectory of all pedestrians.

[0143] In some embodiments, please refer to Figure 4 As shown in part (a), the historical trajectories of all pedestrians in the scene are input into the free pedestrian prediction network, which will predict the free movement results of all pedestrians, i.e., the dashed trajectories in part (b) and the pixel boxes pointed by the dashed trajectories. Subsequently, to predict the social movement trajectory of pedestrian Pedes1, the historical trajectories and predicted trajectories of pedestrian Pedes1 and the nearest pdm people are input into the future social pooling module, and finally the social pedestrian prediction network will use the future social pooling vector output by the future social pooling module and other historical position information, historical speed information, historical action information, and historical optical flow information to predict the social movement trajectory of the pedestrian. In part (c), the lightest dashed trajectory and the lightest pixel box are the output of the social pedestrian prediction network, and it can be found that its trajectory is shifted to the right compared to the free pedestrian prediction network, avoiding collision with pedestrian Pedes3.

[0144] In some embodiments, please refer to Figure 5 , the arrow indicates the input of the vector, MLP represents a multi-layer perceptron, GRU represents a GRU neural network, FC represents a fully connected layer, and RMC represents the calculation of and in step S303 and the process of, denotes vector concatenation. The historical position information, the historical speed information, the historical action information and the historical optical flow information are embedded into vectors using a fully connected layer and a multi-layer perceptron and input into a GRU network for encoding, and the state vector for decoding is obtained after the encoding is completed The state vector for decoding is obtained after the encoding is completed Finally, it is input into another GRU network for decoding, and the predicted next step free movement trajectory is obtained by inputting the state vector of the GRU network at each step into an MLP network.

[0145] In some embodiments, please refer to Figure 6 , Future Pool denotes vector stacking, and CNN denotes a convolutional neural network. First, as described in step S307, for all pedestrians in the current scene, the position prediction information and the speed prediction information are embedded using a fully connected layer and concatenated, and then a fully connected layer is used to obtain a vector describing the state of the pedestrian Subsequently, as shown in Figure 3 , the social pooling matrix SP t is stacked according to the positions of other pedestrians in the scene relative to the pedestrian Pedes1. t Finally, SP t is passed through a convolutional network and a fully connected layer to generate a social pooling vector ES

[0146] In some embodiments, please refer to Figure 7 , the vector ES t output by the future social pooling network is input into a GRU network for encoding, and the state vector for decoding is obtained after the encoding is completed Subsequently, it is concatenated with the remaining historical position information, historical speed information, historical action information and historical optical flow information after the encoding is completed, and the state vector for decoding is obtained The subsequent process is the same as the free pedestrian prediction network.

[0147] The present application also relates to a panoramic video two-frame pedestrian trajectory prediction device for implementing the panoramic video two-frame pedestrian trajectory prediction method described above, and the device comprises:

[0148] a video acquisition device for acquiring and splicing videos;

[0149] a deep learning control board for classifying and learning pedestrians in the videos and analyzing and predicting the walking trajectories of the pedestrians in the videos;

[0150] a general control system for controlling the video acquisition device and the deep learning control board and displaying result information.

[0151] The present application also relates to a computer readable storage medium having stored thereon program instructions which, when executed by a processor, implement a method as described above.

[0152] It should be appreciated that the steps of the methods in the embodiments of the application can be realized or implemented by computer hardware, a combination of hardware and software, or through computer instructions stored in a non-transitory computer readable storage medium. The methods can use standard programming techniques. Each program can be implemented in a high-level procedural or object-oriented programming language to be used in conjunction with a computer system. However, if desired, the programs can be implemented in assembly or machine language, which can be made up of binaries. In any case, the language can be a compiled or interpreted language. Furthermore, the programs can be able to run on the specialized integrated circuits programmed for the purpose.

[0153] Further, the operations of the processes described herein can be performed in any suitable order unless otherwise indicated herein or otherwise clearly contradicted by context. The processes described herein (or variations and / or combinations thereof) can be performed under the control of one or more computer systems configured with executable instructions (e.g., computer-executable instructions, one or more computer programs or one or more applications) to perform the processes of the application. The computer programs include processor-executable instructions that are stored in memory and implement the processes described herein.

[0154] Further, the methods can be implemented in any suitable type of computing platform operably connected to, including but not limited to, a personal computer, mini-computer, mainframe, workstation, networked or distributed computing environment, separate or integrated computer platforms, or in communication with charged particle tools or other imaging devices, and the like. Aspects of the present application can be implemented in machine readable code stored on a non-transitory storage medium or device, whether removable or integrated into a computing platform, such as a hard disk, optical read and / or write storage media, RAM, ROM, and the like, such that it can be read by a programmable computer to configure and operate the computer to perform the processes described herein. Further, the machine readable code, or portions thereof, can be transmitted over wired or wireless networks. The present application described herein includes these and other different types of non-transitory computer readable storage media when such media include instructions or programs that implement the steps described above in conjunction with a microprocessor or other data processor. The present application can also include the computer itself when programmed in accordance with the methods and techniques described herein.

[0155] A computer program can be applied to input data to perform the functions described herein, thereby transforming the input data to generate output data that is stored to non-volatile memory. The output information can also be applied to one or more output devices such as a display. In a preferred embodiment of the application, the transformed data represents a physical and tangible object, including a particular visual depiction of a physical and tangible object produced on a display.

[0156] The above description is only preferred embodiments of the present application, the present application is not limited to the above-described embodiments, as long as the same means to achieve the technical effects of the present application, any modification, equivalent replacement, improvement, etc. within the spirit and principles of the present application, should be included in the scope of protection of the present application. The technical solutions and / or embodiments within the scope of protection of the present application can have various modifications and changes.

Claims

1. A method for two-frame pedestrian trajectory prediction in panoramic video, characterized in that, The method comprises the following steps: S100, acquiring and splicing videos in different directions to obtain a panoramic video, extracting position information, speed information, action information and optical flow information of pedestrians in the panoramic video, and dividing the pedestrians into freely moving pedestrians and socially moving pedestrians; S200, calculating a free pedestrian dataset and a whole pedestrian dataset, the free pedestrian dataset comprising information of all freely moving pedestrians, the whole pedestrian dataset comprising information of all pedestrians, dividing the free pedestrian dataset into a free pedestrian training set, a free pedestrian verification set and a free pedestrian test set, and dividing the whole pedestrian dataset into a whole pedestrian training set, a whole pedestrian verification set and a whole pedestrian test set; S300, using a trained free pedestrian prediction network, a future social pooling network and a social pedestrian prediction network to predict pedestrian trajectories, wherein the free pedestrian prediction network is trained based on the free pedestrian dataset, the future social pooling network and the social pedestrian prediction network are trained based on the whole pedestrian dataset, and input data of the future social pooling network is the trained free pedestrian prediction network; The future social pooling network and the social pedestrian prediction network in step S300 are trained based on the whole pedestrian dataset, comprising: S306, using the trained free pedestrian prediction network to predict free moving pedestrian trajectories of all pedestrians in the whole pedestrian training set to obtain a free moving trajectory prediction sequence list of all pedestrians in the whole pedestrian training set, represented as where i = 1, 2, …, m, t0 is the prediction start time, T obs is the observed two-frame time interval, T gap is the time interval of each prediction, T pred is the maximum time of prediction, m is the number of people contained in all pedestrian samples in the whole pedestrian training set, and T list is the time sequence of all pedestrian samples in the whole pedestrian training set, T list = [t0-T obs ,t0,t0+T gap ,t0+2*T gap ,…,t0+T pred ] ; S307, For the time series T of all pedestrian samples in the entire pedestrian training set. list Each step in the process will determine the total number of pedestrians Pedes at time t. i Location prediction information With speed prediction information Embedded in the fourth fully connected layer φ ex and the fifth fully connected layer φ ev Pedes, the pedestrian i The vector after embedding the position prediction information and velocity prediction information at time t and EV t i ,Bundle and After being connected in series, it passes through the sixth fully connected layer φ ec Obtaining pedestrian Pedes i At time t, the position and velocity information are obtained using the vector embedded by the sixth fully connected layer, which is the vector of all pedestrian states in the sample. wherein i = 1:m, t = T list , φ ex , φ ev are the fourth fully connected layer and the fifth fully connected layer with PRelu activation function respectively, the input layer unit number of the fourth fully connected layer and the fifth fully connected layer is 4, the output unit number is es, the input layer unit number of the sixth fully connected layer φ ec is 2*es, the output unit number is es, are trainable network parameters; S308, embedding the relative position of the pdm pedestrians close to the pedestrian Pedes i to a seventh fully connected layer φ er , obtaining the pedestrian Pedes i and the pedestrian Pedes i The vector of the relative position information at time t is embedded using a fully connected layer for predicting the trajectory of the pedestrian Pedes i , the process is represented as: wherein i = 1, 2, …, pdm, the seventh fully connected layer φ er For a fully connected layer using a PReLu activation function, the input layer unit number of the seventh fully connected layer is 2, and the output unit number is es, is a trainable network parameter, and pdmis the number of other pedestrians closest to the pedestrian Pedes1 in the pixel coordinate system. S309, For the time series T of all pedestrian samples in the entire pedestrian training set. list Each step in the process stacks up to obtain the pedestrian Pedes. i Social pooling matrix at time t Pedestrian Pedes i The social pooling matrix SP at time t t After convolutional network δ sp After extracting the information, obtain the SPC. t SPC t Input to the eighth fully connected layer φ es The pedestrian Pedes are obtained by extracting information through the eighth fully connected layer. i The social pooling vector ES at time t t The process is represented as follows: wherein t=T list , the convolutional network δ sp is a three-layer convolutional network with a PRelu activation function, the convolutional kernel size of the convolutional network δ sp is 3, and the number of output channels is 5, 7, and 9 respectively, the eighth full connection layer φ es is a full connection layer with an output unit number of 6*es, W sp , W es are trainable network parameters; S310, using a sixth GRU network GRU es Pedes i The state vector of the social information of the sixth GRU network used for encoding at t time The process is represented as: Wherein, t=T list , the sixth GRU network GRU es is a GRU recurrent neural network, the number of input layer units is 6*es, the number of state layer units is 2*rs, W ges is a trainable network parameter.

2. The panoramic video two-frame pedestrian trajectory prediction method according to claim 1, characterized in that, Step S100 comprises: S110, acquiring two videos in opposite directions and transversely splicing the two videos to obtain a panoramic video; S120, calculate historical position information of the pedestrian based on the panoramic video Pedestrian Pedes i The position information at the time t is wherein i represents the current i-th pedestrian, and respectively are the pedestrian Pedes i the horizontal and vertical coordinates of the position of the pedestrian Pedes and respectively are the pedestrian Pedes i the length and width of the detection box in the pixel coordinate system; S130, calculate historical speed information of the pedestrian based on the panoramic video Pedestrian Pedes i The speed information at time t is wherein T obs is the time interval between the two frames of the panoramic video; S140, calculate historical action information of the pedestrian based on the panoramic video Pedestrian Pedes i The action information at time t is wherein, are the standardized horizontal and vertical coordinates of the 18 joints of the pedestrian 18; S150, calculate the historical optical flow information of the pedestrian based on the panoramic video Pedestrian Pedes i The optical flow information at time t includes the horizontal axis component and the vertical axis component of the optical flow of the pixels within the pedestrian detection frame. The size of the pedestrian detection frame is adjusted to 10 pixels*10 pixels, which is represented as S160, marking pedestrians in different moving types, including freely moving pedestrians and socially moving pedestrians, the freely moving pedestrians being pedestrians whose movements are not affected by others throughout the whole process, and the socially moving pedestrians being pedestrians whose movements are affected by others in a certain period of time.

3. The panoramic video two-frame pedestrian trajectory prediction method according to claim 1, characterized in that, Step S200 comprises: S210, a free-moving pedestrian sample is made to sample each free-moving pedestrian trajectory at uniform time intervals, numbered as Pedes i The time sequence of the observation frame is: T obslist = [t0-T obs ,t0], where t0 is the prediction start time, T obs is the observed two-frame time interval, The time sequence of the label frame is: T labellist = [t0+T gap ,t0+2*T gap ,…,t0+T pred ], where T pred is the maximum time, T gap is the time interval of each prediction, obtaining each free-moving pedestrian sample, an input of each free-moving pedestrian sample comprising free-moving pedestrian historical position information free-moving pedestrian historical speed information free-moving pedestrian historical motion information and free-moving pedestrian historical optical flow information a label of each free-moving pedestrian sample comprising free-moving pedestrian label position information free-moving pedestrian label speed information free-moving pedestrian label motion information and free-moving pedestrian label optical flow information S220, collecting samples of all freely moving pedestrians to make a free pedestrian dataset, and dividing the samples of the freely moving pedestrians into mutually different free pedestrian training sets, free pedestrian test sets and free pedestrian verification sets according to a preset proportion; S230, make a sample of all pedestrians, sample the trajectories of all pedestrians at uniform time intervals, for a certain sampled pedestrian numbered as Pedes i , all pedestrians are represented as: Pedes 1:m = [Pedes1, Pedes2,..., Pedes m ], Wherein, m is the number of pedestrians in the panoramic video at this moment, input for the all-pedestrian sample includes all-pedestrian historical position information all-pedestrian historical speed information all-pedestrian historical motion information and all-pedestrian historical optical flow information label for the all-pedestrian sample includes all-pedestrian label position information all-pedestrian label speed information all-pedestrian label motion information and all-pedestrian label optical flow information S240, collecting samples of all whole moving pedestrians to make a whole pedestrian dataset, and dividing the samples of the whole moving pedestrians into mutually different whole pedestrian training sets, whole pedestrian test sets and whole pedestrian verification sets according to a preset proportion.

4. The panoramic video two-frame pedestrian trajectory prediction method according to claim 3, characterized in that, The free pedestrian prediction network in step S300 is trained based on the free pedestrian dataset, comprising: S301, respectively, the freely moving pedestrian Pedes i Location information at time t Speed ​​information and action information Embedded into the first fully connected layer φ ex Second fully connected layer φ ev and the third fully connected layer φ ep Free-moving pedestrians Pedes i Optical flow information at time t Embedded into the first multilayer perceptron ψ ef The embedded vectors are obtained respectively. and And embed the vector and Use the first GRU network GRU in sequence ex Second GRU network GRU ev Third GRU network GRU ep and the fourth GRU network GRU ef Encode the freely moving pedestrians Pedes. i The state vector of the GRU network used for encoding at time t, containing position, velocity, motion, and optical flow information. and wherein i = 1, t = T obslist , φ ex , φ ev are respectively a first fully connected layer and a second fully connected layer using a PRelu activation function, the input unit number of the first fully connected layer and the second fully connected layer is 4, the output unit number is es, φ ep is a third fully connected layer using a PRelu activation function, the input unit number of the third fully connected layer is 36, the output unit number is es, ψ ef is a first multi-layer perception of two layers using a PRelu activation function, the input layer unit number of the first multi-layer perception is 200, the hidden layer unit number is 2*es, the output layer unit number is es, the input layer unit number of the first GRU network GRU ex , the second GRU network GRU ev , the third GRU network GRU ep and the fourth GRU network GRU ef is es, the state layer unit number is rs, W x , W v , W p , W f , W ex , W ev , W ep , W ef are trainable network parameters; S302, a state vector of position information, speed information, action information, and optical flow information is obtained and are concatenated and passed through a second multi-layer perceptron ψ gc , outputting a state vector for decoding wherein ψ gc is a second multi-layer perceptron using a PReLu activation function, said second multi-layer perceptron ψ gc is a two-layered multi-layer perceptron MLP network, said second multi-layer perceptron ψ gc has an input layer with 4*rs input units and a hidden layer with 4*rs hidden units, and an output layer with 3*rs output units, W gc is a trainable network parameter; S303、the state vector for decoding inputting a fifth GRU network GRU de and a third multi-layer perception ψ gl , predicting the free moving trajectory of the pedestrian, and the time sequence of the prediction frame is: T predlist = [t0, t0+T gap , t0+2*T gap ,..., t0+T pred ], where T pred is the maximum time, T gap is the time interval of each prediction, The decoding process is represented as: where i = 1, t = T predlist , denotes the pedestrian Pedes i the state vector of the fifth GRU network for decoding at time t, the fifth GRU network GRU de has an input layer unit number of 2*es, a state layer unit number of 3*rs, and is a decoder network, ψ gl is a third multi-layer perceptron MLP network with PReLU activation function using Dropout mechanism, the third multi-layer perceptron has an input layer unit number of 3*rs, hidden layer unit numbers of 3*rs and rs respectively, and an output layer unit number of 4, denotes the predicted free movement position change, and denote the predicted next step free movement position and velocity respectively, W de , W gl , W x , W v are trainable network parameters, The free moving trajectory prediction sequence of the current sample pedestrian is where i = 1, t0is the prediction start time, T obs is the observed time interval between two frames, T pred is the maximum time for prediction, T gap is the time interval for each step of prediction.

5. The panoramic video two-frame pedestrian trajectory prediction method according to claim 4, characterized in that, The free pedestrian prediction network in step S300 is trained based on the free pedestrian dataset, further comprising: S304、train the loss function L of the free pedestrian prediction network CloU , update the free pedestrian prediction network parameters, L CloU = L ACIoU + L FCIoU + L Reg where L ACIoU is the average CIoU error term, L FCIoU is the final CIoU error term, L Reg is the L2 regularization term for the free pedestrian prediction network weights, CIoU(X pred , X gt ) represents the complete intersection ratio of X pred and X gt , length(T labellist ) represents the number of elements in T labellist , and k ACIoU and k FCIoU are set constants. S305, select the free pedestrian prediction network parameters, use the free pedestrian verification set to verify the network parameters of each generation in the training process, select the loss function L of the free pedestrian prediction network FCIoU The network parameters with the minimum item are selected as the final parameters of the free pedestrian prediction network.

6. The panoramic video two-frame pedestrian trajectory prediction method according to claim 5, characterized in that, The future social pooling network and the social pedestrian prediction network in step S300 are trained based on the whole pedestrian dataset, further comprising: S311, respectively, pedestrian Pedes i Location information at time t Speed ​​information and action information Embedded into the ninth fully connected layer 10th Fully Connected Layer and the eleventh fully connected layer Pedestrian Pedes i Optical flow information at time t Embedded into the fourth multilayer perceptron The embedded vectors were obtained respectively. and And embed the vector and Use the seventh GRU network in sequence Eighth GRU Network Ninth GRU Network and the 10th GRU network Encode the pedestrians to obtain the Pedes. i The state vector of the GRU network used for encoding at time t, containing position, velocity, motion, and optical flow information. and The process is represented as follows: wherein i = 1, t = T obslist , the ninth fully connected layer and the tenth fully connected layer use a PRelu activation function, the input layer unit number of the ninth fully connected layer and the tenth fully connected layer is 4, the output unit number is es, the eleventh fully connected layer is a two-layer multi-layer perception MLP using a PRelu activation function, the input unit number of the eleventh fully connected layer is 36, the output unit number is es, the fourth multi-layer perception is a two-layer multi-layer perception MLP using a PRelu activation function, the input layer unit number of the fourth multi-layer perception is 200, the hidden layer unit number is 2*es, the output layer unit number is es, the seventh GRU network the eighth GRU network the ninth GRU network and the tenth GRU network are all GRU recurrent neural networks, the input layer unit number of the seventh GRU network the eighth GRU network the ninth GRU network and the tenth GRU network is es, the state layer unit number is rs, is a trainable network parameter; S312, Pedes i State vectors of position information, speed information, action information, optical flow information and social information of the GRU network for encoding at t time And After being concatenated, pass through the fifth multi-layer perception Get the final historical information encoding vector The process is represented as: The fifth multi-layer perception machine The fifth multi-layer perception machine is a two-layer multi-layer perception machine MLP network using a PRelu activation function The number of input layer units is 6*rs, the number of hidden layer units is 4*rs, and the number of output layer units is 3*rs. S313、encoding the final historical information vector inputting an eleventh GRU network and a sixth multi-layer perception predicting the social moving trajectory of all pedestrians, obtaining a predicted social moving position change a predicted next-step social moving position and a velocity The process is represented as: where i = 1, t = T predlist , Pedes represents a pedestrian i the state vector of the GRU network for decoding at time t, the eleventh GRU network the number of units of the input layer is 2*es, the number of units of the state layer is 3*rs, the sixth multi-layer perception a three-layer multi-layer perception MLP network with a PRelu activation function using a Dropout mechanism, the sixth multi-layer perception the number of units of the input layer is 3*rs, the number of units of the hidden layer is 3*rs and rs respectively, and the number of units of the output layer is 4, is a trainable network parameter, The trajectory prediction sequence of the social pedestrian Pedes1 is obtained, Wherein, i=1; S314, according to step S304, training the loss function of the future social pooling network and the social pedestrian prediction network, updating the network parameters of the future social pooling network and the social pedestrian prediction network, according to step S305, using the whole pedestrian verification set to verify the network parameters of each generation in the training process, selecting the network parameters with the minimum loss function term of the future social pooling network and the social pedestrian prediction network as the final parameters of the future social pooling network and the social pedestrian prediction network.

7. The panoramic video two-frame pedestrian trajectory prediction method according to claim 1, characterized in that, Step S300 uses the trained free pedestrian prediction network, future social pooling network and social pedestrian prediction network to perform pedestrian trajectory prediction, including: For two frames of panoramic video data of time interval T obs , obtain historical position information of all pedestrians in the panoramic video historical speed information historical motion information and historical optical flow information using the free pedestrian prediction network to predict the free movement trajectory of all pedestrians in the image, and then using the future social pooling network and the social pedestrian prediction network to sequentially predict the social movement trajectory of all pedestrians.

8. A panoramic video two-frame pedestrian trajectory prediction apparatus for implementing the panoramic video two-frame pedestrian trajectory prediction method according to any one of claims 1 to 7, characterized in that, The device comprises: a video acquisition device for acquiring and splicing videos; a deep learning control board for classifying and learning pedestrians in the video and analyzing and predicting the walking trajectory of the pedestrians in the video; a general control system for controlling the video acquisition device and the deep learning control board and displaying result information.

9. A computer readable storage medium having program instructions stored thereon, the program instructions being executed by a processor to implement the method of any one of claims 1 to 7.

Images