Trajectory prediction method and device, computer equipment and storage medium

Abstract

The invention relates to a trajectory prediction method and device, computer equipment and a storage medium. The method comprises the following steps: inputting a received to-be-processed image into a prediction model; wherein the prediction model comprises a first local convolutional neural network, a second local convolutional neural network, a first global convolutional neural network and a second global convolutional neural network; in the prediction model, performing difference processing on the first local feature map and the first global feature map to obtain a difference feature map; based on the difference feature map, determining a second local feature map and a second global feature map, and fusing the second local feature map and the second global feature map to obtain a first fused feature map; and predicting track information of a target in the to-be-processed image based on the first fusion feature map. By adopting the method, the accuracy of a target trajectory prediction result can be improved.

Description

Technical Field

[0001] The present application relates to the technical field of target detection, and in particular to a trajectory prediction method, apparatus, computer equipment and storage medium. Background Art

[0002] With the development of large models, trajectory prediction achieved with the help of large models has also been applied to more and more fields. For example, drone technology, robot control, weather forecasting, intelligent driving, etc. Taking intelligent driving as an example, trajectory prediction can be used to adjust the driving route of the two vehicles in time to avoid accidents. The specific prediction method is: use the images collected by the on-board camera to perform target detection to obtain the location information of the target (i.e. pedestrian), and then combine the target's location information with the vehicle's location, speed and other information to predict the target's location at the next specified time or the next specified time period. Since this method essentially treats the target as a static obstacle for trajectory prediction and ignores the uncertainty of pedestrian movement itself, there is a problem of insufficient accuracy of the prediction results. Summary of the invention

[0003] Based on this, it is necessary to provide a trajectory prediction method, device, computer equipment and storage medium to address the above technical problems in order to improve the accuracy of target trajectory prediction results.

[0004] In a first aspect, an embodiment of the present application provides a trajectory prediction method, comprising:

[0005] Inputting the received image to be processed into a prediction model; wherein the prediction model includes a first local convolutional neural network, a second local convolutional neural network, a first global convolutional neural network, and a second global convolutional neural network;

[0006] In the prediction model, a difference process is performed on the first local feature map and the first global feature map to obtain a difference feature map; wherein the first local feature map and the first global feature map have the same size, the first local feature map is obtained by the first local convolutional neural network, and the first global feature map is obtained by the first global convolutional neural network;

[0007] Based on the difference feature map, determining a second local feature map and a second global feature map, and fusing the second local feature map and the second global feature map to obtain a first fused feature map; wherein the second local feature map is obtained by the second local convolutional neural network, and the second global feature map is obtained by the second global convolutional neural network;

[0008] Based on the first fused feature map, trajectory information of the target in the image to be processed is predicted.

[0009] In one embodiment, performing difference processing on the first local feature map and the first global feature map to obtain a difference feature map includes:

[0010] Based on the image to be processed, determining an initial feature map of a preset size;

[0011] Inputting the initial feature map into the first local convolutional neural network to obtain a first local feature map, and inputting the initial feature map into the first global convolutional neural network to obtain a first global feature map; wherein the first local feature map and the first global map are both of the preset size;

[0012] Subtract the first local feature map from the first global feature map to obtain the difference feature map.

[0013] In one embodiment, inputting the initial feature map into the first local convolutional neural network to obtain the first local feature map includes:

[0014] In the first local convolutional neural network, the initial feature map is locally convolved and zero-filled to obtain a first intermediate feature map of the preset size;

[0015] Determining a first attention vector corresponding to the first intermediate feature map;

[0016] Performing a product operation on the first attention vector and the first intermediate feature map to obtain a second intermediate feature map;

[0017] The second intermediate feature map is added to the initial feature map to obtain the first local feature map.

[0018] In one embodiment, inputting the initial feature map into the first global convolutional neural network to obtain a first global feature map includes:

[0019] In the first global convolutional neural network, performing global convolution processing on the initial feature map along a width direction to obtain a third intermediate feature map, and performing global convolution processing on the initial feature map along a height direction to obtain a fourth intermediate feature map;

[0020] Fusing the third intermediate feature map and the fourth intermediate feature map to obtain a fifth intermediate feature map of the preset size;

[0021] Determining a second attention vector corresponding to the fifth intermediate feature map;

[0022] Performing a product operation on the second attention vector and the fifth intermediate feature map to obtain a sixth intermediate feature map;

[0023] The sixth intermediate feature map is added to the initial feature map to obtain the first global feature map.

[0024] In one embodiment, the fusing the second local feature map and the second global feature map to obtain a first fused feature map includes:

[0025] Adding the second local feature map to the first local feature map to obtain a third local feature map, and adding the second global feature map to the first global feature map to obtain a third global feature map;

[0026] In the channel dimension, the third local feature map and the third global feature map are concatenated to obtain the first fused feature map.

[0027] In one embodiment, the prediction model is trained by the following method:

[0028] Acquire a training image set; wherein the training image set includes a plurality of training images and preset labels corresponding to targets in the training images;

[0029] Inputting the training image into the model to be trained, and obtaining a second fused feature map through the local convolutional neural network to be trained and the global convolutional neural network to be trained in the model to be trained;

[0030] Based on the second fused feature map, determining a model label corresponding to the target;

[0031] Based on the difference between the model label and the preset label, the parameters of the local convolutional neural network to be trained and the global convolutional neural network to be trained are adjusted in reverse until the accuracy of the model label meets the preset threshold, thereby obtaining the prediction model.

[0032] In one embodiment, the prediction model includes a long short-term memory network LSTM;

[0033] Then, predicting the trajectory information of the target in the image to be processed based on the first fused feature map includes:

[0034] Processing the plurality of the first fused feature maps through the LSTM to obtain target hidden layer features; wherein the first fused feature maps correspond one to one to the images to be processed;

[0035] The target hidden layer features are processed through a fully connected layer to obtain the trajectory information.

[0036] In one embodiment, the step of processing the plurality of the first fused feature maps by the LSTM to obtain target hidden layer features includes:

[0037] Inputting the plurality of the first fusion feature maps into the first layer LSTM to obtain a plurality of intermediate hidden layer features; wherein the intermediate hidden layer features include the posture information of the target, and the LSTM includes the first layer LSTM and the second layer LSTM;

[0038] Fusion of the intermediate hidden layer features and the first feature vector to obtain a second feature vector; wherein the first feature vector includes speed information and / or position information of the target, and the first feature vector corresponds to the intermediate hidden layer features one by one;

[0039] The second feature vector is input into the second layer LSTM to obtain the target hidden layer feature.

[0040] 8. A trajectory prediction device, comprising:

[0041] An input module, used to input the received image to be processed into a prediction model; wherein the prediction model includes a first local convolutional neural network, a second local convolutional neural network, a first global convolutional neural network and a second global convolutional neural network;

[0042] A difference module, used for performing difference processing on the first local feature map and the first global feature map in the prediction model to obtain a difference feature map; wherein the first local feature map and the first global feature map have the same size, the first local feature map is obtained by the first local convolutional neural network, and the first global feature map is obtained by the first global convolutional neural network;

[0043] A fusion module, based on the difference feature map, determines a second local feature map and a second global feature map, and fuses the second local feature map and the second global feature map to obtain a first fused feature map; wherein the second local feature map is obtained by the second local convolutional neural network, and the second global feature map is obtained by the second global convolutional neural network;

[0044] A trajectory module is used to predict the trajectory information of the target in the image to be processed based on the first fused feature map.

[0045] In one embodiment, the difference module is specifically used to determine an initial feature map of a preset size based on the image to be processed; input the initial feature map into the first local convolutional neural network to obtain a first local feature map, and input the initial feature map into the first global convolutional neural network to obtain a first global feature map; wherein the first local feature map and the first global map are both of the preset size; and subtract the first local feature map from the first global feature map to obtain the difference feature map.

[0046] In one embodiment, the difference module is specifically used to perform local convolution processing on the initial feature map in the first local convolutional neural network, and fill it with zeros to obtain the first intermediate feature map of the preset size; determine a first attention vector corresponding to the first intermediate feature map; perform a product operation on the first attention vector and the first intermediate feature map to obtain a second intermediate feature map; add the second intermediate feature map and the initial feature map to obtain the first local feature map.

[0047] In one embodiment, the difference module is also used to perform global convolution processing on the initial feature map along the width direction in the first global convolutional neural network to obtain a third intermediate feature map, and perform global convolution processing on the initial feature map along the height direction to obtain a fourth intermediate feature map; fuse the third intermediate feature map and the fourth intermediate feature map to obtain a fifth intermediate feature map of the preset size; determine a second attention vector corresponding to the fifth intermediate feature map; perform a product operation on the second attention vector and the fifth intermediate feature map to obtain a sixth intermediate feature map; add the sixth intermediate feature map to the initial feature map to obtain the first global feature map.

[0048] In one embodiment, the fusion module is specifically used to add the second local feature map to the first local feature map to obtain a third local feature map, and to add the second global feature map to the first global feature map to obtain a third global feature map; in the channel dimension, the third local feature map and the third global feature map are spliced ​​to obtain the first fused feature map.

[0049] In one embodiment, the prediction model includes a long short-term memory network (LSTM); the trajectory module is specifically used to process multiple first fused feature maps through the LSTM to obtain target hidden layer features; wherein the first fused feature maps correspond one-to-one to the image to be processed; and the target hidden layer features are processed through a fully connected layer to obtain the trajectory information.

[0050] In one embodiment, the trajectory module is also used to input multiple of the first fused feature maps into the first layer LSTM to obtain multiple intermediate hidden layer features; wherein the intermediate hidden layer features include the posture information of the target, and the LSTM includes the first layer LSTM and the second layer LSTM; the intermediate hidden layer features and the first feature vector are fused to obtain the second feature vector; wherein the first feature vector includes the speed information and / or position information of the target, and the first feature vector corresponds one-to-one to the intermediate hidden layer features; the second feature vector is input into the second layer LSTM to obtain the target hidden layer features.

[0051] In a third aspect, an embodiment of the present application provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method described in the first aspect and any possible implementation manner when executing the computer program.

[0052] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the method described in the first aspect and any possible implementation manner are implemented.

[0053] The above-mentioned trajectory prediction method, device, computer equipment and storage medium perform difference processing on the first local feature map output by the first local convolutional neural network and the first global feature map output by the first global convolutional neural network in the prediction model to obtain a difference feature map to highlight the key features of the key parts of the target. Then, the second global convolutional neural network and the second local convolutional neural network are used to continue processing the difference feature map to capture the key features of the above-mentioned key parts from the global and local scales, so as to form a more accurate feature expression of the key features of the target in the first fused feature map. Therefore, the prediction model performs more accurate feature calculation and prediction of the moving direction and moving speed of the target by processing the feature expression in the first fused feature map, so as to effectively improve the accuracy of the predicted trajectory information. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0055] Figure 1 A diagram showing an application environment of a trajectory prediction method in an embodiment;

[0056] Figure 2 A schematic diagram of a flow chart of a trajectory prediction method in one embodiment;

[0057] Figure 3 A schematic flow chart of the steps of determining a difference feature map in one embodiment;

[0058] Figure 4 is a schematic diagram of the structure of a first local convolutional neural network in one embodiment;

[0059] Figure 5 is a schematic diagram of the structure of a first global convolutional neural network in one embodiment;

[0060] Figure 6 A schematic diagram of a method for generating a first fusion feature map in one embodiment;

[0061] Figure 7 A schematic diagram of a flow chart of a method for generating target hidden layer features by using a double-layer LSTM in one embodiment;

[0062] Figure 8 is a structural block diagram of a trajectory prediction device in one embodiment;

[0063] Fig. 9 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0064] The technical solution of the present application is described in detail below through the accompanying drawings and specific embodiments. It should be understood that the embodiments of the present application and the specific features in the embodiments are detailed descriptions of the technical solution of the present application, rather than limitations on the technical solution of the present application. In the absence of conflict, the embodiments of the present application and the technical features in the embodiments can be combined with each other.

[0065] The terms "first" and "second" in the specification and claims of the present application and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the term "comprising" and any of their variations are intended to cover non-exclusive protection. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the listed steps or units, but optionally also includes steps or units that are not listed, or optionally also includes other steps or units inherent to these processes, methods, products or devices. "Multiple" in the present application can mean at least two, for example, two, three or more, and the embodiments of the present application are not limited.

[0066] The trajectory prediction method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown, the terminal 102 communicates with the server 104 via the network.

[0067] In one embodiment, the terminal 102 collects the image to be processed and sends the image to be processed to the server 104 through the network, so that the server 104 processes the received image to be processed using the prediction model to predict the trajectory information of the target in the image to be processed.

[0068] Alternatively, in one embodiment, the terminal 102 collects the image to be processed, and processes the image to be processed in real time using the prediction model pre-installed in the terminal 102 to predict the trajectory information of the target in the image to be processed. The server 104 maintains and updates the prediction model in the terminal 102 through the network.

[0069] The terminal 102 may be, but is not limited to, various personal computers, laptop computers, smart phones, tablet computers, and portable wearable devices, and the server 104 may be implemented as an independent server or a server cluster consisting of multiple servers.

[0070] In one embodiment, Figure 2 As shown, a trajectory prediction method is provided, the method comprising the following steps:

[0071] Step 201: input the received image to be processed into the prediction model.

[0072] Among them, the prediction model includes a local convolutional neural network and a global convolutional neural network, the local convolutional neural network is a first local convolutional neural network and a second local convolutional neural network; the global convolutional neural network is a first global convolutional neural network and a second global convolutional neural network.

[0073] Specifically, the above-mentioned images to be processed may be multiple images, specifically, continuous and / or discontinuous video frames in a video, or may be images with the same and / or different acquisition time intervals.

[0074] The time information corresponding to the image to be processed may be the acquisition time of the image to be processed, or may be the time when the image to be processed is received.

[0075] The above-mentioned image to be processed contains at least one target. The target described in the embodiment of the present application has motion attributes, such as pedestrians, pets, etc.

[0076] The above-mentioned local convolutional neural network focuses on the local features in the corresponding feature map, identifies and extracts detailed features such as the local texture of the target, as well as the joint features of the target (for example, arms and legs), thereby paying attention to the details of the target movement and the joint transformation, thereby accurately extracting the target's movement information.

[0077] The above-mentioned global convolutional neural network recognizes and extracts target features, target posture and motion features (from a holistic perspective) by focusing on the global features in the corresponding feature map, which helps to extract and recognize target posture information from dimensions such as motion continuity.

[0078] Step 202: In the prediction model, difference processing is performed on the first local feature map and the first global feature map to obtain a difference feature map.

[0079] Among them, the first local feature map and the first global feature map have the same size, the first local feature map is obtained through the above-mentioned first local convolutional neural network, and the above-mentioned first global feature map is obtained through the first global convolutional neural network.

[0080] Specifically, the feature maps processed by the first local convolutional neural network and the first global convolutional neural network are both initial feature maps of a preset size. The initial feature map is a shallow feature map obtained by shallow convolution processing.

[0081] The first local feature map includes the joint positions and posture features of the target. For example, the positions and posture features of the head, elbows, hands, feet, etc. The first local feature map also includes the local texture and detail features of the target, such as clothing texture, etc. The first global feature map at least includes the position information and overall posture features of the target relative to other non-target objects.

[0082] The difference processing helps to highlight the difference between the first local feature map and the first global feature map, thereby obtaining a difference feature map that can reflect the key features of the action change. Compared with the first local feature map and the first global feature map, the edge and detail features in the difference feature map are enhanced through the difference processing, and the features with low correlation (such as background features, etc.) are removed.

[0083] Taking the target as a pedestrian in a running state, the pedestrian in the collected image to be processed is leaning forward, with his arms raised and one foot raised. Then, the first local feature map at least includes feature information that the pedestrian's elbow is slightly raised, the knee is bent, one foot is raised, and the other foot is retreated. The first global feature map at least includes the position information and overall posture features of the pedestrian relative to other non-targets (such as green belts) in the collected image. In the difference feature map obtained by performing difference processing on the above first global feature map and the first local feature map, at least the background information in the first global feature map is removed, and the change features such as one foot raised and the arm slightly bent are enhanced. Therefore, the purpose of enhancing important features can be achieved through the difference feature map, which prompts the difference feature map to highlight feature information that is closer to the running posture of the pedestrian.

[0084] Step 203: Based on the difference feature map, determine a second local feature map and a second global feature map, and fuse the second local feature map and the second global feature map to obtain a first fused feature map.

[0085] Among them, the second local feature map is obtained through a second local convolutional neural network, and the second global feature map is obtained through a second global convolutional neural network.

[0086] Specifically, the second local feature map and the second global feature map can respectively capture the feature information in the difference feature map from the local scale and the global scale, thereby further highlighting the difference between the global features and the local features and highlighting the important feature information.

[0087] The first fused feature map obtained by fusing the second local feature map and the second global feature map here can provide a more abstract and generalized feature representation of the target, thereby highlighting the difference between the local features and the global features of the target, thereby improving the accuracy of the prediction model while reducing the risk of overfitting of the prediction model.

[0088] Step 204: predicting the trajectory information of the target in the image to be processed based on the first fused feature map.

[0089] Specifically, the trajectory information includes the position information of the target at a specified time, and the specified time is after the time corresponding to the image to be processed, so as to achieve the purpose of prediction.

[0090] The trajectory information of the target in the image to be processed can be predicted based on the multiple first fused feature maps. The time corresponding to the first fused feature map is the time corresponding to the image to be processed to which it corresponds.

[0091] The trajectory information may also include speed information of the target corresponding to the specified time and position information, that is, the speed of the target at a certain time and a certain coordinate (ie, position).

[0092] If there are multiple images to be processed, then there are also multiple first fused images corresponding to the images to be processed. Then, the time in the trajectory information may be later than the latest time among the times corresponding to the multiple images to be processed.

[0093] In one embodiment, the position information, speed information and motion information of the target at different times can be directly identified and extracted from the first fusion feature map, so as to predict the trajectory information of the target in the image to be processed.

[0094] In another embodiment, the position information and speed information of the target at the time corresponding to the first fusion feature map can be pre-acquired. Then, based on the first fusion feature map and the position information and speed information of the target at the same time corresponding to the first fusion feature map, the position information of the target at a specified time can be predicted, and the position information at different specified times constitutes the trajectory information of the target.

[0095] In the above trajectory prediction method, the local features and global features of the target in the image to be processed are respectively identified and extracted by the local convolutional neural network and the global convolutional neural network in the prediction model. In addition, after the features of the target motion change are enhanced by performing difference processing on the first local feature map and the first global feature map, local features (i.e., the second local feature map) and global features (i.e., the second global feature map) are further extracted from the difference feature map, further highlighting the difference features and reducing irrelevant information, thereby obtaining a first fused feature map. In this way, when processing the first fused feature map to predict the target trajectory, the influence of irrelevant information (i.e., noise) on the processing process can be reduced by processing the first fused feature map in which irrelevant information is reduced, thereby improving the prediction efficiency and improving the accuracy of the predicted trajectory information. On this basis, processing the difference features highlighted by the first fused feature map can further improve the accuracy of the predicted trajectory information.

[0096] The following first describes in detail the determination of the difference feature map, please refer to Figure 3 :

[0097] Step 301: Determine an initial feature map of a preset size based on the image to be processed.

[0098] The preset size may be C×H×W, where C is the number of channels, H is the height, and W is the width.

[0099] The initial feature map can be obtained by processing the image to be processed by a shallow convolution structure. The shallow convolution structure can be, for example, a vanilla convolution.

[0100] Step 302: Input the initial feature map into a first local convolutional neural network to obtain a first local feature map; and input the initial feature map into a first global convolutional neural network to obtain a first global feature map.

[0101] Wherein, the first local feature map and the first global feature map are both of preset sizes C×H×W.

[0102] Specifically, the first local feature map can be obtained by performing feature enhancement processing on the first intermediate feature map obtained by local convolution processing through a first attention vector. The first attention vector is composed of a plurality of attention coefficients corresponding to elements of the first intermediate feature map.

[0103] Furthermore, the first global feature map can be obtained by performing feature enhancement processing on the fifth intermediate feature map obtained by global convolution processing through a second attention vector. The second attention vector is composed of a plurality of attention coefficients corresponding to elements of the fifth intermediate feature map.

[0104] Step 303: Subtract the first local feature map from the first global feature map to obtain a difference feature map C×H×W.

[0105] Specifically, the elements in the difference feature map can be expressed as D(i, j, k), where i represents the row number of the difference feature map, such as the first row, the second row... j represents the column number in the difference feature map, such as the first column, the second column... k represents the corresponding channel number, such as the first channel, the second channel... The element in the difference feature map can be uniquely located through (i, j, k).

[0106] Then D(i,j,k)=L(i,j,k)-G(i,j,k);

[0107] Among them, L(i,j,k) is the element in the first local feature map, and G(i,j,k) is the element in the first global feature map. Similar to the difference feature map, i,j,k can uniquely locate the element in the first local feature map and can uniquely locate the element in the first global feature map.

[0108] Furthermore, an embodiment is provided below to illustrate the generation of the first local feature map:

[0109] like Figure 4 As shown, in the first local convolutional neural network, the initial feature map Hs of the preset size C×H×W is firstly subjected to local convolution processing to extract local features, and zero-padding is performed to obtain the first intermediate feature map H of the preset size C×H×W. L Specifically, a 3×3 kernel function can be used to perform local convolution processing on the initial feature map Hs to extract local features, and the extracted local feature map is processed with a zero padding size of 1 (i.e., padding=1) to obtain a first intermediate feature map H of a preset size (C×H×W). L .

[0110] Then, the first attention vector corresponding to the first intermediate feature map is determined to avoid over-reliance on local features and ignoring the global information in the initial feature map. Figure 4 , the first attention vector is obtained through the fully connected layer and activation function in the first local convolutional neural network, and the attention coefficients in the first attention vector are all between [0, 1].

[0111] Next, the first attention vector is multiplied by the first intermediate feature map to obtain the second intermediate feature map.

[0112] Finally, the second intermediate feature map is added to the initial feature map to capture the action characteristics of the target at a local scale through feature enhancement, and the first local feature map is obtained.

[0113] In one embodiment, the first attention vector a L It can be calculated by the following formula:

[0114] a L =Sigmoid(W 1 H L +b 1 ), where W 1 is the weight parameter matrix in the first local convolutional neural network obtained by training the prediction model, b 1 is the bias matrix in the first local convolutional neural network obtained by training the prediction model.

[0115] Then the second intermediate feature map can be expressed as a L ·H L , so the first local feature map H L1 It can be expressed as:

[0116] H L1 =a L ·H L +H s ; Among them, H s is the initial feature map.

[0117] Furthermore, the generation of the first global feature map is described below:

[0118] like Figure 5 As shown, in the first global convolutional neural network, the initial feature map is globally convolved along the width direction of the initial feature map to obtain a third intermediate feature map M. And, the initial feature map is globally convolved along the height direction of the initial feature map to obtain a fourth intermediate feature map N. For example, the size of the third intermediate feature map M may be C×1×W, and the size of the fourth intermediate feature map N may be C×H×1, M=f kw (H s ), N = f kH (H S ), where f kw Indicates that the C×W×H convolution kernel performs a global convolution operation in the width direction, f kH It means that the C×W×H convolution kernel performs a global convolution operation in the height direction.

[0119] Then, the third intermediate feature map and the fourth intermediate feature map are fused to obtain a fifth intermediate feature map of a preset size (C×H×W). Specifically, the third intermediate feature map and the fourth intermediate feature map can be multiplied to achieve feature fusion, and the fifth intermediate feature map H can be obtained. G Then H G =M·N.

[0120] Next, determine the second attention vector corresponding to the fifth intermediate feature map. For details, please continue to refer to Figure 5 , the second attention vector is obtained by inputting the fifth intermediate feature map into the fully connected layer and activation function in the first global convolutional network.

[0121] Next, the second attention vector is multiplied by the fifth intermediate feature map to obtain the sixth intermediate feature map.

[0122] Finally, the sixth intermediate feature map is added to the initial feature map to capture the motion characteristics of the target at a global scale and obtain the first global feature map.

[0123] In one embodiment, the second attention vector a G Calculated by the following formula:

[0124] a G =Sigmoid(W 2 H G +b 2 ), where W 2 is the weight parameter matrix in the first global convolutional neural network obtained by training the prediction model, b 1 is the bias matrix in the first global convolutional neural network obtained by training the prediction model.

[0125] Then the sixth intermediate feature map can be expressed as a G ·H G , so the first global feature map H G1 It can be expressed as:

[0126] H G1 =a G ·H G +H s ; Among them, H s is the initial feature map.

[0127] Furthermore, when fusing the second local feature map and the second global feature map, the second local feature map and the second global feature map may be directly added to obtain a first fused feature map.

[0128] Preferably, in order to avoid the problem of partial information loss or dilution when directly adding to obtain the first fused feature map, which leads to a decrease in the accuracy of the prediction model, feature combination can be performed in the following manner to improve the expression ability of the prediction model:

[0129] Please refer to Figure 6 , we can first transform the second local feature map H Ld With the first local feature map H L1 Add them together to get the third local feature map, and add the second global feature map H GdWith the first global feature map H G1 Add them together to get the third global feature map.

[0130] Then, the third local feature map and the third global feature map are concatenated to obtain the first fused feature map H all Specifically, the third local feature map and the third global feature map can be spliced ​​in the channel dimension to obtain the first fused feature map. Then the size of the first fused feature map is 2C×H×W. In this way, the first fused feature map can be focused on expressing the features reflecting the target posture information by enhancing the processed features (features in the first global feature map and features in the first local feature map) and the difference features (features in the second global feature map and features in the second local feature map), while the richness and accuracy of the feature representation can also be enhanced.

[0131] Moreover, the first fused feature map, due to its rich and accurate feature representation, helps the prediction model to better cope with complex environmental changes and noise, thereby effectively enhancing the robustness of the prediction model and reducing the risk of overfitting.

[0132] Furthermore, the posture information and position information of the target can be determined by determining the first fused feature map of the image to be processed obtained by continuous acquisition, and the trajectory information of the target can be predicted based on the posture information and position information of the target. Specifically, the posture information and position information of the target at each acquisition time are determined through multiple first fused feature maps and the acquisition time of the corresponding image to be processed. Then, based on this, the trajectory information of the target at the next time or at the next few target times is predicted. Here, the posture information can be, for example, walking, running, jumping, bending, raising hands, and waving; the position information can be, for example, the direction of the target's hands or feet, and the coordinates of the target in the physical coordinate system.

[0133] Preferably, in one embodiment, the long-term dependency of LSTM (Long Short-Term Memory) can be used to extract time series features from multiple first fused feature maps with time sequence (i.e., obtained by continuously collecting the aforementioned images to be processed), thereby further improving the accuracy of trajectory prediction. Specifically, multiple first fused feature maps can be processed by LSTM to obtain target hidden layer features. Among them, the first fused feature map corresponds to the image to be processed one by one, and each of the first fused features corresponds to a different collection time.

[0134] Then, the target hidden layer features are processed by the fully connected layer to predict the target's trajectory information. The trajectory information at least includes: the position of the target at one or more times after the time corresponding to the image to be processed.

[0135] The prediction model includes an LSTM, and the LSTM is connected to a fully connected layer to output the predicted trajectory information.

[0136] In one embodiment, multiple first fused feature maps are input into LSTM and passed through a fully connected layer to obtain trajectory information, wherein the first fused feature maps correspond one-to-one to the image to be processed.

[0137] In order to further improve the expressiveness and robustness of the prediction model and to further improve the accuracy of the predicted trajectory information, preferably, the LSTM includes multiple layers of LSTM.

[0138] The following uses a double-layer LSTM as an example for explanation. The target hidden layer features include the intermediate hidden layer features generated by the first layer LSTM and the target hidden layer features generated by the second layer LSTM. For details, please refer to Figure 7 :

[0139] Step 701, input multiple first fusion feature maps into the first layer LSTM in sequence to obtain multiple intermediate hidden layer features.

[0140] Among them, the intermediate hidden layer features include the posture information of the target, and the LSTM includes the first layer LSTM and the second layer LSTM.

[0141] The first layer LSTM can gradually process the fusion features in each first fusion feature map, and fuse the information in the input of the previous time step (i.e., the first fusion feature map received in the previous time step) into the intermediate hidden layer features of the first fusion feature map of the current time step, thereby capturing the information in the first fusion feature map before the current time step. In this way, multiple intermediate hidden layer features h are obtained. ai The intermediate hidden layer feature h ai The time steps in the first LSTM layer correspond to each other, and the order of each time step corresponds to the timing of receiving the image to be processed, the timing of generating the first fused feature map, and the order of inputting the first fused feature map to the first LSTM layer. ai Where i indicates the time step corresponding to the intermediate hidden layer feature, i = 1, 2, ..., t.

[0142] Among them, the intermediate hidden layer features include the posture information of the target, and the posture information may be, for example, walking, running, jumping, bending, raising hands, and waving.

[0143] Step 702: fuse the intermediate hidden layer features and the first feature vector to obtain a second feature vector.

[0144] The first feature vector includes the speed information and the position information of the target, and the first feature vector corresponds to the intermediate hidden layer feature one by one.

[0145] Specifically, the intermediate hidden layer features correspond to the first fusion features one by one, and the first fusion features correspond to the images to be processed one by one, so the intermediate hidden layer features correspond to each image to be processed. The first feature vector includes the speed information and position information of the target in each corresponding image to be processed.

[0146] Optionally, the speed information and position information can be obtained by performing target detection on continuous images to be processed. If the first feature vector obtained by detection and representing the speed information and position information of the target in the corresponding image to be processed has a different dimension from the intermediate hidden layer feature vector, then in order to facilitate feature fusion, its dimension can be adjusted in advance to obtain a first feature vector with the same dimension as the intermediate hidden layer feature.

[0147] In one embodiment, the second feature vector may be obtained by concatenating the intermediate hidden layer features with the first feature vector.

[0148] In another embodiment, the second feature vector may be obtained by adding the intermediate hidden layer feature to the first feature vector. Here, adding means adding the intermediate hidden layer feature to the element at the corresponding position in the first feature vector.

[0149] Step 703: input the second feature vector into the second layer LSTM to obtain the target hidden layer feature.

[0150] Specifically, each second eigenvector can be used as the input of the corresponding time step in the second layer LSTM. ai And the first eigenvector is obtained, so in the second layer LSTM, the target hidden layer feature h at each time step pi While combining the first eigenvector of the previous time step and the first eigenvector of the current time step to perform feature calculation, the intermediate hidden layer feature h in the corresponding second eigenvector is also calculated. ai Guide the target hidden layer feature h at its time step pi Output.

[0151] Since each time step in the second layer LSTM receives the intermediate hidden layer feature h corresponding to the time step in the first layer LSTM, pi , and the second eigenvector composed of the corresponding first eigenvector. Therefore, the target hidden layer feature h pi Where i indicates the time step corresponding to the target hidden layer feature, i = 1, 2, ..., t.

[0152] And, the target hidden layer feature h pi i and the intermediate hidden layer feature h ai The values ​​of i are equal.

[0153] Furthermore, the above prediction model can be trained on a convolutional neural network by using the acquired training data set and the first loss function.

[0154] Different from the above prediction model obtained by training, a fully connected layer and an activation function are set in the training model to process the second fused feature map obtained in the training process, and then a model label is output. The model label indicates the posture information of the target contained in the first fused feature map. For example, the model label can be walking, running, jumping, bending, raising hands, waving hands, etc. Then the training method of the prediction model at least includes:

[0155] First, a training image set is obtained. The training image set includes multiple training images and preset labels corresponding to the targets in the training images. The training images in the training image set are sorted according to the time information of the training images. The time information can be the acquisition time of the training image or the time when the training image is acquired. The preset label indicates the posture information of the target.

[0156] Then, the training image set is input into the model to be trained, and a second fused feature map is generated through the local convolutional neural network to be trained and the global convolutional neural network to be trained in the model to be trained. The local convolutional neural network to be trained includes a first local convolutional neural network to be trained and a second local convolutional neural network to be trained, and the global convolutional neural network to be trained includes a first global convolutional neural network to be trained and a second global convolutional neural network to be trained.

[0157] Based on the second fused feature map, a model label corresponding to the target is determined. The model label indicates the posture information of the target. Specifically, the second fused feature map is input into a fully connected layer, and the output of the fully connected layer is processed by an activation function Softmax to obtain the model label.

[0158] Based on the difference between the model label and the preset label, the parameters of the local convolutional neural network to be trained and the global convolutional neural network to be trained in the model to be trained are adjusted in reverse until the accuracy of the model label meets the preset label, and the global convolutional neural network (i.e., the first global convolutional neural network and the second global convolutional neural network) and the local convolutional neural network (i.e., the first local convolutional neural network and the second local convolutional neural network) are determined to obtain, and then the prediction model is obtained.

[0159] The difference between the above model label and the preset label can be determined by the first loss function, and whether the training is completed can also be determined by whether the first loss function converges. That is, based on the function value of the first loss function, the parameters of the local convolutional neural network to be trained and the global convolutional neural network to be trained in the training model are reversely adjusted until the first loss function converges or the function value of the first loss function is less than the first threshold, and the global convolutional neural network and the local convolutional neural network are determined to be obtained.

[0160] When the prediction model also includes LSTM, the convolutional neural network in the prediction model can be trained according to the above method, and the first fusion feature map output by the trained convolutional neural network can be used to train the LSTM to be trained in combination with the second loss function. Specifically, the weights in the LSTM can be updated by the second loss function until the second loss function converges or the function value of the second loss function is less than the second threshold, and the LSTM is obtained; then the prediction model is obtained.

[0161] It should be understood that although Figure 2-Figure 3 , Figure 7 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 2-Figure 3 , Figure 7 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0162] In one embodiment, Figure 8 As shown, a trajectory prediction device is provided, including: an input module 801, a difference module 802, a fusion module 803 and a trajectory module 804, wherein:

[0163] The input module 801 is used to input the received image to be processed into the prediction model; wherein the prediction model includes a first local convolutional neural network, a second local convolutional neural network, a first global convolutional neural network and a second global convolutional neural network.

[0164] The difference module 802 is used to perform difference processing on the first local feature map and the first global feature map in the prediction model to obtain a difference feature map; wherein the first local feature map and the first global feature map have the same size, the first local feature map is obtained by the first local convolutional neural network, and the first global feature map is obtained by the first global convolutional neural network.

[0165] The fusion module 803 determines a second local feature map and a second global feature map based on the difference feature map, and fuses the second local feature map and the second global feature map to obtain a first fused feature map; wherein the second local feature map is obtained by the second local convolutional neural network, and the second global feature map is obtained by the second global convolutional neural network.

[0166] The trajectory module 804 is used to predict the trajectory information of the target in the image to be processed based on the first fused feature map.

[0167] The difference module 802 is specifically used to determine an initial feature map of a preset size based on the image to be processed; input the initial feature map into the first local convolutional neural network to obtain a first local feature map, and input the initial feature map into the first global convolutional neural network to obtain a first global feature map; wherein the first local feature map and the first global map are both of the preset size; and subtract the first local feature map from the first global feature map to obtain the difference feature map.

[0168] The difference module 802 is specifically used to perform local convolution processing on the initial feature map in the first local convolutional neural network, and fill it with zeros to obtain a first intermediate feature map of the preset size; determine a first attention vector corresponding to the first intermediate feature map; perform a product operation on the first attention vector and the first intermediate feature map to obtain a second intermediate feature map; and add the second intermediate feature map to the initial feature map to obtain the first local feature map.

[0169] The difference module 802 is also used to perform global convolution processing on the initial feature map along the width direction in the first global convolutional neural network to obtain a third intermediate feature map, and perform global convolution processing on the initial feature map along the height direction to obtain a fourth intermediate feature map; fuse the third intermediate feature map and the fourth intermediate feature map to obtain a fifth intermediate feature map of the preset size; determine a second attention vector corresponding to the fifth intermediate feature map; perform a product operation on the second attention vector and the fifth intermediate feature map to obtain a sixth intermediate feature map; add the sixth intermediate feature map to the initial feature map to obtain the first global feature map.

[0170] The fusion module 803 is specifically used to add the second local feature map to the first local feature map to obtain a third local feature map, and to add the second global feature map to the first global feature map to obtain a third global feature map; in the channel dimension, splicing the third local feature map and the third global feature map to obtain the first fused feature map.

[0171] The prediction model includes a long short-term memory network LSTM; the trajectory module 804 is specifically used to process multiple first fusion feature maps through the LSTM to obtain target hidden layer features; wherein the first fusion feature maps correspond one-to-one to the image to be processed; and the target hidden layer features are processed through a fully connected layer to obtain the trajectory information.

[0172] The trajectory module 804 is also used to input multiple of the first fused feature maps into the first layer LSTM to obtain multiple intermediate hidden layer features; wherein the intermediate hidden layer features include the posture information of the target, and the LSTM includes the first layer LSTM and the second layer LSTM; fuse the intermediate hidden layer features and the first feature vector to obtain the second feature vector; wherein the first feature vector includes the speed information and / or position information of the target, and the first feature vector corresponds one-to-one to the intermediate hidden layer features; input the second feature vector into the second layer LSTM to obtain the target hidden layer features.

[0173] For the specific definition of the trajectory prediction device, please refer to the definition of the trajectory prediction method above, which will not be repeated here. Each module in the above-mentioned trajectory prediction device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0174] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Fig. 9As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a trajectory prediction method is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0175] Those skilled in the art will understand that the structure shown in FIG. Y is merely a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0176] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:

[0177] Inputting the received image to be processed into a prediction model; wherein the prediction model includes a first local convolutional neural network, a second local convolutional neural network, a first global convolutional neural network, and a second global convolutional neural network;

[0178] In the prediction model, a difference process is performed on the first local feature map and the first global feature map to obtain a difference feature map; wherein the first local feature map and the first global feature map have the same size, the first local feature map is obtained by the first local convolutional neural network, and the first global feature map is obtained by the first global convolutional neural network;

[0179] Based on the difference feature map, determining a second local feature map and a second global feature map, and fusing the second local feature map and the second global feature map to obtain a first fused feature map; wherein the second local feature map is obtained by the second local convolutional neural network, and the second global feature map is obtained by the second global convolutional neural network;

[0180] Based on the first fused feature map, trajectory information of the target in the image to be processed is predicted.

[0181] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0182] Inputting the received image to be processed into a prediction model; wherein the prediction model includes a first local convolutional neural network, a second local convolutional neural network, a first global convolutional neural network, and a second global convolutional neural network;

[0183] In the prediction model, a difference process is performed on the first local feature map and the first global feature map to obtain a difference feature map; wherein the first local feature map and the first global feature map have the same size, the first local feature map is obtained by the first local convolutional neural network, and the first global feature map is obtained by the first global convolutional neural network;

[0184] Based on the difference feature map, determining a second local feature map and a second global feature map, and fusing the second local feature map and the second global feature map to obtain a first fused feature map; wherein the second local feature map is obtained by the second local convolutional neural network, and the second global feature map is obtained by the second global convolutional neural network;

[0185] Based on the first fused feature map, trajectory information of the target in the image to be processed is predicted.

[0186] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0187] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0188] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A trajectory prediction method, characterized in that: include: Inputting the received image to be processed into a prediction model; wherein the prediction model includes a first local convolutional neural network, a second local convolutional neural network, a first global convolutional neural network, and a second global convolutional neural network; In the prediction model, a difference process is performed on the first local feature map and the first global feature map to obtain a difference feature map; wherein the first local feature map and the first global feature map have the same size, the first local feature map is obtained by the first local convolutional neural network, and the first global feature map is obtained by the first global convolutional neural network; Based on the difference feature map, determining a second local feature map and a second global feature map, and fusing the second local feature map and the second global feature map to obtain a first fused feature map; wherein the second local feature map is obtained by the second local convolutional neural network, and the second global feature map is obtained by the second global convolutional neural network; Based on the first fused feature map, trajectory information of the target in the image to be processed is predicted.

2. The method according to claim 1, characterized in that The step of performing difference processing on the first local feature map and the first global feature map to obtain a difference feature map includes: Based on the image to be processed, determining an initial feature map of a preset size; Inputting the initial feature map into the first local convolutional neural network to obtain a first local feature map, and inputting the initial feature map into the first global convolutional neural network to obtain a first global feature map; wherein the first local feature map and the first global map are both of the preset size; Subtract the first local feature map from the first global feature map to obtain the difference feature map.

3. The method according to claim 2, characterized in that The step of inputting the initial feature map into the first local convolutional neural network to obtain a first local feature map comprises: In the first local convolutional neural network, the initial feature map is locally convolved and zero-filled to obtain a first intermediate feature map of the preset size; Determining a first attention vector corresponding to the first intermediate feature map; Performing a product operation on the first attention vector and the first intermediate feature map to obtain a second intermediate feature map; The second intermediate feature map is added to the initial feature map to obtain the first local feature map.

4. The method according to claim 2, characterized in that The step of inputting the initial feature map into the first global convolutional neural network to obtain a first global feature map comprises: In the first global convolutional neural network, performing global convolution processing on the initial feature map along a width direction to obtain a third intermediate feature map, and performing global convolution processing on the initial feature map along a height direction to obtain a fourth intermediate feature map; Fusing the third intermediate feature map and the fourth intermediate feature map to obtain a fifth intermediate feature map of the preset size; Determining a second attention vector corresponding to the fifth intermediate feature map; Performing a product operation on the second attention vector and the fifth intermediate feature map to obtain a sixth intermediate feature map; The sixth intermediate feature map is added to the initial feature map to obtain the first global feature map.

5. The method according to any one of claims 1 to 4, characterized in that: The fusing the second local feature map and the second global feature map to obtain a first fused feature map includes: Adding the second local feature map to the first local feature map to obtain a third local feature map, and adding the second global feature map to the first global feature map to obtain a third global feature map; In the channel dimension, the third local feature map and the third global feature map are concatenated to obtain the first fused feature map.

6. The method according to claim 1, characterized in that The prediction model includes a long short-term memory network LSTM; Then, predicting the trajectory information of the target in the image to be processed based on the first fused feature map includes: Processing the plurality of the first fused feature maps through the LSTM to obtain target hidden layer features; wherein the first fused feature maps correspond one to one to the images to be processed; The target hidden layer features are processed through a fully connected layer to obtain the trajectory information.

7. The method according to claim 6, characterized in that The step of processing the plurality of the first fused feature maps by the LSTM to obtain target hidden layer features includes: Inputting the plurality of the first fusion feature maps into the first layer LSTM to obtain a plurality of intermediate hidden layer features; wherein the intermediate hidden layer features include the posture information of the target, and the LSTM includes the first layer LSTM and the second layer LSTM; Fusion of the intermediate hidden layer features and the first feature vector to obtain a second feature vector; wherein the first feature vector includes speed information and / or position information of the target, and the first feature vector corresponds to the intermediate hidden layer features one by one; The second feature vector is input into the second layer LSTM to obtain the target hidden layer feature.

8. A trajectory prediction device, characterized in that: include: An input module, used to input the received image to be processed into a prediction model; wherein the prediction model includes a first local convolutional neural network, a second local convolutional neural network, a first global convolutional neural network and a second global convolutional neural network; A difference module, used for performing difference processing on the first local feature map and the first global feature map in the prediction model to obtain a difference feature map; wherein the first local feature map and the first global feature map have the same size, the first local feature map is obtained by the first local convolutional neural network, and the first global feature map is obtained by the first global convolutional neural network; A fusion module, based on the difference feature map, determines a second local feature map and a second global feature map, and fuses the second local feature map and the second global feature map to obtain a first fused feature map; wherein the second local feature map is obtained by the second local convolutional neural network, and the second global feature map is obtained by the second global convolutional neural network; A trajectory module is used to predict the trajectory information of the target in the image to be processed based on the first fused feature map.

9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images