A motion trajectory prediction method and device

Abstract

The present disclosure provides a motion trajectory prediction method and device. In the process of determining the motion trajectory prediction result of a pedestrian, not only the pedestrian feature corresponding to the image sequence to be processed is used, but also the pedestrian feature, motion feature and trajectory pedestrian frame corresponding to the image sequence to be processed are combined. In this way, in the process of trajectory motion prediction, not only the pedestrian feature is used for prediction, but also the motion feature and the position of the trajectory pedestrian frame are used for motion prediction, so that the prediction result of the position of the pedestrian in the next frame of image corresponding to the image sequence to be processed is more accurate, thereby the accuracy of the motion trajectory prediction result of the pedestrian can be improved.

Description

Technical Field

[0001] This disclosure relates to the field of data processing technology, and in particular to a method and apparatus for predicting motion trajectories. Background Technology

[0002] Motion trajectory prediction systems are an emerging biometric technology, representing a cutting-edge technology in the international scientific and technological field with broad development prospects. Current motion prediction techniques typically use Kalman filtering for trajectory prediction, but this method does not utilize image features. Therefore, due to missing image information, the predicted trajectory results are not accurate enough. Summary of the Invention

[0003] In view of this, the present disclosure provides a motion trajectory prediction method, apparatus, computer device, and computer-readable storage medium to solve the problem of inaccurate motion trajectory prediction results when facial pose changes rapidly and drastically in the prior art.

[0004] A first aspect of this disclosure provides a motion trajectory prediction method, the method comprising:

[0005] Obtain the image sequence to be processed;

[0006] Based on the image sequence to be processed, determine the pedestrian features, motion features, and trajectory pedestrian bounding boxes corresponding to the image sequence to be processed;

[0007] Based on the pedestrian features, motion features, and trajectory pedestrian bounding boxes corresponding to the image sequence to be processed, generate target interaction features;

[0008] Based on the target interaction features, the predicted pedestrian movement trajectory corresponding to the image sequence to be processed is determined.

[0009] A second aspect of this disclosure provides a motion trajectory prediction device, the device comprising:

[0010] The sequence acquisition unit is used to acquire the sequence of images to be processed.

[0011] The information determination unit is used to determine the pedestrian features, motion features, and trajectory pedestrian bounding boxes corresponding to the image sequence to be processed based on the image sequence to be processed.

[0012] The feature generation unit is used to generate target interaction features based on the pedestrian features, motion features, and trajectory pedestrian bounding boxes corresponding to the image sequence to be processed;

[0013] The result determination unit is used to determine the predicted pedestrian motion trajectory of the image sequence to be processed based on the target interaction features.

[0014] A third aspect of this disclosure provides a computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method described above.

[0015] A fourth aspect of this disclosure provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described method.

[0016] The beneficial effects of this embodiment compared to the prior art are as follows: This embodiment can first obtain a sequence of images to be processed; then, based on the sequence of images to be processed, determine the pedestrian features, motion features, and trajectory pedestrian bounding boxes corresponding to the sequence of images to be processed; next, based on the pedestrian features, motion features, and trajectory pedestrian bounding boxes corresponding to the sequence of images to be processed, generate target interaction features; finally, based on the target interaction features, determine the predicted pedestrian motion trajectory of the sequence of images to be processed. In this embodiment, since the process of determining the predicted pedestrian motion trajectory not only utilizes the pedestrian features corresponding to the sequence of images to be processed, but also combines the pedestrian features, motion features, and trajectory pedestrian bounding boxes corresponding to the sequence of images to be processed, the motion prediction process can be based not only on pedestrian features, but also on motion features and the position of the trajectory pedestrian bounding boxes. This makes the prediction result of the pedestrian position in the next frame of the sequence of images to be processed more accurate, thereby improving the accuracy of the predicted pedestrian motion trajectory. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this disclosure, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a schematic diagram illustrating an application scenario of an embodiment of this disclosure;

[0019] Figure 2 This is a flowchart of the motion trajectory prediction method provided in the embodiments of this disclosure;

[0020] Figure 3 This is a schematic diagram of the network architecture of the motion feature extraction network provided in the embodiments of this disclosure;

[0021] Figure 4 This is a flowchart of the calculation method of the interactive calculation module provided in the embodiments of this disclosure;

[0022] Figure 5 This is a block diagram of the motion trajectory prediction device provided in the embodiments of this disclosure;

[0023] Figure 6 This is a schematic diagram of a computer device provided in an embodiment of this disclosure. Detailed Implementation

[0024] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, so as to provide a thorough understanding of the embodiments of this disclosure. However, those skilled in the art will understand that this disclosure may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this disclosure with unnecessary detail.

[0025] A motion trajectory prediction method and apparatus according to embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings.

[0026] In existing technologies, motion prediction techniques typically use Kalman filtering to predict motion trajectories. However, this method does not utilize image features, which can lead to inaccurate motion trajectory predictions due to missing image information.

[0027] To address the aforementioned problems, this invention provides a motion trajectory prediction method. In this method, a sequence of images to be processed is first acquired. Then, based on the sequence, pedestrian features, motion features, and trajectory pedestrian bounding boxes corresponding to the sequence are determined. Next, target interaction features are generated based on these features. Finally, the predicted pedestrian motion trajectory for the sequence is determined based on these target interaction features. In this embodiment, because the process of determining the predicted pedestrian motion trajectory utilizes not only the pedestrian features corresponding to the sequence but also the pedestrian features, motion features, and trajectory pedestrian bounding boxes, the prediction is based not only on pedestrian features but also on the motion features and the position of the trajectory pedestrian bounding boxes. This makes the prediction of the pedestrian position in the next frame of the sequence more accurate, thereby improving the accuracy of the predicted pedestrian motion trajectory.

[0028] For example, embodiments of the present invention can be applied to, for example... Figure 1 The application scenario shown can include terminal device 1 and server 2.

[0029] Terminal device 1 can be hardware or software. When terminal device 1 is hardware, it can be various electronic devices with image acquisition capabilities and supporting communication with server 2, including but not limited to smartphones, tablets, laptops, and desktop computers; when terminal device 1 is software, it can be installed in the aforementioned electronic devices. Terminal device 1 can be implemented as multiple software programs or software modules, or as a single software program or software module; this embodiment of the disclosure does not impose any limitations on this. Server 2 can be a server that provides various services, such as a backend server that receives requests sent by terminal devices with which it has established communication connections. This backend server can receive and analyze the requests sent by the terminal devices and generate processing results. Server 2 can be a single server, a server cluster consisting of several servers, or a cloud computing service center; this embodiment of the disclosure does not impose any limitations on this.

[0030] It should be noted that server 2 can be either hardware or software. When server 2 is hardware, it can be various electronic devices that provide various services to terminal device 1. When server 2 is software, it can be multiple software programs or software modules that provide various services to terminal device 1, or it can be a single software program or software module that provides various services to terminal device 1. This disclosure does not impose any limitations on this aspect.

[0031] Terminal device 1 and server 2 can communicate via a network. The network can be a wired network using coaxial cable, twisted pair, or fiber optic connection, or a wireless network that enables interconnection of various communication devices without wiring, such as Bluetooth, Near Field Communication (NFC), or Infrared. This disclosure does not limit the scope of the embodiments.

[0032] Specifically, the user can input a sequence of images to be processed through terminal device 1, which then sends the sequence to server 2. Server 2 first determines the pedestrian features, motion features, and trajectory pedestrian bounding boxes corresponding to the sequence. Then, based on these features, server 2 generates target interaction features. Next, based on these target interaction features, server 2 determines the predicted pedestrian motion trajectory for the sequence. Finally, server 2 returns the predicted trajectory to terminal device 1, allowing terminal device 1 to display the predicted trajectory to the user. This approach allows motion prediction to be based not only on pedestrian features but also on motion features and the position of the trajectory pedestrian bounding boxes, resulting in more accurate predictions of the pedestrian's position in the next frame and thus improving the accuracy of the predicted trajectory.

[0033] It should be noted that the specific types, quantities, and combinations of terminal device 1, server 2, and network can be adjusted according to the actual needs of the application scenario, and this disclosure embodiment does not impose any restrictions on this.

[0034] It should be noted that the above application scenarios are shown only for the purpose of understanding this disclosure, and the implementation of this disclosure is not limited in any way. On the contrary, the implementation of this disclosure can be applied to any applicable scenario.

[0035] Figure 2 This is a flowchart of a motion trajectory prediction method provided in an embodiment of this disclosure. Figure 2 A motion trajectory prediction method can be derived from Figure 1 The terminal device or server executes the command. For example... Figure 2 As shown, the motion trajectory prediction method includes:

[0036] S201: Obtain the image sequence to be processed.

[0037] In this embodiment, the image sequence to be processed can be understood as several images for which motion trajectory prediction needs to be performed. These images are multiple images ordered in a certain sequence, such as multiple consecutive video frames from a video (e.g., eight consecutive video frames) or multiple non-consecutive video frames (e.g., obtained by extracting multiple keyframes from a video). As an example, the image sequence to be processed can be captured by a surveillance camera installed in a fixed location, captured by a mobile terminal device, or read from a storage device that pre-stores images.

[0038] S202: Based on the image sequence to be processed, determine the pedestrian features, motion features, and trajectory pedestrian bounding boxes corresponding to the image sequence to be processed.

[0039] After obtaining the image sequence to be processed, the pedestrian features, motion features, and trajectory pedestrian bounding boxes corresponding to each image in the sequence can be extracted first. Then, according to the sorting sequence of the images, the pedestrian features, motion features, and trajectory pedestrian bounding boxes corresponding to each image are sorted to obtain the pedestrian features, motion features, and trajectory pedestrian bounding boxes corresponding to the image sequence to be processed.

[0040] It should be noted that pedestrian features can be understood as discriminative features that reflect the identity of pedestrians in an image. These features may include, for example, the pedestrian's shape, posture, face, eyes, nose, mouth, and clothing. Motion features can be understood as features that reflect the pedestrian's motion state. These features may include attributes such as the pedestrian's speed, posture, and angle. A trajectory pedestrian bounding box can be understood as information in the image that reflects the pedestrian's position. For example, the rectangular box used to mark the pedestrian's position in the image can be considered a trajectory pedestrian bounding box. In one implementation, the trajectory pedestrian bounding box may include: the x-coordinate and y-coordinate of the center point of the bounding box, and the height h and width w of the bounding box.

[0041] S203: Generate target interaction features based on the pedestrian features, motion features, and trajectory pedestrian bounding boxes corresponding to the image sequence to be processed.

[0042] After obtaining the pedestrian features, motion features, and trajectory pedestrian bounding boxes corresponding to the image sequence to be processed, the information such as pedestrian features, motion features, and trajectory pedestrian bounding boxes can be fused, that is, the pedestrian's identity discrimination features, motion features, and location features are fused to obtain the target interaction features.

[0043] Understandably, target interaction features can be features that reflect the movement and positional changes of pedestrians in the image sequence to be processed. It should be noted that if the image sequence to be processed includes multiple pedestrians, then the target interaction features corresponding to the image sequence can include the target interaction features of multiple pedestrians.

[0044] S204: Based on the target interaction features, determine the predicted pedestrian movement trajectory of the image sequence to be processed.

[0045] Since target interaction features can reflect the trends of pedestrian movement and position changes in the image sequence to be processed, after determining the target interaction features, pedestrian trajectory prediction can be performed using these features. This allows us to predict the pedestrian's position in the next frame by utilizing the trends of pedestrian movement and position changes in the image sequence, as well as the pedestrian features in each image, thus obtaining the predicted pedestrian trajectory for the image sequence to be processed.

[0046] As can be seen, the embodiments of this disclosure can first obtain a sequence of images to be processed; then, based on the sequence of images to be processed, determine the pedestrian features, motion features, and trajectory pedestrian bounding boxes corresponding to the sequence of images to be processed; next, generate target interaction features based on the pedestrian features, motion features, and trajectory pedestrian bounding boxes corresponding to the sequence of images to be processed; finally, determine the pedestrian motion trajectory prediction result corresponding to the sequence of images to be processed based on the target interaction features. In this embodiment, since the process of determining the pedestrian motion trajectory prediction result utilizes not only the pedestrian features corresponding to the sequence of images to be processed, but also the pedestrian features, motion features, and trajectory pedestrian bounding boxes corresponding to the sequence of images to be processed, the trajectory motion prediction process can be based not only on pedestrian features, but also on motion features and the position of the trajectory pedestrian bounding boxes, making the prediction result of the pedestrian position in the next frame of the sequence of images to be processed more accurate, thereby improving the accuracy of the pedestrian motion trajectory prediction result.

[0047] Next, we will introduce one implementation of S202, namely, how to determine the pedestrian features, motion features, and trajectory pedestrian bounding boxes corresponding to the image sequence to be processed. In this embodiment, S202 may include the following steps:

[0048] S202a: Input the image sequence to be processed into the pedestrian re-identification network to obtain the pedestrian features and motion features corresponding to the image sequence.

[0049] The pedestrian re-identification network may include a first backbone network, a second backbone network, a third backbone network, a fourth backbone network, a pedestrian feature extraction network, and a motion feature extraction network. The first backbone network, the second backbone network, the third backbone network, and the fourth backbone network are connected in sequence. The pedestrian feature extraction network is connected to the fourth backbone network, and the motion feature extraction network is connected to the third backbone network and the fourth backbone network.

[0050] It should be noted that the backbone of the person re-identification network can be divided into four stages: the first backbone network, the second backbone network, the third backbone network, and the fourth backbone network. These four backbone networks can be sequential, for example, stage 1 -> stage 2 -> stage 3 -> stage 4. The backbone of the person re-identification network can be a classic form of mainstream residual networks, meaning each stage consists of several "bottleneck modules," reusing classic networks (ResNet). It should be noted that the backbone of the person re-identification network can use any four-stage network, such as ResNet, Resnest, SENet, DenseNet, MobileNet, RegNet, etc.

[0051] The pedestrian feature extraction network can be connected to the fourth backbone network, meaning the features output by the fourth backbone network can be used as input to the pedestrian feature extraction network. In one implementation, the pedestrian feature extraction network can include pooling layers and fully connected layers, where the pooling layers are connected to the fourth backbone network and the fully connected layers are connected, with the output of the fully connected layers being the pedestrian features.

[0052] In one implementation, the motion feature extraction network includes a first convolutional network, a second convolutional network, a third convolutional network, a logistic regression function layer, an average pooling layer, a first matrix network, and a second matrix network. For example... Figure 3 As shown, the input to the first convolutional network is the fourth-stage feature map output by the fourth backbone network, and the output is the first convolutional feature map; wherein, the first convolutional network includes two 3x3 convolutional layers, a batch normalization layer, and a ReLU activation function layer. The input to the second convolutional network is the third-stage feature map output by the third backbone network, and the output is the second convolutional feature map; wherein, the second convolutional network includes one 3x3 convolutional layer with a downsampling of 2. The input to the third convolutional network is the stacked feature map obtained by stacking the first convolutional feature map and the second convolutional feature map, and the output is a shallow and deep semantic feature map; wherein, the third convolutional network includes two 3x3 convolutional layers, a batch normalization layer, and a ReLU activation function layer. The input to the convolutional regression function layer is the shallow and deep semantic feature map, and the output is the third convolutional feature map; wherein, the convolutional regression function layer may include a 1x1 convolutional layer with 1 channel and a softmax function layer. The input to the average pooling layer is the shallow and deep semantic feature maps and the third convolutional feature map, and the output is the spatial motion perception feature map.

[0053] The first matrix network takes the spatial motion-aware feature map as input and outputs the motion features corresponding to the image sequence to be processed. The second matrix network takes the spatial motion-aware feature map as input and outputs the predicted spatial displacement difference. The first matrix network is a (512, 256) matrix, and the second matrix network is a (256, 4) matrix. It should be noted that the predicted spatial displacement difference includes the x-coordinate, y-coordinate, height (h), and width (w) of the detection box center point. That is, the spatial displacement difference is the position of the pedestrian in the current frame (x, y, h, w) and in the next frame (x1, y1, h1, w1), which is (x1-x, y1-y, h1-h, w1-w).

[0054] It should be noted that the second matrix network is only used during the training phase, not during the usage phase, because predicting spatial displacement differences is used to assist network training, making the extraction of motion features and pedestrian features more accurate. It should also be noted that the pedestrian re-identification network is trained under supervised conditions using smooth_l1 loss, and the training data consists of a dataset and annotations. Specifically, the dataset consists of pedestrian images, and the annotations are the user IDs associated with each image.

[0055] S202b: Input the pedestrian features corresponding to the image sequence to be processed into the detector to obtain the trajectory pedestrian bounding boxes corresponding to the image sequence to be processed.

[0056] In this embodiment, the pedestrian features corresponding to the image sequence to be processed can be input into a trained detector to obtain the trajectory pedestrian bounding boxes corresponding to the image sequence to be processed. It should be noted that the pedestrian features of each image can be input into the detector separately to obtain the trajectory pedestrian bounding boxes corresponding to the pedestrian features of each image.

[0057] It should be noted that in one implementation, the dimensions of the motion features can be (8, 512), the dimensions of the pedestrian features are (8, 512), and the dimensions of the trajectory pedestrian bounding box are (8, 4).

[0058] Next, we will introduce one implementation of S203, namely, how to generate target interaction features based on the pedestrian features, motion features, and trajectory pedestrian bounding boxes corresponding to the image sequence to be processed. In this embodiment, S203 may include the following steps:

[0059] The pedestrian features, motion features, and trajectory pedestrian bounding boxes corresponding to the image sequence to be processed are input into the interaction network to obtain the target interaction features;

[0060] The interactive network comprises N serially connected interactive computation modules, where N is a positive integer. Each interactive computation module can be a PFS module (i.e., an interactive computation module for pedestrian fusion features and space), meaning the interactive network can contain multiple PFS modules. It should be noted that the purpose of multiple interactive computation modules is to deepen the network structure, thereby enabling better computation of motion features.

[0061] Specifically, for the i-th interactive calculation module, the pedestrian features, motion features, and trajectory pedestrian bounding boxes corresponding to the image sequence to be processed are input into the i-th interactive calculation module to obtain the i-th interactive feature; where i is a positive integer less than or equal to N. If i is 1, the pedestrian feature is the pedestrian feature corresponding to the image sequence to be processed; if i is greater than 1 and less than or equal to N, the pedestrian feature is the (i-1)-th interactive feature. It can be understood that, finally, the N-th interactive feature is used as the target interactive feature.

[0062] For example, suppose the interaction network can include four PFS modules. The input of the first PFS module is the pedestrian features, motion features, and trajectory pedestrian bounding boxes corresponding to the image sequence to be processed, and the output is the first interaction feature f′2. The pedestrian features input to the second PFS module change to f′2, but the trajectory pedestrian motion features and trajectory pedestrian bounding boxes remain unchanged, which are the motion features and trajectory pedestrian bounding boxes corresponding to the image sequence to be processed. After the second PFS module, the second interaction feature f″2 between the pedestrian fusion features and space is obtained. The third PFS module, with the pedestrian features input to f″2, retains the motion features and trajectory pedestrian bounding boxes, resulting in the third interaction feature f″′2 between the pedestrian fusion features and space. The fourth PFS module, again with the pedestrian features input to f″′2, maintains the same motion features and trajectory pedestrian bounding boxes, yielding the fourth interaction feature f″″2 between the pedestrian fusion features and space. Finally, the fourth interaction feature f″″2 can be used as the target interaction feature.

[0063] Next, the processing method of each interactive computing module will be introduced. In this embodiment, the i-th interactive computing module includes: a first parameter matrix, a second parameter matrix, a logistic regression function layer, and a matrix network layer; that is, each interactive computing module includes: a first parameter matrix, a second parameter matrix, a logistic regression function layer, and a matrix network layer.

[0064] Specifically, such as Figure 4As shown, the motion feature (e.g., a pedestrian motion feature f1 of size 8×512) can be input into the first parameter matrix W1 to obtain the first trajectory embedding feature e1; specifically, the motion feature f1 can be processed by the first parameter matrix W1 (whose matrix size is 512×512) to obtain the first trajectory embedding feature e1, whose dimension is (8, 512). Then, the pedestrian feature (i.e., the pedestrian trajectory feature f2) can be input into the second parameter matrix (i.e., the parameter matrix W2) to obtain the second trajectory embedding feature (i.e., the trajectory embedding feature e2); specifically, the pedestrian trajectory feature f2 can be processed by the parameter matrix W2 (whose matrix size is 512×512) to obtain the second parameter matrix e2, whose dimension is (8, 512).

[0065] Then, the first trajectory embedding feature and the second trajectory embedding feature can be input into the logistic regression function layer to obtain the feature motion correlation matrix. As an example, the logistic regression function layer can divide both the first parameter matrix e1 and the second parameter matrix e2 into 4 groups along the channel dimension, each group having a dimension of (8, 128). Performing matrix multiplication (i.e., pairwise matrix multiplication) on each group yields 4 correlation matrices with a dimension of (8, 8). The logistic regression function layer can then add the 4 correlation matrices and perform softmax calculation along the second dimension to obtain the feature motion correlation matrix r1, with a dimension of (8, 8).

[0066] Next, the feature motion correlation matrix r1 and the preset learnable matrix r2 are added together to obtain the two-factor fused pedestrian feature r3. Specifically, the pedestrian feature f2 is multiplied by the feature motion correlation matrix r1 and the learnable matrix r2 respectively, and the two products are added together to obtain the two-factor fused pedestrian feature r3, which has a dimension of (8, 512). The specific formula is: r3 = r1*f2 + r2*f2.

[0067] Finally, the two-factor fused pedestrian feature and the trajectory pedestrian bounding box are input into the matrix network layer to obtain the i-th interaction feature. This matrix network layer includes matrix W3 (512×512) and matrix W4 (512×512). Specifically, the two-factor fused pedestrian feature r3 and the trajectory pedestrian bounding box f3 can be concatted. After concatting, the feature is then processed by matrices W3 and W4 to obtain the fused pedestrian feature and the first spatial interaction feature, resulting in the i-th interaction feature. The specific formula is as follows: fi = concat(r3,f3)*W3*W4, where fi is the i-th interaction feature. Thus, the pedestrian feature, motion feature, and trajectory pedestrian bounding box have undergone the PFS module, completing one interaction calculation.

[0068] Next, we will introduce one implementation of S204, namely, how to determine the predicted pedestrian motion trajectory of the image sequence to be processed based on the target interaction features. In this embodiment, S203 may include the following steps:

[0069] The target interaction features are input into the motion trajectory prediction model to obtain the pedestrian motion trajectory prediction results corresponding to the image sequence to be processed.

[0070] The motion trajectory prediction model includes an average pooling layer, a third parameter matrix, an activation function layer, and a fourth parameter matrix. Specifically, the target interaction features are first average-pooled using an average pooling layer to obtain a feature vector y1 of dimension (1, 512). This feature vector y1 is then sequentially input into a third parameter matrix of size (512, 256), an activation function layer (e.g., ReLU activation), and a fourth parameter matrix of size (512, 4) to obtain the predicted pedestrian motion trajectory for the image sequence. This predicted trajectory can have four values ​​(e.g., the x-coordinate and y-coordinate of the detection box center point, and the height h and width w of the detection box), representing the predicted pedestrian position in the next frame. By using pedestrian re-identification to extract pedestrian and motion features, and feeding multiple frames into the interaction network in batches, the spatial position of the pedestrian in the next frame can be predicted, thereby improving the accuracy of motion prediction and making the trajectory prediction task more precise.

[0071] It should be noted that the interactive network can be trained using smooth_l1 loss during the training process.

[0072] All of the above-mentioned optional technical solutions can be combined in any way to form optional embodiments of this disclosure, and will not be described in detail here.

[0073] The following are embodiments of the apparatus disclosed herein, which can be used to execute embodiments of the method disclosed herein. For details not disclosed in the apparatus embodiments of this disclosure, please refer to the embodiments of the method disclosed herein.

[0074] Figure 5 This is a schematic diagram of the motion trajectory prediction device provided in an embodiment of this disclosure. Figure 5 As shown, the motion trajectory prediction device includes:

[0075] Sequence acquisition unit 501 is used to acquire the image sequence to be processed;

[0076] The information determination unit 502 is used to determine the pedestrian features, motion features, and trajectory pedestrian bounding boxes corresponding to the image sequence to be processed based on the image sequence to be processed.

[0077] The feature generation unit 503 is used to generate target interaction features based on the pedestrian features, motion features and trajectory pedestrian bounding boxes corresponding to the image sequence to be processed;

[0078] The result determination unit 504 is used to determine the pedestrian motion trajectory prediction result corresponding to the image sequence to be processed based on the target interaction features.

[0079] Optionally, the information determining unit 502 is used for:

[0080] The image sequence to be processed is input into a pedestrian re-identification network to obtain pedestrian features and motion features corresponding to the image sequence. The pedestrian re-identification network includes a first backbone network, a second backbone network, a third backbone network, a fourth backbone network, a pedestrian feature extraction network, and a motion feature extraction network. The first backbone network, the second backbone network, the third backbone network, and the fourth backbone network are connected in sequence. The pedestrian feature extraction network is connected to the fourth backbone network, and the motion feature extraction network is connected to the third backbone network and the fourth backbone network.

[0081] The pedestrian features corresponding to the image sequence to be processed are input into the detector to obtain the trajectory pedestrian bounding boxes corresponding to the image sequence to be processed.

[0082] Optionally, the motion feature extraction network includes a first convolutional network, a second convolutional network, a third convolutional network, a logistic regression function layer, an average pooling layer, a first matrix network, and a second matrix network.

[0083] The first convolutional network takes the fourth-stage feature map output by the fourth backbone network as input and outputs the first convolutional feature map. The second convolutional network takes the third-stage feature map output by the third backbone network as input and outputs the second convolutional feature map. The third convolutional network takes a stacked feature map obtained by stacking the first and second convolutional feature maps as input and outputs a shallow and deep semantic feature map. The convolutional regression function layer takes the shallow and deep semantic feature map as input and outputs the third convolutional feature map. The average pooling layer takes the shallow and deep semantic feature map and the third convolutional feature map as input and outputs a spatial motion perception feature map. The first matrix network takes the spatial motion perception feature map as input and outputs the motion features corresponding to the image sequence to be processed. The second matrix network takes the spatial motion perception feature map as input and outputs the predicted spatial displacement difference.

[0084] Optionally, the feature generation unit 503 is used for:

[0085] The pedestrian features, motion features, and trajectory pedestrian bounding boxes corresponding to the image sequence to be processed are input into the interaction network to obtain the target interaction features;

[0086] The interactive network comprises N interactive computing modules connected in series, where N is a positive integer.

[0087] Optionally, the feature generation unit 503 is used for:

[0088] For the i-th interactive calculation module, the pedestrian features, motion features, and trajectory pedestrian boxes corresponding to the image sequence to be processed are input into the i-th interactive calculation module to obtain the i-th interactive feature; where i is a positive integer less than or equal to N. If i is 1, the pedestrian feature is the pedestrian feature corresponding to the image sequence to be processed; if i is greater than 1 and less than or equal to N, the pedestrian feature is the (i-1)-th interactive feature.

[0089] The Nth interaction feature is used as the target interaction feature.

[0090] Optionally, the i-th interactive computation module includes: a first parameter matrix, a second parameter matrix, a logistic regression function layer, and a matrix network layer; the feature generation unit 503 is used for:

[0091] The motion feature is input into the first parameter matrix to obtain the first trajectory embedding feature;

[0092] The pedestrian feature is input into the second parameter matrix to obtain the second trajectory embedding feature;

[0093] The first trajectory embedding feature and the second trajectory embedding feature are input into the logistic regression function layer to obtain the feature motion correlation matrix;

[0094] The feature motion correlation matrix is ​​added to the preset learnable matrix to obtain the two-factor fused pedestrian feature;

[0095] The two-factor fused pedestrian features and the trajectory pedestrian bounding box are input into the matrix network layer to obtain the i-th interactive feature.

[0096] Optionally, the result determination unit 504 is used for

[0097] The target interaction features are input into the motion trajectory prediction model to obtain the pedestrian motion trajectory prediction results corresponding to the image sequence to be processed;

[0098] The motion trajectory prediction model includes: an average pooling layer, a third parameter matrix, an activation function layer, and a fourth parameter matrix.

[0099] The technical solution provided in this disclosure is a motion trajectory prediction device. The device includes: a sequence acquisition unit for acquiring a sequence of images to be processed; an information determination unit for determining pedestrian features, motion features, and trajectory pedestrian bounding boxes corresponding to the sequence of images to be processed; a feature generation unit for generating target interaction features based on the pedestrian features, motion features, and trajectory pedestrian bounding boxes corresponding to the sequence of images to be processed; and a result determination unit for determining the predicted pedestrian motion trajectory result corresponding to the sequence of images to be processed based on the target interaction features. In this embodiment, because the process of determining the predicted pedestrian motion trajectory result utilizes not only the pedestrian features corresponding to the sequence of images to be processed but also the pedestrian features, motion features, and trajectory pedestrian bounding boxes corresponding to the sequence of images to be processed, the motion prediction process can be based not only on pedestrian features but also on motion features and the position of the trajectory pedestrian bounding boxes. This makes the predicted pedestrian position in the next frame of the image sequence more accurate, thereby improving the accuracy of the predicted pedestrian motion trajectory result.

[0100] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this disclosure.

[0101] Figure 6 This is a schematic diagram of the computer device 6 provided in an embodiment of this disclosure. Figure 6 As shown, the computer device 6 in this embodiment includes a processor 601, a memory 602, and a computer program 603 stored in the memory 602 and executable on the processor 601. When the processor 601 executes the computer program 603, it implements the steps in the various method embodiments described above. Alternatively, when the processor 601 executes the computer program 603, it implements the functions of each module / unit in the various device embodiments described above.

[0102] Exemplarily, computer program 603 may be divided into one or more modules / units, which are stored in memory 602 and executed by processor 601 to perform the present disclosure. The one or more modules / units may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of computer program 603 in computer device 6.

[0103] Computer device 6 can be a desktop computer, laptop, handheld computer, cloud server, or other similar computer device. Computer device 6 may include, but is not limited to, processor 601 and memory 602. Those skilled in the art will understand that... Figure 6This is merely an example of computer device 6 and does not constitute a limitation on computer device 6. It may include more or fewer components than shown, or combine certain components, or different components. For example, computer device may also include input / output devices, network access devices, buses, etc.

[0104] The processor 601 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0105] The memory 602 can be an internal storage unit of the computer device 6, such as a hard disk or RAM of the computer device 6. The memory 602 can also be an external storage device of the computer device 6, such as a plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card equipped on the computer device 6. Furthermore, the memory 602 can include both internal and external storage units of the computer device 6. The memory 602 is used to store computer programs and other programs and data required by the computer device. The memory 602 can also be used to temporarily store data that has been output or will be output.

[0106] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this disclosure. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0107] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0108] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this disclosure.

[0109] In the embodiments provided in this disclosure, it should be understood that the disclosed apparatus / computer devices and methods can be implemented in other ways. For example, the apparatus / computer device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. Multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0110] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0111] Furthermore, the functional units in the various embodiments of this disclosure can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0112] If an integrated module / unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program may include computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. A computer-readable medium may include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in a computer-readable medium may be appropriately added to or subtracted according to the requirements of legislation and patent practice in a jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0113] The above embodiments are only used to illustrate the technical solutions of this disclosure, and are not intended to limit it. Although this disclosure has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this disclosure, and should all be included within the protection scope of this disclosure.

Claims

1. A method for predicting motion trajectories, characterized in that, The method includes: Obtain the image sequence to be processed; Based on the image sequence to be processed, determine the pedestrian features, motion features, and trajectory pedestrian bounding boxes corresponding to the image sequence to be processed; the pedestrian features are discriminative features that can reflect the identity of pedestrians in the image, the motion features are features that can reflect the motion state of pedestrians, and the trajectory pedestrian bounding boxes are information in the image that can reflect the location of pedestrians; Based on the pedestrian features, motion features, and trajectory pedestrian bounding boxes corresponding to the image sequence to be processed, target interaction features are generated; the target interaction features are obtained by fusing pedestrian features, motion features, and trajectory pedestrian bounding boxes, and are used to reflect the movement change trend and position change trend of pedestrians in the image sequence to be processed. Based on the target interaction features, determine the pedestrian motion trajectory prediction result corresponding to the image sequence to be processed; The step of determining the pedestrian features, motion features, and trajectory pedestrian bounding boxes corresponding to the image sequence to be processed includes: The image sequence to be processed is input into a pedestrian re-identification network to obtain pedestrian features and motion features corresponding to the image sequence. The pedestrian re-identification network includes a first backbone network, a second backbone network, a third backbone network, a fourth backbone network, a pedestrian feature extraction network, and a motion feature extraction network. The first backbone network, the second backbone network, the third backbone network, and the fourth backbone network are four-stage networks connected in series. The pedestrian feature extraction network is connected to the fourth backbone network, and the motion feature extraction network is connected to the third backbone network and the fourth backbone network. Input the pedestrian features corresponding to the image sequence to be processed into the detector to obtain the trajectory pedestrian bounding box corresponding to the image sequence to be processed. The motion feature extraction network includes a first convolutional network, a second convolutional network, a third convolutional network, a logistic regression function layer, an average pooling layer, a first matrix network, and a second matrix network. The input to the first convolutional network is the fourth-stage feature map output by the fourth backbone network, and the output is the first convolutional feature map; the input to the second convolutional network is the third-stage feature map output by the third backbone network, and the output is the second convolutional feature map; the input to the third convolutional network is a stacked feature map obtained by stacking the first convolutional feature map and the second convolutional feature map, and the output is a shallow and deep semantic feature map; the input to the convolutional regression function layer is the shallow and deep semantic feature map, and the output is the third convolutional feature map; the input to the average pooling layer is the shallow and deep semantic feature map and the third convolutional feature map, and the output is a spatial motion perception feature map; the input to the first matrix network is the spatial motion perception feature map, and the output is the motion feature corresponding to the image sequence to be processed; the input to the second matrix network is the spatial motion perception feature map, and the output is the predicted spatial displacement difference.

2. The method according to claim 1, characterized in that, The step of generating target interaction features based on pedestrian features, motion features, and trajectory pedestrian bounding boxes corresponding to the image sequence to be processed includes: The pedestrian features, motion features, and trajectory pedestrian bounding boxes corresponding to the image sequence to be processed are input into the interactive network to obtain the target interactive features; The interactive network comprises N interactive computing modules connected in series, where N is a positive integer.

3. The method according to claim 2, characterized in that, The step of inputting the pedestrian features, motion features, and trajectory pedestrian bounding boxes corresponding to the image sequence to be processed into the interactive network to obtain the target interactive features includes: For the i-th interactive calculation module, the pedestrian features, motion features, and trajectory pedestrian boxes corresponding to the image sequence to be processed are input into the i-th interactive calculation module to obtain the i-th interactive feature; where i is a positive integer less than or equal to N. If i is 1, the pedestrian feature is the pedestrian feature corresponding to the image sequence to be processed; if i is greater than 1 and less than or equal to N, the pedestrian feature is the (i-1)-th interactive feature. The Nth interaction feature is taken as the target interaction feature.

4. The method according to claim 3, characterized in that, The i-th interactive calculation module includes: a first parameter matrix, a second parameter matrix, a logistic regression function layer, and a matrix network layer; the step of inputting pedestrian features and motion features and trajectory pedestrian bounding boxes corresponding to the image sequence to be processed into the i-th interactive calculation module to obtain the i-th interactive features includes: The motion features are input into the first parameter matrix to obtain the first trajectory embedding features; The pedestrian features are input into the second parameter matrix to obtain the second trajectory embedding features; The first trajectory embedding feature and the second trajectory embedding feature are input into the logistic regression function layer to obtain the feature motion correlation matrix; The feature motion correlation matrix and the preset learnable matrix are added together to obtain the two-factor fused pedestrian features; The two-factor fused pedestrian features and the trajectory pedestrian bounding box are input into the matrix network layer to obtain the i-th interaction feature.

5. The method according to any one of claims 1-4, characterized in that, The step of determining the pedestrian motion trajectory prediction result corresponding to the image sequence to be processed based on the target interaction features includes: The target interaction features are input into the motion trajectory prediction model to obtain the pedestrian motion trajectory prediction result corresponding to the image sequence to be processed; The motion trajectory prediction model includes: an average pooling layer, a third parameter matrix, an activation function layer, and a fourth parameter matrix.

6. A motion trajectory prediction device, characterized in that, The device includes: The sequence acquisition unit is used to acquire the sequence of images to be processed. The information determination unit is used to determine the pedestrian features, motion features, and trajectory pedestrian bounding boxes corresponding to the image sequence to be processed based on the image sequence to be processed; the pedestrian features are discrimination features that can reflect the identity of pedestrians in the image, the motion features are features that can reflect the motion state of pedestrians, and the trajectory pedestrian bounding boxes are information in the image that can reflect the location of pedestrians; The feature generation unit is used to generate target interaction features based on the pedestrian features, motion features, and trajectory pedestrian bounding boxes corresponding to the image sequence to be processed; the target interaction features are obtained by fusing the pedestrian features, motion features, and trajectory pedestrian bounding boxes, and are used to reflect the movement change trend and position change trend of pedestrians in the image sequence to be processed. The result determination unit is used to determine the pedestrian motion trajectory prediction result corresponding to the image sequence to be processed based on the target interaction features. The information determination unit is specifically used for: inputting the image sequence to be processed into a pedestrian re-identification network to obtain pedestrian features and motion features corresponding to the image sequence; wherein, the pedestrian re-identification network includes a first backbone network, a second backbone network, a third backbone network, a fourth backbone network, a pedestrian feature extraction network, and a motion feature extraction network, wherein the first backbone network, the second backbone network, the third backbone network, and the fourth backbone network are serially connected four-stage networks, the pedestrian feature extraction network is connected to the fourth backbone network, and the motion feature extraction network is connected to the third backbone network and the fourth backbone network; inputting the pedestrian features corresponding to the image sequence to be processed into a detector to obtain the trajectory pedestrian bounding box corresponding to the image sequence to be processed; the motion feature extraction network includes a first convolutional network, a second convolutional network, a third convolutional network, a logistic regression function layer, an average pooling layer, and a first moment... The system consists of a first convolutional network and a second matrix network. The first convolutional network takes the fourth-stage feature map output by the fourth backbone network as its input and outputs a first convolutional feature map. The second convolutional network takes the third-stage feature map output by the third backbone network as its input and outputs a second convolutional feature map. The third convolutional network takes a stacked feature map obtained by stacking the first and second convolutional feature maps as its input and outputs a shallow and deep semantic feature map. The convolutional regression function layer takes the shallow and deep semantic feature maps as its input and outputs a third convolutional feature map. The average pooling layer takes the shallow and deep semantic feature maps and the third convolutional feature map as its input and outputs a spatial motion perception feature map. The first matrix network takes the spatial motion perception feature map as its input and outputs the motion features corresponding to the image sequence to be processed. The second matrix network takes the spatial motion perception feature map as its input and outputs the predicted spatial displacement difference.

7. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 5.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 5.

Images