Motion trajectory information generation method and apparatus, electronic device, and readable medium

Abstract

This disclosure provides embodiments of a method, apparatus, electronic device, and readable medium for generating motion trajectory information. One specific implementation of the method includes: acquiring a target image sequence; generating a sequence of adjacent target image groups; acquiring an inertial information sequence; inputting the adjacent target image group sequence into an image feature extraction layer to obtain an image feature tensor sequence; inputting the inertial information sequence into an inertial feature extraction layer to obtain an inertial feature vector sequence; inputting the image feature tensor sequence and the inertial feature vector sequence into an image inertial feature fusion layer to obtain an image inertial fusion feature tensor sequence; inputting the image inertial fusion feature tensor sequence into a pose network layer to obtain a pose information sequence; inputting the target image sequence into a depth network layer to obtain a depth map sequence; and generating motion trajectory information based on the pose information sequence and the depth map sequence. This implementation improves the accuracy of the estimated pose and the correctness of the generated motion trajectory.

Description

Technical Field

[0001] The embodiments disclosed herein relate to the field of computer vision technology, and more specifically to methods, apparatus, electronic devices, and readable media for generating motion trajectory information. Background Technology

[0002] With the rapid development of intelligent mobile robots, self-driving cars, drones, and wearable devices, environmental perception and autonomous localization technologies have become crucial foundations for intelligent navigation. Among these, visual odometry (VO) technology, as an important means of estimating the relative motion of a vehicle and reconstructing its surrounding environment, has been widely researched and applied. Currently, when estimating the pose of generated motion trajectories, the common approach is to acquire image sequences using monocular or binocular cameras, extract feature points, match features, and solve geometric constraints to achieve continuous estimation of the camera pose (position and orientation).

[0003] However, when using the above method, the following technical problems often arise:

[0004] Since pose estimation is obtained through the acquisition of image sequences, when the acquired image sequences are affected by sudden changes in lighting or occlusion, the quality of the acquired images is easily reduced, resulting in lower accuracy of the estimated pose. Consequently, the accuracy of the generated motion trajectory is also lower, leading to more errors in the localization and displacement operations of related intelligent agents (such as robots and drones).

[0005] The information disclosed in this background section is only intended to enhance the understanding of the background of the inventive concept, and therefore may contain information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0006] The summary portion of this disclosure is intended to provide a brief overview of the concepts, which will be described in detail in the detailed description portion. This summary portion is not intended to identify key or essential features of the claimed technical solutions, nor is it intended to limit the scope of the claimed technical solutions.

[0007] Some embodiments of this disclosure provide methods, apparatuses, electronic devices, and computer-readable media for generating motion trajectory information to address the technical problems mentioned in the background section above.

[0008] In a first aspect, some embodiments of this disclosure provide a method for generating motion trajectory information. The method includes: acquiring a target image sequence; generating a sequence of adjacent target image groups based on the target image sequence; acquiring an inertial information sequence, wherein the inertial information in the inertial information sequence corresponds to adjacent target image groups in the adjacent target image group sequence; inputting the adjacent target image group sequence into an image feature extraction layer of a pre-trained pose information generation model to obtain an image feature tensor sequence, wherein the pose information generation model further includes an inertial feature extraction layer, an image inertial feature fusion layer, a pose network layer, and a deep network layer; and inputting the aforementioned inertial information sequence into an image feature extraction layer of a pre-trained pose information generation model to obtain an image feature tensor sequence. The inertial information sequence is input to the aforementioned inertial feature extraction layer to obtain an inertial feature vector sequence, wherein the inertial feature vectors in the aforementioned inertial feature vector sequence correspond to the image feature tensors in the aforementioned image feature tensor sequence; the aforementioned image feature tensor sequence and the aforementioned inertial feature vector sequence are input to the aforementioned image inertial feature fusion layer to obtain an image inertial fusion feature tensor sequence; the aforementioned image inertial fusion feature tensor sequence is input to the aforementioned pose network layer to obtain a pose information sequence; the aforementioned target image sequence is input to the aforementioned depth network layer to obtain a depth map sequence; and motion trajectory information is generated based on the aforementioned pose information sequence and the aforementioned depth map sequence.

[0009] Secondly, some embodiments of this disclosure provide a motion trajectory information generation apparatus, the apparatus comprising: a first acquisition unit configured to acquire a target image sequence; a first generation unit configured to generate a sequence of adjacent target image groups based on the target image sequence; a second acquisition unit configured to acquire an inertial information sequence, wherein the inertial information in the inertial information sequence corresponds to adjacent target image groups in the adjacent target image group sequence; a first input unit configured to input the adjacent target image group sequence into an image feature extraction layer of a pre-trained pose information generation model to obtain an image feature tensor sequence, wherein the pose information generation model further comprises an inertial feature extraction layer, an image inertial feature fusion layer, a pose network layer, and a deep network layer; and a second input unit configured to... The first input unit is configured to input the aforementioned inertial information sequence into the aforementioned inertial feature extraction layer to obtain an inertial feature vector sequence, wherein the inertial feature vectors in the aforementioned inertial feature vector sequence correspond to the image feature tensors in the aforementioned image feature tensor sequence; the second input unit is configured to input the aforementioned image feature tensor sequence and the aforementioned inertial feature vector sequence into the aforementioned image inertial feature fusion layer to obtain an image inertial fusion feature tensor sequence; the third input unit is configured to input the aforementioned image inertial fusion feature tensor sequence into the aforementioned pose network layer to obtain a pose information sequence; the fourth input unit is configured to input the aforementioned image inertial fusion feature tensor sequence into the aforementioned pose network layer to obtain a pose information sequence; the fifth input unit is configured to input the aforementioned target image sequence into the aforementioned depth network layer to obtain a depth map sequence; and the fifth generation unit is configured to generate motion trajectory information based on the aforementioned pose information sequence and the aforementioned depth map sequence.

[0010] Thirdly, some embodiments of this disclosure provide an electronic device, including: one or more processors; and a storage device having one or more programs stored thereon, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation of the first aspect above.

[0011] Fourthly, some embodiments of this disclosure provide a computer-readable medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the method described in any of the implementations of the first aspect above.

[0012] The above embodiments of this disclosure have the following beneficial effects: the motion trajectory information generation method of some embodiments of this disclosure can improve the accuracy of the generated pose and the correctness of the motion trajectory, thereby reducing the number of errors in the positioning and displacement operations of the relevant intelligent agent. Specifically, the reason why the estimated pose accuracy is low, and the generated motion trajectory accuracy is low, leading to more errors in the positioning and displacement operations of the relevant intelligent agent (e.g., robot and drone) is that: since the pose estimation is obtained through the acquisition of image sequences, when the acquired image sequences are affected by sudden changes in illumination or occlusion, the quality of the acquired images is easily reduced, resulting in low accuracy of the estimated pose and low correctness of the generated motion trajectory, thus leading to more errors in the positioning and displacement operations of the relevant intelligent agent (e.g., robot and drone). Based on this, the motion trajectory information generation method of some embodiments of this disclosure first acquires a target image sequence. This yields a target image sequence representing visual information. Then, based on the target image sequence, a sequence of adjacent target image groups is generated. This yields a sequence of adjacent target image groups representing two consecutive adjacent frames. Afterwards, an inertial information sequence is acquired. In this sequence, the inertial information corresponds to the adjacent target image groups in the adjacent target image group sequence. Thus, an inertial information sequence representing the motion state is obtained. Next, the adjacent target image group sequence is input into the image feature extraction layer of a pre-trained pose information generation model to obtain an image feature tensor sequence. The pose information generation model further includes an inertial feature extraction layer, an image inertial feature fusion layer, a pose network layer, and a deep network layer. Thus, an image feature tensor sequence representing the high-level semantic features of the visual data is obtained. Then, the inertial information sequence is input into the inertial feature extraction layer to obtain an inertial feature vector sequence. The inertial feature vectors in the inertial feature vector sequence correspond to the image feature tensors in the image feature tensor sequence. Thus, an inertial feature vector sequence representing the high-level semantic features of the motion state data is obtained. Finally, the image feature tensor sequence and the inertial feature vector sequence are input into the image inertial feature fusion layer to obtain an image inertial fusion feature tensor sequence. Therefore, an image inertial fusion feature tensor sequence can be obtained, enabling the fusion of image features representing visual information and inertial features representing motion state information at the data layer. Subsequently, this image inertial fusion feature tensor sequence is input into the pose network layer to obtain a pose information sequence. This yields pose information sequences representing the transformation state of the subsequent frame relative to the preceding frame in two adjacent frames. Next, the target image sequence is input into the deep network layer to obtain a depth map sequence. This depth map sequence can then be used to reconstruct a realistic physical scene. Finally, motion trajectory information is generated based on the pose information sequence and the depth map sequence.This allows for the acquisition of highly accurate motion trajectory information. Furthermore, by acquiring target image sequences representing visual information, high-accuracy pose information can be obtained when visual effects are good. Additionally, by acquiring inertial information sequences representing motion states, visual features can be compensated for when visual effects are poor, achieving complementarity between the two. Moreover, by tightly coupling visual and inertial features at the feature level, the accuracy and robustness of data fusion can be further improved. The weight ratio of the two types of features can be dynamically adjusted when the environment changes. That is, when visual features are good (e.g., clear scene texture or sufficient lighting), the weight of visual features is automatically increased; when visual features are poor (e.g., visual signals are obstructed or lighting is poor), the influence of inertial features is automatically increased. This improves the accuracy of the generated pose and the correctness of the motion trajectory, thereby reducing the number of errors in the localization and displacement operations performed by the relevant intelligent agent. Attached Figure Description

[0013] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and elements are not necessarily drawn to scale.

[0014] Figure 1 This is a flowchart of some embodiments of the motion trajectory information generation method according to the present disclosure;

[0015] Figure 2 These are schematic diagrams illustrating the structure of some embodiments of the motion trajectory information generation apparatus according to the present disclosure;

[0016] Figure 3 This is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure. Detailed Implementation

[0017] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0018] It should also be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings. Unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other.

[0019] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0020] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0021] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0022] This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0023] Figure 1 A flow 100 of some embodiments of the motion trajectory information generation method according to the present disclosure is shown. The motion trajectory information generation method includes the following steps:

[0024] Step 101: Obtain the target image sequence.

[0025] In some embodiments, the execution entity (e.g., a computing device) of the motion trajectory information generation method can acquire the target image sequence from the image acquisition device via a wired or wireless connection. It should be noted that the aforementioned wireless connection methods may include, but are not limited to, 3G / 4G connections, WiFi connections, Bluetooth connections, WiMAX connections, Zigbee connections, UWB (ultra-wideband) connections, and other currently known or future-developed wireless connection methods. The aforementioned image acquisition device may be a device with shooting or screenshot functions. For example, the aforementioned image acquisition device may be an industrial camera. The aforementioned target image sequence is a sequence of consecutive target images acquired by the aforementioned image acquisition device within a preset time period. The target images in the aforementioned target image sequence may be images of the captured target scene. The aforementioned target scene may be any scene. The aforementioned target scene is not specifically limited here. The aforementioned preset time period may be a pre-set time period. For example, the aforementioned preset time period may be 2 minutes.

[0026] Step 102: Generate a sequence of adjacent target image groups based on the target image sequence.

[0027] In some embodiments, the execution entity may generate a sequence of adjacent target image groups based on the target image sequence.

[0028] In practice, firstly, for each pair of adjacent target images in the aforementioned target image sequence, the executing entity can determine these two adjacent target images as an adjacent target image group. This adjacent target image group can include a first target image and a second target image. The acquisition time of the second target image is later than that of the first target image. The first target image can be the target image at the first position in the adjacent target image group. The second target image can be the target image at the last position in the adjacent target image group. For example, the adjacent target image group can be [first target image, second target image]. Then, the sequence composed of the determined adjacent target image groups can be defined as the adjacent target image group sequence.

[0029] Step 103: Obtain the inertial information sequence.

[0030] In some embodiments, the aforementioned execution entity can acquire inertial information sequences from an IMU (Inertial Measurement Unit) sensor via a wired or wireless connection. The IMU sensor may include a three-axis accelerometer and a three-axis gyroscope. The inertial information in the aforementioned inertial information sequence corresponds one-to-one with the adjacent target image groups in the aforementioned adjacent target image group sequence. Here, the one-to-one corresponding adjacent target image groups and the inertial information are time-synchronized. The inertial information in the aforementioned inertial information sequence may include linear acceleration and angular velocity.

[0031] Step 104: Input the sequence of adjacent target image groups into the image feature extraction layer of the pre-trained pose information generation model to obtain the image feature tensor sequence.

[0032] In some embodiments, the execution entity can input the sequence of adjacent target image groups into the image feature extraction layer of a pre-trained pose information generation model to obtain an image feature tensor sequence. The pose information generation model may further include an inertial feature extraction layer, an image inertial feature fusion layer, a pose network layer, and a deep network layer. The pose information generation model can be a network model that takes adjacent target image groups as input and outputs the pose information and depth maps corresponding to the adjacent target image groups. The pose information can characterize the relative transformation of the position and orientation of two adjacent target images within an adjacent target image group. Specifically, the pose information may include a three-dimensional translation vector (x, y, z) and a three-axis rotation angle (x, y, z). , , In the above (x, y, z), x can represent a translation along the x-axis. y can represent a translation along the y-axis. z can represent a translation along the z-axis. , , In ) This can represent rotation about the x-axis. (The above...) , , In ) This can represent rotation about the y-axis. (The above...) , , In ) This can represent rotation around the z-axis. The depth maps in the aforementioned depth map group correspond one-to-one with the adjacent target images in the aforementioned adjacent target image group. The depth maps in the aforementioned depth map group can be maps representing the vertical distances from each corresponding point in the captured target scene to the camera plane. The aforementioned image feature extraction layer can be a network layer capable of extracting features from the adjacent target images included in the input adjacent target image group. The aforementioned inertial feature extraction layer can be a network layer capable of extracting features from the input inertial information. The image feature fusion layer can be a network layer capable of fusing input image features and inertial features. The aforementioned pose network layer can be a network layer capable of estimating the pose of the input adjacent target image group. The aforementioned depth network layer can be a network layer capable of estimating the depth of the input target image. Depth estimation can be understood as predicting the distances from each point in the target scene corresponding to the captured target image to the camera plane.

[0033] In some optional implementations of certain embodiments, the execution entity may input the sequence of adjacent target images into the image feature extraction layer of a pre-trained pose information generation model through the following steps to obtain an image feature tensor sequence:

[0034] First, for each adjacent target image group in the above sequence of adjacent target image groups, perform the following steps:

[0035] The first sub-step involves performing channel stitching on each adjacent target image within the aforementioned adjacent target image group to generate a channel-stitched image. For example, the data dimension of the first target image in the adjacent target image group can be 512*256*3. The data dimension of the second target image in the adjacent target image group can be 512*256*3. The data dimension of the channel-stitched image can be 512*256*6.

[0036] The second sub-step involves performing convolution processing on the aforementioned channel-stitched image to generate a first convolutional feature map. This first convolutional feature map can represent high-dimensional low-level features. For example, it can be a 512*256*6-dimensional feature map representing the edge and line features of the scene corresponding to the channel-stitched image. "512" represents the number of rows in the first convolutional feature map. "256" represents the number of columns in the first convolutional feature map. "6" represents the number of RGB (red, green, blue) channels corresponding to the first convolutional feature map.

[0037] The third sub-step involves performing convolution processing on the first convolutional feature map to generate a second convolutional feature map. This second convolutional feature map can represent mid-level features of a second-highest dimension. For example, it could be a 256*128*6-dimensional feature map representing the texture features of the scene corresponding to the channel-stitched image.

[0038] The fourth sub-step involves performing convolution processing on the second convolutional feature map to generate a third convolutional feature map. This third convolutional feature map can represent low-dimensional high-level features. For example, it can be a 128*64*6-dimensional feature map representing the local motion features of the scene corresponding to the channel-stitched image.

[0039] The fifth sub-step involves batch normalizing the third convolutional feature map to generate a batch normalized feature map.

[0040] The sixth sub-step involves generating an image feature tensor based on the batch normalized feature map and the second preset activation function. In practice, the executing entity can input the batch normalized feature map into the second preset activation function to obtain the image feature tensor. The second preset activation function can be a pre-defined activation function. For example, the second preset activation function can be a ReLU activation function.

[0041] The second step is to determine the generated image feature tensors as a sequence of image feature tensors.

[0042] In some optional implementations of certain embodiments, the pose information generation model described above can be obtained through the following training methods:

[0043] The first step is to acquire a sample set. This sample set includes sample image groups and sample inertial information. The sample image groups can be two consecutive, adjacent frames of sample target images acquired by the image acquisition device. The sample inertial information can be the inertial information corresponding to the two adjacent sample target images. Here, "correspondence" can be understood as time synchronization. It should be noted that the execution entity for training the pose information generation model can be the aforementioned execution entity or other computing devices.

[0044] The second step is to select samples from the above sample set and perform the following training steps;

[0045] The first sub-step involves inputting the selected sample image set and sample inertial information into the initial pose information generation model to obtain sample pose information and sample depth map set. The initial pose information generation model can be an initial neural network model that takes the sample image set and sample inertial information as input and outputs the sample pose information and sample depth map set. This initial neural network model can be a neural network model to be trained. The sample pose information can characterize the relative transformation of the position and orientation of two adjacent target images within the sample image set. Specifically, the sample pose information can include a three-dimensional translation vector (x, y, z) and a three-axis rotation angle (x, y, z). , , In the above (x, y, z), x can represent a translation along the x-axis. y can represent a translation along the y-axis. z can represent a translation along the z-axis. , , In ) This can represent rotation about the x-axis. (The above...) , , In ) This can represent rotation about the y-axis. (The above...) , , In ) It can represent rotation about the z-axis. The sample depth maps in the above sample depth map group correspond one-to-one with the sample images in the above sample image group. The sample depth maps in the above sample depth map group can be maps representing the vertical distance from each corresponding point in the captured target scene to the camera plane.

[0046] The second sub-step involves determining the loss function value corresponding to the initial pose information generation model based on the aforementioned sample image set, sample inertial information, sample depth map set, and sample pose information. The sample image set includes a first sample image and a second sample image. The first sample image and the second sample image have the same image size. The second sample image was acquired later than the first sample image. The sample depth map set may include a first sample depth map and a second sample depth map. The first sample depth map corresponds to the first sample image. The second sample depth map corresponds to the second sample image. The preset function corresponding to the aforementioned loss function value can be expressed by the following formula:

[0047] .

[0048] Among them, the above This can represent the first weighting coefficient. Here, the first weighting coefficient can be 1. (The above...) This can represent the second weighting coefficient. Here, the range of the aforementioned second weighting coefficient is [0.01, 0.5]. This can represent the third weighting coefficient. The range of the aforementioned third weighting coefficient is [0.1, 1.0]. This can be the fourth weighting coefficient. Here, the range of the aforementioned fourth weighting coefficient can be [0.1, 1.0].

[0049] .

[0050] Among them, the above It can represent the first Frame. Here, it can also be represented as the image frame number corresponding to the first sample image. The above... It can be represented as the first A frame can also be represented as the image frame number corresponding to the second sample image. The above... This can represent the pixel coordinates of the second sample image. (The above...) This can represent the pixel coordinates of the first sample image. (The above...) This can represent the number of pixels in the first sample image. (The above...) This can represent the fifth preset weighting coefficient, used for balancing. Loss and (L1 norm) loss. Here, the fifth preset weighting coefficient mentioned above can be 0.85. The above... This can represent the first sample image in pixels. The image values. (Above) This can represent the projected image of the second sample image onto the first sample image. (The above...) The L1 norm can represent the difference in luminance between the first sample image and the projected image. (The above...) This can represent the structural similarity between the first sample image and the projected image. The above... This can represent photometric loss and is used to measure the error in image reconstruction (between the first sample image and the projected image). The projected image mentioned above... It is obtained through the following formula:

[0051] .

[0052] Among them, the above This can represent the camera intrinsic parameter matrix. (The above...) This can represent the inverse matrix of the camera intrinsic parameter matrix. (The above...) This can represent the pixel coordinates of the projected image reconstructed from the first sample image through projection and sampling. (The above...) This can represent the depth map of the first sample. (The above...) This can represent the depth map of the first sample at the pixel level. The depth value at that location. (The above) It can represent the first Frame image to the The relative pose of the frame image can be represented as sample pose information.

[0053] .

[0054] Among them, the above This can represent the depth map of the second sample. (The above...) This can represent the second sample depth map at the pixel level. The depth value at that location. This can represent the gradient of the second sample depth map in the x-direction. (The above...) This can represent the gradient of the second sample depth map in the y-direction. (The above...) This can represent the gradient of the second sample image in the x-direction. (The above...) This can represent the gradient of the second sample image in the y-direction. (The above...) It can represent a smoothness loss, used to encourage the depth map to smooth at color edges.

[0055] .

[0056] The above This can represent the inverse matrix of the sample pose information. (The above...) This can represent the cumulative matrix of pose information for all samples up to frame t predicted by the model. (The above...) This can be a Lie algebra mapping used to map the SE(3) transformation matrix to the se(3) vector space. The above... It can be the 2-norm of the Lie algebra mapping. (The above...) It can represent pose loss. It is used to constrain the consistency between the predicted pose loss and the actual pose loss.

[0057] .

[0058] The above This can represent the inverse matrix of the camera intrinsic parameter matrix. (The above...) This can represent the second sample depth map at the pixel level. The depth value at that location. (The above) This can represent the depth map of the first sample at the pixel level. The depth value at that location. (The above) It can represent the first Frame image to the The relative pose of the frame images. This can also represent the pose information from the first sample image to the second sample image. (The above...) It can represent the loss of three-dimensional geometric consistency.

[0059] The third sub-step involves determining that the loss function value satisfies a preset optimization objective, and then defining the initial pose information generation model as the trained pose information generation model. The preset optimization objective can be that the absolute value of the difference between the loss function value in this training round and a preset number of consecutive loss function values ​​from previous rounds is less than or equal to a preset loss difference threshold. This preset loss difference threshold can be a pre-set threshold, for example, 10 to the power of negative cube. The preset number can also be a pre-set number, for example, 20.

[0060] Optionally, the steps for training the above pose information generation model may further include:

[0061] The fourth sub-step involves adjusting the network parameters of the initial pose information generation model in response to the determination that the loss function value does not meet the preset optimization objective. It also involves reselecting samples from the aforementioned sample set, using the adjusted initial pose information generation model as the new initial pose information generation model, and repeating the training steps. As an example, the back propagation algorithm (BP algorithm) and gradient descent methods (such as mini-batch gradient descent) can be used to adjust the network parameters of the initial pose information generation model.

[0062] Step 105: Input the inertial information sequence into the inertial feature extraction layer to obtain the inertial feature vector sequence.

[0063] In some embodiments, the execution entity can input the inertial information sequence into the inertial feature extraction layer to obtain an inertial feature vector sequence. The inertial feature vectors in the inertial feature vector sequence correspond to the image feature tensors in the image feature tensor sequence. Here, the correspondence can be understood as a one-to-one correspondence.

[0064] In some optional implementations of certain embodiments, the execution entity may input the inertial information sequence into the inertial feature extraction layer through the following steps to obtain an inertial feature vector sequence:

[0065] The first step is to embed the aforementioned inertial information sequence to generate an embedded inertial information vector sequence. In practice, the executing entity can perform embedding processing on the aforementioned inertial information sequence to generate an embedded inertial information vector sequence.

[0066] The second step involves performing the following first loop step for each embedded inertial information vector in the above sequence of embedded inertial information vectors:

[0067] The first sub-step is to determine the above-mentioned embedded inertial information vector as the input vector at the current moment.

[0068] The second sub-step involves generating a first reset vector based on the previous hidden state vector corresponding to the current input vector and the first preset reset coefficient.

[0069] In practice, the aforementioned executing entity can determine the first reset vector as the product of the previously hidden state vector and the first preset reset coefficient. Wherein, when the current input vector is the first input vector, the previously hidden state vector corresponding to the current input vector can be a zero vector, that is, the initial hidden state vector can be a zero vector.

[0070] The third sub-step involves generating a second reset vector based on the current input vector and the second preset reset coefficient. In practice, the executing entity can determine the second reset vector as the product of the current input vector and the second preset reset coefficient.

[0071] The fourth sub-step involves generating a reset vector based on the first reset vector, the second reset vector, and the first preset activation function. In practice, the executing entity can input the sum of the first and second reset vectors into the first preset function to obtain the reset vector. The first preset activation function can be the Sigmoid function.

[0072] The fifth sub-step involves generating a third reset vector based on the current input vector and the third preset reset coefficient. In practice, the executing entity can determine the third reset vector as the product of the current input vector and the third preset reset coefficient.

[0073] The sixth sub-step involves generating a fourth reset vector based on the previously hidden state vector, the reset vector, and the fourth preset reset coefficient. In practice, firstly, the executing entity can determine the Hadamard vector as the Hadamard product of the previously hidden state vector and the reset vector. Then, the fourth reset vector is determined as the product of the Hadamard vector and the fourth preset reset coefficient.

[0074] The seventh sub-step generates the candidate hidden state vector for the current moment based on the third reset vector, the fourth reset vector, and the third preset activation function. In practice, the executing entity can input the sum of the third reset vector and the fourth reset vector into the third preset activation function to obtain the candidate hidden state vector for the current moment. The third preset activation function can be a hyperbolic tangent function.

[0075] The eighth sub-step involves generating a first update vector based on the current input vector and the first preset update coefficient. In practice, the executing entity can determine the first update vector as the product of the current input vector and the first preset update coefficient.

[0076] The ninth sub-step involves generating a second update vector based on the previously hidden state vector and the second preset update coefficient. In practice, the executing entity can determine the second update vector as the product of the previously hidden state vector and the second preset update coefficient.

[0077] The tenth sub-step involves generating an update vector based on the first update vector, the second update vector, and the first preset activation function. In practice, the executing entity can input the sum of the first update vector and the second update vector into the first preset activation function to obtain the update vector.

[0078] The eleventh sub-step involves generating a first current-time hidden state vector based on the aforementioned update vector and the aforementioned current-time candidate hidden state vector. In practice, the executing entity can determine the first current-time hidden state vector by the Hadamard product of the aforementioned update vector and the aforementioned current-time candidate hidden state vector.

[0079] The twelfth sub-step involves generating a second current-time hidden state vector based on the previously hidden state vector and the aforementioned update vector. In practice, firstly, the executing entity can determine the forgetting vector as the difference between 1 and the update vector. Then, the Hadamard product of the forgetting vector and the previously hidden state vector can be used to determine the second current-time hidden state vector.

[0080] The thirteenth sub-step involves generating the current hidden state vector based on the first and second current hidden state vectors. In practice, the executing entity can determine the current hidden state vector as the sum of the first and second current hidden state vectors.

[0081] The fourteenth sub-step involves determining the current hidden state vector as the previous hidden state vector of the next inertial feature vector of the aforementioned inertial feature vector, and then executing the first loop step again.

[0082] In some embodiments, the executing entity may determine the current hidden state vector as the previous hidden state vector of the next inertial feature vector of the inertial feature vector, and then execute the first loop step again. Wherein, when the current hidden state vector is the current hidden state vector corresponding to the last inertial feature vector in the inertial feature vector sequence, the next inertial feature vector of the inertial feature vector does not exist.

[0083] The third step is to determine the generated hidden state vectors at each current time as a sequence of inertial feature vectors.

[0084] Step 106: Input the image feature tensor sequence and the inertial feature vector sequence into the image inertial feature fusion layer to obtain the image inertial fusion feature tensor sequence.

[0085] In some embodiments, the execution entity may input the image feature tensor sequence and the inertial feature vector sequence into the image inertial feature fusion layer to obtain the image inertial fusion feature tensor sequence.

[0086] In some optional implementations of certain embodiments, the execution entity may input the image feature tensor sequence and the inertial feature vector sequence into the image inertial feature fusion layer through the following steps to obtain the image inertial fusion feature tensor sequence:

[0087] First, for each image feature tensor in the above image feature tensor sequence, perform the following steps:

[0088] The first sub-step involves concatenating the image feature tensor with the inertial feature vectors corresponding to the image feature tensor in the inertial feature vector sequence to generate a concatenated feature tensor.

[0089] The second sub-step involves multiplying the concatenated feature tensor with a preset weight matrix to determine the weight feature matrix. The preset weight matrix can be a pre-set, trained weight matrix.

[0090] The third sub-step involves determining the channel information based on the aforementioned weight feature matrix and preset bias information. In practice, the executing entity can determine the channel information by summing the aforementioned weight feature matrix and preset bias information. The preset bias information can be pre-trained bias information.

[0091] The fourth sub-step involves generating channel weights based on the first preset activation function and the aforementioned channel information. In practice, the executing entity can input the aforementioned channel information into the first preset activation function to generate the channel weights. The first preset activation function can be a pre-defined activation function. Here, the first preset activation function can be the Sigmoid activation function.

[0092] The fifth sub-step involves determining the image inertial fusion feature tensor based on the aforementioned channel weights, the aforementioned image feature tensor, and the corresponding inertial feature vector of the aforementioned image feature tensor. In practice, firstly, the executing entity can concatenate the aforementioned image feature tensor with the aforementioned inertial feature tensor to generate a concatenated feature tensor. Then, the Hadamard product of the concatenated feature tensor and the aforementioned channel weights is determined as the image inertial fusion feature tensor. The aforementioned image inertial fusion feature tensor can be expressed by the following formula:

[0093] .

[0094] Among them, the above This can represent channel weights. (The above...) It can represent image feature tensors. The above... This can represent an inertial eigenvector. (The above...) This can represent the image inertial fusion feature tensor. (The above...) This can represent a channel-dimensional concatenation operation. (The above...) It can represent the Hadamard product (element-by-element multiplication).

[0095] The second step is to determine the sequence of image inertial fusion feature tensors as the image inertial fusion feature tensor sequence.

[0096] Step 107: Input the image inertial fusion feature tensor sequence into the pose network layer to obtain the pose information sequence.

[0097] In some embodiments, the execution entity may input the image inertial fusion feature tensor sequence into the pose network layer to obtain a pose information sequence.

[0098] In some optional implementations of certain embodiments, the execution entity may input the image inertial fusion feature tensor sequence into the pose network layer through the following steps to obtain a pose information sequence:

[0099] The first step is to extract temporal features from the aforementioned image-inertial fusion feature tensor sequence, obtaining a temporal feature tensor sequence. In practice, the aforementioned execution entity can use a temporal recursive network to extract temporal features from the aforementioned image-inertial fusion feature tensor sequence, obtaining a temporal feature tensor sequence. The aforementioned temporal recursive network can be any network capable of extracting temporal features from the input data. For example, the aforementioned temporal recursive network can be an LSTM (Long Short-Term Memory) network or a GRU (Gated Recurrent Unit) network.

[0100] The second step is to perform fully connected processing on the above temporal feature tensor sequence to obtain the fully connected temporal feature tensor sequence as the pose information sequence.

[0101] Step 108: Input the target image sequence into the deep network layer to obtain the depth map sequence.

[0102] In some embodiments, the execution entity can input the target image sequence into the deep network layer to obtain a depth map sequence. The deep network layer can include an encoder layer, a decoder layer, and a depth map head layer. The encoder layer can be a pre-trained ResNet50 (with fully connected layers removed) that extracts features at different scales through multi-level feature extraction. The decoder layer can be a network layer consisting of at least two upsampling modules. These at least two upsampling modules are used to upsample the feature maps of different scales obtained from the encoder layer and to fuse them with feature maps of corresponding sizes from the feature maps of different scales (skip connections). The depth head layer can include a convolutional module and a sigmoid activation function. Here, the convolutional module is used to reduce the number of channels to 1.

[0103] Step 109: Generate motion trajectory information based on the pose information sequence and depth map sequence.

[0104] In some embodiments, the execution entity can generate motion trajectory information based on the pose information sequence and the depth map sequence. In practice, firstly, the execution entity can accumulate and process each pose information in the pose information sequence to obtain an absolute pose information sequence. Then, motion trajectory information can be generated by calling a preset trajectory generation function interface, using the absolute pose information sequence and the depth map sequence as input parameters to the preset trajectory generation function interface. The preset trajectory generation function interface can be a pre-defined function that can convert the input pose information sequence and depth map sequence into motion trajectory information.

[0105] Optionally, the aforementioned executing entity can also control an associated intelligent agent to perform positioning and displacement operations corresponding to the aforementioned motion trajectory information, based on the aforementioned motion trajectory information. The associated intelligent agent can be a device capable of positioning and displacement based on motion trajectory information. For example, the intelligent agent can be a robot, a drone, or a robotic arm. In practice, the aforementioned executing entity can control the associated intelligent agent to perform positioning and displacement operations corresponding to the aforementioned motion trajectory information by calling a preset automatic control function interface, using the motion trajectory information as input parameters. The preset automatic control function interface can be a pre-defined function that generates control commands based on the input motion trajectory information.

[0106] The above embodiments of this disclosure have the following beneficial effects: the motion trajectory information generation method of some embodiments of this disclosure can improve the accuracy of the generated pose and the correctness of the motion trajectory, thereby reducing the number of errors in the positioning and displacement operations of the relevant intelligent agent. Specifically, the reason why the estimated pose accuracy is low, and the generated motion trajectory accuracy is low, leading to more errors in the positioning and displacement operations of the relevant intelligent agent (e.g., robot and drone) is that: since the pose estimation is obtained through the acquisition of image sequences, when the acquired image sequences are affected by sudden changes in illumination or occlusion, the quality of the acquired images is easily reduced, resulting in low accuracy of the estimated pose and low correctness of the generated motion trajectory, thus leading to more errors in the positioning and displacement operations of the relevant intelligent agent (e.g., robot and drone). Based on this, the motion trajectory information generation method of some embodiments of this disclosure first acquires a target image sequence. This yields a target image sequence representing visual information. Then, based on the target image sequence, a sequence of adjacent target image groups is generated. This yields a sequence of adjacent target image groups representing two consecutive adjacent frames. Afterwards, an inertial information sequence is acquired. In this sequence, the inertial information corresponds to the adjacent target image groups in the adjacent target image group sequence. Thus, an inertial information sequence representing the motion state is obtained. Next, the adjacent target image group sequence is input into the image feature extraction layer of a pre-trained pose information generation model to obtain an image feature tensor sequence. The pose information generation model further includes an inertial feature extraction layer, an image inertial feature fusion layer, a pose network layer, and a deep network layer. Thus, an image feature tensor sequence representing the high-level semantic features of the visual data is obtained. Then, the inertial information sequence is input into the inertial feature extraction layer to obtain an inertial feature vector sequence. The inertial feature vectors in the inertial feature vector sequence correspond to the image feature tensors in the image feature tensor sequence. Thus, an inertial feature vector sequence representing the high-level semantic features of the motion state data is obtained. Finally, the image feature tensor sequence and the inertial feature vector sequence are input into the image inertial feature fusion layer to obtain an image inertial fusion feature tensor sequence. Therefore, an image inertial fusion feature tensor sequence can be obtained, enabling the fusion of image features representing visual information and inertial features representing motion state information at the data layer. Subsequently, this image inertial fusion feature tensor sequence is input into the pose network layer to obtain a pose information sequence. This yields pose information sequences representing the transformation state of the subsequent frame relative to the preceding frame in two adjacent frames. Next, the target image sequence is input into the deep network layer to obtain a depth map sequence. This depth map sequence can then be used to reconstruct a realistic physical scene. Finally, motion trajectory information is generated based on the pose information sequence and the depth map sequence.This allows for the acquisition of highly accurate motion trajectory information. Furthermore, by acquiring target image sequences representing visual information, high-accuracy pose information can be obtained when visual effects are good. Additionally, by acquiring inertial information sequences representing motion states, visual features can be compensated for when visual effects are poor, achieving complementarity between the two. Moreover, by tightly coupling visual and inertial features at the feature level, the accuracy and robustness of data fusion can be further improved. The weight ratio of the two types of features can be dynamically adjusted when the environment changes. That is, when visual features are good (e.g., clear scene texture or sufficient lighting), the weight of visual features is automatically increased; when visual features are poor (e.g., visual signals are obstructed or lighting is poor), the influence of inertial features is automatically increased. This improves the accuracy of the generated pose and the correctness of the motion trajectory, thereby reducing the number of errors in the localization and displacement operations performed by the relevant intelligent agent.

[0107] Further reference Figure 2 As an implementation of the methods shown in the above figures, this disclosure provides some embodiments of a motion trajectory information generation device, which are similar to... Figure 1 Corresponding to the method embodiments shown, this motion trajectory information generation device can be specifically applied to various electronic devices.

[0108] like Figure 2As shown, a motion trajectory information generation device 200 in some embodiments includes: a first acquisition unit configured to acquire a target image sequence; a first generation unit configured to generate a sequence of adjacent target image groups based on the target image sequence; a second acquisition unit configured to acquire an inertial information sequence, wherein the inertial information in the inertial information sequence corresponds to the adjacent target image groups in the adjacent target image group sequence; a first input unit configured to input the adjacent target image group sequence into the image feature extraction layer of a pre-trained pose information generation model to obtain an image feature tensor sequence, wherein the pose information generation model further includes an inertial feature extraction layer, an image inertial feature fusion layer, a pose network layer, and a deep network layer; and a second input unit configured to input the inertial information sequence into the image feature extraction layer of a pre-trained pose information generation model to obtain an image feature tensor sequence. The inertial information sequence is input to the inertial feature extraction layer to obtain an inertial feature vector sequence, wherein the inertial feature vectors in the inertial feature vector sequence correspond to the image feature tensors in the image feature tensor sequence; the third input unit is configured to input the image feature tensor sequence and the inertial feature vector sequence to the image inertial feature fusion layer to obtain an image inertial fusion feature tensor sequence; the fourth input unit is configured to input the image inertial fusion feature tensor sequence to the pose network layer to obtain a pose information sequence; the fifth input unit is configured to input the target image sequence to the depth network layer to obtain a depth map sequence; and the second generation unit is configured to generate motion trajectory information based on the pose information sequence and the depth map sequence.

[0109] It is understandable that the units recorded in the motion trajectory information generation device 200 and the reference Figure 1 The steps in the described method correspond to each other. Therefore, the operations, features, and beneficial effects described above for the method also apply to the motion trajectory information generation device 200 and the units contained therein, and will not be repeated here.

[0110] The following is for reference. Figure 3 It shows a schematic diagram of the structure of an electronic device 300 (e.g., a computing device) suitable for implementing some embodiments of the present disclosure. Figure 3 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of this disclosure.

[0111] like Figure 3As shown, the electronic device 300 may include a processing unit 301 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 302 or a program loaded from a storage device 308 into a random access memory (RAM) 303. The RAM 303 also stores various programs and data required for the operation of the electronic device 300. The processing unit 301, ROM 302, and RAM 303 are interconnected via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0112] Typically, the following devices can be connected to I / O interface 305: input devices 306 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 307 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 308 including, for example, magnetic tapes, hard disks, etc.; and communication devices 309. Communication device 309 allows electronic device 300 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 3 An electronic device 300 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively. Figure 3 Each box shown can represent a device or multiple devices as needed.

[0113] In particular, according to some embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, some embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication device 309, or installed from storage device 308, or installed from ROM 302. When the computer program is executed by processing device 301, it performs the functions defined in the methods of some embodiments of this disclosure.

[0114] It should be noted that, in some embodiments of this disclosure, the computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In some embodiments of this disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In some embodiments of this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0115] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0116] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device. The aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to: acquire a target image sequence; generate a sequence of adjacent target image groups based on the target image sequence; acquire an inertial information sequence, wherein the inertial information in the inertial information sequence corresponds to adjacent target image groups in the adjacent target image group sequence; and input the adjacent target image group sequence into the image feature extraction layer of a pre-trained pose information generation model to obtain an image feature tensor sequence, wherein the pose information generation model further includes an inertial feature extraction layer, an image inertial feature fusion layer, a pose network layer, and a deep network. The image network layer is used to obtain an inertial feature vector sequence by inputting the inertial information sequence into the inertial feature extraction layer. The inertial feature vectors in the inertial feature vector sequence correspond to the image feature tensors in the image feature tensor sequence. The image feature tensor sequence and the inertial feature vector sequence are then input into the image inertial feature fusion layer to obtain an image inertial fusion feature tensor sequence. The image inertial fusion feature tensor sequence is then input into the pose network layer to obtain a pose information sequence. The target image sequence is then input into the depth network layer to obtain a depth map sequence. Motion trajectory information is generated based on the pose information sequence and the depth map sequence.

[0117] Computer program code for performing operations of some embodiments of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0118] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0119] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0120] The above description is merely a selection of preferred embodiments of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in the embodiments of this disclosure.

Claims

1. A method for generating motion trajectory information, characterized in that, include: Obtain the target image sequence; Based on the target image sequence, generate a sequence of adjacent target image groups; Obtain an inertial information sequence, wherein the inertial information in the inertial information sequence corresponds to the adjacent target image group in the adjacent target image group sequence; The adjacent target image group sequence is input into the image feature extraction layer of a pre-trained pose information generation model to obtain an image feature tensor sequence. The pose information generation model further includes an inertial feature extraction layer, an image inertial feature fusion layer, a pose network layer, and a deep network layer. The inertial information sequence is input into the inertial feature extraction layer to obtain an inertial feature vector sequence, wherein the inertial feature vectors in the inertial feature vector sequence correspond to the image feature tensors in the image feature tensor sequence; The image feature tensor sequence and the inertial feature vector sequence are input into the image inertial feature fusion layer to obtain the image inertial fusion feature tensor sequence; The image inertial fusion feature tensor sequence is input into the pose network layer to obtain the pose information sequence; The target image sequence is input into the deep network layer to obtain a depth map sequence; Motion trajectory information is generated based on the pose information sequence and the depth map sequence.

2. The method according to claim 1, characterized in that, The method further includes: Based on the motion trajectory information, the associated intelligent agent is controlled to perform positioning and displacement operations corresponding to the motion trajectory information.

3. The method according to claim 1, characterized in that, The step of inputting the image feature tensor sequence and the inertial feature vector sequence into the image inertial feature fusion layer to obtain the image inertial fusion feature tensor sequence includes: For each image feature tensor in the image feature tensor sequence, perform the following steps: The image feature tensor and the inertial feature vector corresponding to the image feature tensor in the inertial feature vector sequence are concatenated through channels to generate a concatenated feature tensor. The product of the spliced ​​feature tensor and the preset weight matrix is ​​determined as the weight feature matrix; The channel information is determined based on the weight feature matrix and the preset bias information; Based on the first preset activation function and the channel information, channel weights are generated; Based on the channel weights, the image feature tensor, and the inertial feature vector corresponding to the image feature tensor, the image inertial fusion feature tensor is determined; The sequence of the obtained image inertial fusion feature tensors is defined as the image inertial fusion feature tensor sequence.

4. The method according to claim 1, characterized in that, The step of inputting the sequence of adjacent target images into the image feature extraction layer of a pre-trained pose information generation model to obtain an image feature tensor sequence includes: For each adjacent target image group in the adjacent target image group sequence, perform the following steps: Channel stitching is performed on each adjacent target image in the adjacent target image group to generate a channel stitched image; The channel-stitched image is convolved to generate a first convolutional feature map; The first convolutional feature map is convolved to generate a second convolutional feature map; The second convolutional feature map is convolved to generate the third convolutional feature map; The third convolutional feature map is batch normalized to generate a batch normalized feature map. Based on the batch normalized feature map and the second preset activation function, an image feature tensor is generated; The generated image feature tensors are defined as an image feature tensor sequence.

5. The method according to claim 1, characterized in that, The pose information generation model was obtained through the following training method: Obtain a sample set, wherein the samples in the sample set include sample image groups and sample inertial information; Samples are selected from the sample set, and the following training steps are performed; The selected sample image group and sample inertial information are input into the initial pose information generation model to obtain sample pose information and sample depth map group; Based on the sample image group, the sample inertial information, the sample depth map group, and the sample pose information, determine the loss function value corresponding to the initial pose information generation model; In response to determining that the loss function value satisfies the preset optimization objective, the initial pose information generation model is determined as the trained pose information generation model.

6. The method according to claim 5, characterized in that, The method further includes: In response to the determination that the loss function value does not meet the preset optimization objective, the network parameters of the initial pose information generation model are adjusted, and samples are reselected from the sample set. The adjusted initial pose information generation model is used as the initial pose information generation model, and the training steps are executed again.

7. The method according to claim 1, characterized in that, The inertial information sequence is input into the inertial feature extraction layer to obtain an inertial feature vector sequence: The inertial information sequence is embedded to generate an embedded inertial information vector sequence; For each embedded inertial information vector in the sequence of embedded inertial information vectors, perform the following first loop step: The embedded inertial information vector is determined as the input vector at the current moment; A first reset vector is generated based on the previous hidden state vector corresponding to the current input vector and the first preset reset coefficient; Generate a second reset vector based on the current input vector and the second preset reset coefficient; A reset vector is generated based on the first reset vector, the second reset vector, and the first preset activation function; Based on the current input vector and the third preset reset coefficient, a third reset vector is generated; The fourth reset vector is generated based on the previously hidden state vector, the reset vector, and the fourth preset reset coefficient. Based on the third reset vector, the fourth reset vector, and the third preset activation function, generate the candidate hidden state vector at the current moment; Based on the current input vector and the first preset update coefficient, a first update vector is generated; Based on the previously hidden state vector and the second preset update coefficient, a second update vector is generated; An update vector is generated based on the first update vector, the second update vector, and the first preset activation function; Based on the update vector and the candidate hidden state vector at the current time, a first hidden state vector at the current time is generated; Based on the hidden state vector of the previous time step and the update vector, a second hidden state vector of the current time step is generated; Generate a current hidden state vector based on the first current hidden state vector and the second current hidden state vector; The current hidden state vector is determined as the previous hidden state vector of the next embedded inertial information vector of the embedded inertial information vector, and the first loop step is executed again. The generated hidden state vectors at each current time step are determined as a sequence of inertial feature vectors.

8. A motion trajectory information generation device, applied to the motion trajectory information generation method according to any one of claims 1 to 7, characterized in that, include: The first acquisition unit is configured to acquire the target image sequence; The first generation unit is configured to generate a sequence of adjacent target image groups based on the target image sequence; The second acquisition unit is configured to acquire an inertial information sequence, wherein the inertial information in the inertial information sequence corresponds to the adjacent target image group in the adjacent target image group sequence; The first input unit is configured to input the sequence of adjacent target images into the image feature extraction layer of a pre-trained pose information generation model to obtain an image feature tensor sequence. The pose information generation model further includes an inertial feature extraction layer, an image inertial feature fusion layer, a pose network layer, and a deep network layer. The second input unit is configured to input the inertial information sequence into the inertial feature extraction layer to obtain an inertial feature vector sequence, wherein the inertial feature vectors in the inertial feature vector sequence correspond to the image feature tensors in the image feature tensor sequence. The third input unit is configured to input the image feature tensor sequence and the inertial feature vector sequence into the image inertial feature fusion layer to obtain the image inertial fusion feature tensor sequence; The fourth input unit is configured to input the image inertial fusion feature tensor sequence into the pose network layer to obtain a pose information sequence; The fifth input unit is configured to input the target image sequence into the deep network layer to obtain a depth map sequence; The second generation unit is configured to generate motion trajectory information based on the pose information sequence and the depth map sequence.

9. An electronic device, characterized in that, include: One or more processors; A storage device on which one or more programs are stored; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1 to 7.

10. A computer-readable medium, characterized in that, It stores a computer program thereon, wherein the computer program, when executed by a processor, implements the method as described in any one of claims 1 to 7.

Images