Motion tracking methods, devices and self-moving devices

Abstract

This application provides a motion tracking method, apparatus, and self-moving device, relating to the field of image processing technology. The method includes acquiring a first image and a second image of adjacent frames during the movement of the self-moving device; if there is a jump between the first image and the second image, acquiring the previous frame of the first image and determining the motion direction of the self-moving device based on the camera pose of the first image relative to the previous frame; performing frame interpolation between the first image and the second image based on the first image and the motion direction to generate an interpolated frame image; and generating the motion trajectory of the self-moving device based on the first image, the interpolated frame image, and the second image. This allows for indirect optical flow tracking from the first image to the second image through the interpolated frame image, thereby improving the stability of optical flow tracking.

Description

Technical Field

[0001] This application relates to the field of motion tracking technology, and more particularly to a motion tracking method, device, and self-moving device. Background Technology

[0002] In recent years, the use of VSLAM (Visual Simultaneous Localization and Mapping) systems has gradually become a research hotspot for positioning and navigation of self-moving devices. VSLAM systems can not only ensure that self-moving devices are in a safe working state, but also maintain robust control for a long time in complex operating environments.

[0003] The robustness of a VSLAM system typically depends on its stability when using optical flow to track images. However, in practical applications, frame drops often occur, causing optical flow tracing to fail to accurately track the motion trajectory of the image, thus affecting the stability of optical flow tracing. Summary of the Invention

[0004] In view of this, embodiments of this application provide a motion tracking method, apparatus, and self-moving device to improve the stability of optical flow tracking.

[0005] To achieve the above objectives, in a first aspect, embodiments of this application provide a motion tracking method, including:

[0006] Acquire the first and second images of adjacent frames during the movement of the mobile device;

[0007] In the case of a jump between the first image and the second image, the previous frame of the first image is obtained, and the motion direction of the self-moving device is determined according to the camera pose of the first image relative to the previous frame.

[0008] Based on the first image and the direction of motion, frame interpolation is performed between the first image and the second image to generate an interpolated image;

[0009] The movement trajectory of the self-moving device is generated based on the first image, the interpolated frame image, and the second image.

[0010] Secondly, embodiments of this application provide a motion tracking device, characterized in that it includes:

[0011] The acquisition module is used to acquire the first and second images of adjacent frames during the movement of the self-moving device;

[0012] The determination module is used to acquire the previous frame of the first image when there is a jump between the first image and the second image, and determine the motion direction of the self-moving device based on the camera pose of the first image relative to the previous frame image.

[0013] The generation module is used to perform frame interpolation based on the first image and the direction of motion, and to perform frame interpolation between the first image and the second image to generate an interpolated image; and to generate the movement trajectory of the self-moving device based on the first image, the interpolated image, and the second image.

[0014] Thirdly, embodiments of this application provide a self-moving device, including: a memory and a processor, wherein the memory is used to store a computer program; and the processor is used to execute the method described in the first aspect or any embodiment of the first aspect when the computer program is invoked.

[0015] Fourthly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the method described in the first aspect or any of the embodiments of the first aspect.

[0016] Fifthly, embodiments of this application provide a computer program product that, when run on a self-moving device, causes the self-moving device to execute the motion tracking method of any one of the first aspects described above.

[0017] The technical solution provided in this application acquires a first image and a second image of adjacent frames during the movement of the self-moving device; when there is a jump between the first image and the second image, the previous frame image of the first image is acquired, and the movement direction of the self-moving device is determined according to the camera pose of the first image relative to the previous frame image; based on the first image and the movement direction, frame interpolation is performed between the first image and the second image to generate an interpolated frame image; in this way, the tracking from the first image to the second image can be indirectly achieved through the interpolated frame image, making up for the problem of frame loss. At the same time, the movement direction of the self-moving device can be predicted based on the camera pose between the images, and then the corresponding interpolated frame image is generated according to the movement direction of the self-moving device, thereby improving the accuracy of image tracking; in addition, the movement trajectory of the self-moving device is generated based on the first image, the interpolated frame image and the second image, so that the generated movement trajectory has continuity, thereby improving the stability of trajectory tracking. Attached Figure Description

[0018] Figure 1 A flowchart illustrating the motion tracking method provided in this application embodiment;

[0019] Figure 2 A schematic flowchart illustrating a method for generating a frame-interpolated image provided in an embodiment of this application;

[0020] Figure 3 This is a schematic diagram of the upsampling process provided in an embodiment of this application;

[0021] Figure 4A schematic flowchart illustrating a method for generating a frame-interpolated image provided in an embodiment of this application;

[0022] Figure 5 This is a schematic diagram of the downsampling process provided in an embodiment of this application;

[0023] Figure 6 A schematic flowchart illustrating a method for generating a frame-interpolated image provided in an embodiment of this application;

[0024] Figure 7 This is a schematic diagram of the structure of the motion tracking device provided in the embodiments of this application;

[0025] Figure 8 This is a schematic diagram of the structure of the self-moving device provided in an embodiment of this application. Detailed Implementation

[0026] The embodiments of this application are described below with reference to the accompanying drawings. The terminology used in the implementation section of this application is for explaining specific embodiments only and is not intended to limit the scope of this application.

[0027] Figure 1 This is a flowchart illustrating the motion tracking method provided in the embodiments of this application, as shown below. Figure 1 As shown, the motion tracking method provided in this application embodiment may include the following steps:

[0028] S101. Acquire the first and second images of adjacent frames during the movement of the self-moving device.

[0029] The self-moving device can be a device that includes self-movement assistance functions. These functions can be implemented via an in-vehicle terminal, and the corresponding self-moving device can be a vehicle equipped with that terminal. The self-moving device can also be a semi-self-moving device or a fully autonomous device. Examples include lawnmowers, robot vacuums, and robots with navigation capabilities.

[0030] Specifically, the self-moving device is equipped with an image acquisition device, which can acquire real-time images during the movement of the self-moving device.

[0031] The first image is the image captured by the mobile device at time t1, and the second image is the image captured by the mobile device at time t2. Time t1 and time t2 represent two adjacent image capture times on the time axis. Time t1 is earlier than time t2. Correspondingly, the first image and the second image are two adjacent frames.

[0032] S102. Determine whether there is a jump between the first image and the second image.

[0033] If yes, proceed to step S103; otherwise, proceed to step S106.

[0034] In some embodiments, timestamps can be used to determine whether there is a jump between the first image and the second image.

[0035] Specifically, the time interval between the timestamp of the first image and the timestamp of the second image can be determined; if the time interval is greater than a preset time threshold, it is determined that there is a jump between the first image and the second image.

[0036] For example, σt can be used to represent the time threshold. If the time interval (t2-t1) between the timestamp t1 of the first image and the timestamp t2 of the second image is ≤ σt, then it can be considered that there is no jump between the first image and the second image; if the time interval (t2-t1) between the timestamp t1 of the first image and the timestamp t2 of the second image is > σt, then it can be considered that there is a jump between the first image and the second image.

[0037] In some embodiments, optical flow tracing can also be used to determine whether there is a jump between the first image and the second image.

[0038] Specifically, optical flow tracing can be performed on the first image and the second image. If the tracing is successful, it is considered that there is no transition between the first image and the second image. If the optical flow tracing fails, it is considered that there is a transition between the first image and the second image.

[0039] By determining whether there is a jump between the first and second images, it is possible to determine whether frame interpolation is needed between the first and second images, which facilitates the implementation of different frame interpolation strategies based on different situations.

[0040] S103. Obtain the previous frame image of the first image, and determine the motion direction of the self-moving device based on the camera pose of the first image relative to the previous frame image.

[0041] The previous frame of the first image is the image captured by the mobile device at time t0 (i.e., the time before t1); the camera pose includes a rotation matrix and a translation vector.

[0042] If there is a jump between the first image and the second image, then frame interpolation is performed between the first image and the second image to indirectly achieve optical flow tracing from the first image to the second image through the interpolated image.

[0043] During frame interpolation, the interpolation strategy can be determined based on the motion direction of the mobile device to improve the accuracy of the interpolation results, thereby improving the accuracy of optical flow tracing.

[0044] Specifically, since the image acquisition device is mounted on the self-moving device, the direction of motion of the self-moving device can be approximated as the direction of motion of the image acquisition device. When determining the direction of motion of the image acquisition device, optical flow tracing can be performed on the first image and the previous frame image. Based on the optical flow tracing results, the camera pose of the first image relative to the previous frame image can be obtained. The camera pose is the position and orientation of the image acquisition device in space. The camera pose can be viewed as a transformation of the image acquisition device from its original position to its current position, and can be transformed using the translation vector and rotation moment in the camera pose.

[0045] In addition, optical flow tracing is performed continuously during the movement of the self-moving device. In the previous process of determining the movement trajectory of the self-moving device, the camera pose has been determined based on the optical flow tracing results between the first image and the previous frame image. Therefore, when determining the movement trajectory of the self-moving device this time, the camera pose of the first image determined last time relative to the previous frame image can also be directly read.

[0046] Then, the direction of camera motion, i.e. the direction of motion of the self-moving device at time t1, can be determined by analyzing the rotation matrix and translation vector.

[0047] The movement direction of the self-moving device can be classified in different ways. For example, it can be roughly divided into two types: forward and backward movement and turning movement; or it can be divided into three types: backward movement, forward movement, and turning movement, to improve the accuracy of subsequent frame interpolation results. This application embodiment will use this as an example for subsequent illustrative description. It is understood that the movement direction can also be divided into more types to further improve the accuracy of the frame interpolation results, and this application embodiment does not particularly limit this.

[0048] Specifically, at time t1, the camera's rotation matrix is ​​0, and the translation vector's value on the x-axis is decreasing compared to the previous time. Therefore, the image acquisition device at time t1 can be considered to be moving backward.

[0049] Specifically, at time t1, the camera's rotation matrix is ​​0, and the translation vector's value on the x-axis is increasing compared to the previous time. Therefore, the camera's motion direction at time t1 can be considered as forward motion.

[0050] Specifically, at time t1, the camera's translation vector remains unchanged, but the rotation matrix changes, indicating that the self-moving device's direction of motion at time t1 is turning.

[0051] S104. Based on the first image and the movement direction of the self-moving device, perform frame interpolation between the first image and the second image to generate an interpolated image.

[0052] After determining the motion direction of the mobile device, frame interpolation can be performed between the first image and the second image based on the first image and the motion direction of the mobile device to generate an interpolated image. Different interpolation strategies can be used for different motion directions. The interpolation strategies corresponding to the three motion directions mentioned above are described below.

[0053] For cases where the direction of motion is backward, the specific details can be determined according to... Figure 2 The method shown is used for frame interpolation. Figure 2 This is a schematic flowchart of a method for generating a frame-padding image provided in an embodiment of this application, as shown below. Figure 2 As shown, the method may include the following steps:

[0054] S201. When the direction of motion is backward, determine the upsampling ratio coefficient based on the pixel motion speed corresponding to the first image.

[0055] Due to relative motion, the direction of pixel movement in the image is opposite to the direction of movement of the image acquisition device (i.e., the direction of movement of the mobile device). When the direction of movement of the mobile device is backward, the direction of pixel movement is forward. In this case, the first image can be upsampled to determine the supplementary frame image.

[0056] Specifically, the upsampling ratio coefficient can be determined first based on the pixel motion speed of the first image, and then upsampling can be performed based on the upsampling ratio coefficient. The upsampling ratio coefficient can be determined by formula (1).

[0057] C1=F(V1) (1)

[0058] Where C1 represents the upsampling ratio coefficient, and V1 represents the pixel motion speed of the corresponding pixel in the first image when it moves forward. The pixel motion speed can be obtained by dividing the position change of the same pixel point between adjacent frames by the tracking time interval between image frames. For example, if the previous frame of the first image and the feature point matched by the first image are determined, and the position change s of the matched feature point is determined, and the tracking time interval from the previous frame of the first image to the first image is t, then the pixel motion speed of the first image is V1 = s / t. According to F(V1), the corresponding functional relationship between C1 and V1 can be C1 = V1 + at, where a represents acceleration and t represents time; or it can be other corresponding functional relationships, which are not particularly limited in this embodiment.

[0059] S202. Upsample the first image according to the upsampling ratio coefficient to obtain the first target image.

[0060] Upsampling can magnify an image. Specifically, upsampling methods such as interpolation can be used to interpolate the pixels in the first image according to the upsampling ratio coefficient to obtain the first target image.

[0061] S203. Using the center of the first image as the origin, crop the first target image according to the size of the first image to obtain a padded frame image.

[0062] The first target image obtained after upsampling is larger in size than the first image. In order to facilitate subsequent optical flow tracing, the first target image can be cropped according to the size of the first image to obtain a padded frame image.

[0063] Specifically, Figure 3 This is a schematic diagram of the upsampling process provided in an embodiment of this application, as shown below. Figure 3 As shown, the first target image can be cropped according to the size of the first image, with the center point of the first target image as the center point of the interpolated image, to obtain the interpolated image.

[0064] For cases where the direction of motion is forward, the specific details can be determined according to... Figure 4 The method shown is used for frame interpolation. Figure 4 This is a schematic flowchart of a method for generating a frame-padding image provided in an embodiment of this application, as shown below. Figure 4 As shown, the method may include the following steps:

[0065] S401. When the direction of motion is forward, determine the downsampling ratio coefficient based on the pixel motion speed corresponding to the first image.

[0066] As mentioned earlier, the direction of pixel movement in the image is opposite to the direction of movement of the mobile device. When the mobile device moves forward, the direction of pixel movement is backward. In this case, the first image can be downsampled to determine the interpolated frame image.

[0067] Specifically, the downsampling ratio coefficient can be determined first based on the pixel motion speed of the first image, and then downsampling can be performed based on the downsampling ratio coefficient. The upsampling ratio coefficient can be determined by formula (2).

[0068] C2=F(V2) (2)

[0069] Wherein, C2 represents the downsampling ratio coefficient, V2 represents the motion speed of the pixel corresponding to the first image when it moves backward, and F(V2) represents the functional relationship between C2 and V2, which can be C2 = V2 + at, or other corresponding functional relationships. This application does not impose any special limitations on this.

[0070] S402. The first image is downsampled according to the downsampling ratio coefficient to obtain the second target image.

[0071] Downsampling, also known as image downsampling, reduces the size of the original image. Specifically, it can be achieved by extracting feature points and other methods, which extract pixels from the first image according to a downsampling ratio to obtain the second target image.

[0072] S403. Generate a frame interpolation image based on the second target image.

[0073] The second target image obtained after downsampling is smaller than the first image. To facilitate subsequent optical flow tracing, the second target image can be padded according to the size of the first image to obtain a padded frame image. Of course, if the second target image and the first image are the same size, the size of the second target image is not adjusted, and the second target image is directly used as the padded frame image.

[0074] Specifically, Figure 5 This is a schematic diagram of the downsampling process provided in an embodiment of this application, such as... Figure 5 As shown, based on the size of the first image, the center point of the second target image can be used as the center point of the interpolated frame image to fill the second target image with pixels, thus obtaining the interpolated frame image. The pixel color can be gray, black, or other colors; this embodiment does not impose any particular limitation, and gray pixels are used as an example for illustration.

[0075] For cases where the direction of motion is turning, the specific details can be determined according to... Figure 6 The method shown is used for frame interpolation. Figure 6 This is a schematic flowchart of a method for generating a frame-padding image provided in an embodiment of this application, as shown below. Figure 6 As shown, the method may include the following steps when the direction of motion is steering motion:

[0076] S601. Determine the displacement of the target feature point from the previous frame to the first image, and the pixel movement speed of the target feature point in the first image.

[0077] The target feature points can be corner pixels or edge pixels of the target object in the previous frame image. The target feature points can be feature points of static objects in the image. Since the position of static objects changes very little in physical space, the feature points of static objects can be extracted as target feature points. This application does not make any special limitation on this.

[0078] Specifically, the displacement of the target feature point from the previous frame to the first image can be determined based on the pixel coordinates of the target feature point in the previous frame and the first image; then, the pixel movement speed of the target feature point in the first image can be determined based on the displacement and the time interval between the previous frame and the first image.

[0079] S602. Based on the displacement and pixel movement speed, perform a translation operation on the first image to generate a third target image.

[0080] Specifically, the uniform velocity model can be used to perform a translation operation on the first image based on the displacement and pixel movement speed to generate the third target image.

[0081] S603, Perform optical flow tracing on the third target image and the second image.

[0082] If tracing fails, proceed to step S604; if tracing succeeds, proceed to step S605.

[0083] Specifically, optical flow tracing can be performed on the third target image and the second image. If the tracing is successful, it means that the motion amplitude between the first image and the second image is not large, and only one frame interpolation operation is needed to determine the motion trajectory of the mobile device from the first image to the second image. At this time, the third target image can be determined as the interpolated image. If the tracing fails, it means that the motion amplitude between the first image and the second image is too large. Even with only one frame interpolation, it is still impossible to know the motion trajectory of the mobile device from the first image to the second image. At this time, multiple frame interpolation operations can be performed.

[0084] S604. Use the first image as the new previous frame image, use the third target image as the new first image, and return to step S601.

[0085] Specifically, when performing the frame interpolation operation again, the first image can be used as the new previous frame image, the third target image can be determined as the new first image, and then the execution of step S601 is returned to determine the displacement of the target feature points from the new previous frame image to the new first image, as well as the pixel movement speed of the target feature points in the new first image, until the tracking is successful.

[0086] S605. The generated third target images are determined as interpolated frame images.

[0087] If optical flow tracing is successfully performed on the third target image and the second image, then each of the generated third target images can be identified as a frame interpolation image.

[0088] If one frame interpolation is performed, one third target image can be obtained; if multiple frame interpolations are performed, multiple third target images can be obtained.

[0089] S105. Generate the movement trajectory of the self-moving device based on the first image, the interpolated frame image, and the second image.

[0090] Specifically, a first optical flow tracing can be performed on the first image and the interpolated frame image to obtain the first camera pose of the target image relative to the first image; then, a second optical flow tracing can be performed on the interpolated frame image and the second image to obtain the second camera pose of the second image relative to the interpolated frame image; finally, the movement trajectory of the self-moving device can be generated based on the first camera pose and the second camera pose.

[0091] Correspondingly, if there are n (n≥2) frame interpolations, the first optical flow tracing can be performed on the first image and the first interpolated frame image to obtain the first camera pose information of the first interpolated frame image relative to the first image; then, the second optical flow tracing can be performed on the first interpolated frame image and the second interpolated frame image to obtain the second camera pose information of the second interpolated frame image relative to the first interpolated frame image, and so on, performing optical flow tracing on each adjacent interpolated frame image to obtain the camera pose between each adjacent interpolated frame image; then, the (n+1)th optical flow tracing can be performed on the nth interpolated frame image and the second image to obtain the (n+1)th camera pose of the second image relative to the last interpolated frame image; finally, based on the transformation matrix from the camera coordinate system to the self-moving device coordinate system, each relative pose is transformed into the self-moving device coordinate system to obtain the movement trajectory of the self-moving device.

[0092] S106. Generate the movement trajectory of the self-moving device based on the first image and the second image.

[0093] Specifically, optical flow tracing is performed on the first and second images to obtain the camera pose, and the movement trajectory of the self-moving device is obtained through the camera pose information.

[0094] The technical solution provided in this application acquires a first image and a second image of adjacent frames during the movement of the self-moving device; when there is a jump between the first image and the second image, the previous frame image of the first image is acquired, and the movement direction of the self-moving device is determined according to the camera pose of the first image relative to the previous frame image; based on the first image and the movement direction, frame interpolation is performed between the first image and the second image to generate an interpolated frame image; in this way, the tracking from the first image to the second image can be indirectly achieved through the interpolated frame image, making up for the problem of frame loss. At the same time, the movement direction of the self-moving device can be predicted based on the camera pose between the images, and then the corresponding interpolated frame image is generated according to the movement direction of the self-moving device, thereby improving the accuracy of image tracking; in addition, the movement trajectory of the self-moving device is generated based on the first image, the interpolated frame image and the second image, so that the generated movement trajectory has continuity, thereby improving the stability of trajectory tracking.

[0095] Those skilled in the art will understand that the above embodiments are exemplary and not intended to limit this application. Where possible, the execution order of one or more of the above steps can be adjusted, or they can be selectively combined to obtain one or more other embodiments. Those skilled in the art can arbitrarily select and combine the above steps as needed, and all those that do not depart from the essence of this application fall within the protection scope of this application.

[0096] Based on the same inventive concept, as an implementation of the above method, this application provides a motion tracking device. This device embodiment corresponds to the aforementioned method embodiment. For ease of reading, this device embodiment will not repeat the details of the aforementioned method embodiment one by one, but it should be clear that the device in this embodiment can implement all the contents of the aforementioned method embodiment.

[0097] Figure 7 This is a schematic diagram of the structure of the motion tracking device provided in the embodiments of this application, such as... Figure 7 As shown, the apparatus provided in this embodiment includes:

[0098] The acquisition module 701 is used to acquire the first and second images of adjacent frames during the movement of the self-moving device;

[0099] The determination module 702 is used to determine the motion direction of the self-moving device based on the camera pose of the first image relative to the previous frame image when there is a jump between the first image and the second image.

[0100] The generation module 703 is used to perform frame interpolation based on the first image and the direction of motion, perform frame interpolation between the first image and the second image, and generate an interpolated image; and generate the movement trajectory of the self-moving device based on the first image, the interpolated image, and the second image.

[0101] As an optional implementation of this application, the determining module 702 is further configured to:

[0102] Determine the time interval between the timestamp of the first image and the timestamp of the second image;

[0103] If the time interval is greater than a preset time threshold, it is determined that there is a jump between the first image and the second image.

[0104] As an optional implementation of this application, the determining module 702 is further configured to:

[0105] If optical flow tracing of the first and second images fails, it is determined that there is a jump between the first and second images.

[0106] As an optional implementation of this application, the generation module 703 is specifically used for:

[0107] When the direction of motion is backward, the upsampling ratio coefficient is determined based on the pixel motion speed corresponding to the first image;

[0108] The first image is upsampled according to the upsampling ratio coefficient to obtain the first target image;

[0109] Using the center of the first image as the origin, the first target image is cropped according to the size of the first image to obtain the padded frame image.

[0110] As an optional implementation of this application, the generation module 703 is specifically used for:

[0111] When the direction of motion is forward, the downsampling ratio coefficient is determined based on the pixel motion speed corresponding to the first image;

[0112] The first image is downsampled according to the downsampling ratio coefficient to obtain the second target image;

[0113] Generate a padded frame image based on the second target image.

[0114] As an optional implementation of this application, the generation module 703 is specifically used for:

[0115] When the direction of motion is turning motion, determine the displacement of the target feature point from the previous frame to the first image, and the pixel motion speed of the target feature point in the first image.

[0116] Based on the displacement and pixel motion speed, the first image is translated to generate the third target image;

[0117] If the second image cannot be tracked based on the third target image, the third target image is used as the new first image, the first image is used as the new previous frame image, and the process returns to determine the displacement of the target feature point from the previous frame image to the first image, as well as the pixel motion speed of the target feature point in the first image, until the tracking is successful, and each generated third target image is determined as a supplementary frame image.

[0118] As an optional implementation of this application, the generation module 703 is specifically used for:

[0119] Perform first optical flow tracing on the first image and the interpolated frame image to obtain the first camera pose of the target image relative to the first image;

[0120] Perform second optical flow tracing on the interpolated frame image and the second image to obtain the second camera pose of the second image relative to the interpolated frame image;

[0121] The movement trajectory of the self-moving device is generated based on the poses of the first and second cameras.

[0122] The motion tracking device provided in this embodiment can execute the above method embodiment, and its implementation principle and technical effect are similar, so it will not be described again here.

[0123] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0124] Based on the same inventive concept, this application also provides a self-moving device. Figure 8 This is a schematic diagram of the structure of the self-moving device provided in the embodiments of this application, such as... Figure 8 As shown, the self-moving device provided in this embodiment includes: a memory 810 and a processor 820. The memory 810 is used to store computer programs; the processor 820 is used to execute the method of the above method embodiment when the computer program is invoked.

[0125] The self-moving device provided in this embodiment can execute the above method embodiment. Its implementation principle and technical effect are similar, and will not be described again here.

[0126] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the method described in the above method embodiments.

[0127] This application also provides a computer program product that, when run on a self-moving device, enables the self-moving device to implement the method described in the above method embodiments.

[0128] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in or transmitted through a computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, or magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

[0129] Those skilled in the art will understand that implementing all or part of the processes in the above embodiments can be accomplished by a computer program instructing related hardware. This program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The aforementioned storage medium can include various media capable of storing program code, such as ROM or random access memory (RAM), magnetic disks, or optical disks.

[0130] The naming or numbering of steps in this application does not mean that the steps in the method flow must be executed in the time / logical order indicated by the naming or numbering. The execution order of the named or numbered process steps can be changed according to the technical purpose to be achieved, as long as the same or similar technical effect can be achieved.

[0131] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0132] In the embodiments provided in this application, it should be understood that the disclosed apparatus / device and method can be implemented in other ways. For example, the apparatus / device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0133] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0134] In the description of this application, unless otherwise stated, " / " indicates that the objects before and after are in an "or" relationship. For example, A / B can mean A or B. "And / or" in this application is merely a description of the relationship between the related objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist simultaneously, and B exists alone. A and B can be singular or plural.

[0135] Furthermore, in the description of this application, unless otherwise stated, "multiple" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0136] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0137] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein.

[0138] References to "one embodiment" or "some embodiments" in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized.

[0139] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A motion tracking method, characterized in that, include: Acquire the first and second images of adjacent frames during the movement of the mobile device; In the case of a jump between the first image and the second image, the previous frame of the first image is obtained, and the movement direction of the self-moving device is determined according to the camera pose of the first image relative to the previous frame. Based on the first image and the direction of motion, frame interpolation is performed between the first image and the second image to generate an interpolated image; Based on the first image, the interpolated image, and the second image, a movement trajectory of the mobile device is generated; The step of performing frame interpolation between the first image and the second image based on the first image and the motion direction to generate an interpolated image includes: When the direction of motion is backward, an upsampling ratio coefficient is determined based on the pixel motion speed corresponding to the first image; the first image is upsampled according to the upsampling ratio coefficient to obtain a first target image; the first target image is cropped with the center of the first image as the origin and according to the size of the first image to obtain the interpolated frame image; When the direction of motion is forward, a downsampling ratio coefficient is determined based on the pixel motion speed corresponding to the first image; the first image is downsampled according to the downsampling ratio coefficient to obtain a second target image; and the interpolated frame image is generated based on the second target image. When the direction of motion is a turning motion: determine the displacement of the target feature point from the previous frame image to the first image, and the pixel motion speed of the target feature point in the first image; perform a translation operation on the first image based on the displacement and pixel motion speed to generate a third target image; perform optical flow tracing on the third target image and the second image; if optical flow tracing on the third target image and the second image fails, then take the first image as the new previous frame image, take the third target image as the new first image, and return to execute the step of determining the displacement of the target feature point from the previous frame image to the first image, and the pixel motion speed of the target feature point in the first image, until optical flow tracing on the third target image and the second image is successful, and determine each of the generated third target images as the supplementary frame image.

2. The method according to claim 1, characterized in that, The method further includes: Determine the time interval between the timestamp of the first image and the timestamp of the second image; If the time interval is greater than a preset time threshold, it is determined that there is a jump between the first image and the second image.

3. The method according to claim 1, characterized in that, The method further includes: If optical flow tracing of the first image and the second image fails, it is determined that there is a jump between the first image and the second image.

4. The method according to any one of claims 1-3, characterized in that, The step of generating the movement trajectory of the mobile device based on the first image, the interpolated image, and the second image includes: Perform first optical flow tracing on the first image and the interpolated frame image to obtain the first camera pose of the target image relative to the first image; Perform second optical flow tracing on the interpolated image and the second image to obtain the second camera pose of the second image relative to the interpolated image; The movement trajectory of the self-moving device is generated based on the first camera pose and the second camera pose.

5. A motion tracking device, characterized in that, include: The acquisition module is used to acquire a first image and a second image of adjacent frames during the movement of the self-moving device; and to acquire the previous frame image of the first image when there is a jump between the first image and the second image. The determining module is used to determine the motion direction of the self-moving device based on the camera pose of the first image relative to the previous frame image. The generation module is used to perform frame interpolation based on the first image and the direction of motion, and to perform frame interpolation between the first image and the second image to generate an interpolated image; and to generate the movement trajectory of the self-moving device based on the first image, the interpolated image and the second image. The step of performing frame interpolation between the first image and the second image based on the first image and the motion direction to generate an interpolated image includes: When the direction of motion is backward, an upsampling ratio coefficient is determined based on the pixel motion speed corresponding to the first image; the first image is upsampled according to the upsampling ratio coefficient to obtain a first target image; the first target image is cropped with the center of the first image as the origin and according to the size of the first image to obtain the interpolated frame image; When the direction of motion is forward, a downsampling ratio coefficient is determined based on the pixel motion speed corresponding to the first image; the first image is downsampled according to the downsampling ratio coefficient to obtain a second target image; and the interpolated frame image is generated based on the second target image. When the direction of motion is a turning motion: determine the displacement of the target feature point from the previous frame image to the first image, and the pixel motion speed of the target feature point in the first image; perform a translation operation on the first image based on the displacement and pixel motion speed to generate a third target image; perform optical flow tracing on the third target image and the second image; if optical flow tracing on the third target image and the second image fails, then take the first image as the new previous frame image, take the third target image as the new first image, and return to execute the step of determining the displacement of the target feature point from the previous frame image to the first image, and the pixel motion speed of the target feature point in the first image, until optical flow tracing on the third target image and the second image is successful, and determine each of the generated third target images as the supplementary frame image.

6. A self-moving device, characterized in that, include: A memory and a processor, the memory being used to store a computer program; the processor being used to execute the method as described in any one of claims 1-4 when the computer program is invoked.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-4.

Images