Track determination method and device, storage medium and computer program product

Abstract

The embodiment of the invention provides a track determination method and device, a storage medium and a computer program product, and the method comprises the steps: obtaining a first image of continuous frames, and determining first information corresponding to each frame of image through a first model; wherein the first information comprises first position information and first descriptors corresponding to the N first feature points, and the first descriptors are used for quantifying the feature information corresponding to the first feature points; determining L pairs of matching feature points based on second position information of the second feature points, a second descriptor and a second model; screening wrong matching points in the L pairs of matching feature points to obtain M pairs of matching feature points, and determining first pose changes corresponding to the two continuous frames of images based on the M pairs of matching feature points; and generating the motion track of the camera based on the first pose change, thereby improving the positioning precision.

Description

Technical Field

[0001] The present application relates to the field of simultaneous visual mapping and positioning (SLAM) technology, and in particular to a trajectory determination method, device, storage medium and computer program product. Background Art

[0002] In the field of vision-based SLAM, the front-end processing of visual data generally requires restoring the camera's posture information at different sampling times through continuous image frames. There are currently two ways to restore the camera's posture information at different sampling times. One way is through feature matching, and the other way is through the direct method and the use of epipolar geometry to restore the camera's relative posture between the previous and next frames. This is the premise of vision-based SLAM technology. Only when the relative posture of the camera between the previous and next frames is restored can synchronous positioning and mapping be performed.

[0003] However, the above two methods are less effective in restoring the camera's pose information at different sampling times in low-texture and violent motion scenes. There may be mismatching of feature points, resulting in poor or even erroneous pose estimation results, which in turn reduces the accuracy of positioning. Summary of the invention

[0004] The embodiments of the present application provide a trajectory determination method, device, storage medium and computer program product, which can ensure the reliability of restoring camera posture changes and improve positioning accuracy.

[0005] The technical solution of the embodiment of the present application is implemented as follows:

[0006] In a first aspect, an embodiment of the present application provides a trajectory determination method, the method comprising:

[0007] Acquire a first image of continuous frames, and determine first information corresponding to each frame of the image through a first model; wherein the first information includes first position information and a first descriptor corresponding to N first feature points, the first descriptor is used to quantify feature information corresponding to the first feature points, the first model is at least used to extract the first feature points of the first image, and N is a positive integer;

[0008] Determine L pairs of matching feature points based on second position information of the second feature point, the second descriptor and the second model; wherein the second feature point includes feature points of any two consecutive frames of images in the first feature point set, the first feature point set includes feature points after filtering the N first feature points, and the second model is at least used for matching the input feature points;

[0009] Screening the wrong matching points in the L pairs of matching feature points to obtain M pairs of matching feature points, and determining the first position change corresponding to the two consecutive frames of images based on the M pairs of matching feature points;

[0010] A motion trajectory of the camera is generated based on the first posture change.

[0011] In a second aspect, an embodiment of the present application provides a trajectory determination device, the trajectory determination device comprising: an acquisition unit, a determination unit, a screening unit, and a generation unit; wherein:

[0012] The acquisition unit is used to acquire the first image of the continuous frames;

[0013] The determining unit is used to determine the first information corresponding to each frame of the image through a first model; wherein the first information includes first position information and a first descriptor corresponding to N first feature points, the first descriptor is used to quantify the feature information corresponding to the first feature points, the first model is used at least to extract the first feature points of the first image, and N is a positive integer; and determine L pairs of matching feature points based on the second position information of the second feature points, the second descriptor and the second model; wherein the second feature points include feature points of any two consecutive frames of images in the first feature point set, the first feature point set includes feature points after filtering the N first feature points, and the second model is used at least to match the input feature points;

[0014] The screening unit is used to screen the erroneous matching points in the L pairs of matching feature points to obtain M pairs of matching feature points;

[0015] The determining unit is further used to determine the first pose change corresponding to the two consecutive frames of images based on the M pairs of matching feature points;

[0016] The generating unit is used to generate a motion trajectory of the camera based on the first posture change.

[0017] In a third aspect, an embodiment of the present application provides a trajectory determination device, the trajectory determination device comprising: a processor and a memory; wherein:

[0018] The memory is used to store a computer program that can be run on the processor;

[0019] The processor is used to execute the trajectory determination method as described above when running the computer program.

[0020] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a computer program code is stored. When the computer program code is executed by a computer, the trajectory determination method as described above is implemented.

[0021] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the trajectory determination method as described above.

[0022] The embodiments of the present application provide a trajectory determination method, device, storage medium and computer program product, the method comprising: acquiring a first image of continuous frames, and determining first information corresponding to each frame of the image through a first model; wherein the first information comprises first position information and a first descriptor corresponding to N first feature points, the first descriptor is used to quantify feature information corresponding to the first feature points, the first model is at least used to extract the first feature points of the first image, N is a positive integer; based on the second position information of the second feature points, the second descriptor and the second model, L pairs of matching feature points are determined; wherein the second feature points include feature points of any two consecutive frames of images in the first feature point set, the first feature point set includes feature points after filtering the N first feature points, and the second model is at least used to match the input feature points; filtering the erroneous matching points in the L pairs of matching feature points to obtain M pairs of matching feature points, and determining the first position change corresponding to the two consecutive frames of images based on the M pairs of matching feature points; generating the motion trajectory of the camera based on the first position change. It can be seen that after determining the first position information and the first descriptor of the N first feature points corresponding to each frame of the image through the first model, L pairs of matching feature points can be determined based on the second position information, the second descriptor and the second model of the second feature points, the second feature points include the feature points of any two consecutive frames of images in the first feature point set, and the first feature point set includes the feature points after the N first feature points are screened, that is, the present application can screen the first feature points to obtain higher quality second feature points, thereby improving the matching accuracy of determining the L pairs of matching feature points, and then the erroneous matching points in the L pairs of matching feature points can be screened to obtain M pairs of matching feature points, and the first position change corresponding to the two consecutive frames of images can be determined based on the M pairs of matching feature points; finally, the motion trajectory of the camera can be generated based on the first position change, that is, the present application can further eliminate the erroneous matching points in the L pairs of matching feature points, thereby ensuring the reliability of the subsequent use of the eliminated M pairs of matching feature points to generate the motion trajectory of the camera, thereby improving the accuracy of positioning. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 The trajectory determination method proposed in the embodiment of the present application is shown in FIG. Figure 1 ;

[0024] Figure 2 A schematic diagram of the first model structure proposed in the embodiment of the present application;

[0025] Figure 3A schematic diagram of the second model structure proposed in the embodiment of the present application;

[0026] Figure 4 This is a schematic diagram of IMU pre-integration proposed in an embodiment of the present application;

[0027] Figure 5 A schematic diagram of the actual motion trajectory proposed in the embodiment of the present application;

[0028] Figure 6 The trajectory determination method proposed in the embodiment of the present application is shown in FIG. Figure 2 ;

[0029] Figure 7 Schematic diagram of the structure of the trajectory determination device proposed in the embodiment of the present application Figure 1 ;

[0030] Figure 8 Schematic diagram of the structure of the trajectory determination device proposed in the embodiment of the present application Figure 2 . DETAILED DESCRIPTION

[0031] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. It is understood that the specific embodiments described herein are only used to explain the related applications, rather than to limit the applications. It should also be noted that, for ease of description, only the parts related to the related applications are shown in the drawings.

[0032] In the field of vision-based SLAM, the front-end processing of visual data generally requires restoring the camera's posture information at different sampling times through continuous image frames. There are currently two ways to restore the camera's posture information at different sampling times. One way is through feature matching, and the other way is through the direct method and the use of epipolar geometry to restore the camera's relative posture between the previous and next frames. This is the premise of vision-based SLAM technology. Only when the relative posture of the camera between the previous and next frames is restored can synchronous positioning and mapping be performed.

[0033] The feature matching method extracts a set of sparse significant image features in each image, uses feature descriptors to match them in continuous image frames, and then restores the camera motion and scene structure through epipolar geometry. Compared with the feature matching method, the direct method does not extract feature points, but directly estimates the camera motion and environmental structure by minimizing the error metric of image pixel intensity based on the grayscale invariance assumption. The local intensity gradient size and direction are used in the optimization process, which can act directly on the image or on the feature points (semi-direct). These methods have poor effects in some complex situations, such as low texture, outdoor scenes with drastic lighting changes, etc., resulting in poor or even wrong pose estimation results, which leads to operational errors of the entire visual SLAM system.

[0034] Although there are many current visual front-end processing methods, there are relatively few monocular visual front-end processing methods that fully integrate deep learning models. Some deep learning-based visual front-end processing methods have also been proposed in related technologies. Although some deep learning networks have been used and visual front-end processing methods have been explored to a certain extent, these methods add deep learning networks to some links of the front-end processing, or use traditional methods to fuse with deep learning networks. Although they have achieved certain results in some environments, there are still some problems such as mismatching of some features, which leads to poor or even wrong pose estimation results, and thus reduces the accuracy of positioning.

[0035] In order to solve the problem of low positioning accuracy in current visual front-end processing methods, embodiments of the present application provide a trajectory determination method, device, storage medium and computer program product, the method comprising: acquiring a first image of continuous frames, and determining first information corresponding to each frame of the image through a first model; wherein the first information comprises first position information and a first descriptor corresponding to N first feature points, the first descriptor is used to quantify feature information corresponding to the first feature points, the first model is at least used to extract the first feature points of the first image, N is a positive integer; based on the second position information, the second descriptor and the second model of the second feature points, L pairs of matching feature points are determined; wherein the second feature points include feature points of any two consecutive frames of images in the first feature point set, the first feature point set includes feature points after filtering the N first feature points, and the second model is at least used to match the input feature points; filtering the erroneous matching points in the L pairs of matching feature points to obtain M pairs of matching feature points, and determining the first position change corresponding to the two consecutive frames of images based on the M pairs of matching feature points; generating the motion trajectory of the camera based on the first position change. It can be seen that after determining the first position information and the first descriptor of the N first feature points corresponding to each frame of the image through the first model, L pairs of matching feature points can be determined based on the second position information, the second descriptor and the second model of the second feature points, the second feature points include the feature points of any two consecutive frames of images in the first feature point set, and the first feature point set includes the feature points after the N first feature points are screened, that is, the present application can screen the first feature points to obtain higher quality second feature points, thereby improving the matching accuracy of determining the L pairs of matching feature points, and then the erroneous matching points in the L pairs of matching feature points can be screened to obtain M pairs of matching feature points, and the first position change corresponding to the two consecutive frames of images can be determined based on the M pairs of matching feature points; finally, the motion trajectory of the camera can be generated based on the first position change, that is, the present application can further eliminate the erroneous matching points in the L pairs of matching feature points, thereby ensuring the reliability of the subsequent use of the eliminated M pairs of matching feature points to generate the motion trajectory of the camera, thereby improving the accuracy of positioning.

[0036] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0037] The present application embodiment provides a trajectory determination method, Figure 1 The trajectory determination method proposed in the embodiment of the present application is shown in FIG. Figure 1 ,like Figure 1 As shown, the trajectory determination method may include the following steps:

[0038] Step 101, obtain a first image of continuous frames, and determine the first information corresponding to each frame of the image through a first model; wherein the first information includes first position information and a first descriptor corresponding to N first feature points, the first descriptor is used to quantify the feature information corresponding to the first feature points, the first model is at least used to extract the first feature points of the first image, and N is a positive integer.

[0039] In an embodiment of the present application, the trajectory determination device may acquire first images of consecutive frames, and determine first information corresponding to each frame of the image through a first model.

[0040] It should be noted that, in the embodiments of the present application, the first model may be a SuperPoint model, and the present application does not specifically limit the type of the first model.

[0041] It should be noted that, in an embodiment of the present application, the first information may include the first position information and the first descriptor of N first feature points corresponding to each frame of image, and may also include other parameter information of the first feature points. The present application does not specifically limit the amount and type of information included in the first information.

[0042] It should be noted that, in the embodiments of the present application, the first position information may include the two-dimensional (2D) position of each first feature point, and the present application does not specifically limit the type of information included in the first position information.

[0043] It should be noted that, in an embodiment of the present application, the first descriptor can be used to quantify feature information corresponding to the first feature point. For example, the feature information may be an appearance feature. The present application does not specifically limit the type of feature information.

[0044] It should be noted that, in the embodiments of the present application, Figure 2 This is a schematic diagram of the first model structure proposed in the embodiment of the present application, such as Figure 2 As shown, the first model includes at least one first encoder and at least two first decoders. The present application does not specifically limit the number and type of modules included in the first model.

[0045] It should be noted that, in an embodiment of the present application, when the trajectory determination device determines the first information corresponding to each frame of the image through the first model, it can perform convolution processing on each frame of the image based on the first encoder to obtain a second image; then, the second image can be respectively extracted through two first decoders to obtain the first position information, first descriptor and first score value of N first feature points corresponding to each frame of the image; wherein the first score value includes the quality score corresponding to the first feature point.

[0046] Step 102: determine L pairs of matching feature points based on the second position information of the second feature point, the second descriptor and the second model; wherein the second feature point includes feature points of any two consecutive frames of images in the first feature point set, the first feature point set includes feature points after N first feature points are screened, and the second model is at least used to match the input feature points.

[0047] In an embodiment of the present application, after acquiring the first image of continuous frames and determining the first information corresponding to each frame of the image through the first model, the trajectory determination device can determine L pairs of matching feature points based on the second position information of the second feature point, the second descriptor and the second model.

[0048] It should be noted that, in an embodiment of the present application, the second feature points include feature points of any two consecutive frames of images in the first feature point set, and the first feature point set includes feature points after screening N first feature points, that is, the second feature points are part of the first feature points, and the present application does not specifically limit the number of second feature points.

[0049] It should be noted that, in the embodiment of the present application, the second descriptor is a descriptor corresponding to the second feature point.

[0050] It should be noted that, in the embodiments of the present application, the second model may be a SuperGlue model, and the present application does not specifically limit the model type of the second model.

[0051] It should be noted that, in an embodiment of the present application, when the trajectory determination device determines L pairs of matching feature points based on the second position information, the second descriptor and the second model of the second feature point, it can determine the first distance distribution value based on the distance value between each first feature point and the adjacent feature point; then the first feature points can be screened based on the first distance distribution value and the first score value to obtain the screened feature points; then, the feature points corresponding to any two consecutive frames of images can be selected from the screened feature points as the second feature points; finally, the second position information and the second descriptor of the second feature point can be input into the second model to generate L pairs of matching feature points.

[0052] It should be noted that, in the embodiment of the present application, when the trajectory determination device determines the first distance distribution value based on the distance value between each first feature point and the adjacent feature point, it can be calculated by the following formula (1).

[0053]

[0054] Wherein, i represents each first feature point, j represents the feature points adjacent to each first feature point, and n represents the total number of feature points.

[0055] It should be noted that, in the embodiment of the present application, the trajectory determination device determines the first distance distribution value; then the first feature point can be screened based on the first distance distribution value and the first score value to obtain the screened feature points.

[0056] Furthermore, in an embodiment of the present application, when the trajectory determination device screens the first feature points based on the first distance distribution value and the first score value to obtain the screened feature points, the second score value can be determined based on the first distance distribution value, the first score value and the preset proportional coefficient; then the second score values ​​can be sorted according to a preset sorting rule, and the first S third score values ​​can be selected; wherein the second score value includes the third score value; and then the screened feature points can be generated based on the feature points corresponding to the third score values.

[0057] It should be noted that, in the embodiment of the present application, when the trajectory determination device determines the second score value based on the first distance distribution value, the first score value and the preset proportional coefficient, it can be calculated by the following formula (2).

[0058] Point i =score i +r·dist i (2)

[0059] Among them, score i represents the first fraction value, r represents the proportional coefficient, dist i Represents the first distance distribution value.

[0060] It should be noted that, in an embodiment of the present application, after determining the second score value, the trajectory determination device can sort the second score value according to a preset sorting rule and select the first S third score values. For example, the second score values ​​can be sorted in order from large to small, and then the first 100 third score values ​​can be selected, and then the filtered feature points can be generated based on the feature points corresponding to the third score values. The present application does not specifically limit the size of S.

[0061] That is to say, in an embodiment of the present application, the present application selects the feature points corresponding to the first S third score values ​​as the filtered feature points, so that the output feature points can have a more reasonable spatial allocation while meeting higher quality, further improving the accuracy of the odometer system.

[0062] Furthermore, in an embodiment of the present application, after obtaining the filtered feature points, the trajectory determination device can select feature points corresponding to any two consecutive frames of images from the filtered feature points as second feature points, and then input the second position information and the second descriptor of the second feature points into the second model to generate L pairs of matching feature points.

[0063] It should be noted that, in the embodiments of the present application, Figure 3 A schematic diagram of the second model structure proposed in the embodiment of the present application is shown in FIG. Figure 3 As shown, the second model includes at least a first network, and the first network includes a feature point encoder, a self-attention layer, and a cross-attention layer. This application does not specifically limit the number and type of networks included in the second model.

[0064] It should be noted that, in an embodiment of the present application, when the trajectory determination device inputs the second position information and the second descriptor of the second feature point into the second model to generate L pairs of matching feature points, the second position information and the second descriptor of the second feature point can be input into the feature point encoder to obtain a feature descriptor corresponding to two consecutive frames of images; wherein the feature descriptor is used to describe the local features of the image; then the feature descriptor can be input into the self-attention layer and the cross-attention layer to obtain a first matrix; wherein the first matrix represents the matching score of the second feature point; and then the first matrix can be optimized by a first preset algorithm to obtain a second matrix; wherein the second matrix represents the optimal matching degree of the second feature point; finally, L pairs of matching feature points can be obtained based on the element values ​​corresponding to the second matrix.

[0065] It should be noted that, in the embodiments of the present application, the first preset algorithm may be a Sinkhorn algorithm, and the present application does not specifically limit the algorithm type of the first preset algorithm.

[0066] Exemplarily, in an embodiment of the present application, the trajectory determination device can input the second position information and the second descriptor of the second feature point into the feature point encoder, and can obtain the encoded matching symbol as a feature descriptor for subsequent feature matching, and then the feature descriptor can be input into the self-attention layer and the cross-attention layer to obtain a first matrix, and then the first matrix can be optimized and iteratively processed through the Sinkhorn algorithm to obtain a second matrix, so that L pairs of matching feature points can be obtained based on the element values ​​corresponding to the second matrix.

[0067] Step 103: Screen the wrong matching points in the L pairs of matching feature points to obtain M pairs of matching feature points, and determine the first position change corresponding to two consecutive frames of images based on the M pairs of matching feature points.

[0068] In an embodiment of the present application, after determining L pairs of matching feature points based on the second position information of the second feature point, the second descriptor and the second model, the trajectory determination device can filter out the erroneous matching points in the L pairs of matching feature points to obtain M pairs of matching feature points, and determine the first pose change corresponding to two consecutive frames of images based on the M pairs of matching feature points.

[0069] It should be noted that, in an embodiment of the present application, when the trajectory determination device screens out the erroneous matching points in L pairs of matching feature points and obtains M pairs of matching feature points, the model parameters of the third model can be determined based on the M pairs of matching feature points; then, for each pair of matching feature points in the M pairs of matching feature points, it can be determined whether the matching feature points meet the preset requirements based on the model parameters, one of the feature points in the matching feature points, and another feature point; and then, M pairs of matching feature points can be determined based on the matching feature points that meet the requirements.

[0070] It should be noted that, in the embodiments of the present application, the third model may be a random sample consensus algorithm (RANSAC) model, and the present application does not specifically limit the type of the third model.

[0071] It should be noted that in an embodiment of the present application, when the trajectory determination device determines the model parameters of the third model based on M pairs of matching feature points, it can determine the second posture change corresponding to two consecutive frames of images based on any pair of matching feature points among the M pairs of matching feature points and the epipolar geometry principle; and then the model parameters can be determined based on the second posture change.

[0072] Exemplarily, in an embodiment of the present application, the trajectory determination device can randomly extract any pair of matching feature points from M pairs of matching feature points, and use the extracted any pair of matching feature points to form a set as potential internal points, and then use the epipolar geometry principle to calculate the second pose change corresponding to two consecutive frames of images, and use the second pose change as a model parameter.

[0073] It should be noted that, in an embodiment of the present application, after determining the model parameters of the third model, the trajectory determination device can judge, for each pair of M pairs of matching feature points, whether the matching feature points meet the preset requirements based on the model parameters, one of the matching feature points, and another feature point.

[0074] It should be noted that, in an embodiment of the present application, when the trajectory determination device determines whether the matching feature point meets the preset requirements based on the model parameters, one of the matching feature points and another feature point, it can first determine whether the distance between one of the matching feature points and the other feature point after the conversion processing with the model parameters is less than or equal to the first preset threshold; if the distance is less than or equal to the first preset threshold, it is determined that the matching feature point meets the preset requirements; or, if the distance is greater than the first preset threshold, it is determined that the matching feature point does not meet the preset requirements. The present application does not specifically limit the size of the first preset threshold.

[0075] It should be noted that, in the embodiment of the present application, when the trajectory determination device determines whether the distance between one of the feature points in the matching feature points and another feature point after the conversion processing with the model parameters is less than or equal to the first preset threshold, the distance between one of the feature points in the matching feature points and another feature point after the conversion processing with the model parameters is calculated by the following formula (3).

[0076]

[0077] in, represents the model parameters, f i,k-1 Represents one of the feature points in the matching feature points, f i,k Indicates another feature point in the matched feature point.

[0078] It should be noted that, in the embodiment of the present application, after determining that the matching feature points meet the preset requirements, the trajectory determination device can determine M pairs of matching feature points based on the matching feature points that meet the requirements.

[0079] It should be noted that, in an embodiment of the present application, when the trajectory determination device determines M pairs of matching feature points based on matching feature points that meet the requirements, it can determine a first ratio based on the matching feature points that meet the requirements and L pairs of matching feature points; and further, when the first ratio is greater than or equal to a second preset threshold, it can determine M pairs of matching feature points based on the matching feature points that meet the requirements.

[0080] It should be noted that, in the embodiments of the present application, the second preset threshold value may be any value between 0 and 1, and the present application does not specifically limit the size of the second preset threshold value.

[0081] Exemplarily, in an embodiment of the present application, the second preset threshold is 0.75. Assuming that there are 30 pairs of matching feature points that meet the requirements and the number of L pairs of matching feature points is 40, the first proportion value can be obtained based on the following formula (4). The first proportion value is 0.75, that is, the first proportion value is equal to the second preset threshold value, and then M pairs of matching feature points can be determined based on the matching feature points that meet the requirements.

[0082] ratio=m / L (4)

[0083] Among them, ratio represents the first ratio value, and m represents the matching feature points that meet the requirements.

[0084] Exemplarily, in an embodiment of the present application, the trajectory determination device may re-estimate the first pose change (model parameters) between two frames of images based on M pairs of matching feature points and using the epipolar geometry principle.

[0085] It should be noted that, in the embodiment of the present application, if the first proportion value is less than the second preset threshold, the above step 103 can be repeated until the optimal model parameters (the first pose change) are obtained.

[0086] That is to say, in the embodiment of the present application, the trajectory determination device can determine whether the matching feature points meet the preset requirements, and after screening out the matching feature points that meet the requirements, it can further determine the first ratio based on the matching feature points that meet the requirements and L pairs of matching feature points; when the first ratio is greater than or equal to the second preset threshold, M pairs of matching feature points are determined based on the matching feature points that meet the requirements, thereby ensuring the reliability of the subsequent use of M pairs of matching feature points to restore the camera posture change, thereby improving the positioning accuracy. That is, after being processed by the random sample consistency algorithm, the erroneous matching points in the matching results are further eliminated, ensuring the reliability of the subsequent use of the matching results to restore the camera posture change, and improving the positioning accuracy.

[0087] Step 104: Generate a motion trajectory of the camera based on the first pose change.

[0088] In an embodiment of the present application, after the trajectory determination device screens out the erroneous matching points in L pairs of matching feature points to obtain M pairs of matching feature points, and determines the first pose change corresponding to two consecutive frames of images based on the M pairs of matching feature points, the camera's motion trajectory can be generated based on the first pose change.

[0089] It should be noted that, in an embodiment of the present application, when the trajectory determination device generates the motion trajectory of the camera based on the first pose change, it can optimize the initial pose of the camera based on the first pose change and the sliding window optimization module to obtain the optimized camera pose, and generate the motion trajectory based on the optimized camera pose; wherein, the initial pose of the camera is determined based on the first pose change.

[0090] It should be noted that in an embodiment of the present application, when the trajectory determination device optimizes the initial pose of the camera based on the first pose change and the sliding window optimization module to obtain the optimized camera pose, the collected inertial measurement unit (IMU) measurement data can be pre-integrated to obtain the IMU pre-integration result; then, the initial pose of the camera can be optimized based on the status information of the simultaneous mapping and positioning (SLAM) system, the IMU pre-integration result, the first pose change and the sliding window optimization module to obtain the optimized camera pose.

[0091] It should be noted that, in the embodiments of the present application, the SLAM system may include one or more of the first model, the second model, and the sliding window optimization module, and the present application does not specifically limit the content included in the SLAM system.

[0092] It should be noted that, in the embodiments of the present application, Figure 4 This is a schematic diagram of the IMU pre-integration proposed in the embodiment of the present application, such as Figure 4 As shown, all IMUs between the Kth frame and the K+1th frame are integrated to obtain the position, velocity, and rotation of the K+1th frame as the initial value of the visual estimation; if the IMU is integrated in the world coordinate system, the integral term includes the instantaneous rotation matrix of the body coordinate system with respect to the world coordinate system. However, when the initial pose is optimized, the rotation matrix of the body coordinate system relative to the world coordinate system at the key frame moment will change, then the IMU needs to be re-integrated, and pre-integration is to avoid repeated integration. IMU pre-integration changes the reference coordinate system to the body coordinate system of the previous frame, thereby integrating the relative motion between the two frames.

[0093] Exemplarily, in an embodiment of the present application, the trajectory determination device can input IMU measurement data into the SLAM system and obtain the IMU pre-integration result based on the following formulas (5)-(7).

[0094]

[0095] Where Δv ij Represents the pre-integrated speed measurement value, ΔP ij Represents the pre-integrated displacement measurement, ΔR ij represents the pre-integrated rotation measurement value, The bias of the gyroscope, represents the noise of the gyroscope, represents the bias of the accelerometer, represents the noise of the accelerometer, represents the true value of the accelerometer, Represents the true value of the gyroscope, ΔR ik Represents the rotation matrix of the IMU relative to the world coordinate system at the i-th moment, Δv ik Indicates the velocity of the IMU relative to the world coordinate system at the i-th moment.

[0096] It should be noted that in an embodiment of the present application, when the trajectory determination device optimizes the initial pose of the camera based on the state information of the SLAM system, the IMU pre-integration result, the first pose change and the sliding window optimization module, it can align the first pose change based on the IMU pre-integration result when the state information representation of the SLAM system is not initialized, and obtain the aligned third pose change; then the initial pose of the camera can be optimized based on the third pose change and the sliding window optimization module to obtain the optimized camera pose; or, when the state information representation of the SLAM system has been initialized, the initial pose of the camera can be optimized based on the IMU pre-integration result, the first pose change and the sliding window optimization module to obtain the optimized camera pose.

[0097] Exemplarily, in an embodiment of the present application, when the state information representation of the SLAM system is not initialized, the trajectory determination device can align the first pose change based on the IMU pre-integration result and calculate the gyroscope zero bias. Subsequently, the initial speed, gravity direction and scale information of the system can be restored based on the gyroscope zero bias to complete the initialization of the system. Furthermore, the initial pose of the camera can be optimized based on the aligned third pose change and the sliding window optimization module to obtain the optimized camera pose.

[0098] Exemplarily, in an embodiment of the present application, when the state information representation of the SLAM system has been initialized, the trajectory determination device uses the zero bias state obtained by the system state estimation to correct the IMU measurement data, so that the IMU pre-integration result between the two frames of images can be calculated based on the corrected IMU measurement data and the above formulas (5)-(7), and then the initial posture of the camera can be optimized based on the IMU pre-integration result, the first posture change and the sliding window optimization module; the optimized camera posture is obtained.

[0099] That is to say, in an embodiment of the present application, when the state information representation of the SLAM system has been initialized, the trajectory determination device can use the zero bias state obtained by the system state estimation to correct the IMU measurement data, that is, the IMU measurement data can be integrated and corrected and then pre-integrated, and finally the sliding window optimization algorithm is used to optimize the initial posture of the camera, thereby effectively ensuring the reliability and accuracy of the camera posture.

[0100] It should be noted that, in the embodiment of the present application, the SLAM system is obtained by training the initial SLAM system based on the first loss function, that is, the state estimation of the camera and the IMU is optimized by minimizing the first loss function, thereby realizing the fusion of visual and inertial information and improving the accuracy and robustness of the SLAM system; wherein, the first loss function includes the marginalized prior error, the IMU pre-integration error and the visual image reprojection error; wherein, the formula of the first loss function is shown in the following formula (8).

[0101]

[0102] Among them, ‖r p -H p X‖ 2 represents the marginalized prior error, represents the IMU pre-integration error, Represents the visual image reprojection error.

[0103] It should be noted that in an embodiment of the present application, the trajectory determination device can marginalize the key frame pose of the exit sliding window, maintain the marginalized data in the form of Schur's complement, and convert the historical frame data of the exit window into the system's prior information through Schur's complement, thereby reducing the computational load while ensuring that the historical data participates in the optimization process.

[0104] Furthermore, in an embodiment of the present application, the trajectory determination device optimizes the initial posture of the camera based on the IMU pre-integration result, the first posture change and the sliding window optimization module, and can obtain an optimized camera posture, and then can generate an actual motion trajectory based on the optimized camera posture, optimized IMU posture, etc.

[0105] It should be noted that, in the embodiments of the present application, Figure 5 This is a schematic diagram of the actual motion trajectory proposed in the embodiment of the present application, such as Figure 5 As shown in the figure, the black solid line (VINS-SuperGlue) is the trajectory of this application, which is better than VINS-Mono and OpenVINS in trajectory accuracy. Compared with VINS-Mono, the trajectory accuracy is improved by more than 100%. The root mean square error (RMSE) of the algorithm of this application and some visual inertial navigation algorithms on public data sets is shown in Table 1 below. The RMSE of this application (Ours) is the smallest.

[0106] Table 1

[0107] OpenVINS VINS-Mono Ours RMSE(m) 0.096 0.227 0.081

[0108] In summary, the trajectory determination device can select the feature points corresponding to the first S third score values ​​as the feature points after screening, so that the output feature points can have a relatively reasonable spatial allocation while meeting the high quality, further improving the accuracy of the odometer system, and then the feature points corresponding to any two consecutive frames of images can be selected from the filtered feature points as the second feature points, and then the second position information and the second descriptor of the second feature point can be input into the second model to generate L pairs of matching feature points, and then the wrong matching points in the L pairs of matching feature points can be screened to obtain M pairs of matching feature points, and the first position change corresponding to the two consecutive frames of images can be determined based on the M pairs of matching feature points, thereby ensuring the reliability of the subsequent use of M pairs of matching feature points to restore the camera pose change, thereby improving the positioning accuracy. That is, after being processed by the random sample consistency algorithm, the wrong matching points in the matching results are further eliminated, ensuring the reliability of the subsequent use of the matching results to restore the camera pose change, and improving the positioning accuracy; and the present application can integrate and correct the IMU measurement data and then pre-integrate, and finally use the sliding window optimization algorithm to optimize the initial pose of the camera, effectively ensuring the reliability and accuracy of the camera pose.

[0109] An embodiment of the present application provides a trajectory determination method, which includes: acquiring a first image of continuous frames, and determining first information corresponding to each frame of the image through a first model; wherein the first information includes first position information and a first descriptor corresponding to N first feature points, the first descriptor is used to quantify the feature information corresponding to the first feature point, and the first model is at least used to extract the first feature point of the first image, where N is a positive integer; determining L pairs of matching feature points based on the second position information of the second feature point, the second descriptor and the second model; wherein the second feature point includes feature points of any two consecutive frames of images in the first feature point set, the first feature point set includes feature points after filtering the N first feature points, and the second model is at least used to match the input feature points; filtering the erroneous matching points in the L pairs of matching feature points to obtain M pairs of matching feature points, and determining the first position change corresponding to the two consecutive frames of images based on the M pairs of matching feature points; and generating the motion trajectory of the camera based on the first position change. It can be seen that after determining the first position information and the first descriptor of the N first feature points corresponding to each frame of the image through the first model, L pairs of matching feature points can be determined based on the second position information, the second descriptor and the second model of the second feature points, the second feature points include the feature points of any two consecutive frames of images in the first feature point set, and the first feature point set includes the feature points after the N first feature points are screened, that is, the present application can screen the first feature points to obtain higher quality second feature points, thereby improving the matching accuracy of determining the L pairs of matching feature points, and then the erroneous matching points in the L pairs of matching feature points can be screened to obtain M pairs of matching feature points, and the first position change corresponding to the two consecutive frames of images can be determined based on the M pairs of matching feature points; finally, the motion trajectory of the camera can be generated based on the first position change, that is, the present application can further eliminate the erroneous matching points in the L pairs of matching feature points, thereby ensuring the reliability of the subsequent use of the eliminated M pairs of matching feature points to generate the motion trajectory of the camera, thereby improving the accuracy of positioning.

[0110] Based on the above embodiments, another embodiment of the present application provides a trajectory determination method, which can be applied to a visual front end composed of a deep learning method, extracting image feature points (i.e., first feature points) through the SuperPoint (i.e., first model) method, and then using the SuperGlue (i.e., second model) method for feature tracking to restore the relative pose of the camera between consecutive image frames (i.e., the first pose change), and then using the back-end sliding window optimization to obtain the final pose (i.e., optimized camera pose) output. The monocular vision front-end processing method does not directly use the matching results of SuperGlue. In order to improve the quality of feature pairs used for pose estimation, the present application improves the quality of feature matching results by retaining high-quality matching results and further eliminating errors in the structure.

[0111] It should be noted that in an embodiment of the present application, the trajectory determination device can use the SuperPoint (i.e., the first model) deep learning network model to extract feature points (i.e., the first feature points) from the image, and use the SuperGlue (i.e., the second model) method to perform feature matching to obtain the relative pose between consecutive frames of the image, and then use the back-end nonlinear optimization method to obtain the final pose output, thereby improving the stability and accuracy of the visual front end in complex environments (such as weak textures and strong lighting changes).

[0112] It should be noted that in an embodiment of the present application, the trajectory determination device can improve the quality of the feature matching results by retaining high-quality matching results and further eliminating errors in the structure based on a combination of a deep learning visual front end and an inertial measurement unit (IMU), and perform visual front-end preprocessing, IMU measurement pre-integration, monocular vision and IMU initialization, back-end sliding window nonlinear optimization, loop detection optimization, and finally output the posture changes of the entire system.

[0113] It should be noted that in the embodiments of the present application, the present application performs feature extraction and matching based on the improved SuperPoint and SuperGlue networks, uses a fully convolutional network to extract image feature points, adds spatial distribution restrictions to the feature extraction results, and uses an attention neural network to perform feature matching after jointly encoding the image feature points and feature vectors in the matching stage. Then, the Sinkhorn algorithm (the first preset algorithm) is used to solve the matching results. The present application further designs an error removal algorithm based on the feature matching of the network to remove errors from the feature matching results, effectively ensuring the reliability and accuracy of the feature matching results, and on the basis of the improved method, realizes a monocular visual inertial navigation system to realize the positioning and navigation of the robot.

[0114] It should be noted that, in the embodiments of the present application, Figure 6 The trajectory determination method proposed in the embodiment of the present application is shown in FIG. Figure 2 ,like Figure 6As shown, the trajectory determination method may include the following steps: step 1: input a monocular visual image, perform image distortion correction and grayscale, and output the image after distortion correction and grayscale (i.e., the first image) to facilitate subsequent further processing of the image; step 2: use SuperPoint (i.e., the first model) to extract image feature points from the monocular visual image (i.e., the first image) output in the previous step, use a fully convolutional encoder (the first encoder) to perform convolution encoding on the input image, and then use two decoders (the first decoder) to extract the 2D position of the feature point (the first position information of the first feature point) and the descriptor (i.e., the first descriptor), and finally output the 2D position and descriptor of the feature point and the feature point The quality score (i.e., the first score value) is used to improve the strategy for outputting feature points. Different from the original method that directly outputs feature points according to the score, the present application here also considers the distribution of feature points, and calculates the first distance distribution value based on the distance between the feature point and the surrounding feature points (the smaller the distance, the higher the score, and the score is calculated based on the average distance). The proportional coefficient is set for the first distance distribution value and then summed with the quality score to obtain the final score of the feature point (i.e., the second score value). The 100 feature extraction results with the highest output scores (i.e., the third score value) are used for subsequent matching. In this way, the output feature points (i.e., the filtered feature points) can have a more reasonable spatial allocation while meeting higher quality, thereby further improving the accuracy of the odometer system.

[0115] That is to say, in an embodiment of the present application, the present application selects the feature points corresponding to the first S third score values ​​as the filtered feature points, so that the output feature points can have a more reasonable spatial allocation while meeting higher quality, further improving the accuracy of the odometer system.

[0116] It should be noted that in an embodiment of the present application, the trajectory determination method further includes the following steps: Step 3: Input the result of the feature extraction in the previous step, use SuperGlue (i.e., the second model) to perform feature matching between the two frames of images, and obtain a preliminary feature matching result (i.e., L pairs of matching feature points). Specifically, input the 2D position (i.e., the second descriptor) and descriptor (the second descriptor) of the feature points (i.e., the second feature points) of the two frames of images to the feature point encoder, obtain the encoded matching symbol as the feature descriptor for subsequent feature matching, and then input the feature descriptors in the two consecutive image frames into an attention neural network that alternates between self-attention and cross-attention for matching processing to obtain a feature matching score matrix (i.e., the first matrix), optimize and iterate the characteristics of the score matrix (i.e., the first matrix) through the Sinkhorn algorithm (i.e., the first preset algorithm), obtain the matching score matrix (i.e., the second matrix), and obtain the matching feature point pairs (i.e., L pairs of matching feature points) according to the element values ​​of the corresponding positions of the matrix (i.e., the second matrix) and output them.

[0117] It should be noted that, in the embodiments of the present application, the first preset algorithm may be a Sinkhorn algorithm, and the present application does not specifically limit the algorithm type of the first preset algorithm.

[0118] Exemplarily, in an embodiment of the present application, the trajectory determination device can input the second position information and the second descriptor of the second feature point into the feature point encoder, and can obtain the encoded matching symbol as a feature descriptor for subsequent feature matching, and then the feature descriptor can be input into the self-attention layer and the cross-attention layer to obtain a first matrix, and then the first matrix can be optimized and iteratively processed through the Sinkhorn algorithm to obtain a second matrix, so that L pairs of matching feature points can be obtained based on the element values ​​corresponding to the second matrix.

[0119] It should be noted that, in an embodiment of the present application, the trajectory determination method further includes the following steps: Step 4: Input the preliminary matching feature point pairs (L pairs of matching feature points) of the previous step, use the random sample consensus algorithm to select the matching results, eliminate the matching results of poor quality or even errors, and obtain the pose change between the two frames of images (i.e., the first pose change). The random sample consensus algorithm (RANSAC) is a general term for a class of methods, and the idea of ​​random sample consistency can be used to set specific algorithms according to different usage environments and target requirements; the specific algorithm steps may include the following: 1. Assuming that there are L pairs of matched feature point pairs, randomly select l feature point pairs to form a set as potential internal points, and use the antipodal geometry principle to calculate the pose transformation between the two frames of images as the model parameters of the algorithm. 2. Model parameters calculated in the previous step Use the above formula (3) to calculate the distance between each feature point after model parameter conversion and the corresponding feature point of the next frame as the basis for judging whether the feature point pair meets the model parameters, and count the number of feature point pairs that meet the model parameters m. 3. If the result ratio ratio (i.e., the first proportion value) calculated in the previous step exceeds the second preset threshold (i.e., the number of point pairs that meet the model parameters reaches a certain ratio), use all the inliers (M pairs of matching feature points) to re-estimate the model parameters (i.e. the first pose change). If the result calculated in 2. does not meet the threshold requirement, repeat steps 1. to 3. to obtain the optimal model parameters.

[0120] That is to say, in the embodiment of the present application, the trajectory determination device can determine whether the matching feature points meet the preset requirements, and after screening out the matching feature points that meet the requirements, it can further determine the first proportion value based on the matching feature points that meet the requirements and L pairs of matching feature points; when the first proportion value is greater than or equal to the second preset threshold, M pairs of matching feature points are determined based on the matching feature points that meet the requirements, thereby ensuring the reliability of the subsequent use of M pairs of matching feature points to restore the camera pose change, thereby improving the positioning accuracy. That is, after being processed by the random sample consistency algorithm, the erroneous matching points in the matching results are further eliminated, ensuring the reliability of the subsequent use of the matching results to restore the camera pose change, improving the positioning accuracy, further enhancing the loop detection rate under large-scale motion, improving the comprehensive performance of the monocular visual inertial navigation system in complex environments, and providing more accurate and robust navigation location services.

[0121] It should be noted that, in the embodiment of the present application, the trajectory determination method also includes the following steps: step 5: proceed to the next step according to the specific state of the system, if the system is not initialized, proceed to step 6, if the system has been successfully initialized, proceed to step 7; wherein, step 6: the trajectory determination device inputs the IMU measurement data, uses the above formulas (5)-(7) to directly pre-integrate the three-axis accelerometer and three-axis gyroscope data between the two frames of images involved in the initialization, uses the result of the integral calculation (IMU pre-integration result) to align with the image posture change (i.e., the first posture change) obtained in step 4, calculates the gyroscope zero bias, and then brings the gyroscope zero bias into the calculation to restore the initial speed and gravity direction. The system is initialized by obtaining the direction and scale information of the system; Step 7: Input the IMU measurement data, use the zero bias state obtained by the system state estimation to correct the IMU data, and then use the above (5)-(7) formulas to integrate the three-axis accelerometer and three-axis gyroscope data between the two frames of images, calculate and output the IMU pre-integration result and the corresponding covariance between the two frames of images; Step 8: Input the IMU pre-integration result output by step 7 and the visual feature (first pose change) matching result output by step 4, run the back-end sliding window optimization, and construct the sliding window optimization loss function. The back-end system optimization loss function (i.e., the first loss function) is shown in the above formula (8).

[0122] It should be noted that in an embodiment of the present application, when the trajectory determination device optimizes the initial pose of the camera based on the state information of the SLAM system, the IMU pre-integration result, the first pose change and the sliding window optimization module, it can align the first pose change based on the IMU pre-integration result when the state information representation of the SLAM system is not initialized, and obtain the aligned third pose change; then the initial pose of the camera can be optimized based on the third pose change and the sliding window optimization module to obtain the optimized camera pose; or, when the state information representation of the SLAM system has been initialized, the initial pose of the camera can be optimized based on the IMU pre-integration result, the first pose change and the sliding window optimization module to obtain the optimized camera pose.

[0123] Exemplarily, in an embodiment of the present application, when the state information representation of the SLAM system is not initialized, the trajectory determination device can align the first pose change based on the IMU pre-integration result and calculate the gyroscope zero bias. Subsequently, the initial speed, gravity direction and scale information of the system can be restored based on the gyroscope zero bias to complete the initialization of the system. Furthermore, the initial pose of the camera can be optimized based on the aligned third pose change and the sliding window optimization module to obtain the optimized camera pose.

[0124] Exemplarily, in an embodiment of the present application, when the state information representation of the SLAM system has been initialized, the trajectory determination device uses the zero bias state obtained by the system state estimation to correct the IMU measurement data, so that the IMU pre-integration result between the two frames of images can be calculated based on the corrected IMU measurement data and the above formulas (5)-(7), and then the initial posture of the camera can be optimized based on the IMU pre-integration result, the first posture change and the sliding window optimization module; the optimized camera posture is obtained.

[0125] That is to say, in an embodiment of the present application, when the state information representation of the SLAM system has been initialized, the trajectory determination device can use the zero bias state obtained by the system state estimation to correct the IMU measurement data, that is, the IMU measurement data can be integrated and corrected and then pre-integrated, and finally the sliding window optimization algorithm is used to optimize the initial posture of the camera, thereby effectively ensuring the reliability and accuracy of the camera posture.

[0126] It should be noted that, in the embodiment of the present application, the SLAM system is obtained by training the initial SLAM system based on the first loss function, that is, the state estimation of the camera and the IMU is optimized by minimizing the first loss function, thereby realizing the fusion of visual and inertial information and improving the accuracy and robustness of the SLAM system; wherein, the first loss function includes the marginalized prior error, the IMU pre-integration error and the visual image reprojection error; wherein, the formula of the first loss function is as shown in the above formula (8).

[0127] It should be noted that in an embodiment of the present application, the trajectory determination device can marginalize the key frame pose of the exit sliding window, maintain the marginalized data in the form of Schur's complement, and convert the historical frame data of the exit window into the system's prior information through Schur's complement, thereby reducing the computational load while ensuring that the historical data participates in the optimization process.

[0128] Furthermore, in an embodiment of the present application, the trajectory determination device optimizes the initial posture of the camera based on the IMU pre-integration result, the first posture change and the sliding window optimization module, and can obtain an optimized camera posture, and then can generate an actual motion trajectory based on the optimized camera posture, optimized IMU posture, etc.

[0129] It should be noted that, in the embodiments of the present application, Figure 5 As shown in the figure, the black solid line is the trajectory of this application, which is better than VINS-Mono and OpenVINS in trajectory accuracy; compared with VINS-Mono, the trajectory accuracy is improved by more than 100%. The RMSE of the algorithm of this application and some visual inertial navigation algorithms on the public data set are shown in Table 1 above, and the RMSE of this application (Ours) is the smallest.

[0130] In summary, the trajectory determination device can select the feature points corresponding to the first S third score values ​​as the feature points after screening, so that the output feature points can have a relatively reasonable spatial allocation while meeting the high quality, further improving the accuracy of the odometer system, and then the feature points corresponding to any two consecutive frames of images can be selected from the filtered feature points as the second feature points, and then the second position information and the second descriptor of the second feature point can be input into the second model to generate L pairs of matching feature points, and then the wrong matching points in the L pairs of matching feature points can be screened to obtain M pairs of matching feature points, and the first position change corresponding to the two consecutive frames of images can be determined based on the M pairs of matching feature points, thereby ensuring the reliability of the subsequent use of M pairs of matching feature points to restore the camera pose change, thereby improving the positioning accuracy. That is, after being processed by the random sample consistency algorithm, the wrong matching points in the matching results are further eliminated, ensuring the reliability of the subsequent use of the matching results to restore the camera pose change, and improving the positioning accuracy; and the present application can integrate and correct the IMU measurement data and then pre-integrate, and finally use the sliding window optimization algorithm to optimize the initial pose of the camera, effectively ensuring the reliability and accuracy of the camera pose.

[0131] An embodiment of the present application provides a trajectory determination method, which includes: acquiring a first image of continuous frames, and determining first information corresponding to each frame of the image through a first model; wherein the first information includes first position information and a first descriptor corresponding to N first feature points, the first descriptor is used to quantify the feature information corresponding to the first feature point, and the first model is at least used to extract the first feature point of the first image, where N is a positive integer; determining L pairs of matching feature points based on the second position information of the second feature point, the second descriptor and the second model; wherein the second feature point includes feature points of any two consecutive frames of images in the first feature point set, the first feature point set includes feature points after filtering the N first feature points, and the second model is at least used to match the input feature points; filtering the erroneous matching points in the L pairs of matching feature points to obtain M pairs of matching feature points, and determining the first position change corresponding to the two consecutive frames of images based on the M pairs of matching feature points; and generating the motion trajectory of the camera based on the first position change. It can be seen that after determining the first position information and the first descriptor of the N first feature points corresponding to each frame of the image through the first model, L pairs of matching feature points can be determined based on the second position information, the second descriptor and the second model of the second feature point. The second feature point is the feature point of any two consecutive frames of images after the first feature point is screened, that is, the present application can screen the first feature point to obtain a higher quality second feature point, thereby improving the matching accuracy of determining the L pairs of matching feature points, and then the erroneous matching points in the L pairs of matching feature points can be screened to obtain M pairs of matching feature points, and the first position change corresponding to the two consecutive frames of images can be determined based on the M pairs of matching feature points; finally, the motion trajectory of the camera can be generated based on the first position change, that is, the present application can further eliminate the erroneous matching points in the L pairs of matching feature points, thereby ensuring the reliability of the subsequent use of the eliminated M pairs of matching feature points to generate the motion trajectory of the camera, thereby improving the accuracy of positioning.

[0132] Based on the above embodiments, the present application provides a trajectory determination device. Figure 7 Schematic diagram of the structure of the trajectory determination device Figure 1 ,like Figure 7 As shown, the trajectory determination device 10 includes: an acquisition unit 11, a determination unit 12, a screening unit 13 and a generation unit 14; wherein,

[0133] The acquisition unit 11 is used to acquire a first image of a continuous frame;

[0134] The determining unit 12 is used to determine the first information corresponding to each frame of the image through a first model; wherein the first information includes first position information and a first descriptor corresponding to N first feature points, the first descriptor is used to quantify the feature information corresponding to the first feature points, the first model is used at least to extract the first feature points of the first image, and N is a positive integer; and determine L pairs of matching feature points based on the second position information of the second feature points, the second descriptor and the second model; wherein the second feature points include feature points of any two consecutive frames of images in the first feature point set, the first feature point set includes feature points after filtering the N first feature points, and the second model is used at least to match the input feature points;

[0135] The screening unit 13 is used to screen the wrong matching points in the L pairs of matching feature points to obtain M pairs of matching feature points;

[0136] The determining unit 12 is further configured to determine the first pose change corresponding to the two consecutive frames of images based on the M pairs of matching feature points;

[0137] The generating unit 14 is configured to generate a motion trajectory of the camera based on the first posture change.

[0138] In the embodiments of the present application, further, Figure 8 Schematic diagram of the structure of the trajectory determination device Figure 2 ,like Figure 8 As shown, the trajectory determination device 10 proposed in the embodiment of the present application may also include a processor 15, a memory 16 storing executable instructions of the processor 15, and further, the device 10 may also include a communication interface 17, and a bus 18 for connecting the processor 15, the memory 16 and the communication interface 17.

[0139] In the embodiment of the present application, the processor 15 can be at least one of an application-specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field programmable gate array (FPGA), a central processing unit (CPU), a controller, a microcontroller, and a microprocessor. It can be understood that for different devices, the electronic device used to implement the function of the processor can also be other, and the embodiment of the present application is not specifically limited. The device 10 can also include a memory 16, which can be connected to the processor 15, wherein the memory 16 is used to store executable program code, the program code includes computer operation instructions, and the memory 16 may include a high-speed RAM memory, and may also include a non-volatile memory, for example, at least two disk memories.

[0140] In the embodiment of the present application, the bus 18 is used to connect the communication interface 17, the processor 15 and the memory 16, and the mutual communication between these devices.

[0141] In the embodiment of the present application, the memory 16 is used to store instructions and data.

[0142] Further, in an embodiment of the present application, the above-mentioned processor 15 is used to obtain a first image of continuous frames, and determine the first information corresponding to each frame of the image through a first model; wherein the first information includes first position information and a first descriptor corresponding to N first feature points, the first descriptor is used to quantify the feature information corresponding to the first feature point, and the first model is at least used to extract the first feature point of the first image, and N is a positive integer; based on the second position information, the second descriptor and the second model of the second feature point, L pairs of matching feature points are determined; wherein the second feature point includes feature points of any two consecutive frames of images in the first feature point set, the first feature point set includes feature points after filtering the N first feature points, and the second model is at least used to match the input feature points; the erroneous matching points in the L pairs of matching feature points are filtered to obtain M pairs of matching feature points, and the first pose change corresponding to the two consecutive frames of images is determined based on the M pairs of matching feature points; the motion trajectory of the camera is generated based on the first pose change.

[0143] In practical applications, the memory 16 may be a volatile memory, such as a random access memory (RAM); or a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk (HDD) or a solid-state drive (SSD); or a combination of the above types of memory, and provide instructions and data to the processor 15.

[0144] An embodiment of the present application provides a trajectory determination device, which obtains a first image of continuous frames and determines first information corresponding to each frame of the image through a first model; wherein the first information includes first position information and a first descriptor corresponding to N first feature points, the first descriptor is used to quantify feature information corresponding to the first feature points, and the first model is at least used to extract the first feature points of the first image, where N is a positive integer; based on the second position information of the second feature points, the second descriptor and the second model, L pairs of matching feature points are determined; wherein the second feature points include feature points of any two consecutive frames of images in the first feature point set, the first feature point set includes feature points after filtering the N first feature points, and the second model is at least used to match the input feature points; the erroneous matching points in the L pairs of matching feature points are filtered to obtain M pairs of matching feature points, and the first position change corresponding to the two consecutive frames of images is determined based on the M pairs of matching feature points; and the motion trajectory of the camera is generated based on the first position change. It can be seen that after determining the first position information and the first descriptor of the N first feature points corresponding to each frame of the image through the first model, L pairs of matching feature points can be determined based on the second position information, the second descriptor and the second model of the second feature point. The second feature point is the feature point of any two consecutive frames of images after the first feature point is screened, that is, the present application can screen the first feature point to obtain a higher quality second feature point, thereby improving the matching accuracy of determining the L pairs of matching feature points, and then the erroneous matching points in the L pairs of matching feature points can be screened to obtain M pairs of matching feature points, and the first position change corresponding to the two consecutive frames of images can be determined based on the M pairs of matching feature points; finally, the motion trajectory of the camera can be generated based on the first position change, that is, the present application can further eliminate the erroneous matching points in the L pairs of matching feature points, thereby ensuring the reliability of the subsequent use of the eliminated M pairs of matching feature points to generate the motion trajectory of the camera, thereby improving the accuracy of positioning.

[0145] An embodiment of the present application provides a computer-readable storage medium having a program stored thereon, and when the program is executed by a processor, the trajectory determination method as described above is implemented.

[0146] Specifically, the program instructions corresponding to a trajectory determination method in this embodiment can be stored in a storage medium such as a CD, a hard disk, a USB flash drive, etc. When the program instructions corresponding to a trajectory determination method in the storage medium are read or executed by an electronic device, the following steps are included:

[0147] Acquire a first image of continuous frames, and determine first information corresponding to each frame of the image through a first model; wherein the first information includes first position information and a first descriptor corresponding to N first feature points, the first descriptor is used to quantify feature information corresponding to the first feature points, the first model is at least used to extract the first feature points of the first image, and N is a positive integer;

[0148] Determine L pairs of matching feature points based on second position information of the second feature point, the second descriptor and the second model; wherein the second feature point includes feature points of any two consecutive frames of images after filtering the first feature point, and the second model is at least used for matching the input feature points;

[0149] Screening the wrong matching points in the L pairs of matching feature points to obtain M pairs of matching feature points, and determining the first position change corresponding to the two consecutive frames of images based on the M pairs of matching feature points;

[0150] A motion trajectory of the camera is generated based on the first posture change.

[0151] The embodiment of the present application further provides a computer program product, including a computer program, which can be executed by the processor 15 of the trajectory determination device 10 to complete the steps of any of the aforementioned methods.

[0152] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of hardware embodiments, software embodiments, or embodiments in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) that contain computer-usable program code.

[0153] The present application is described with reference to implementation flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the process in the flowchart. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0154] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which is implemented in the implementation flow diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0155] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for executing the steps in the flowchart. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0156] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application.

Claims

1. A trajectory determination method, characterized in that: The method comprises: Acquire a first image of continuous frames, and determine first information corresponding to each frame of the image through a first model; wherein the first information includes first position information and a first descriptor corresponding to N first feature points, the first descriptor is used to quantify feature information corresponding to the first feature points, the first model is at least used to extract the first feature points of the first image, and N is a positive integer; Determine L pairs of matching feature points based on second position information of the second feature point, the second descriptor and the second model; wherein the second feature point includes feature points of any two consecutive frames of images in the first feature point set, the first feature point set includes feature points after filtering the N first feature points, and the second model is at least used for matching the input feature points; Screening the wrong matching points in the L pairs of matching feature points to obtain M pairs of matching feature points, and determining the first position change corresponding to the two consecutive frames of images based on the M pairs of matching feature points; A motion trajectory of the camera is generated based on the first posture change.

2. The method according to claim 1, characterized in that The first model includes at least one first encoder and at least two first decoders, and determining the first information corresponding to each frame of the image by using the first model includes: Performing convolution processing on each frame of the image based on the first encoder to obtain a second image; The second image is subjected to feature extraction by two of the first decoders to obtain first position information, first descriptors and first score values ​​of N first feature points corresponding to each frame of the image; wherein the first score value includes a quality score corresponding to the first feature point.

3. The method according to claim 1, characterized in that: The determining L pairs of matching feature points based on the second position information of the second feature point, the second descriptor, and the second model includes: Determine a first distance distribution value based on the distance value between each of the first feature points and an adjacent feature point; Performing screening processing on the first feature points based on the first distance distribution value and the first score value to obtain screened feature points; Selecting feature points corresponding to any two consecutive frames of images from the filtered feature points as second feature points; The second position information of the second feature point and the second descriptor are input into the second model to generate the L pairs of matching feature points.

4. The method according to claim 3, characterized in that The filtering the first feature points based on the first distance distribution value and the first score value to obtain the filtered feature points includes: Determine a second score value based on the first distance distribution value, the first score value, and a preset proportionality coefficient; Sorting the second score values ​​according to a preset sorting rule, and selecting the first S third score values; wherein the second score value includes the third score value, and S is a positive integer; The filtered feature points are generated based on the feature points corresponding to the third score value.

5. The method according to claim 3, characterized in that: The second model at least includes a first network, the first network includes a feature point encoder, a self-attention layer, and a cross-attention layer, and the second position information of the second feature point and the second descriptor are input into the second model to generate the L pairs of matching feature points, including: Inputting the second position information of the second feature point and the second descriptor into the feature point encoder to obtain a feature descriptor corresponding to the two consecutive frames of images; wherein the feature descriptor is used to describe the local features of the image; Input the feature descriptor into the self-attention layer and the cross-attention layer to obtain a first matrix; wherein the first matrix represents the matching score of the second feature point; Optimizing the first matrix by a first preset algorithm to obtain a second matrix; wherein the second matrix represents the optimal matching degree of the second feature point; The L pairs of matching feature points are obtained based on the element values ​​corresponding to the second matrix.

6. The method according to claim 1, characterized in that The step of screening the wrong matching points in the L pairs of matching feature points to obtain M pairs of matching feature points includes: Determining model parameters of a third model based on the M pairs of matching feature points; For each pair of matching feature points in the M pairs of matching feature points, judging whether the matching feature points meet preset requirements based on the model parameters, one of the matching feature points, and another feature point; The M pairs of matching feature points are determined based on the matching feature points that meet the requirements.

7. The method according to claim 6, characterized in that The determining model parameters of the third model based on the M pairs of matching feature points includes: Determine a second posture change corresponding to the two consecutive frames of images based on any pair of matching feature points among the M pairs of matching feature points and epipolar geometry principle; The model parameters are determined based on the second pose change.

8. The method according to claim 6, characterized in that The determining whether the matching feature point meets the preset requirement based on the model parameter, one of the matching feature points, and another feature point includes: Determine whether a distance between one of the matching feature points and another feature point after conversion processing with the model parameter is less than or equal to a first preset threshold; If the distance is less than or equal to the first preset threshold, it is determined that the matching feature point meets the preset requirement; or, If the distance is greater than the first preset threshold, it is determined that the matching feature point does not meet the preset requirement.

9. The method according to claim 6, characterized in that The determining of the M pairs of matching feature points based on the matching feature points that meet the requirements includes: Determine a first proportion value based on the matching feature points that meet the requirements and the L pairs of matching feature points; When the first proportion is greater than or equal to a second preset threshold, the M pairs of matching feature points are determined based on the matching feature points that meet the requirements.

10. The method according to claim 1, characterized in that The step of generating a motion trajectory of the camera based on the first posture change includes: Based on the first posture change, the sliding window optimization module optimizes the initial posture of the camera to obtain an optimized camera posture, and generates the motion trajectory based on the optimized camera posture; wherein the initial posture of the camera is determined based on the first posture change.

11. The method according to claim 10, characterized in that The step of optimizing the initial posture of the camera based on the first posture change and the sliding window optimization module to obtain an optimized camera posture includes: Perform pre-integration processing on the collected inertial measurement unit IMU measurement data to obtain an IMU pre-integration result; Based on the status information of the simultaneous mapping and positioning SLAM system, the IMU pre-integration result, the first pose change and the sliding window optimization module, the initial pose of the camera is optimized to obtain the optimized camera pose; wherein the SLAM system includes one or more of the first model, the second model and the sliding window optimization module.

12. The method according to claim 11, characterized in that The optimizing process of the initial posture of the camera based on the state information of the SLAM system, the IMU pre-integration result, the first posture change and the sliding window optimization module includes: When the state information representation of the SLAM system is not initialized, aligning the first posture change based on the IMU pre-integration result to obtain an aligned third posture change; Based on the third posture change, the sliding window optimization module optimizes the initial posture of the camera to obtain the optimized camera posture; or, When the state information representation of the SLAM system has been initialized, the initial pose of the camera is optimized based on the IMU pre-integration result, the first pose change and the sliding window optimization module; and the optimized camera pose is obtained.

13. The method according to claim 11, characterized in that The SLAM system is obtained by training an initial SLAM system based on a first loss function; wherein the first loss function includes a marginalized prior error, an IMU pre-integration error, and a visual image reprojection error.

14. A trajectory determination device, characterized in that: The trajectory determination device comprises: an acquisition unit, a determination unit, a screening unit and a generation unit; wherein, The acquisition unit is used to acquire the first image of the continuous frames; The determining unit is used to determine the first information corresponding to each frame of the image through a first model; wherein the first information includes first position information and a first descriptor corresponding to N first feature points, the first descriptor is used to quantify the feature information corresponding to the first feature points, the first model is used at least to extract the first feature points of the first image, and N is a positive integer; and determine L pairs of matching feature points based on the second position information of the second feature points, the second descriptor and the second model; wherein the second feature points include feature points of any two consecutive frames of images in the first feature point set, the first feature point set includes feature points after filtering the N first feature points, and the second model is used at least to match the input feature points; The screening unit is used to screen the erroneous matching points in the L pairs of matching feature points to obtain M pairs of matching feature points; The determining unit is further used to determine the first pose change corresponding to the two consecutive frames of images based on the M pairs of matching feature points; The generating unit is used to generate a motion trajectory of the camera based on the first posture change.

15. A trajectory determination device, characterized in that: The trajectory determination device comprises: a processor and a memory; wherein, The memory is used to store a computer program that can be run on the processor; The processor is configured to execute the method according to any one of claims 1 to 13 when running the computer program.

16. A computer-readable storage medium, characterized in that: The storage medium stores computer program codes, and when the computer program codes are executed by a computer, the method according to any one of claims 1 to 13 is executed.

17. A computer program product comprising a computer program, characterized in that The computer program implements the method according to any one of claims 1-13 when executed by a processor.

Images