Mileage calculation method, device and equipment and medium

Abstract

This application discloses a method, apparatus, device, and medium for constructing reprojection residual blocks in an odometry algorithm. In this embodiment, the visual acquisition direction of the visual radar has at least one degree of freedom, allowing the visual radar to acquire environmental images from different visual acquisition directions during the movement of the mobile device. This enables the visual radar to obtain more environmental information, thereby improving the performance of the visual algorithm. Furthermore, to avoid errors arising from combining traditional odometry with the visual radar of this application, this embodiment incorporates the visual acquisition direction of the visual radar as one of the bases for constructing the reprojection residual blocks, thus mitigating the negative impact of changes in the visual acquisition direction of the visual radar on the odometry algorithm.

Description

Technical Field

[0001] This application relates to the field of SLAM (simultaneous localization and mapping) technology, and in particular to a method, apparatus, device and medium for constructing reprojection residual blocks using odometry. Background Technology

[0002] Currently, traditional visual-inertial SLAM algorithms, such as visual-inertial odometry, are often used in fields such as ground mobile robots, quadruped robots, and autonomous driving. In these cases, the camera responsible for image acquisition is usually fixed on the mobile device. In this situation, because the camera is fixed, it may be limited by factors such as field of view, mechanical obstruction, and installation position, resulting in a decrease in the performance of the SLAM algorithm. In other words, current traditional SLAM algorithms have a technical problem of performance limitation. Summary of the Invention

[0003] The main purpose of this application is to provide a method for constructing reprojection residual blocks in the odometry calculation method, which aims to solve the technical problem of performance limitations in current traditional SLAM algorithms.

[0004] To achieve the above objectives, this application provides a method for constructing reprojection residual blocks for odometry calculation. The method is applied to a visual radar, which is installed on a mobile device and has at least one degree of freedom in its visual acquisition direction. The odometry calculation method is a visual inertial encoder odometry calculation method.

[0005] The method for constructing reprojection residual blocks in the odometry calculation method includes:

[0006] Acquire the viewing angle information of the visual acquisition direction relative to the mobile device, and the pose information of the mobile device;

[0007] A reprojection residual block is constructed based on the viewpoint information, the pose information, the image data acquired by the visual radar, and the depth information in the image data.

[0008] Optionally, the image data includes two image frames with matching feature points, the two image frames being a first image frame and a second image frame, the matching feature points including a first feature point on the first image frame and a second feature point on the second image frame, the first feature point and the second feature point representing the same feature in the actual scene;

[0009] The step of constructing the reprojection residual block based on the viewpoint information, pose information, image data acquired by the visual radar, and depth information in the image data includes:

[0010] For any matching feature point on the two image frames, based on the first coordinate data of the first feature point relative to the first image frame, the viewpoint information, and the pose information, estimated coordinate data of the first feature point relative to the second image frame is generated, wherein the first coordinate data is generated based on the position of the first feature point on the first image frame and the depth information of the position.

[0011] The reprojection residual block is constructed based on the deviation between the estimated coordinate data and the second coordinate data of the second feature point relative to the second image frame.

[0012] Optionally, the visual radar includes a camera, a motor for controlling the rotation of the camera, and a base, the base being fixed to the mobile device, and the camera being rotatably connected to the base via the motor;

[0013] The steps of acquiring the viewing angle information of the visual acquisition direction relative to the mobile device and the pose information of the mobile device include:

[0014] The position state of the motor rotation mechanism is obtained based on the encoder built into the base, and the position state is converted into the view information;

[0015] The angular velocity and acceleration of the mobile device are obtained based on the IMU built into the base, wherein the angular velocity and acceleration are used for the estimation of the pose information.

[0016] Optionally, the generation time of the first image frame is a first moment, the generation time of the second image frame is a second moment, the viewpoint information includes a first viewpoint angle at the first moment and a second viewpoint angle at the second moment, the pose information includes a first pose at the first moment and a second pose at the second moment, and the camera is configured with a camera coordinate system.

[0017] The step of generating estimated coordinate data of the first feature point relative to the second image frame based on the first coordinate data of the first feature point relative to the first image frame, the viewpoint information, and the pose information includes:

[0018] Based on the first viewpoint and the first pose, the first coordinate data of the first feature point in the camera coordinate system at the first moment is converted into world coordinate data in the world coordinate system.

[0019] Based on the second viewpoint and the second pose, the world coordinate data is converted into estimated coordinate data of the first feature point in the camera coordinate system at the second moment.

[0020] Optionally, the encoder is configured with an encoder coordinate system, and the IMU is configured with an IMU coordinate system;

[0021] The step of converting the first coordinate data of the first feature point in the camera coordinate system at the first moment into world coordinate data in the world coordinate system based on the first viewpoint angle and the first pose includes:

[0022] Based on the first viewpoint, the first coordinate data is converted into the first intermediate coordinate data of the first feature point in the encoder coordinate system at the first moment;

[0023] Based on the relative pose between the encoder and the IMU, the first intermediate coordinate data is converted into the second intermediate coordinate data of the first feature point in the IMU coordinate system at the first moment.

[0024] Based on the first pose, the second intermediate coordinate data is converted into the world coordinate data of the first feature point in the world coordinate system at the first moment.

[0025] Optionally, the encoder is configured with an encoder coordinate system, and the IMU is configured with an IMU coordinate system;

[0026] The step of converting the world coordinate data into estimated coordinate data of the first feature point in the camera coordinate system at the second moment based on the second view angle and the second pose includes:

[0027] Based on the second pose, the world coordinate data is converted into the third intermediate coordinate data of the second feature point in the IMU coordinate system at the second time.

[0028] Based on the relative pose between the encoder and the IMU, the third intermediate coordinate data is transformed into the fourth intermediate coordinate data of the first feature point in the encoder coordinate system at the second time.

[0029] Based on the second viewpoint, the fourth intermediate coordinate data is converted into the estimated coordinate data of the first feature point in the camera coordinate system at the second moment.

[0030] Optionally, the odometry calculation method further includes marginalized residual blocks and IMU residual blocks, which are used together with the reprojection residual blocks to participate in the least squares optimization of the odometry calculation method.

[0031] To achieve the above objectives, this application also provides a reprojection residual block construction device for odometry calculation, wherein the reprojection residual block construction device for odometry calculation is applied to a visual radar, wherein the visual radar is installed on a mobile device and the visual acquisition direction of the visual radar has at least one degree of freedom, and the odometry calculation method is a visual inertial encoder odometry calculation method.

[0032] The reprojection residual block construction device for the odometry calculation method includes:

[0033] The acquisition module is used to acquire the viewing angle information of the visual acquisition direction relative to the mobile device, and the pose information of the mobile device;

[0034] A construction module is used to construct a reprojection residual block based on the viewpoint information, the pose information, the image data acquired by the visual radar, and the depth information in the image data.

[0035] To achieve the above objectives, this application also provides a reprojection residual block construction device for odometry calculation. The odometry calculation reprojection residual block construction device includes: a memory, a processor, and an odometry calculation reprojection residual block construction program stored in the memory and executable on the processor. When the odometry calculation reprojection residual block construction program is executed by the processor, it implements the steps of the odometry calculation reprojection residual block construction method as described above.

[0036] To achieve the above objectives, this application also provides a medium, which is a computer-readable storage medium, on which a reprojection residual block construction program for the odometer calculation method is stored. When the reprojection residual block construction program for the odometer calculation method is executed by a processor, it implements the steps of the reprojection residual block construction method for the odometer calculation method as described above.

[0037] This application proposes a method, apparatus, device, and medium for constructing reprojection residual blocks in an odometry algorithm. In this embodiment, the method is applied to a visual radar, which is mounted on a mobile device and has at least one degree of freedom in its visual acquisition direction. The odometry algorithm is a visual inertial encoder odometry algorithm. The method acquires the viewing angle information of the visual acquisition direction relative to the mobile device, as well as the pose information of the mobile device. A reprojection residual block is constructed based on the viewing angle information, the pose information, the image data acquired by the visual radar, and the depth information in the image data. Because the visual radar has at least one degree of freedom in its visual acquisition direction, it can acquire environmental images from different visual acquisition directions during the movement of the mobile device, allowing it to obtain more environmental information and thus improving the performance of the visual algorithm. Furthermore, to avoid errors that may occur when traditional algorithms are combined with the visual radar of this application, this embodiment incorporates the visual radar's visual acquisition direction as one of the bases for constructing the reprojection residual block, thereby addressing the negative impact of changes in the visual radar's visual acquisition direction on the visual algorithm. Attached Figure Description

[0038] Figure 1 This is a schematic diagram of the device structure of the hardware operating environment involved in the embodiments of this application;

[0039] Figure 2 This is a flowchart illustrating the first embodiment of the reprojection residual block construction method of the odometer calculation method of this application.

[0040] Figure 3 This is a schematic diagram of the visual radar structure in the reprojection residual block construction method of the odometer calculation method in this application.

[0041] Figure 4 This is a detailed flowchart of step S20 in the reprojection residual block construction method of the mileage calculation method of this application;

[0042] Figure 5 This is a schematic diagram of the reprojection residual block construction device for the mileage calculation method in the reprojection residual block construction method of this application.

[0043] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0044] It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit this application.

[0045] like Figure 1 As shown, Figure 1This is a schematic diagram of the device structure of the hardware operating environment involved in the embodiments of this application.

[0046] The device in this application embodiment can be a robot, or it can be an electronic terminal device such as a PC, smartphone, tablet computer, or portable computer.

[0047] like Figure 1 As shown, the device may include: a processor 1001, such as a CPU; a network interface 1004; a user interface 1003; a memory 1005; and a communication bus 1002. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen or an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be high-speed RAM or non-volatile memory, such as a disk drive. Optionally, the memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0048] Optionally, the device may also include a camera, RF (Radio Frequency) circuitry, sensors, audio circuitry, a WiFi module, and so on. Sensors may include light sensors, motion sensors, and other sensors. Specifically, light sensors may include ambient light sensors and proximity sensors. The ambient light sensor can adjust the display brightness according to the ambient light level, while the proximity sensor can turn off the display and / or backlight when the mobile terminal is moved to the ear. As a type of motion sensor, a gravity accelerometer can detect the magnitude of acceleration in various directions (generally three axes). When stationary, it can detect the magnitude and direction of gravity, and can be used for applications that identify the mobile terminal's posture (such as landscape / portrait switching, related games, magnetometer posture calibration), vibration recognition functions (such as pedometers, taps), etc. Of course, the mobile terminal may also be equipped with other sensors such as gyroscopes, barometers, hygrometers, thermometers, and infrared sensors, which will not be elaborated here.

[0049] Those skilled in the art will understand that Figure 1 The device structure shown does not constitute a limitation on the device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0050] like Figure 1 As shown, the memory 1005, which serves as a computer storage medium, may include an operating system, a network communication module, a user interface module, and a reprojection residual block construction program for the odometer calculation method.

[0051] exist Figure 1 In the device shown, network interface 1004 is mainly used to connect to the backend server and communicate data with it; user interface 1003 is mainly used to connect to the client (user end) and communicate data with it; processor 1001 can be used to call the reprojection residual block construction program of the odometer calculation method stored in memory 1005. The working data of the robot arm is recorded to the first preset storage area in a cyclic overwrite manner. The method is applied to a visual radar, which is set on a mobile device and has at least one degree of freedom in the visual acquisition direction. The odometer calculation method is a visual inertial encoder odometer calculation method, and the following operations are performed:

[0052] Acquire the viewing angle information of the visual acquisition direction relative to the mobile device, and the pose information of the mobile device;

[0053] A reprojection residual block is constructed based on the viewpoint information, the pose information, the image data acquired by the visual radar, and the depth information in the image data.

[0054] Furthermore, the processor 1001 can call the reprojection residual block construction program for the odometry calculation method stored in the memory 1005, and also perform the following operations:

[0055] The image data includes two image frames with matching feature points. The two image frames are a first image frame and a second image frame. The matching feature points include a first feature point on the first image frame and a second feature point on the second image frame. The first feature point and the second feature point represent the same feature in the actual scene.

[0056] The step of constructing the reprojection residual block based on the viewpoint information, pose information, image data acquired by the visual radar, and depth information in the image data includes:

[0057] For any matching feature point on the two image frames, based on the first coordinate data of the first feature point relative to the first image frame, the viewpoint information, and the pose information, estimated coordinate data of the first feature point relative to the second image frame is generated, wherein the first coordinate data is generated based on the position of the first feature point on the first image frame and the depth information of the position.

[0058] The reprojection residual block is constructed based on the deviation between the estimated coordinate data and the second coordinate data of the second feature point relative to the second image frame.

[0059] Furthermore, the processor 1001 can call the reprojection residual block construction program for the odometry calculation method stored in the memory 1005, and also perform the following operations:

[0060] The visual radar includes a camera, a motor that controls the rotation of the camera, and a base. The base is fixed to the mobile device, and the camera is rotatably connected to the base via the motor.

[0061] The steps of acquiring the viewing angle information of the visual acquisition direction relative to the mobile device and the pose information of the mobile device include:

[0062] The position state of the motor rotation mechanism is obtained based on the encoder built into the base, and the position state is converted into the view information;

[0063] The angular velocity and acceleration of the mobile device are obtained based on the IMU built into the base, wherein the angular velocity and acceleration are used for the estimation of the pose information.

[0064] Furthermore, the processor 1001 can call the reprojection residual block construction program for the odometry calculation method stored in the memory 1005, and also perform the following operations:

[0065] The first image frame is generated at a first moment, the second image frame is generated at a second moment, the viewpoint information includes the first viewpoint angle at the first moment and the second viewpoint angle at the second moment, the pose information includes the first pose at the first moment and the second pose at the second moment, and the camera is configured with a camera coordinate system.

[0066] The step of generating estimated coordinate data of the first feature point relative to the second image frame based on the first coordinate data of the first feature point relative to the first image frame, the viewpoint information, and the pose information includes:

[0067] Based on the first viewpoint and the first pose, the first coordinate data of the first feature point in the camera coordinate system at the first moment is converted into world coordinate data in the world coordinate system.

[0068] Based on the second viewpoint and the second pose, the world coordinate data is converted into estimated coordinate data of the first feature point in the camera coordinate system at the second moment.

[0069] Furthermore, the processor 1001 can call the reprojection residual block construction program for the odometry calculation method stored in the memory 1005, and also perform the following operations:

[0070] The encoder is configured with an encoder coordinate system, and the IMU is configured with an IMU coordinate system;

[0071] The step of converting the first coordinate data of the first feature point in the camera coordinate system at the first moment into world coordinate data in the world coordinate system based on the first viewpoint angle and the first pose includes:

[0072] Based on the first viewpoint, the first coordinate data is converted into the first intermediate coordinate data of the first feature point in the encoder coordinate system at the first moment;

[0073] Based on the relative pose between the encoder and the IMU, the first intermediate coordinate data is converted into the second intermediate coordinate data of the first feature point in the IMU coordinate system at the first moment.

[0074] Based on the first pose, the second intermediate coordinate data is converted into the world coordinate data of the first feature point in the world coordinate system at the first moment.

[0075] Furthermore, the processor 1001 can call the reprojection residual block construction program for the odometry calculation method stored in the memory 1005, and also perform the following operations:

[0076] The encoder is configured with an encoder coordinate system, and the IMU is configured with an IMU coordinate system;

[0077] The step of converting the world coordinate data into estimated coordinate data of the first feature point in the camera coordinate system at the second moment based on the second view angle and the second pose includes:

[0078] Based on the second pose, the world coordinate data is converted into the third intermediate coordinate data of the second feature point in the IMU coordinate system at the second time.

[0079] Based on the relative pose between the encoder and the IMU, the third intermediate coordinate data is transformed into the fourth intermediate coordinate data of the first feature point in the encoder coordinate system at the second time.

[0080] Based on the second viewpoint, the fourth intermediate coordinate data is converted into the estimated coordinate data of the first feature point in the camera coordinate system at the second moment.

[0081] Furthermore, the odometry calculation method also includes marginalized residual blocks and IMU residual blocks, which are used together with the reprojection residual blocks to participate in the least squares optimization of the odometry calculation method.

[0082] Reference Figure 2 The first embodiment of the reprojection residual block construction method for odometry calculation in this application is applied to a visual radar, which is installed on a mobile device and has at least one degree of freedom in its visual acquisition direction. The odometry calculation method is a visual inertial encoder odometry calculation method. The reprojection residual block construction method of the odometry calculation method includes:

[0083] Step S10: Obtain the viewing angle information of the visual acquisition direction relative to the mobile device, and the pose information of the mobile device;

[0084] It should be noted that in this embodiment, the odometry calculation method in the reprojection residual block construction method of the above odometry calculation method is a visual inertial encoder odometry calculation method. This method is applied to a visual radar, which is installed on a mobile device to undertake the task of acquiring images of the surrounding environment and its own posture. The mobile device can be a ground mobile robot, a quadruped robot, a drone, or a car, etc., any device with mobility. To ensure that the image acquisition is not limited by factors such as the field of view, mechanical obstruction, and installation position, in this embodiment, the visual acquisition direction of the visual radar is set to have at least one degree of freedom. For example, the visual acquisition direction of the visual radar can be controlled to rotate in different directions, so that the visual radar can acquire environmental images from different visual acquisition directions during the movement of the mobile device. That is, compared with existing solutions, in this embodiment, the visual radar can acquire more environmental information, thereby improving the performance of the odometry calculation method. Furthermore, it should be noted that since the visual acquisition direction of the visual radar in this embodiment has at least one degree of freedom, traditional VIO (visual-inertial odometry) and its SLAM algorithm will encounter errors when combined with this visual radar, such as incorrect residual construction and Jacobian matrix calculation, leading to incorrect output results from the SLAM algorithm. To address this issue, this embodiment uses the visual acquisition direction as the basis for constructing the optimization parameters of the visual algorithm, thereby resolving the impact of variations in the visual radar's visual acquisition direction.

[0085] For example, the visual acquisition direction of the visual radar is acquired relative to the viewing angle of the mobile device. This includes, for instance, the angle at which the visual radar rotates in its visual acquisition direction on the mobile device. The pose information of the mobile device is also acquired. For example, the acceleration and angular velocity of the mobile device can be acquired through an IMU (Inertial Measurement Unit) to estimate the pose information. It should be noted that in practical applications, in addition to using the acceleration and angular velocity obtained from the IMU, the pose information of the mobile device can also be estimated using the acquisition results of other types of sensors; these will not be elaborated upon here.

[0086] Step S20: Construct a reprojection residual block based on the viewpoint information, the pose information, the image data acquired by the visual radar, and the depth information in the image data.

[0087] For example, the visual radar also includes a camera or video camera to acquire image data of the surrounding environment. The camera can be a depth camera, so the acquired image data may contain depth information. A reprojection residual block is constructed based on the viewpoint information, pose information, image data, and depth information in the image data. This reprojection residual block can be used for least-squares optimization of the odometry algorithm. For example, the Jacobian matrix is ​​constructed based on the reprojection residual block using the Ceres library (a nonlinear optimization library). During the visual algorithm optimization process, the parameters are updated and improved using the Jacobian matrix. The Ceres library is open source, so its usage will not be described in detail.

[0088] In this embodiment, the above method is applied to the visual radar, which is installed on a mobile device and has multiple degrees of freedom in its visual acquisition direction. The optimization parameters include at least a reprojection residual block. The method involves acquiring the viewing angle information of the visual acquisition direction relative to the mobile device, as well as the pose information of the mobile device. A reprojection residual block is constructed based on the viewing angle information, the pose information, the image data acquired by the visual radar, and the depth information in the image data. Because the visual radar has multiple degrees of freedom in its visual acquisition direction, it can acquire environmental images from different visual acquisition directions during the movement of the mobile device, allowing it to obtain more environmental information and thus improving the performance of the visual algorithm. Furthermore, to avoid errors that may occur when traditional algorithms are combined with the visual radar of this application, this embodiment incorporates the visual acquisition direction of the visual radar as one of the bases for constructing the reprojection residual block, thereby addressing the negative impact of changes in the visual radar's visual acquisition direction on the visual algorithm.

[0089] In one feasible implementation, the image data includes two image frames with matching feature points, the two image frames being a first image frame and a second image frame, the matching feature points including a first feature point on the first image frame and a second feature point on the second image frame, the first feature point and the second feature point representing the same feature in the actual scene;

[0090] The step of constructing the reprojection residual block based on the viewpoint information, pose information, image data acquired by the visual radar, and depth information in the image data includes:

[0091] Step S21: For any matching feature point on the two image frames, based on the first coordinate data of the first feature point relative to the first image frame, the viewpoint information, and the pose information, generate estimated coordinate data of the first feature point relative to the second image frame, wherein the first coordinate data is generated based on the position of the first feature point on the first image frame and the depth information of the position.

[0092] Step S22: Based on the deviation between the estimated coordinate data and the second coordinate data of the second feature point relative to the second image frame, construct the reprojection residual block.

[0093] It should be noted that the image data mentioned above refers to image data acquired by the camera. For example, in practical applications, image data typically includes image frames within a sliding window period. Two image frames generated close to each other may have matching feature points. For instance, if two image frames are a first image frame and a second image frame, and a first feature point on the first image frame and a second feature point on the second image frame represent the same feature in the actual scene, then the first and second feature points are considered matching feature points. In practical applications, feature point extraction algorithms can be set by technicians according to actual needs, such as the FAST (Features From Accelerated Segment Test) algorithm. Typically, multiple pairs of matching feature points are stored between two image frames. In this embodiment, a single matching feature point will be used as an example.

[0094] For example, for any matching feature point on two image frames, including a first feature point on the first image frame and a second feature point on the second image frame, the first coordinate data of the first feature point relative to the first image frame is determined. This first coordinate data is the coordinate data of the first feature point in the camera coordinate system at the time the first image frame is generated. The first coordinate data can be generated from the position of the first feature point on the first image frame and the corresponding depth information. It should also be noted that the depth information of the first feature point on the first image frame will be used as a parameter to be optimized. The first coordinate data is then transformed into estimated coordinate data relative to the second image frame. This estimated coordinate data is the coordinate data of the first feature point in the camera coordinate system at the time the second image frame is generated. During the transformation, multiple transformation matrices are generated based on viewpoint and pose information to transform the first coordinate data. It is understood that the first and second feature points represent the same feature point in the actual environment. Therefore, ideally (when the viewpoint and pose information are completely accurate), the estimated coordinate data and the second coordinate data of the second feature point relative to the second image frame should be consistent; that is, the estimated coordinate data and the second coordinate data are the same. Therefore, in this embodiment, a reprojection residual block is constructed based on the deviation between the estimated coordinate data and the second coordinate data, and the reprojection residual block can be used for optimization in the vision algorithm.

[0095] In one possible implementation, the visual radar includes a camera, a motor for controlling the rotation of the camera, and a base, the base being fixed to the mobile device, and the camera being rotatably connected to the base via the motor;

[0096] The steps of acquiring the viewing angle information of the visual acquisition direction relative to the mobile device and the pose information of the mobile device include:

[0097] Step S11: Obtain the position state of the motor rotation mechanism based on the encoder built into the base, and convert the position state into the viewing angle information;

[0098] Step S12: Obtain the angular velocity and acceleration of the mobile device based on the IMU built into the base, wherein the angular velocity and acceleration are used for the estimation of the pose information.

[0099] For example, in this embodiment, the visual radar includes a camera, a motor controlling the rotation of the camera, and a base. The base is fixed to the mobile device, and the camera is rotatably connected to the base via the motor. The motor can be a miniature hollow brushless DC motor, and a miniature slip ring is provided between the motor and the camera. The motor includes a stator and a rotor; similarly, the miniature slip ring also has a stator and a rotor. The rotor of the miniature slip ring is fixed to the rotor of the motor, and the stator of the miniature slip ring is fixed to the stator of the motor. The camera and the rotor of the motor are fixed. The power supply and communication lines of the camera pass through the miniature slip ring. The motor can control the rotation of the camera according to control commands, enabling the acquisition of surrounding environmental images from different perspectives. For example, refer to... Figure 3 The diagram shows the structure of the visual radar in this application. The visual radar includes a camera 1, a motor 3, and a base 4. A miniature slip ring 2 can be set between the camera 1 and the motor 3 for wiring. An encoder and an IMU (not shown in the figure) can be built into the base 4.

[0100] For example, the position state of the motor rotation mechanism is obtained based on the encoder built into the base, that is, the rotor position of the motor is read. The shooting angle (i.e., angle information) of the camera can be determined based on the rotor position. The conversion relationship between the rotor position and the angle information can be obtained through pre-calibration, which will not be elaborated here. Then, the rotor position is converted into angle information according to the conversion relationship. At the same time, an IMU can be built into the base. Since the base and the mobile device are fixed, the pose information estimated by the angular velocity and acceleration obtained by the IMU is also the pose information of the mobile device. It should be noted that the estimation of pose information can refer to the acquisition results of other sensors in addition to angular velocity and acceleration.

[0101] In one feasible implementation, the generation time of the first image frame is a first moment, the generation time of the second image frame is a second moment, the viewpoint information includes a first viewpoint angle at the first moment and a second viewpoint angle at the second moment, the pose information includes a first pose at the first moment and a second pose at the second moment, and the camera is configured with a camera coordinate system.

[0102] The step of generating estimated coordinate data of the first feature point relative to the second image frame based on the first coordinate data of the first feature point relative to the first image frame, the viewpoint information, and the pose information includes:

[0103] Step S210: Based on the first viewpoint angle and the first pose, convert the first coordinate data of the first feature point in the camera coordinate system at the first moment into world coordinate data in the world coordinate system.

[0104] Step S220: Based on the second viewpoint angle and the second pose, the world coordinate data is converted into estimated coordinate data of the first feature point in the camera coordinate system at the second moment.

[0105] It should be noted that the generation time of the first image frame is the first moment, and the generation time of the second image frame is the second moment; the viewpoint information includes the first viewpoint angle obtained at the first moment and the second viewpoint angle obtained at the second moment; the pose information includes the first pose obtained at the first moment and the second pose obtained at the second moment, and the camera is configured with a camera coordinate system.

[0106] For example, in practical applications, the world coordinate system of mobile devices and visual radar remains unchanged. Therefore, when converting the first coordinate data into estimated coordinate data, the world coordinate system can be used as a transition during the conversion. Specifically, a transformation matrix is ​​first generated based on the first viewpoint angle and first position obtained at the first moment. Then, the transformation matrix is ​​used to convert the first coordinate data of the first feature point in the camera coordinate system to the world coordinate system to obtain world coordinate data. Next, a transformation matrix is ​​generated based on the second viewpoint angle and second pose obtained at the second moment. This transformation matrix is ​​then used to convert the first feature point from the world coordinate system to the camera coordinate system at the second moment to obtain the estimated coordinate data.

[0107] In one feasible implementation, the encoder is configured with an encoder coordinate system, and the IMU is configured with an IMU coordinate system;

[0108] The step of converting the first coordinate data of the first feature point in the camera coordinate system at the first moment into world coordinate data in the world coordinate system based on the first viewpoint angle and the first pose includes:

[0109] Step S211: Based on the first viewing angle, convert the first coordinate data into the first intermediate coordinate data of the first feature point in the encoder coordinate system at the first moment.

[0110] Step S212: Based on the relative pose between the encoder and the IMU, the first intermediate coordinate data is converted into the second intermediate coordinate data of the first feature point in the IMU coordinate system at the first moment.

[0111] Step S213: Based on the first pose, convert the second intermediate coordinate data into the world coordinate data of the first feature point in the world coordinate system at the first moment.

[0112] It should be noted that the encoder is configured with an encoder coordinate system, and the IMU is configured with an IMU coordinate system.

[0113] For example, let the first coordinate data be:

[0114]

[0115] In the formula, λ is the projection of the first feature point onto the camera's normalized plane coordinate system in the first image frame. l The inverse depth of the first feature point.

[0116] Using the first perspective angle θ, the first coordinate data is transformed into the first intermediate coordinate data in the encoder coordinate system:

[0117]

[0118] In the formula, exp is the axis-angle representation characterizing rotation, and θ i Let 'a' be the first viewpoint angle corresponding to the first image frame, and 'a' be the visual radar rotation axis coordinate. λ is the projection of the first feature point onto the camera's normalized plane coordinate system in the first image frame. l The inverse depth of the first feature point. This represents the displacement from the camera coordinate system to the encoder coordinate system.

[0119] By using the relative pose between the encoder and the IMU (since both the encoder and the IMU are integrated in the base, the relative pose is usually fixed and can be pre-calibrated), the first intermediate coordinate data is converted into the second intermediate coordinate data of the first feature point in the IMU coordinate system at the first moment:

[0120]

[0121] In the formula, This represents the rotation matrix from the encoder coordinate system to the IMU coordinate system, where exp is the axis-angle representation of the rotation, and θ is the rotation matrix. i Let 'a' be the first viewpoint angle corresponding to the first image frame, and 'a' be the visual radar rotation axis coordinate. λ is the projection of the first feature point onto the camera's normalized plane coordinate system in the first image frame. l The inverse depth of the first feature point. This represents the displacement from the camera coordinate system to the encoder coordinate system. This represents the displacement from the encoder coordinate system to the IMU coordinate system.

[0122] Using the first pose at the first moment, the second intermediate coordinate data is converted into the world coordinate data of the first feature point in the world coordinate system at the first moment:

[0123]

[0124] In the formula, This represents the rotation matrix from the IMU coordinate system to the world coordinate system at the first moment. This represents the rotation matrix from the encoder coordinate system to the IMU coordinate system, where exp is the axis-angle representation of the rotation, and θ is the rotation matrix. i Let 'a' be the first viewpoint angle corresponding to the first image frame, and 'a' be the visual radar rotation axis coordinate. λ is the projection of the first feature point onto the camera's normalized plane coordinate system in the first image frame. l The inverse depth of the first feature point. This represents the displacement from the camera coordinate system to the encoder coordinate system. This represents the displacement from the encoder coordinate system to the IMU coordinate system. This represents the displacement from the IMU coordinate system to the world coordinate system.

[0125] It is understood that, in this embodiment, the world coordinate data can be obtained by sequentially processing the first coordinate data based on the first viewpoint angle, the relative pose between the encoder and the IMU, and the first position.

[0126] In one feasible implementation, the encoder is configured with an encoder coordinate system, and the IMU is configured with an IMU coordinate system;

[0127] The step of converting the world coordinate data into estimated coordinate data of the first feature point in the camera coordinate system at the second moment based on the second view angle and the second pose includes:

[0128] Step S221: Based on the second pose, the world coordinate data is converted into the third intermediate coordinate data of the second feature point in the IMU coordinate system at the second time.

[0129] Step S222: Based on the relative pose between the encoder and the IMU, the third intermediate coordinate data is converted into the fourth intermediate coordinate data of the first feature point in the encoder coordinate system at the second time.

[0130] Step S223: Based on the second viewing angle, the fourth intermediate coordinate data is converted into the estimated coordinate data of the first feature point in the camera coordinate system at the second moment.

[0131] For example, also based on the above example, based on the second pose, the world coordinate data is converted into the third intermediate coordinate data of the second feature point in the IMU coordinate system at the second time:

[0132]

[0133] In the formula, This represents the transpose of the rotation matrix from the world coordinate system to the IMU coordinate system at the second moment. This represents the rotation matrix from the IMU coordinate system to the world coordinate system at the first moment. This represents the rotation matrix from the encoder coordinate system to the IMU coordinate system, where exp is the axis-angle representation of the rotation, and θ is the rotation matrix. i Let 'a' be the first viewpoint angle corresponding to the first image frame, and 'a' be the visual radar rotation axis coordinate. λ is the projection of the first feature point onto the camera's normalized plane coordinate system in the first image frame. l The inverse depth of the first feature point. This represents the displacement from the camera coordinate system to the encoder coordinate system. This represents the displacement from the encoder coordinate system to the IMU coordinate system. This represents the displacement from the IMU coordinate system to the world coordinate system at the first moment. This represents the displacement from the world coordinate system to the IMU coordinate system at the second moment.

[0134] Based on the relative pose between the encoder and the IMU, the third intermediate coordinate data is transformed into the fourth intermediate coordinate data of the first feature point in the encoder coordinate system at the second time step:

[0135]

[0136] In the formula, This represents the transpose of the rotation matrix from the IMU coordinate system to the encoder coordinate system at the second moment. This represents the transpose of the rotation matrix from the world coordinate system to the IMU coordinate system at the second moment. This represents the rotation matrix from the IMU coordinate system to the world coordinate system at the first moment. This represents the rotation matrix from the encoder coordinate system to the IMU coordinate system, where exp is the axis-angle representation of the rotation, and θ is the rotation matrix. i Let 'a' be the first viewpoint angle corresponding to the first image frame, and 'a' be the visual radar rotation axis coordinate. λ is the projection of the first feature point onto the camera's normalized plane coordinate system in the first image frame. l The inverse depth of the first feature point. This represents the displacement from the camera coordinate system to the encoder coordinate system. This represents the displacement from the encoder coordinate system to the IMU coordinate system. This represents the displacement from the IMU coordinate system to the world coordinate system at the first moment. This represents the displacement from the world coordinate system to the IMU coordinate system at the second moment.

[0137] Based on the second perspective angle, the fourth intermediate coordinate data is converted into estimated coordinate data of the first feature point in the camera coordinate system at the second time point:

[0138]

[0139] In the formula, θ j This refers to the second viewpoint angle corresponding to the second image frame. Corresponding to the estimated coordinate data, This represents the transpose of the rotation matrix from the IMU coordinate system to the encoder coordinate system at the second moment. This represents the transpose of the rotation matrix from the world coordinate system to the IMU coordinate system at the second moment. This represents the rotation matrix from the IMU coordinate system to the world coordinate system at the first moment. This represents the rotation matrix from the encoder coordinate system to the IMU coordinate system, where exp is the axis-angle representation of the rotation, and θ is the rotation matrix. i Let 'a' be the first viewpoint angle corresponding to the first image frame, and 'a' be the visual radar rotation axis coordinate. λ is the projection of the first feature point onto the camera's normalized plane coordinate system in the first image frame. l The inverse depth of the first feature point. This represents the displacement from the camera coordinate system to the encoder coordinate system. This represents the displacement from the encoder coordinate system to the IMU coordinate system. This represents the displacement from the IMU coordinate system to the world coordinate system at the first moment. This represents the displacement from the world coordinate system to the IMU coordinate system at the second moment.

[0140] It is understood that, in this embodiment, the estimated coordinate data can be obtained by sequentially transforming the estimated coordinate data based on the second pose, the relative pose between the encoder and the IMU, and the second viewpoint.

[0141] Based on the above example, the estimated deviation between the coordinate data and the second coordinate data is:

[0142]

[0143] In the formula, e is the expression for the visual reprojection residual block. It should be noted that the visual reprojection residual is the projection of the residual on the unit sphere onto the tangent plane, while B is the matrix formed by the basis of the tangent plane. Corresponding to the estimated coordinate data, This corresponds to the second coordinate data.

[0144] Furthermore, the partial derivatives of the reprojection residuals with respect to the variables to be optimized will be used as part of constructing the Jacobian matrix of the residual block. The derivative of the reprojection residuals with respect to the translation vector can be obtained directly. The derivative with respect to the rotation matrix can be obtained using the perturbation model of the SO(3) manifold. The derivative with respect to the rotation angle θ at the first and second time moments can be obtained by expanding the rotation matrix in terms of axis angle using the Rodrigues formula. The derivative with respect to the rotation axis a can be obtained using the perturbation model of the spherical manifold. The approximate BCH (Baker-Campbell-Hausdorff) formula is used as the conversion between the infinitesimal transformation relationship of the Lie group and the corresponding Lie algebra.

[0145] The Jacobian matrix part of the reprojection residual is calculated and solved as follows:

[0146]

[0147] In the formula, x is any variable to be optimized, e is the expression of the visual reprojection residual block, and the result of taking the derivative of e with respect to the variable to be optimized is the Jacobian matrix part corresponding to the above reprojection residual.

[0148]

[0149]

[0150]

[0151]

[0152] The above, y, C, J r (φ), R x (φ) are all custom intermediate values, which facilitate the expression of the Jacobian matrix part corresponding to the reprojection residual in the later stage.

[0153] The formula for expressing the Jacobian matrix part of the reprojection residual can be derived as follows:

[0154]

[0155]

[0156]

[0157]

[0158]

[0159]

[0160]

[0161]

[0162]

[0163]

[0164]

[0165] In practical applications, the Jacobian matrix expression formula corresponding to the above reprojection residual block can be imported into the Ceres library. The Ceres library is a least squares optimization algorithm library that can be used for backend optimization of visual odometry.

[0166] In one feasible implementation, the odometer calculation method further includes marginalized residual blocks and IMU residual blocks, which are used together with the reprojection residual blocks to participate in the least squares optimization of the odometer calculation method.

[0167] For example, odometry also includes marginalized residual blocks and IMU residual blocks, which, along with reprojection residual blocks, participate in the optimization of odometry. For instance, based on the Ceres open-source least squares optimization library, the backend optimization problem of the VIEO (visual-inertial-encoder-odometry) algorithm is established by combining reprojection residual blocks with marginalized and IMU residuals, and solved iteratively. The final solution yields the optimal state estimate that minimizes the residual function, which serves as the parameter estimate in the VINES system (visual-inertial encoder system). The Ceres library constructs the final Jacobian matrix using the chain rule of residual blocks. When defining the residual function and optimization variables, the Ceres library optimizes the structure of the Jacobian matrix calculated based on the aforementioned marginalized, IMU, and reprojection residual blocks, along with its related storage and computation. This allows for parameter updates and improvements during the optimization process using the Jacobian matrix.

[0168] In addition, refer to Figure 5 This application also proposes a reprojection residual block construction device 100 for odometer calculation. The device is applied to a visual radar, which is installed on a mobile device and has at least one degree of freedom in the visual acquisition direction. The odometer calculation method is a visual inertial encoder odometer calculation method.

[0169] The reprojection residual block construction device 100 for the odometry calculation method includes:

[0170] The acquisition module 10 is used to acquire the viewing angle information of the visual acquisition direction relative to the mobile device, and the pose information of the mobile device.

[0171] The construction module 20 is used to construct a reprojection residual block based on the viewpoint information, the pose information, the image data acquired by the visual radar, and the depth information in the image data.

[0172] Optionally, the image data includes two image frames with matching feature points, the two image frames being a first image frame and a second image frame, the matching feature points including a first feature point on the first image frame and a second feature point on the second image frame, the first feature point and the second feature point representing the same feature in the actual scene; the construction module 20 is further configured to:

[0173] For any matching feature point on the two image frames, based on the first coordinate data of the first feature point relative to the first image frame, the viewpoint information, and the pose information, estimated coordinate data of the first feature point relative to the second image frame is generated, wherein the first coordinate data is generated based on the position of the first feature point on the first image frame and the depth information of the position.

[0174] The reprojection residual block is constructed based on the deviation between the estimated coordinate data and the second coordinate data of the second feature point relative to the second image frame.

[0175] Optionally, the visual radar includes a camera, a motor for controlling the rotation of the camera, and a base. The base is fixed to the mobile device, and the camera is rotatably connected to the base via the motor. The acquisition module 10 is further configured to:

[0176] The steps of acquiring the viewing angle information of the visual acquisition direction relative to the mobile device and the pose information of the mobile device include:

[0177] The position state of the motor rotation mechanism is obtained based on the encoder built into the base, and the position state is converted into the view information;

[0178] The angular velocity and acceleration of the mobile device are obtained based on the IMU built into the base, wherein the angular velocity and acceleration are used for the estimation of the pose information.

[0179] Optionally, the generation time of the first image frame is a first moment, the generation time of the second image frame is a second moment, the viewpoint information includes a first viewpoint angle at the first moment and a second viewpoint angle at the second moment, the pose information includes a first pose at the first moment and a second pose at the second moment, and the camera is configured with a camera coordinate system; the construction module 20 is further configured to:

[0180] Based on the first viewpoint and the first pose, the first coordinate data of the first feature point in the camera coordinate system at the first moment is converted into world coordinate data in the world coordinate system.

[0181] Based on the second viewpoint and the second pose, the world coordinate data is converted into estimated coordinate data of the first feature point in the camera coordinate system at the second moment.

[0182] Optionally, the encoder is configured with an encoder coordinate system, and the IMU is configured with an IMU coordinate system; the construction module 20 is further configured to:

[0183] Based on the first viewpoint, the first coordinate data is converted into the first intermediate coordinate data of the first feature point in the encoder coordinate system at the first moment;

[0184] Based on the relative pose between the encoder and the IMU, the first intermediate coordinate data is converted into the second intermediate coordinate data of the first feature point in the IMU coordinate system at the first moment.

[0185] Based on the first pose, the second intermediate coordinate data is converted into the world coordinate data of the first feature point in the world coordinate system at the first moment.

[0186] Optionally, the encoder is configured with an encoder coordinate system, and the IMU is configured with an IMU coordinate system; the construction module 20 is further configured to:

[0187] Based on the second pose, the world coordinate data is converted into the third intermediate coordinate data of the second feature point in the IMU coordinate system at the second time.

[0188] Based on the relative pose between the encoder and the IMU, the third intermediate coordinate data is transformed into the fourth intermediate coordinate data of the first feature point in the encoder coordinate system at the second time.

[0189] Based on the second viewpoint, the fourth intermediate coordinate data is converted into the estimated coordinate data of the first feature point in the camera coordinate system at the second moment.

[0190] Optionally, the odometry calculation method further includes marginalized residual blocks and IMU residual blocks, which are used together with the reprojection residual blocks to participate in the least squares optimization of the odometry calculation method.

[0191] The odometry reprojection residual block construction apparatus provided in this application adopts the odometry reprojection residual block construction method in the above embodiments, aiming to solve the technical problem of performance limitations in current traditional SLAM algorithms. Compared with the prior art, the beneficial effects of the odometry reprojection residual block construction apparatus provided in this application are the same as those of the odometry reprojection residual block construction method provided in Embodiment 1 above, and other technical features in this odometry reprojection residual block construction apparatus are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0192] Furthermore, this application also proposes a reprojection residual block construction device for odometer calculation. The reprojection residual block construction device for odometer calculation includes: a memory, a processor, and a reprojection residual block construction program for odometer calculation stored in the memory and executable on the processor. When the reprojection residual block construction program for odometer calculation is executed by the processor, it implements the steps of the reprojection residual block construction method for odometer calculation as described above.

[0193] The specific implementation of the reprojection residual block construction device for the odometer calculation method of this application is basically the same as the various embodiments of the reprojection residual block construction method of the above-mentioned odometer calculation method, and will not be repeated here.

[0194] Furthermore, this application also proposes a medium, which is a computer-readable storage medium, on which a reprojection residual block construction program for the odometer calculation method is stored. When the reprojection residual block construction program for the odometer calculation method is executed by a processor, it implements the steps of the reprojection residual block construction method for the odometer calculation method as described above.

[0195] The specific implementation of the medium in this application is basically the same as the various embodiments of the reprojection residual block construction method of the above-mentioned mileage calculation method, and will not be described again here.

[0196] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0197] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0198] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device (which may be a vehicle, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0199] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A method for constructing a reprojection residual block for a mileage calculation method, characterized by, The method is applied to a visual radar, which is installed on a mobile device and has at least one degree of freedom in the visual acquisition direction. The odometer calculation method is a visual inertial encoder odometer calculation method. The method for constructing reprojection residual blocks in the odometry calculation method includes: Acquire the viewing angle information of the visual acquisition direction relative to the mobile device, and the pose information of the mobile device; A reprojection residual block is constructed based on the viewpoint information, the pose information, the image data acquired by the visual radar, and the depth information in the image data; The image data includes two image frames with matching feature points, namely a first image frame and a second image frame. The matching feature points include a first feature point on the first image frame and a second feature point on the second image frame. The first feature point and the second feature point represent the same feature in the actual scene. The step of constructing the reprojection residual block based on the viewpoint information, pose information, image data acquired by the visual radar, and depth information in the image data includes: For any matching feature point on the two image frames, based on the first coordinate data of the first feature point relative to the first image frame, the viewpoint information, and the pose information, estimated coordinate data of the first feature point relative to the second image frame is generated, wherein the first coordinate data is generated based on the position of the first feature point on the first image frame and the depth information of the position. The reprojection residual block is constructed based on the deviation between the estimated coordinate data and the second coordinate data of the second feature point relative to the second image frame.

2. The reprojection residual block construction method of claim 1, wherein, The visual radar includes a camera, a motor that controls the rotation of the camera, and a base. The base is fixed to the mobile device, and the camera is rotatably connected to the base via the motor. The steps of acquiring the viewing angle information of the visual acquisition direction relative to the mobile device and the pose information of the mobile device include: The position state of the motor rotation mechanism is obtained based on the encoder built into the base, and the position state is converted into the view information; The angular velocity and acceleration of the mobile device are obtained based on the IMU built into the base, wherein the angular velocity and acceleration are used for the estimation of the pose information.

3. The reprojection residual block construction method of claim 2, wherein, The first image frame is generated at a first moment, the second image frame is generated at a second moment, the viewpoint information includes the first viewpoint angle at the first moment and the second viewpoint angle at the second moment, the pose information includes the first pose at the first moment and the second pose at the second moment, and the camera is configured with a camera coordinate system. The step of generating estimated coordinate data of the first feature point relative to the second image frame based on the first coordinate data of the first feature point relative to the first image frame, the viewpoint information, and the pose information includes: Based on the first viewpoint and the first pose, the first coordinate data of the first feature point in the camera coordinate system at the first moment is converted into world coordinate data in the world coordinate system. Based on the second viewpoint and the second pose, the world coordinate data is converted into estimated coordinate data of the first feature point in the camera coordinate system at the second moment.

4. The reprojection residual block construction method of claim 3, wherein, The encoder is configured with an encoder coordinate system, and the IMU is configured with an IMU coordinate system; The step of converting the first coordinate data of the first feature point in the camera coordinate system at the first moment into world coordinate data in the world coordinate system based on the first viewpoint angle and the first pose includes: Based on the first viewpoint, the first coordinate data is converted into the first intermediate coordinate data of the first feature point in the encoder coordinate system at the first moment; Based on the relative pose between the encoder and the IMU, the first intermediate coordinate data is converted into the second intermediate coordinate data of the first feature point in the IMU coordinate system at the first moment; Based on the first pose, the second intermediate coordinate data is converted into the world coordinate data of the first feature point in the world coordinate system at the first moment.

5. The reprojection residual block construction method of claim 3, wherein, The encoder is configured with an encoder coordinate system, and the IMU is configured with an IMU coordinate system; The step of converting the world coordinate data into estimated coordinate data of the first feature point in the camera coordinate system at the second moment based on the second view angle and the second pose includes: Based on the second pose, the world coordinate data is converted into the third intermediate coordinate data of the second feature point in the IMU coordinate system at the second time. Based on the relative pose between the encoder and the IMU, the third intermediate coordinate data is transformed into the fourth intermediate coordinate data of the first feature point in the encoder coordinate system at the second time. Based on the second viewpoint, the fourth intermediate coordinate data is converted into the estimated coordinate data of the first feature point in the camera coordinate system at the second moment.

6. The reprojection residual block construction method of claim 1 to 5, wherein, The odometer calculation method also includes marginalized residual blocks and IMU residual blocks, which are used together with the reprojection residual blocks to participate in the least squares optimization of the odometer calculation method.

7. A re-projection residual block construction apparatus of a mileage calculation method, characterized by, The device is applied to a visual radar, which is installed on a mobile device and has at least one degree of freedom in the visual acquisition direction. The odometer calculation method is a visual inertial encoder odometer calculation method. The reprojection residual block construction device for the odometry calculation method includes: The acquisition module is used to acquire the viewing angle information of the visual acquisition direction relative to the mobile device, and the pose information of the mobile device; The construction module is used to construct a reprojection residual block based on the viewpoint information, the pose information, the image data acquired by the visual radar, and the depth information in the image data; The image data includes two image frames with matching feature points, the two image frames being a first image frame and a second image frame. The matching feature points include a first feature point on the first image frame and a second feature point on the second image frame, the first feature point and the second feature point representing the same feature in the actual scene; the construction module is further configured to: For any matching feature point on the two image frames, based on the first coordinate data of the first feature point relative to the first image frame, the viewpoint information, and the pose information, estimated coordinate data of the first feature point relative to the second image frame is generated, wherein the first coordinate data is generated based on the position of the first feature point on the first image frame and the depth information of the position. The reprojection residual block is constructed based on the deviation between the estimated coordinate data and the second coordinate data of the second feature point relative to the second image frame.

8. A reprojection residual block construction device of a mileage calculation method, characterized by, The reprojection residual block construction device for the odometer calculation method includes: a memory, a processor, and a reprojection residual block construction program for the odometer calculation method stored in the memory and executable on the processor. When the reprojection residual block construction program for the odometer calculation method is executed by the processor, it implements the steps of the reprojection residual block construction method for the odometer calculation method as described in any one of claims 1 to 6.

9. A medium characterized by, The medium is a computer-readable storage medium, on which a reprojection residual block construction program for the odometer calculation method is stored. When the reprojection residual block construction program for the odometer calculation method is executed by a processor, it implements the steps of the reprojection residual block construction method for the odometer calculation method as described in any one of claims 1 to 6.

Images