Visual odometer measurement method and device, electronic equipment and medium

Abstract

The invention relates to the technical field of intelligent robots, and provides a visual odometer measurement method and device, electronic equipment and a medium. Comprising the following steps: performing time synchronization on each sensor; an image sequence of the first sensor is obtained and preprocessed, and visual motion recovery is obtained; acquiring depth information of the environment through a second sensor, and performing point cloud fusion, feature registration and scale recovery to acquire visual motion recovery; and obtaining a visual odometer according to the visual motion recovery and the IMU motion between the image sequence frames. According to the method, the time of each sensor is synchronized, so that the time consumption uncertainty of data transmission is avoided; and point cloud fusion, feature registration and scale recovery are carried out, so that the problem of scale uncertainty of feature points is solved.

Description

Technical Field

[0001] The present disclosure relates to the field of intelligent robot technology, and in particular to a visual odometer measurement method, device, electronic device and medium. Background Art

[0002] Robot localization technology is mainly based on AMCL, but in order to handle complex environments, it is necessary to integrate other localization technologies, such as visual odometry. Visual odometry (VO) is a process of determining the position and orientation of a robot by analyzing a sequence of related images. Visual odometry enhances the navigation accuracy of a robot when it moves in any way on any surface.

[0003] Existing visual odometry measurements generally obtain image sequences through a monocular camera, but the camera's VO still has many problems. For example, in areas with little texture, the number of feature points is insufficient and VO cannot work; when the camera moves quickly, the image is blurred and it is difficult for the visual front end to extract feature points; in dynamic scenes, moving objects and pedestrians will cause feature point matching errors; monocular cameras cannot solve scale problems, etc. Summary of the invention

[0004] The present disclosure provides a visual odometer measurement method, device, electronic device and medium, including: time synchronization of each sensor; acquiring an image sequence of a first sensor, performing preprocessing, and obtaining visual motion recovery; acquiring depth information of an environment through a second sensor, and performing point cloud fusion, feature registration and scale recovery to obtain the visual motion recovery; obtaining a visual odometer according to the visual motion recovery and the IMU motion between frames of the image sequence. Compared with the prior art, the present disclosure synchronizes each sensor time, thereby avoiding uncertainty in the time consumption of data transmission; and performing point cloud fusion, feature registration and scale recovery to solve the uncertainty in the scale of feature points.

[0005] In order to achieve the above objectives, the present invention adopts the following technical solutions:

[0006] The first aspect of the present disclosure provides a visual odometer measurement method, comprising:

[0007] Time synchronization of each sensor.

[0008] An image sequence of the first sensor is acquired, preprocessed, and visual motion restoration is obtained.

[0009] The depth information of the environment is acquired through a second sensor, and point cloud fusion, feature registration and scale restoration are performed to acquire the visual motion restoration.

[0010] A visual odometer is obtained according to the visual motion restoration and the IMU motion between frames of the image sequence.

[0011] Furthermore, the visual odometer measurement method, after obtaining the visual odometer according to the visual motion restoration and the IMU motion between the image sequence frames, further includes:

[0012] The third sensor provides global positioning information for the visual odometer.

[0013] Furthermore, in the visual odometer measurement method, the scale recovery includes:

[0014] The image feature points and the normalized spherical coordinates of the point cloud are managed by kdtree, the three neighboring points of the graphic feature points are searched, and the distance from each point to the plane is calculated to restore the depth information of the image feature points.

[0015] Furthermore, the visual odometer measurement method manages the normalized spherical coordinates of the image feature points and the point cloud through kdtree, searches for three neighboring points of the graphic feature points, and calculates the distance from each point to the plane to restore the depth information of the image feature points, including:

[0016] For visual features without depth information, the Z axis is set as the initial value and projected to the camera body coordinate system and normalized.

[0017] The point cloud data of each of the second sensors is projected into the camera body coordinate system, and projected into the camera image and downsampled.

[0018] The second sensor is fused, filtered, projected to the point cloud data in the camera body coordinate system and normalized.

[0019] The visual feature point cloud is managed by KD Tree, and three adjacent points of the normalized feature point V are found. The threshold of 6 pixels corresponds to the square of the distance in Cartesian space. The three-dimensional coordinates of each point in the camera body coordinate system are calculated, and then the normal vector N determined by ABC is calculated:

[0020] N=(AB)X(BC)

[0021] In the formula, A, B, and C are the three neighboring points of the feature point V.

[0022] Calculate the distance L1 from the origin of the camera body coordinate system to the plane determined by ABC:

[0023] L1=(N(0)*A(0)+N(1)*A(1)+N(2)*A(2))

[0024] Wherein, N(0) is the horizontal coordinate component of the normal vector N, A(0) is the projection distance on N(0), N(1) is the vertical coordinate component of the normal vector N, A(1) is the projection distance on N(1), N(2) is the vertical coordinate component of the normal vector N, and A(2) is the projection distance on N(2).

[0025] Calculate the distance L2 from the origin to the plane determined by the normalized feature point and the normal vector N:

[0026] L2=(N(0)*V(0)+N(1)*V(1)+N(2)*V(2))

[0027] In the formula, N(0) is the horizontal coordinate component of the normal vector N, V(0) is the horizontal coordinate of the feature point, N(1) is the vertical coordinate component of the normal vector N, V(1) is the vertical coordinate of the feature point, N(2) is the vertical coordinate component of the normal vector N, and V(2) is the vertical coordinate of the feature point.

[0028] Calculate the depth s of the restored feature points:

[0029] s=L1 / L2

[0030] Wherein, L1 is the distance from the origin of the camera body coordinate system to the plane determined by ABC, and L2 is the distance from the origin to the plane determined by the normalized feature point and the normal vector N.

[0031] Furthermore, the visual odometer measurement method includes: each of the sensors includes:

[0032] Each of the sensors includes an RGB camera, an IMU, a depth camera, a lidar and WIFI.

[0033] Furthermore, in the visual odometer measurement method, the first sensor includes:

[0034] The first sensor is an RGB camera and an IMU.

[0035] Furthermore, in the visual odometer measurement method, the second sensor includes:

[0036] The second sensor includes a depth camera and a lidar.

[0037] Furthermore, in the visual odometer measurement method, the third sensor includes:

[0038] The third sensor is WIFI.

[0039] Furthermore, in the visual odometer measurement method, the preprocessing includes:

[0040] The preprocessing includes feature point extraction and tracking, semantic target detection and IMU pre-integration of the image sequence.

[0041] Furthermore, in the visual odometer measurement method, the preprocessing further includes:

[0042] The preprocessing includes global image retrieval, feature point matching, scale restoration and visual global positioning.

[0043] A second aspect of the present disclosure provides a visual odometer measurement device, comprising:

[0044] The synchronization unit is used to synchronize the time of each sensor.

[0045] The first acquisition unit is used to acquire an image sequence of the first sensor, perform preprocessing, and obtain visual motion restoration.

[0046] The second acquisition unit is used to acquire the depth information of the environment through a second sensor, and perform point cloud fusion, feature registration and scale restoration to obtain the visual motion restoration.

[0047] The third acquisition unit is used to acquire the visual odometer according to the visual motion restoration and the IMU motion between the image sequence frames.

[0048] Furthermore, the visual odometer measurement device further includes:

[0049] A unit is provided, for a third sensor to provide global positioning information for the visual odometer.

[0050] Furthermore, in the visual odometer measurement device, the second acquisition unit includes:

[0051] The first projection module is used to set the Z axis as an initial value for the visual features without depth information, and project it to the camera body coordinate system and normalize it.

[0052] The second projection module is used to project the point cloud data of each of the second sensors into the camera body coordinate system, and project them into the camera image and downsample them.

[0053] The third projection module is used to fuse, filter, project and normalize the point cloud data of the second sensor to the camera body coordinate system.

[0054] The first calculation module is used to manage the visual feature point cloud in KD Tree and find three adjacent points of the normalized feature point V. The threshold of 6 pixels corresponds to the square of the distance in Cartesian space. The three-dimensional coordinates of each point in the camera body coordinate system are calculated, and then the normal vector N determined by ABC is calculated:

[0055] N=(AB)X(BC)

[0056] In the formula, A, B, and C are the three neighboring points of the feature point V.

[0057] The second calculation module is used to calculate the distance L1 from the origin of the camera body coordinate system to the plane determined by ABC:

[0058] L1=(N(0)*A(0)+N(1)*A(1)+N(2)*A(2))

[0059] Wherein, N(0) is the horizontal coordinate component of the normal vector N, A(0) is the projection distance on N(0), N(1) is the vertical coordinate component of the normal vector N, A(1) is the projection distance on N(1), N(2) is the vertical coordinate component of the normal vector N, and A(2) is the projection distance on N(2).

[0060] The third calculation module is used to calculate the distance L2 from the origin to the plane determined by the normalized feature point and the normal vector N:

[0061] L2=(N(0)*V(0)+N(1)*V(1)+N(2)*V(2))

[0062] In the formula, N(0) is the horizontal coordinate component of the normal vector N, V(0) is the horizontal coordinate of the feature point, N(1) is the vertical coordinate component of the normal vector N, V(1) is the vertical coordinate of the feature point, N(2) is the vertical coordinate component of the normal vector N, and V(2) is the vertical coordinate of the feature point.

[0063] The fourth calculation module is used to calculate the depth s of the restored feature points:

[0064] s=L1 / L2

[0065] Wherein, L1 is the distance from the origin of the camera body coordinate system to the plane determined by ABC, and L2 is the distance from the origin to the plane determined by the normalized feature point and the normal vector N.

[0066] A third aspect of the present disclosure provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.

[0067] A fourth aspect of the present disclosure provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0068] The present disclosure provides a visual odometer measurement method, device, electronic device and medium, including: time synchronization of each sensor; acquiring an image sequence of a first sensor, performing preprocessing, and obtaining visual motion recovery; acquiring depth information of an environment through a second sensor, and performing point cloud fusion, feature registration and scale recovery to obtain the visual motion recovery; obtaining a visual odometer according to the visual motion recovery and the IMU motion between frames of the image sequence. Compared with the prior art, the present disclosure synchronizes each sensor time, thereby avoiding uncertainty in the time consumption of data transmission; and performing point cloud fusion, feature registration and scale recovery to solve the uncertainty in the scale of feature points. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0070] Figure 1 A flow chart of a visual odometer measurement method according to an embodiment of the present disclosure;

[0071] Figure 2 It is a flowchart of another visual odometer measurement method in an embodiment of the present disclosure;

[0072] Figure 3 A schematic diagram of the structure of a visual odometer measurement device according to an embodiment of the present disclosure;

[0073] Figure 4 It is a schematic diagram of the structure of another visual odometer measurement device in an embodiment of the present disclosure;

[0074] Figure 5 The present invention is a schematic diagram of the structure of a visual odometer measurement electronic device in an embodiment of the present invention. DETAILED DESCRIPTION

[0075] The technical solutions in the embodiments of the present disclosure will be described clearly and completely below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0076] Unless otherwise defined, all technical and scientific terms used in the present disclosure have the same meanings as those commonly understood by technicians in the field of the present disclosure; the terms used in the present disclosure are only for the purpose of describing specific embodiments, and are not intended to limit the terms "including" and "having" and any variations thereof in the specification and claims of the present disclosure and the above-mentioned drawings, and are intended to cover non-exclusive inclusions.

[0077] In the description of the embodiments of the present disclosure, the technical terms "first", "second", etc. are only used to distinguish different objects, and cannot be understood as indicating or implying relative importance or implicitly indicating the number, specific order or primary and secondary relationship of the indicated technical features. In the description of the embodiments of the present disclosure, the meaning of "multiple" is more than two, unless otherwise clearly and specifically defined.

[0078] In the description of the embodiments of the present disclosure, the term "and / or" is only a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B may represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0079] In the description of the embodiments of the present disclosure, the term "multiple" refers to more than two (including two). Similarly, "multiple groups" refers to more than two groups (including two groups), and "multiple pieces" refers to more than two pieces (including two pieces).

[0080] In the description of the embodiments of the present disclosure, the orientations or positional relationships indicated by technical terms such as "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise", "axial", "radial", and "circumferential" are based on the orientations or positional relationships shown in the accompanying drawings and are only for the convenience of describing the embodiments of the present disclosure and simplifying the description. They do not indicate or imply that the referred device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as limiting the embodiments of the present disclosure.

[0081] In the description of the embodiments of the present disclosure, unless otherwise clearly specified and limited, technical terms such as "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, and it can be the internal connection of two elements or the interaction relationship between two elements. For ordinary technicians in this field, the specific meanings of the above terms in the embodiments of the present disclosure can be understood according to the specific circumstances.

[0082] Example 1

[0083] The present disclosure provides a visual odometer measurement method, such as Figure 1 As shown, including:

[0084] 101. Perform time synchronization on each sensor.

[0085] A sensor (transducer / sensor) is a detection device that can sense the information being measured and convert the sensed information into electrical signals or other required forms of information output according to certain rules to meet the requirements of information transmission, processing, storage, display, recording and control.

[0086] Each sensor has its own clock source, and different sensors have different sampling frequencies. In addition, data transmission, camera exposure, etc. will produce uncontrollable delays. In order to effectively integrate the perception data of multiple sensors, time synchronization must be performed.

[0087] 102. Acquire an image sequence of a first sensor, perform preprocessing, and obtain visual motion restoration.

[0088] Image sequence: is a group of images associated with time or spatial positions. Image sequence is also called image stack or video.

[0089] 103. Acquire depth information of the environment through a second sensor, and perform point cloud fusion, feature registration, and scale restoration to obtain the visual motion restoration.

[0090] Point cloud fusion: It connects multiple images into a large image. Feature registration: It finds similar images, but the deformed images need some correction transformations such as rotation.

[0091] 104. Obtain a visual odometer according to the visual motion restoration and the IMU motion between frames of the image sequence.

[0092] IMU: (Inertial measurement unit) is a device that measures the three-axis attitude angle (or angular rate) and acceleration of an object.

[0093] The present disclosure provides a visual odometer measurement method, including: time synchronization of each sensor; acquiring an image sequence of a first sensor, performing preprocessing, and obtaining visual motion restoration; acquiring depth information of an environment through a second sensor, performing point cloud fusion, feature registration, and scale recovery, and obtaining the visual motion restoration; obtaining a visual odometer according to the visual motion recovery and the IMU motion between frames of the image sequence. Compared with the prior art, the present disclosure synchronizes each sensor time, thereby avoiding uncertainty in the time consumption of data transmission; and performing point cloud fusion, feature registration, and scale recovery, thereby solving the uncertainty in the scale of feature points.

[0094] Example 2

[0095] The present disclosure provides a visual odometer measurement method, such as Figure 2 As shown, including:

[0096] 201. Perform time synchronization on each sensor.

[0097] Specifically, a unified clock source provides the same reference time to each sensor, and each sensor calibrates its own clock time based on the provided reference time, thus achieving time synchronization in hardware. Since clock sources all have clock drift, and the clock drift of each clock source is different, even if the timestamps of each sensor are aligned at the initial moment, after running for a period of time, the previously aligned results will deviate. The solution to this problem is to unify the clock source in hardware.

[0098] Among them, the sensors include RGB camera, IMU, depth camera, lidar and WIFI.

[0099] 202. Acquire an image sequence of a first sensor, perform preprocessing, and obtain visual motion restoration.

[0100] Among them, the first sensor is an RGB camera and an IMU.

[0101] The preprocessing includes feature point extraction and tracking, semantic target detection and IMU pre-integration of the image sequence; in addition, it also includes global image retrieval, feature point matching, scale recovery and visual global positioning.

[0102] 203. Acquire depth information of the environment through a second sensor, and perform point cloud fusion, feature registration, and scale restoration to acquire the visual motion restoration.

[0103] Wherein, the second sensor includes a depth camera and a laser radar.

[0104] Among them, scale recovery manages the normalized spherical coordinates of the image feature points and the point cloud through kdtree, searches for the three neighboring points of the graphic feature points, and calculates the distance from each point to the plane to restore the depth information of the image feature points.

[0105] 2031. For visual features without depth information, set the Z axis as the initial value, and project it to the camera body coordinate system and normalize it.

[0106] 2032. Project the point cloud data of each of the second sensors into the camera body coordinate system, and project into the camera image and downsample.

[0107] 2033. Fuse, filter, and project the point cloud data of the second sensor to the camera body coordinate system and normalize the point cloud data.

[0108] 2034. Manage the visual feature point cloud in KD Tree, and find 3 adjacent points of the normalized feature point V therein, the threshold of 6 pixels corresponds to the square of the distance in Cartesian space, 3 point data: A, B, C, the 3D coordinates of each point in the camera body coordinate system, and then calculate the normal vector N determined by ABC:

[0109] N=(AB)X(BC)

[0110] In the formula, A, B, and C are the three neighboring points of the feature point V.

[0111] 2035. Calculate the distance L1 from the origin of the camera body coordinate system to the plane defined by ABC:

[0112] L1=(N(0)*A(0)+N(1)*A(1)+N(2)*A(2))

[0113] Wherein, N(0) is the horizontal coordinate component of the normal vector N, A(0) is the projection distance on N(0), N(1) is the vertical coordinate component of the normal vector N, A(1) is the projection distance on N(1), N(2) is the vertical coordinate component of the normal vector N, and A(2) is the projection distance on N(2).

[0114] 2036. Calculate the distance L2 from the origin to the plane determined by the normalized feature point and the normal vector N:

[0115] L2=(N(0)*V(0)+N(1)*V(1)+N(2)*V(2))

[0116] In the formula, N(0) is the horizontal coordinate component of the normal vector N, V(0) is the horizontal coordinate of the feature point, N(1) is the vertical coordinate component of the normal vector N, V(1) is the vertical coordinate of the feature point, N(2) is the vertical coordinate component of the normal vector N, and V(2) is the vertical coordinate of the feature point.

[0117] 2037. Calculate the depth s of the restored feature points:

[0118] s=L1 / L2

[0119] Wherein, L1 is the distance from the origin of the camera body coordinate system to the plane determined by ABC, and L2 is the distance from the origin to the plane determined by the normalized feature point and the normal vector N.

[0120] 204. Obtain a visual odometer according to the visual motion restoration and the IMU motion between frames of the image sequence.

[0121] 205. A third sensor provides global positioning information for the visual odometer.

[0122] Among them, the third sensor is WIFI.

[0123] The present disclosure provides a visual odometer measurement method, including: time synchronization of each sensor; acquiring an image sequence of a first sensor, performing preprocessing, and obtaining visual motion restoration; acquiring depth information of an environment through a second sensor, performing point cloud fusion, feature registration, and scale recovery, and obtaining the visual motion restoration; obtaining a visual odometer according to the visual motion recovery and the IMU motion between frames of the image sequence. Compared with the prior art, the present disclosure synchronizes each sensor time, thereby avoiding uncertainty in the time consumption of data transmission; and performing point cloud fusion, feature registration, and scale recovery, thereby solving the uncertainty in the scale of feature points.

[0124] In addition, the third sensor in this embodiment provides global positioning information for the visual odometer, thereby stabilizing the visual odometer.

[0125] Example 3

[0126] The present disclosure provides a visual odometer measurement device, such as Figure 3 As shown, including:

[0127] The synchronization unit 31 is used to perform time synchronization on each sensor.

[0128] The first acquisition unit 32 is used to acquire an image sequence of the first sensor, perform preprocessing, and obtain visual motion restoration.

[0129] The second acquisition unit 33 is used to acquire the depth information of the environment through a second sensor, and perform point cloud fusion, feature registration and scale restoration to obtain the visual motion restoration.

[0130] The third acquisition unit 34 is used to acquire the visual odometer according to the visual motion restoration and the IMU motion between the image sequence frames.

[0131] The present disclosure provides a visual odometer measurement device, comprising: a synchronization unit that performs time synchronization on each sensor; a first acquisition unit that acquires an image sequence of the first sensor, performs preprocessing, and obtains visual motion restoration; a second acquisition unit that acquires depth information of the environment through the second sensor, performs point cloud fusion, feature registration, and scale recovery, and obtains the visual motion restoration; a third acquisition unit that acquires a visual odometer according to the visual motion recovery and the IMU motion between frames of the image sequence. Compared with the prior art, the present disclosure synchronizes each sensor in time, thereby avoiding uncertainty in the time consumption of data transmission; and performs point cloud fusion, feature registration, and scale recovery, thereby solving the uncertainty in the scale of feature points.

[0132] Example 4

[0133] The present disclosure provides a visual odometer measurement device, such as Figure 4 As shown, including:

[0134] The synchronization unit 41 is used to perform time synchronization on each sensor.

[0135] The first acquisition unit 42 is used to acquire an image sequence of the first sensor, perform preprocessing, and obtain visual motion restoration.

[0136] The second acquisition unit 43 is used to acquire the depth information of the environment through a second sensor, and perform point cloud fusion, feature registration and scale restoration to obtain the visual motion restoration.

[0137] The first projection module 431 is used to set the Z axis as an initial value for the visual feature without depth information, and project it to the camera body coordinate system and normalize it.

[0138] The second projection module 432 is used to project the point cloud data of each of the second sensors into the camera body coordinate system, and project them into the camera image and downsample them.

[0139] The third projection module 433 is used to fuse, filter, project and normalize the point cloud data of the second sensor to the camera body coordinate system.

[0140] The first calculation module 434 is used to manage the visual feature point cloud in KD Tree, and find three adjacent points of the normalized feature point V therein, the threshold of 6 pixels corresponds to the square of the distance in the Cartesian space, 3 point data: A, B, C, the three-dimensional coordinates of each point in the camera body coordinate system, and then calculate the normal vector N determined by ABC:

[0141] N=(AB)X(BC)

[0142] In the formula, A, B, and C are the three neighboring points of the feature point V.

[0143] The second calculation module 435 is used to calculate the distance L1 from the origin of the camera body coordinate system to the plane determined by ABC:

[0144] L1=(N(0)*A(0)+N(1)*A(1)+N(2)*A(2))

[0145] Wherein, N(0) is the horizontal coordinate component of the normal vector N, A(0) is the projection distance on N(0), N(1) is the vertical coordinate component of the normal vector N, A(1) is the projection distance on N(1), N(2) is the vertical coordinate component of the normal vector N, and A(2) is the projection distance on N(2).

[0146] The third calculation module 436 is used to calculate the distance L2 from the origin to the plane determined by the normalized feature point and the normal vector N:

[0147] L2=(N(0)*V(0)+N(1)*V(1)+N(2)*V(2))

[0148] In the formula, N(0) is the horizontal coordinate component of the normal vector N, V(0) is the horizontal coordinate of the feature point, N(1) is the vertical coordinate component of the normal vector N, V(1) is the vertical coordinate of the feature point, N(2) is the vertical coordinate component of the normal vector N, and V(2) is the vertical coordinate of the feature point.

[0149] The fourth calculation module 437 is used to calculate the depth s of the restored feature points:

[0150] s=L1 / L2

[0151] Wherein, L1 is the distance from the origin of the camera body coordinate system to the plane determined by ABC, and L2 is the distance from the origin to the plane determined by the normalized feature point and the normal vector N.

[0152] The third acquisition unit 44 is used to acquire the visual odometer according to the visual motion restoration and the IMU motion between the image sequence frames.

[0153] The providing unit 45 is configured to use a third sensor to provide global positioning information for the visual odometer.

[0154] The present disclosure provides a visual odometer measurement device, comprising: a synchronization unit that performs time synchronization on each sensor; a first acquisition unit that acquires an image sequence of the first sensor, performs preprocessing, and obtains visual motion restoration; a second acquisition unit that acquires depth information of the environment through the second sensor, performs point cloud fusion, feature registration, and scale recovery, and obtains the visual motion restoration; a third acquisition unit that acquires a visual odometer according to the visual motion recovery and the IMU motion between frames of the image sequence. Compared with the prior art, the present disclosure synchronizes each sensor in time, thereby avoiding the uncertainty of data transmission time consumption; and performs point cloud fusion, feature registration, and scale recovery, thereby solving the uncertainty of feature point scale.

[0155] In addition, the providing unit provides global positioning information for the visual odometer, thereby stabilizing the visual odometer.

[0156] Example 5

[0157] The present disclosure provides a visual odometer measurement electronic device, such as Figure 5 As shown, it includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the above-mentioned visual odometer measurement method are implemented.

[0158] A processing device (e.g., a central processing unit, a graphics processing unit, etc.) 51 can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 52 or a program loaded from a storage device 58 into a random access memory (RAM) 53. Various programs and data required for the operation of the electronic device are also stored in the RAM 53. The processing device 51, the ROM 52, and the RAM 53 are connected to each other via a bus 54. An input / output (I / O) interface 55 is also connected to the bus 54.

[0159] Typically, the following devices may be connected to the I / O interface 55: an input device 56 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 57 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 58 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 59. The communication device 59 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Figure 5 An electronic device having various devices is shown, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead. Figure 5 Each block shown in the figure may represent one device, or may represent multiple devices as required.

[0160] In particular, according to some embodiments of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, some embodiments of the present disclosure include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In some such embodiments, the computer program can be downloaded and installed from the network through the communication device 59, or installed from the storage device 58, or installed from the ROM 52. When the computer program is executed by the processing device 51, the above-mentioned functions defined in the method of some embodiments of the present disclosure are executed.

[0161] The present disclosure provides a visual odometer measurement electronic device, comprising: performing time synchronization on each sensor; acquiring an image sequence of a first sensor, performing preprocessing, and obtaining visual motion restoration; acquiring depth information of an environment through a second sensor, and performing point cloud fusion, feature registration, and scale recovery to obtain the visual motion recovery; obtaining a visual odometer according to the visual motion recovery and the IMU motion between frames of the image sequence. Compared with the prior art, the present disclosure performs time synchronization on each sensor, thereby avoiding uncertainty in the time consumption of data transmission; and performs point cloud fusion, feature registration, and scale recovery to solve the uncertainty in the scale of feature points.

[0162] In addition, the third sensor in this embodiment provides global positioning information for the visual odometer, thereby stabilizing the visual odometer.

[0163] Example 6

[0164] The embodiment of the present disclosure provides a visual odometer measurement, wherein a computer program is stored on the computer-readable storage medium, and when the computer program is executed by one or more processors, the above-mentioned visual odometer measurement method is implemented.

[0165] It should be noted that the computer-readable medium in some embodiments of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In some embodiments of the present disclosure, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In some embodiments of the present disclosure, the computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0166] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0167] The computer-readable medium may be included in the device; or it may exist independently without being installed in the electronic device. The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: synchronizes the time of each sensor; obtains the image sequence of the first sensor, performs preprocessing, and obtains visual motion restoration; obtains the depth information of the environment through the second sensor, and performs point cloud fusion, feature registration, and scale restoration to obtain the visual motion restoration; obtains the visual odometer based on the visual motion restoration and the IMU motion between the image sequence frames.

[0168] Computer program code for performing the operations of some embodiments of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0169] The present disclosure provides a computer-readable medium for measuring visual odometers, including: performing time synchronization on each sensor; acquiring an image sequence of a first sensor, performing preprocessing, and obtaining visual motion recovery; acquiring depth information of an environment through a second sensor, and performing point cloud fusion, feature registration, and scale recovery to obtain the visual motion recovery; obtaining a visual odometer according to the visual motion recovery and the IMU motion between frames of the image sequence. Compared with the prior art, the present disclosure performs time synchronization on each sensor, thereby avoiding uncertainty in the time consumption of data transmission; and performing point cloud fusion, feature registration, and scale recovery to solve the uncertainty in the scale of feature points.

[0170] In addition, the third sensor in this embodiment provides global positioning information for the visual odometer, thereby stabilizing the visual odometer.

[0171] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0172] The units described in some embodiments of the present disclosure may be implemented by software or hardware. The units described may also be set in a processor, for example, it may be described as: a processor includes a receiving unit, an information acquisition unit, a target determination unit, and a distribution unit. The names of these units do not constitute limitations on the units themselves in some cases.

[0173] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0174] The above descriptions are only some preferred embodiments of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but should also cover other technical solutions formed by any combination of the above-mentioned technical features or their equivalent features without departing from the above-mentioned inventive concept. For example, the above-mentioned features are replaced with the technical features with similar functions disclosed in the embodiments of the present disclosure (but not limited to) and the technical solutions formed.

[0175] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, not to limit them; although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present invention, and they should all be included in the scope of the claims and specification of the present invention. In particular, as long as there is no structural conflict, the various technical features mentioned in each embodiment can be combined in any way. The present invention is not limited to the specific embodiments disclosed herein, but includes all technical solutions that fall within the scope of the claims.

Claims

1. A visual odometer measurement method, characterized in that: include: Time synchronization of each sensor; acquiring an image sequence of a first sensor, performing preprocessing, and obtaining visual motion restoration; Acquire the depth information of the environment through a second sensor, and perform point cloud fusion, feature registration and scale recovery to obtain the visual motion recovery; A visual odometer is obtained according to the visual motion restoration and the IMU motion between frames of the image sequence.

2. The visual odometer measurement method according to claim 1, characterized in that: After obtaining the visual odometer according to the visual motion restoration and the IMU motion between the image sequence frames, the method further includes: The third sensor provides global positioning information for the visual odometer.

3. The visual odometer measurement method according to claim 1, characterized in that: The scale restoration includes: The image feature points and the normalized spherical coordinates of the point cloud are managed by kdtree, the three neighboring points of the graphic feature points are searched, and the distance from each point to the plane is calculated to restore the depth information of the image feature points.

4. The visual odometer measurement method according to claim 3, characterized in that: The image feature points and the normalized spherical coordinates of the point cloud are managed by kdtree, three neighboring points of the graphic feature points are searched, and the distance from each point to the plane is calculated to restore the depth information of the image feature points, including: For visual features without depth information, the Z axis is set as the initial value, projected to the camera body coordinate system and normalized; Projecting the point cloud data of each of the second sensors into the camera body coordinate system, and projecting into the camera image and downsampling; fusing, filtering, projecting and normalizing the point cloud data of the second sensor to the camera body coordinate system; The visual feature point cloud is managed by KDTree, and three adjacent points of the normalized feature point V are found in it. The threshold of 6 pixels corresponds to the square of the distance in Cartesian space. The three-dimensional coordinates of each point in the camera body coordinate system are calculated, and then the normal vector N determined by ABC is calculated: N=(AB)X(BC) In the formula, A, B, and C are the three adjacent points of the feature point V; Calculate the distance L1 from the origin of the camera body coordinate system to the plane determined by ABC: L1=(N(0)*A(0)+N(1)*A(1)+N(2)*A(2)) Wherein, N(0) is the horizontal coordinate component of the normal vector N, A(0) is the projection distance on N(0), N(1) is the vertical coordinate component of the normal vector N, A(1) is the projection distance on N(1), N(2) is the vertical coordinate component of the normal vector N, and A(2) is the projection distance on N(2); Calculate the distance L2 from the origin to the plane determined by the normalized feature point and the normal vector N: L2=(N(0)*V(0)+N(1)*V(1)+N(2)*V(2)) Wherein, N(0) is the horizontal coordinate component of the normal vector N, V(0) is the horizontal coordinate of the feature point, N(1) is the vertical coordinate component of the normal vector N, V(1) is the vertical coordinate of the feature point, N(2) is the vertical coordinate component of the normal vector N, and V(2) is the vertical coordinate of the feature point; Calculate the depth s of the restored feature points: s=L1 / L2 Wherein, L1 is the distance from the origin of the camera body coordinate system to the plane determined by ABC, and L2 is the distance from the origin to the plane determined by the normalized feature point and the normal vector N.

5. The visual odometer measurement method according to claim 1, characterized in that: include: Each of the sensors comprises: Each of the sensors includes an RGB camera, an IMU, a depth camera, a lidar and WIFI.

6. The visual odometer measurement method according to claim 1 or 5, characterized in that: The first sensor comprises: The first sensor is an RGB camera and an IMU.

7. The visual odometer measurement method according to claim 1 or 5, characterized in that: The second sensor comprises: The second sensor includes a depth camera and a lidar.

8. The visual odometer measurement method according to claim 1 or 5, characterized in that: The third sensor comprises: The third sensor is WIFI.

9. The visual odometer measurement method according to claim 1, characterized in that: The pre-processing comprises: The preprocessing includes feature point extraction and tracking, semantic target detection and IMU pre-integration of the image sequence.

10. The visual odometer measurement method according to claim 1, characterized in that: The pre-processing further comprises: The preprocessing includes global image retrieval, feature point matching, scale restoration and visual global positioning.

11. A visual odometer measuring device, characterized in that: include: A synchronization unit, used for time synchronization of each sensor; A first acquisition unit is used to acquire an image sequence of a first sensor, perform preprocessing, and obtain visual motion restoration; A second acquisition unit, used to acquire depth information of the environment through a second sensor, and perform point cloud fusion, feature registration and scale restoration to acquire the visual motion restoration; The third acquisition unit is used to acquire the visual odometer according to the visual motion restoration and the IMU motion between the image sequence frames.

12. The visual odometer measuring device according to claim 11, characterized in that: Also includes: A unit is provided, for a third sensor to provide global positioning information for the visual odometer.

13. The visual odometer measuring device according to claim 11, characterized in that: The second acquiring unit includes: The first projection module is used to set the Z axis as an initial value for the visual features without depth information, and project it to the camera body coordinate system and normalize it; A second projection module, used for projecting the point cloud data of each of the second sensors into the camera body coordinate system, and projecting into the camera image and downsampling; A third projection module, used for fusing, filtering, projecting and normalizing the point cloud data of the second sensor to the camera body coordinate system; The first calculation module is used to manage the visual feature point cloud in KDTree and find three adjacent points of the normalized feature point V. The threshold of 6 pixels corresponds to the square of the distance in Cartesian space. The three-dimensional coordinates of each point in the camera body coordinate system are calculated, and then the normal vector N determined by ABC is calculated: N=(AB)X(BC) In the formula, A, B, and C are the three adjacent points of the feature point V; The second calculation module is used to calculate the distance L1 from the origin of the camera body coordinate system to the plane determined by ABC: L1=(N(0)*A(0)+N(1)*A(1)+N(2)*A(2)) Wherein, N(0) is the horizontal coordinate component of the normal vector N, A(0) is the projection distance on N(0), N(1) is the vertical coordinate component of the normal vector N, A(1) is the projection distance on N(1), N(2) is the vertical coordinate component of the normal vector N, and A(2) is the projection distance on N(2); The third calculation module is used to calculate the distance L2 from the origin to the plane determined by the normalized feature point and the normal vector N: L2=(N(0)*V(0)+N(1)*V(1)+N(2)*V(2)) Wherein, N(0) is the horizontal coordinate component of the normal vector N, V(0) is the horizontal coordinate of the feature point, N(1) is the vertical coordinate component of the normal vector N, V(1) is the vertical coordinate of the feature point, N(2) is the vertical coordinate component of the normal vector N, and V(2) is the vertical coordinate of the feature point; The fourth calculation module is used to calculate the depth s of the restored feature points: s=L1 / L2 Wherein, L1 is the distance from the origin of the camera body coordinate system to the plane determined by ABC, and L2 is the distance from the origin to the plane determined by the normalized feature point and the normal vector N.

14. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 10 are implemented.

15. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 10 are implemented.

Images