Visual odometer method and electronic equipment

Abstract

The invention provides a visual odometer method and electronic equipment. A terminal obtains a real-time image sequence through a camera, obtains a positioning image from the image sequence at a first sampling frequency, and sends the positioning image to a server, and the server processes the positioning image to obtain global pose information of the terminal corresponding to the positioning image. And the terminal determines relative pose information of an image between the positioning image and the next positioning image based on a visual mileage calculation method, and obtains global pose information corresponding to each frame of image in the image sequence in combination with global pose information of the terminal corresponding to the positioning image obtained by the server. The problem of accumulative error drift is solved, and the accuracy of the visual odometer is improved. And the terminal equipment can perform visual odometer based on one camera, so that the scale uncertainty of monocular vision is solved. The complexity of terminal calculation is reduced, the accuracy of scale calculation is improved, and the real-time performance of calculation is ensured.

Description

Technical Field

[0001] The present disclosure relates to the field of computer vision technology, and in particular to a visual odometer method and electronic device. Background Art

[0002] Visual Odometry (VO) is a technology that estimates the position changes of a camera (or robot, drone, etc.) in space by analyzing the image sequence acquired by the camera.

[0003] The visual odometry usually needs to cooperate with other sensors to obtain the corresponding global pose. For example, the sensor can be an inertial measurement unit (IMU) or a laser radar (LiDAR).

[0004] However, current terminals and other devices, such as smartphones, may have more than one camera, but these multiple cameras are used for photo-taking scenes, with different field of view (FOV), focal length, etc., and at the operating system level, users can only obtain one of the main cameras for building a visual odometer. Most terminals and other devices do not have sensors that can cooperate with visual odometers to determine global pose, making it impossible to achieve corresponding global pose determination on the terminal. Summary of the invention

[0005] The present disclosure provides a visual odometer method and an electronic device to solve the problem that the global posture cannot be determined on the current terminal.

[0006] In a first aspect, the present disclosure provides a visual odometer method, the method comprising:

[0007] Acquire image sequence through camera;

[0008] Acquire a positioning image from the image sequence at a first sampling frequency, and send the positioning image to a server, so that the server determines global position information of the positioning image according to the positioning image and map data;

[0009] Receiving global pose information of the positioning image sent by the server;

[0010] Respectively obtain images other than the positioning image in the image sequence as first images, and perform the following processing: determine first relative position information between the first image and a previous frame image; the previous frame image is an image in the image sequence that is located before the first image and adjacent to the first image;

[0011] The global pose information of the first image is obtained according to the global pose information of the positioning image corresponding to the first image and the first relative pose information; the positioning image corresponding to the first image is a positioning image that is located before the first image in the image sequence and is closest to the first image.

[0012] In some embodiments, the method further comprises:

[0013] Determine the speed information of the positioning image according to the global pose information of the positioning image and the global pose information of a positioning image of a previous frame of the positioning image;

[0014] The obtaining the global pose information of the first image according to the global pose information of the positioning image corresponding to the first image and the first relative pose information includes:

[0015] Acquire an absolute displacement between the first image and the previous frame image, wherein the absolute displacement is determined according to speed information of a positioning image corresponding to the first image, the first sampling frequency, and a frequency at which the camera acquires the image sequence;

[0016] Determine a scale factor according to the first relative pose information, the absolute displacement between the first image and the previous frame image, and the global pose information of the previous frame image;

[0017] The global pose information of the first image is determined according to the scale factor, the first relative pose information and the global pose information of the previous frame image.

[0018] In some embodiments, determining the speed information of the positioning image according to the global pose information of the positioning image and the global pose information of a previous frame positioning image of the positioning image includes:

[0019] Determine a first movement distance according to the global position information of the positioning image and the global position information of a positioning image of a previous frame of the positioning image; the first movement distance refers to the movement distance between the moment when the positioning image is acquired and the moment when the previous positioning image of the positioning image is acquired;

[0020] The speed information of the positioning image is determined according to the first movement distance, the average movement speed, the total time length corresponding to the previous positioning image of the positioning image and the first sampling frequency; the average movement speed refers to the average speed from the moment when the image sequence is started to be acquired to the moment when the positioning image is acquired.

[0021] In some embodiments, determining the first movement distance according to the global pose information of the positioning image and the global pose information of a previous frame positioning image of the positioning image includes:

[0022] The first movement distance is determined by the following formula:

[0023]

[0024] Among them, S i is the first movement distance; R i is the posture information in the global pose information of the positioning image; t i is the position information in the global pose information of the positioning image; R i-1 is the posture information in the global posture information of the previous positioning image of the positioning image; i-1 It is the position information in the global pose information of the previous positioning image.

[0025] In some embodiments, determining the speed information of the positioning image according to the first movement distance, the average movement speed, the total duration corresponding to the previous positioning image of the positioning image, and the first sampling frequency includes:

[0026] The velocity information of the positioning image is determined by the following formula:

[0027]

[0028] Among them, V i V is the velocity value of the positioning image; i-1 is the average movement speed before obtaining the positioning image; Tt is the total time length corresponding to the previous positioning image; S i is the first movement distance; dt is the first sampling period.

[0029] In some embodiments, determining the scale factor according to the first relative pose information, the absolute displacement between the first image and the previous frame image, and the global pose information of the previous frame image includes:

[0030] The scale factor is determined according to the following formula:

[0031]

[0032] Among them, λ is the scale factor; R is the rotation information in the global pose information of the previous frame image; ΔR is the relative rotation information in the first relative pose information of the first image, and Δt is the relative position information in the first relative pose information of the first image.

[0033] In some embodiments, determining the global pose information of the first image according to the scale factor, the first relative pose information and the global pose information of the previous frame image includes:

[0034] The global pose information of the first image is determined according to the following formula:

[0035] R I =ΔR·R J

[0036] t I =ΔR·t J +λ·Δt

[0037] Among them, R I is the rotation information in the global pose information of the first image; t I is the position information in the global pose information of the first image; λ is the scale factor; ΔR is the relative rotation information in the first relative pose information of the first image, and Δt is the relative position information in the first relative pose information of the first image; R J is the rotation information in the global pose information of the previous frame image; t J It is the position information in the global pose information of the previous frame image.

[0038] In some embodiments, determining first relative pose information between the first image and a previous frame of image includes:

[0039] Performing feature extraction on the first image to obtain feature points of the first image;

[0040] Performing optical flow tracking processing on feature points of the first image and feature points of an image frame before the first image to obtain tracking point pairs;

[0041] First relative position information between the first image and the previous frame image is determined according to the tracking point pair.

[0042] In some embodiments, extracting features from the first image to obtain feature points of the first image includes:

[0043] Performing uniform grid division processing on the first image to obtain N image blocks, where N is an integer greater than 1;

[0044] Feature extraction is performed on each of the N image blocks to obtain feature points of the first image.

[0045] In some embodiments, performing optical flow tracking on the feature points of the first image and the feature points of a frame of an image before the first image to obtain tracking point pairs includes:

[0046] determining an inter-frame motion offset according to speed information of a positioning image corresponding to the first image;

[0047] Based on the inter-frame motion offset, optical flow tracking processing is performed on the feature points of the first image and the feature points of a frame image before the first image to obtain tracking point pairs.

[0048] In some embodiments, performing optical flow tracking processing on feature points of the first image and feature points of a frame image before the first image based on the inter-frame motion offset to obtain tracking point pairs includes:

[0049] Based on the inter-frame motion offset, forward optical flow tracking is performed on the feature points of the first image and the feature points of the previous frame image to obtain forward tracking point pairs;

[0050] Based on the feature points in the first image in the forward tracking point pair, performing reverse optical flow tracking processing on the feature points of the first image and the feature points of the previous frame image to obtain a reverse tracking point pair;

[0051] The intersection between the forward tracking point pair and the reverse tracking point pair is obtained to obtain a tracking point pair.

[0052] In a second aspect, the present disclosure provides a visual odometer method, the method comprising:

[0053] receiving a positioning image sent by a terminal; the positioning image is acquired by the terminal from an image sequence at a first sampling frequency, and the image sequence is acquired by the terminal through a camera;

[0054] Determining global position information of the positioning image according to the positioning image and the map data;

[0055] The global pose information of the positioning image is sent to the terminal, so that the terminal obtains images other than the positioning image in the image sequence as first images, and performs the following processing: determining the first relative pose information between the first image and a previous frame image; the previous frame image is an image that is located before the first image and adjacent to the first image in the image sequence; obtaining the global pose information of the first image according to the speed information of the positioning image corresponding to the first image and the first relative pose information; the positioning image corresponding to the first image is a positioning image that is located before the first image and closest to the first image in the image sequence.

[0056] In a third aspect, the present disclosure provides an electronic device, including a memory and a processor, wherein the memory stores a computer program executable on the processor, and when the processor executes the program, the method described in the first aspect is implemented.

[0057] In a fourth aspect, the present disclosure provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program executable on the processor, and when the processor executes the program, the method described in the second aspect is implemented.

[0058] In a fifth aspect, the present disclosure provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in the first aspect above.

[0059] In a sixth aspect, the present disclosure provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in the second aspect above.

[0060] The visual odometer method and electronic device provided by the embodiment of the present disclosure, the terminal obtains a real-time image sequence through a camera, obtains a positioning image from the image sequence at a first sampling frequency, and sends the positioning image to a server, the server processes the positioning image, obtains the global position and posture information of the terminal corresponding to the positioning image, the terminal determines the relative position and posture information of the image between the positioning image and the next positioning image, and obtains the global position and posture information corresponding to each frame of the image in the image sequence in combination with the global position and posture information of the terminal corresponding to the positioning image obtained by the server. Each frame of the image is processed based on the global position and posture information of the corresponding positioning image, which solves the problem of cumulative error drift and improves the accuracy of the visual odometer. The global position and posture information of the positioning image is obtained by the server, so that the terminal device can perform visual odometer calculation based on one camera, which solves the scale uncertainty of monocular vision. In addition, the complex calculation process of obtaining the global position and posture information is completed by the server, which does not occupy the computing resources of the terminal, reduces the complexity of the terminal calculation, improves the accuracy of the scale calculation, and ensures the real-time performance of the calculation. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 A schematic diagram of the structure of a visual odometer system provided in an embodiment of the present disclosure;

[0062] Figure 2 An interactive schematic diagram of a visual odometer method provided by an embodiment of the present disclosure;

[0063] Figure 3 An interactive schematic diagram of another visual odometer method provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0064] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0065] In the present disclosure, "at least one" refers to one or more, and "plurality" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a alone, b alone, or c alone can represent: a alone, b alone, c alone, a and b in combination, a and c in combination, b and c in combination, or a, b, and c in combination, where a, b, and c can be single or multiple. In addition, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance.

[0066] The directions or positional relationships indicated by terms such as "center", "longitudinal", "lateral", "up", "down", "left", "right", "front", and "back" are based on the directions or positional relationships shown in the accompanying drawings and are only for the convenience of describing the present disclosure and simplifying the description. They do not indicate or imply that the device or element referred to must have a specific direction, be constructed and operated in a specific direction, and therefore should not be understood as a limitation on the present disclosure.

[0067] The terms "connected" and "connected" should be understood in a broad sense. For example, the "connected" or "connected" of a circuit structure may refer to not only a physical connection, but also an electrical connection or a signal connection. For example, it may be a direct connection, i.e. a physical connection, or it may be an indirect connection through at least one intermediate element, as long as the circuit is connected. It may also be the internal connection of two elements. In addition to signal connection through a circuit, signal connection may also refer to signal connection through a media medium, such as radio waves. For those of ordinary skill in the art, the specific meanings of the above terms in this disclosure may be understood according to specific circumstances.

[0068] The concepts involved in the present disclosure are introduced below.

[0069] Visual Odometry (VO) calculates the relative motion of a device relative to the previous frame and is often used for short-term motion estimation. Visual Localization is a technology that uses image data acquired by visual sensors (such as cameras) to determine the absolute position and posture of a device in an environment. Visual localization not only focuses on relative motion, but also accurately calculates the position of a device in a global coordinate system based on a known map or reference. That is, visual localization is usually based on a global map or a known reference, providing global pose information of the device in a global environment. Typically, global pose information includes global position and pose, which can be a six-degree-of-freedom pose.

[0070] The basic principle of visual odometry is to calculate the relative motion of the camera with respect to the environment by matching image features (such as key points and edges) between consecutive frame images.

[0071] Visual odometry is widely used in unmanned driving, robot navigation, augmented reality and other fields.

[0072] Although visual odometry is very useful in many application scenarios, it also has some obvious disadvantages and limitations. The following are the determinations of visual odometry:

[0073] 1. Sensitive to light and environmental changes:

[0074] Visual odometry relies on image features, so they may be difficult to extract or match in environments with varying lighting, shadows, strong light, or dim lighting. Poor lighting, reflections, strong shadows, etc. can all lead to degraded performance.

[0075] 2. Cumulative error (drift):

[0076] Visual odometry works by estimating relative motion frame by frame, and errors accumulate over long periods of time. This error is called drift, which can cause the final pose estimate to deviate significantly from the actual position, especially when there is no external reference (such as GPS, map) for correction.

[0077] 3. Dynamic scene problem:

[0078] If there are moving objects in the scene (such as vehicles, pedestrians, etc.), the movement of these objects will affect the stability of visual features, resulting in inaccurate estimated motion information. In complex dynamic environments, the robustness of visual odometry is poor.

[0079] 4. Computational complexity and real-time challenges:

[0080] Image processing and feature matching are computationally intensive tasks, especially when high-resolution images and complex algorithms are used, which places high demands on processor performance. If hardware resources are limited or real-time processing is required, visual odometry may not be able to meet real-time requirements.

[0081] 5. Not friendly to environments lacking texture:

[0082] Visual odometry relies on texture features in the environment, and it is difficult to accurately estimate motion if the scene lacks sufficient identifiable features (e.g., smooth walls, large areas of a single color).

[0083] 6. Scale uncertainty:

[0084] In monocular visual odometry (i.e., only one camera), the actual scale of the object cannot be directly measured, resulting in inaccurate scale information in pose estimation. This requires additional sensors (such as IMU or binocular cameras) or specific environmental information to solve.

[0085] Due to the above shortcomings, visual odometry systems are usually used in combination with other sensors to improve the overall robustness and accuracy of the system.

[0086] However, on mobile phones, although today's smartphones have more than one camera, these multiple cameras are used for photo-taking scenes, with different fields of view FOV, focal lengths, etc., and at the operating system level, users can only obtain one of the main cameras to build the algorithm for the visual odometer. Therefore, on smartphones, the visual odometer can only be regarded as a monocular camera. In addition, except for the flagship smartphones in the market, which dominate the sales and ownership, the vast majority of smartphones do not have very good IMU devices, or only have gyroscopes, or even no IMU. Therefore, the traditional algorithms used by fusing with other sensors are not available on most smartphones.

[0087] The visual odometer method provided in this disclosure can be applied to the visual odometer system. Figure 1 , Figure 1 The structure diagram of a visual odometer system provided by an embodiment of the present disclosure is as follows. The visual odometer system provided by this embodiment includes a terminal 101 and a server 102, and the terminal 101 and the server 102 are connected via a network communication.

[0088] Among them, terminal 101 can refer to user equipment, access terminal, user unit, user station, mobile station, mobile station, remote station, remote terminal, mobile device, user terminal, wireless communication device, user agent or user device. The terminal device may also be a satellite phone, a cellular phone, a smart phone, a wireless data card, a wireless modem, a machine type communication device, a cordless phone, a session initiation protocol (SIP) phone, a wireless local loop (WLL) station, a personal digital assistant (PDA), a handheld device with wireless communication function, a computing device or other processing device connected to a wireless modem, a vehicle-mounted device or a wearable device, a virtual reality (VR) terminal device, an augmented reality (AR) terminal device, a wireless terminal in industrial control, a wireless terminal in self-driving, a wireless terminal in remote medical, a wireless terminal in a smart grid, a wireless terminal in transportation safety, a wireless terminal in a smart city, a wireless terminal in a smart home, a terminal device in a 5G network, a terminal device in a 6G network, or a terminal device in a future communication network, etc. In addition, the terminal device may also be a terminal device in an Internet of Things (IoT) system.

[0089] The server 102 may be a single server or a server cluster consisting of two or more servers, which is not limited in the present disclosure.

[0090] The visual odometer method provided by the present disclosure is that the terminal obtains a real-time image sequence through a camera, obtains a positioning image from the image sequence at a first sampling frequency, and sends the positioning image to a server, the server processes the positioning image, obtains the global position and posture information of the terminal corresponding to the positioning image, the terminal determines the relative position and posture information of the image between the positioning image and the next positioning image, and combines the global position and posture information of the terminal corresponding to the positioning image obtained by the server to obtain the global position and posture information corresponding to each frame of the image in the image sequence. Each frame of the image is processed based on the global position and posture information of the corresponding positioning image, which solves the problem of cumulative error drift and improves the accuracy of the visual odometer. The global position and posture information of the positioning image is obtained by the server, so that the terminal device can perform visual odometer calculation based on one camera, which solves the scale uncertainty of monocular vision. In addition, the complex calculation process of obtaining the global position and posture information is completed by the server, which does not occupy the computing resources of the terminal, reduces the complexity of the terminal calculation, improves the accuracy of the scale calculation, and ensures the real-time performance of the calculation.

[0091] The technical solution provided by the present disclosure is described in detail below with specific embodiments.

[0092] See also Figure 2 , Figure 2 This is an interactive schematic diagram of a visual odometer method provided by an embodiment of the present disclosure. The method provided by this embodiment can be applied to a visual odometer system, which can be the above-mentioned Figure 1 The visual odometer system in the embodiment shown. The terminal in this embodiment can be the above Figure 1 The terminal 101 in the visual odometer system in the embodiment shown. The server in this embodiment can be the above Figure 1 The server 102 in the visual odometer system in the illustrated embodiment. The method provided in this embodiment may include the following steps 201 to 206.

[0093] Step 201: The terminal obtains an image sequence through a camera.

[0094] The camera may belong to the terminal or be connected to the terminal. The camera may shoot the surrounding scene to obtain a video. The screen portion of the video may be regarded as an image sequence obtained by arranging multiple images in the order in which they are acquired, and each image in the image sequence may be referred to as a frame image.

[0095] Step 202: The terminal obtains a positioning image from the image sequence at a first sampling frequency, and sends the positioning image to the server.

[0096] The camera shoots at a preset frequency to obtain an image sequence. Generally, the preset frequency when the camera shoots can be set according to the actual application scenario. Exemplarily, the preset frequency can be a frequency between 30Hz and 60Hz. The first sampling frequency is less than the preset frequency. Exemplarily, the first sampling frequency can be 1Hz. In the present disclosure, the sampling period corresponding to the first sampling frequency is referred to as the first sampling period.

[0097] In actual applications, the terminal uses a camera to shoot the environment in real time to obtain an image sequence. The terminal obtains a positioning image from the image sequence at a first sampling frequency. Each time a positioning image is obtained, the terminal sends the positioning image to the server.

[0098] Step 203: The server determines the global position information of the positioning image according to the positioning image and the map data.

[0099] After receiving the positioning image sent by the terminal, the server processes the positioning image and determines the global position information of the positioning image based on the positioning image and the map data stored in the server.

[0100] The global pose information of the positioning image refers to the global pose information when the terminal device obtains the positioning image through the camera. The global pose information is used to characterize the pose of the terminal in the map. Generally, the global pose information includes the global position and pose, and the global pose information can be a six-degree-of-freedom pose.

[0101] The map data may include: a plurality of key frame images, positions and postures corresponding to the key frame images, a three-dimensional (3D) point cloud, and possible auxiliary information.

[0102] Exemplarily, the server can obtain and store map data. The map data can be obtained through a mapping process. Several possible mapping processes are introduced below. The mapping process is to map the collected scene data through a mapping system to obtain offline map data. Ordinary smart phones, panoramic cameras or laser scanning devices can be used to collect scene data. The mapping process is to map the collected data, and its core algorithm varies depending on the input source of the collected data. For data collected using camera devices such as mobile phones or panoramic cameras, algorithms such as Structure from Motion or Visual Simultaneous Localization and Mapping (Visual SLAM) can be used for mapping. Data collected using laser light equipment is often mapped using algorithms such as Lidar-SLAM or truncated signed distance function (TSDF). After the mapping process, the map data is obtained.

[0103] Exemplarily, step 203 may be implemented through the following steps.

[0104] Step 2031: The server extracts features from the positioning image to obtain feature points of the positioning image.

[0105] The feature point can be represented by a two-dimensional (2D) point.

[0106] Step 2032: The server performs feature matching in the map data based on the feature points of the positioning image to obtain matching feature points.

[0107] The matched feature points may be the correspondence between the 2D points in the positioning image and the 3D points in the map data, that is, 2D-3D point pairs.

[0108] Step 2033: The server obtains the global pose information of the positioning image based on the matched feature points and the geometric information.

[0109] Furthermore, the global position information of the positioning image in the map can be obtained through a perspective-n-Point (PnP) solution method.

[0110] Step 204: The server sends the global pose information of the positioning image to the terminal.

[0111] After obtaining the global pose information of the positioning image, the server sends the global pose information of the positioning image to the terminal. Since the server can handle a large number of complex computing tasks, the server can quickly obtain the global pose information of the positioning image based on the positioning image, and then send the global pose information of the positioning image to the terminal.

[0112] The terminal needs to obtain the global pose information of all images in the image sequence. Since the server obtains the global pose information of the positioning image and sends it to the terminal, the terminal does not need to process the positioning image. For the images between every two positioning images in the image sequence, the terminal needs to refer to the global pose information of the positioning image and process the images after the positioning image and before the next positioning image to obtain the global pose information of each image, which is described in detail below.

[0113] The terminal obtains images other than the positioning image in the image sequence as first images, and performs the following processing of step 205 and step 206.

[0114] Step 205: The terminal determines first relative posture information between the first image and the previous frame image.

[0115] The previous frame image is an image that is located before the first image and adjacent to the first image in the image sequence.

[0116] From the above discussion, the visual odometry method can determine the relative posture at a short distance. Therefore, each time the terminal obtains a first image, the terminal determines the first relative posture information based on the visual odometry method according to the first image and the previous frame image.

[0117] Step 206: The terminal obtains the global pose information of the first image according to the global pose information of the positioning image corresponding to the first image and the first relative pose information.

[0118] The positioning image corresponding to the first image is a positioning image that is located before the first image in the image sequence and is closest to the first image.

[0119] Since the positioning image has been processed by the server and the global pose information of the positioning image has been obtained, for the images in the first sampling period after the positioning image, the global pose information of each image can be determined based on the global pose information of the positioning image and the relative pose between two adjacent images.

[0120] In the visual odometer method provided in this embodiment, the terminal obtains a real-time image sequence through a camera, obtains a positioning image from the image sequence at a first sampling frequency, and sends the positioning image to a server. The server processes the positioning image to obtain the global position information of the terminal corresponding to the positioning image. The terminal determines the relative position information of the image between the positioning image and the next positioning image, and combines the global position information of the terminal corresponding to the positioning image obtained by the server to obtain the global position information corresponding to each frame of the image in the image sequence. Each frame of the image is processed based on the global position information of the corresponding positioning image, which solves the problem of cumulative error drift and improves the accuracy of the visual odometer. The global position information of the positioning image is obtained by the server, so that the terminal device can perform the calculation of the visual odometer based on one camera, which solves the scale uncertainty of monocular vision. In addition, the complex calculation process of obtaining the global position information is completed by the server, which does not occupy the computing resources of the terminal, reduces the complexity of the terminal calculation, improves the accuracy of the scale calculation, and ensures the real-time performance of the calculation.

[0121] See also Figure 3 , Figure 3 An interactive schematic diagram of another visual odometer method provided by an embodiment of the present disclosure, Figure 3 is Figure 2 On the basis of the illustrated embodiment, further, the following step 207 may be included after step 204 , and correspondingly, step 206 may be implemented through the following steps 2061 and 2062 .

[0122] Step 207: The terminal determines the speed information of the positioning image according to the global position information of the positioning image and the global position information of the previous frame positioning image.

[0123] The terminal can store the global pose information of the positioning image locally each time after receiving the global pose information of the positioning image sent by the server. The terminal determines the speed information of the positioning image based on the global pose information of the positioning image and the global pose information of the positioning image of the previous frame of the positioning image. The speed information of the positioning image is used to indicate the speed of the terminal when the terminal acquires the positioning image. For the time period within the first sampling period after acquiring the positioning image, since the time to acquire the positioning image is relatively close, the global pose information of the terminal can be calculated based on the speed information of the terminal determined by the positioning image.

[0124] Furthermore, the terminal may process the global position information of the received positioning image through an internal motion model module to obtain speed information of the positioning image.

[0125] Further, step 207 can be implemented through the following steps 2071 and 2072.

[0126] Step 2071: The terminal determines a first movement distance according to the global position information of the positioning image and the global position information of the previous frame positioning image.

[0127] The first movement distance refers to the movement distance of the terminal between the moment when the positioning image is acquired and the moment when the previous positioning image of the positioning image is acquired.

[0128] Furthermore, the first movement distance can be determined by the following formula (1).

[0129]

[0130] Among them, S i is the first movement distance; R i is the posture information in the global pose information of the positioning image; t i is the position information in the global pose information of the positioning image; R i-1 is the posture information in the global posture information of the previous positioning image of the positioning image; i-1 is the position information in the global pose information of the previous positioning image of the positioning image. “||||” represents the norm.

[0131] Step 2072: The terminal determines speed information of the positioning image according to the first movement distance, the average movement speed, the total duration corresponding to the previous positioning image of the positioning image, and the first sampling frequency.

[0132] The average motion speed refers to the average speed of the terminal from the moment when the image sequence is acquired to the moment when the positioning image is acquired.

[0133] The total duration corresponding to the previous positioning image of the positioning image refers to the time length from the moment when the image sequence is started to be acquired to the moment when the previous positioning image of the positioning image is acquired. The total duration corresponding to the previous positioning image of the positioning image is also: the duration of operation of the visual odometer system in the terminal until the previous positioning image of the positioning image is acquired.

[0134] The first sampling period can be determined by the first sampling frequency, and the first sampling period is the time length between acquiring the positioning image and acquiring the previous positioning image of the positioning image.

[0135] In this embodiment, after receiving the global position information of the positioning image sent by the server, the terminal can determine the speed information of the positioning image, thereby determining the current speed. Since the speed of the terminal is not a sudden change, the speed of the current terminal can be estimated by the accumulated speed change from the moment when the image sequence is first acquired.

[0136] Furthermore, the speed information of the positioning image can be determined by the following formula (2).

[0137]

[0138] Among them, V i V is the velocity value of the positioning image; i-1 is the average movement speed before obtaining the positioning image; Tt is the total time length corresponding to the previous positioning image; S i is the first movement distance; dt is the first sampling period.

[0139] Step 2061: Obtain the absolute displacement between the first image and the previous frame image.

[0140] The absolute displacement between the first image and the previous frame image refers to the displacement of the terminal in the time period between the moment of acquiring the previous frame image and the moment of acquiring the first image. The absolute displacement is determined based on the speed information of the positioning image corresponding to the first image, the first sampling frequency, and the frequency of the camera acquiring the image sequence. The frequency of the camera acquiring the image sequence is the preset frequency in the above embodiment.

[0141] Furthermore, assuming that the first sampling frequency is 1 Hz, the absolute displacement between the first image and the previous frame image can be determined by the following formula (3).

[0142] s=V i / F Formula (3)

[0143] Where s is the absolute displacement between the first image and the previous frame, V i is the speed information of the positioning image corresponding to the first image, and F is the frequency at which the camera acquires the image sequence.

[0144] Step 2062: The terminal determines a scale factor according to the first relative posture information, the absolute displacement between the first image and the previous frame image, and the global posture information of the previous frame image.

[0145] For the first image and the previous frame image, the terminal can determine the scale factor based on the global pose information of the previous frame image and in combination with the first relative pose information of the two frames of images and the absolute displacement between the two frames of images.

[0146] Furthermore, the scale factor can be determined by the following formula (4).

[0147]

[0148] Among them, λ is the scale factor; R is the rotation information in the global pose information of the previous frame image; ΔR is the relative rotation information in the first relative pose information of the first image, and Δt is the relative position information in the first relative pose information of the first image.

[0149] Step 2063: The terminal determines the global pose information of the first image according to the scale factor, the first relative pose information and the global pose information of the previous frame image.

[0150] The terminal can determine the global pose information of the first image based on the global pose information of the previous frame image, the scale factor obtained in the above steps, and the first relative pose information between the first image and the previous frame image.

[0151] Furthermore, the global pose information of the first image is determined according to the following formula (5) and formula (6):

[0152] R I =ΔR·R J Formula (5)

[0153] t I =ΔR·t I +λ·Δt Formula (6)

[0154] Among them, R I is the rotation information in the global pose information of the first image; t I is the position information in the global pose information of the first image; λ is the scale factor; ΔR is the relative rotation information in the first relative pose information of the first image, and Δt is the relative position information in the first relative pose information of the first image; R J is the rotation information in the global pose information of the previous frame image; t J It is the position information in the global pose information of the previous frame image.

[0155] In this embodiment, the terminal can calculate the speed when acquiring the positioning image by using the global pose information of the positioning image obtained by the server. And on the basis of this speed, in the process of extrapolating the image in the first sampling period after the positioning image, the global pose information of the image in the first sampling period after the positioning image is obtained by combining the relative pose information of the image and its previous frame image. The complexity of terminal calculation is reduced, the accuracy of scale calculation is improved, and the real-time calculation is guaranteed. The problem of cumulative error drift is solved, and the accuracy of the visual odometer is improved. By obtaining the global pose information of the positioning image through the server, the terminal device can perform the calculation of the visual odometer based on one camera, which solves the scale uncertainty of monocular vision.

[0156] In some scenarios, there are multiple ways to implement the visual mileage calculation method in step 205. The following is an exemplary implementation of a visual mileage calculation method.

[0157] In some embodiments, step 205 may be implemented through steps 2051 to 2053 as follows.

[0158] Step 2051: The terminal extracts features from the first image to obtain feature points of the first image.

[0159] The feature extraction is performed on the first image, and the feature points of the first image obtained are 2D points.

[0160] There are multiple ways for the terminal to extract features from the first image, wherein the feature extraction can usually be performed on the first image as a whole.

[0161] However, in dynamic scenes such as pedestrians and scene changes, such as posting posters, if the texture features in the first image are relatively concentrated, then the extracted feature points will be concentrated in the area where the texture features are concentrated, while for areas where the texture features are not obvious, fewer feature points may be extracted, and the relative pose information determined based on the feature points is not accurate enough.

[0162] The following introduces a method of feature extraction.

[0163] Furthermore, step 2051 can be implemented through the following steps 20511-20512.

[0164] Step 20511: The terminal performs uniform grid division processing on the first image to obtain N image blocks.

[0165] Wherein, N is an integer greater than 1.

[0166] The uniform grid division process refers to performing grid division on the first image, and the sizes of the obtained N image blocks are the same.

[0167] Step 20512: The terminal performs feature extraction on each of the N image blocks to obtain feature points of the first image.

[0168] In this embodiment, the terminal performs uniform grid division processing on the first image and extracts features from each obtained image block, thereby ensuring that feature points can be extracted from each area of ​​the first image, ensuring that the feature points extracted from the first image can reflect the features of each area of ​​the first image, making the relative posture information subsequently determined based on the feature points more accurate, solving the stable output of dynamic scenes and real-time calculation. The accuracy of the visual odometer is improved.

[0169] Step 2052: The terminal performs optical flow tracking processing on the feature points of the first image and the feature points of the previous frame image of the first image to obtain tracking point pairs.

[0170] The terminal may perform optical flow tracking processing on the feature points of the first image to the feature points of the previous frame image of the first image to obtain a tracking point pair.

[0171] The optical flow tracking process can be implemented by various optical flow methods such as the LK optical flow method (Lucas-Kanade Optical Flow), which is not limited in the present disclosure.

[0172] The following introduces an optical flow tracking processing method.

[0173] Further, step 2052 can be implemented through the following steps 20521-20522.

[0174] Step 20521: The terminal determines the inter-frame motion offset according to the speed information of the positioning image corresponding to the first image.

[0175] The speed information of the positioning image is the speed information of the positioning image determined according to the global posture information of the positioning image mentioned in the above embodiment.

[0176] The inter-frame motion offset refers to the offset of the feature points of the first image relative to the feature points of the previous frame image.

[0177] The terminal can determine the time difference between each two adjacent images in the image sequence, that is, the time difference between the first image and the previous frame image, based on the preset frequency. The terminal then calculates the inter-frame motion offset based on the speed information of the positioning image.

[0178] Step 20522: The terminal performs optical flow tracking processing on the feature points of the first image and the feature points of the previous frame image of the first image based on the inter-frame motion offset to obtain tracking point pairs.

[0179] The terminal can predict the positions of the feature points of the first image based on the inter-frame motion offset.

[0180] Furthermore, the predicted positions of the feature points of the first image can be obtained by the following formula (7).

[0181]

[0182] in, is the predicted position of the feature point of the first image; {P=i|i=1~N} is the position of the feature point of the previous frame image, i represents the i-th feature point, N is the total number of feature points; offset is the inter-frame motion offset.

[0183] After the terminal obtains the predicted positions of the feature points of the first image, in the subsequent optical flow tracking processing, the search range of the optical flow tracking visual search can be reduced based on the predicted positions of the feature points of the first image, thereby reducing the amount of optical flow tracking calculations, improving processing efficiency, and improving the real-time performance of the calculations.

[0184] Furthermore, if there are some similar textures or local repetitions in the image, the problem of tracking misalignment is likely to occur during the optical flow tracking process. In the method provided in this embodiment, step 20522 can be implemented by the following steps 205221 to 205223.

[0185] Step 205221: The terminal performs forward optical flow tracking on the feature points of the first image and the feature points of the previous frame image based on the inter-frame motion offset to obtain forward tracking point pairs.

[0186] The terminal may perform forward optical flow tracking, that is, perform optical flow tracking on feature points of the first image and feature points of a previous frame image to obtain forward tracking point pairs.

[0187] Step 205222: The terminal performs reverse optical flow tracking processing on the feature points of the first image and the feature points of the previous frame of the first image based on the feature points in the first image in the forward tracking point pair to obtain a reverse tracking point pair.

[0188] The terminal may perform reverse optical flow tracking, that is, perform optical flow tracking on feature points of the previous frame image and feature points of the first image to obtain reverse tracking point pairs.

[0189] Step 205223: The terminal obtains the intersection of the forward tracking point pair and the reverse tracking point pair to obtain the tracking point pair.

[0190] The terminal obtains the intersection between the forward tracking point pair and the reverse tracking point pair, that is, the repeated tracking point pair in the forward tracking point pair and the reverse tracking point pair, to obtain the tracking point pair.

[0191] Exemplarily, based on the inter-frame motion offset, the terminal performs optical flow tracking on the feature points of the first image and the feature points of the previous frame image, that is, based on the feature point set A of the previous frame image, the corresponding feature point set B, that is, the forward tracking point set B, is found on the first image, thereby obtaining one or more forward tracking point pairs (A, B). Based on the feature point set B on the first image, the terminal finds the tracking point set A' on the previous frame image, thereby obtaining one or more reverse tracking point pairs (A', B). The forward tracking point pairs (A, B) and the reverse tracking point pairs (A', B) are intersected in pairs to obtain the tracking point set B'.

[0192] In this embodiment, through bidirectional optical flow tracking, the point pairs tracked by the bidirectional optical flow tracking are determined as tracking point pairs. When processing images with similar textures or local repetitions, tracking misalignment can be avoided, making the obtained tracking point pairs more accurate and more robust, thereby improving the overall robustness.

[0193] Step 2053: The terminal determines first relative pose information between the first image and the previous frame image based on the tracking point pair.

[0194] The terminal can obtain a basic matrix E according to the tracking point pair. The basic matrix E is decomposed to obtain first relative position information between the first image and the previous frame image.

[0195] This embodiment provides a specific method for obtaining the relative posture information between two adjacent frames of images in an image sequence. The processing amount is small and the processing can be completed quickly on a terminal with limited processing resources, thereby ensuring the real-time nature of the calculation.

[0196] An embodiment of the present disclosure provides an electronic device, including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the program, the steps executed by the terminal in the visual odometer method described in any of the above embodiments are implemented.

[0197] The implementation principle and beneficial effects of the electronic device provided in this embodiment are similar to those of the above method embodiment, and will not be described in detail here.

[0198] An embodiment of the present disclosure provides an electronic device, including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the program, the steps executed by the server in the visual odometer method described in any of the above embodiments are implemented.

[0199] The implementation principle and beneficial effects of the electronic device provided in this embodiment are similar to those of the above method embodiment, and will not be described in detail here.

[0200] Based on the visual odometer method described in any of the above embodiments, the embodiment of the present disclosure further provides a computer-readable storage medium, for example, the non-temporary computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, and an optical data storage device, etc. The storage medium stores computer instructions for executing the steps executed by the terminal in the visual odometer method described in any of the above embodiments, which will not be described in detail here.

[0201] Based on the visual odometer method described in any of the above embodiments, the embodiment of the present disclosure further provides a computer-readable storage medium, for example, the non-temporary computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, and an optical data storage device, etc. The storage medium stores computer instructions for executing the steps executed by the server in the visual odometer method described in any of the above embodiments, which will not be described in detail here.

[0202] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0203] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the disclosure disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

Claims

1. A visual odometry method, characterized in that: The method comprises: Acquire image sequence through camera; Acquire a positioning image from the image sequence at a first sampling frequency, and send the positioning image to a server, so that the server determines global position information of the positioning image according to the positioning image and map data; Receiving global pose information of the positioning image sent by the server; Respectively obtain images other than the positioning image in the image sequence as first images, and perform the following processing: determine first relative position information between the first image and a previous frame image; the previous frame image is an image in the image sequence that is located before the first image and adjacent to the first image; The global pose information of the first image is obtained according to the global pose information of the positioning image corresponding to the first image and the first relative pose information; the positioning image corresponding to the first image is a positioning image that is located before the first image in the image sequence and is closest to the first image.

2. The method according to claim 1, characterized in that: The method further comprises: Determine the speed information of the positioning image according to the global pose information of the positioning image and the global pose information of a positioning image of a previous frame of the positioning image; The obtaining the global pose information of the first image according to the global pose information of the positioning image corresponding to the first image and the first relative pose information includes: Acquire an absolute displacement between the first image and the previous frame image, wherein the absolute displacement is determined according to speed information of a positioning image corresponding to the first image, the first sampling frequency, and a frequency at which the camera acquires the image sequence; Determine a scale factor according to the first relative pose information, the absolute displacement between the first image and the previous frame image, and the global pose information of the previous frame image; The global pose information of the first image is determined according to the scale factor, the first relative pose information and the global pose information of the previous frame image.

3. The method according to claim 2, characterized in that The determining the speed information of the positioning image according to the global pose information of the positioning image and the global pose information of a positioning image of a previous frame of the positioning image comprises: Determine a first movement distance according to the global position information of the positioning image and the global position information of a positioning image of a previous frame of the positioning image; the first movement distance refers to the movement distance between the moment when the positioning image is acquired and the moment when the previous positioning image of the positioning image is acquired; The speed information of the positioning image is determined according to the first movement distance, the average movement speed, the total time length corresponding to the previous positioning image of the positioning image and the first sampling frequency; the average movement speed refers to the average speed from the moment when the image sequence is started to be acquired to the moment when the positioning image is acquired.

4. The method according to claim 3, characterized in that The determining of the first movement distance according to the global pose information of the positioning image and the global pose information of a positioning image of a previous frame of the positioning image comprises: The first movement distance is determined by the following formula: Among them, S i is the first movement distance; R i is the posture information in the global pose information of the positioning image; t i is the position information in the global pose information of the positioning image; R i-1 is the posture information in the global posture information of the previous positioning image of the positioning image; i-1 It is the position information in the global pose information of the previous positioning image.

5. The method according to claim 3 or 4, characterized in that: The determining the speed information of the positioning image according to the first movement distance, the average movement speed, the total duration corresponding to the previous positioning image of the positioning image, and the first sampling frequency includes: The velocity information of the positioning image is determined by the following formula: Among them, V i V is the velocity value of the positioning image; i-1 is the average motion speed before obtaining the positioning image; Tt is the total time length corresponding to the previous positioning image; S i is the first movement distance; dt is the first sampling period.

6. The method according to any one of claims 2 to 4, characterized in that: The determining of the scale factor according to the first relative pose information, the absolute displacement between the first image and the previous frame image, and the global pose information of the previous frame image includes: The scale factor is determined according to the following formula: Among them, λ is the scale factor; R J is the rotation information in the global pose information of the previous frame image; ΔR is the relative rotation information in the first relative pose information of the first image, and Δt is the relative position information in the first relative pose information of the first image.

7. The method according to claim 2, characterized in that The determining the global pose information of the first image according to the scale factor, the first relative pose information and the global pose information of the previous frame image includes: The global pose information of the first image is determined according to the following formula: R I =ΔR·R J t I =ΔR·t J +λ·Δt Among them, R I is the rotation information in the global pose information of the first image; t I is the position information in the global pose information of the first image; λ is the scale factor; ΔR is the relative rotation information in the first relative pose information of the first image, and Δt is the relative position information in the first relative pose information of the first image; R J is the rotation information in the global pose information of the previous frame image; t J It is the position information in the global pose information of the previous frame image.

8. The method according to claim 2, characterized in that: The determining of first relative position information between the first image and a previous frame image includes: Performing feature extraction on the first image to obtain feature points of the first image; Performing optical flow tracking processing on feature points of the first image and feature points of an image frame before the first image to obtain tracking point pairs; First relative position information between the first image and the previous frame image is determined according to the tracking point pair.

9. The method according to claim 8, characterized in that The extracting features from the first image to obtain feature points of the first image includes: Performing uniform grid division processing on the first image to obtain N image blocks, where N is an integer greater than 1; Feature extraction is performed on each of the N image blocks to obtain feature points of the first image.

10. The method according to claim 8, characterized in that The step of performing optical flow tracking on the feature points of the first image and the feature points of a frame of an image before the first image to obtain a tracking point pair includes: determining an inter-frame motion offset according to speed information of a positioning image corresponding to the first image; Based on the inter-frame motion offset, optical flow tracking processing is performed on the feature points of the first image and the feature points of a frame image before the first image to obtain tracking point pairs.

11. The method according to claim 10, characterized in that The step of performing optical flow tracking processing on the feature points of the first image and the feature points of a frame image before the first image based on the inter-frame motion offset to obtain a tracking point pair includes: Based on the inter-frame motion offset, forward optical flow tracking is performed on the feature points of the first image and the feature points of the previous frame image to obtain forward tracking point pairs; Based on the feature points in the first image in the forward tracking point pair, performing reverse optical flow tracking processing on the feature points of the first image and the feature points of the previous frame image to obtain a reverse tracking point pair; The intersection between the forward tracking point pair and the reverse tracking point pair is obtained to obtain a tracking point pair.

12. A visual odometer method, characterized in that: The method comprises: receiving a positioning image sent by a terminal; the positioning image is acquired by the terminal from an image sequence at a first sampling frequency, and the image sequence is acquired by the terminal through a camera; Determining global position information of the positioning image according to the positioning image and the map data; The global pose information of the positioning image is sent to the terminal, so that the terminal obtains images other than the positioning image in the image sequence as first images, and performs the following processing: determining the first relative pose information between the first image and a previous frame image; the previous frame image is an image that is located before the first image and adjacent to the first image in the image sequence; obtaining the global pose information of the first image according to the speed information of the positioning image corresponding to the first image and the first relative pose information; the positioning image corresponding to the first image is a positioning image that is located before the first image and closest to the first image in the image sequence.

13. An electronic device comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, characterized in that: When the processor executes the program, the steps of the method according to any one of claims 1 to 11 are implemented; or, the steps of the method according to claim 12 are implemented.

Images