Device positioning method, computer device and medium

Abstract

The present application relates to a device positioning method, a computer device and a medium, the method comprising: obtaining the state parameters of the previous frame image, the current frame image and the previous frame image to the current frame image; associating the feature points of the current frame image with the feature points of the previous frame image to obtain the first type of feature points, the second type of feature points and the current frame local map space of the current frame image; associating the second type of feature points in the current frame image with the feature points in the point cloud library to obtain the third type of feature points and / or the fourth type of feature points of the current frame image; the third type of feature points are historically marked feature points; updating the current frame local map space according to the above association results to obtain the updated current frame local map space; calculating the device's posture according to the state parameters and the updated current frame local map space. The present application improves the accuracy and stability of device positioning by retrieving historically marked feature points.

Description

Technical Field

[0001] The present application relates to the technical field of visual positioning, and in particular to a positioning method for a device, a computer device, and a medium. Background Art

[0002] Autonomous driving, mobile robots, augmented reality (AR), virtual reality (VR) and other fields all need to solve the key technical problem of simultaneous localization and mapping (SLAM) of their smart devices. At present, common SLAM mostly uses a single sensor (such as visual systems such as cameras and lidar) and an inertial measurement unit (IMU), but the visual system has a large drift in fast motion and cannot restore the scale, which limits its practical application in smart devices. IMU sensors have the advantages of low cost, high accuracy in fast motion and the ability to provide scale information. The functions of IMU sensors and visual sensors can complement each other.

[0003] Therefore, in recent years, the trend of integrating vision systems and IMU sensors has become increasingly obvious. The technology of using the measurement data of vision systems and IMU sensors to estimate the position and direction of smart devices over time is called visual inertial odometry (VIO). Existing smart devices gradually use a visual inertial odometry module to complete the SLAM of their devices. VIO has been widely used in various potential applications due to its small size and low price of sensor modules. However, VIO still has the problem of poor positioning accuracy. Summary of the invention

[0004] Based on this, it is necessary to provide a positioning method, computer equipment and medium for a device whose feature points can be associated across frames and have high positioning accuracy in order to address the above technical problems.

[0005] In a first aspect, the present application provides a method for positioning a device, the method comprising:

[0006] Obtaining a previous frame image, a current frame image, and state parameters from the previous frame image to the current frame image captured by the device;

[0007] By adopting the optical flow tracking method, the feature points of the current frame image are associated with the feature points of the previous frame image to obtain the first type of feature points, the second type of feature points and the current frame local map space of the current frame image; the first type of feature points refer to the feature points in the current frame image that can be associated with the feature points of the previous frame image; the second type of feature points refer to the feature points in the current frame image that cannot be associated with the feature points of the previous frame image;

[0008] Associating the second-category feature points in the current frame image with the feature points in the pre-stored point cloud library to obtain the third-category feature points and / or the fourth-category feature points of the current frame image; the second-category feature points include the third-category feature points and the fourth-category feature points; the third-category feature points refer to the feature points in the current frame image that cannot be associated with the feature points in the previous frame image but can be associated with the feature points in the pre-stored point cloud library; the fourth-category feature points refer to the feature points in the current frame image that cannot be associated with the feature points in the previous frame image and cannot be associated with the feature points in the pre-stored point cloud library; the point cloud library stores all map points in the historical local map space;

[0009] Updating the fourth type of feature points into the point cloud library;

[0010] According to the association result of the feature points in the current frame image, the feature points in the previous frame image and the feature points in the point cloud library, the current frame local map space is updated to obtain an updated current frame local map space;

[0011] The position and posture of the device in the updated local map space of the current frame is calculated based on the state parameters and the updated local map space of the current frame.

[0012] In one embodiment, the second type of feature points in the current frame image are associated with the feature points in a pre-stored point cloud library to obtain the third type of feature points and / or the fourth type of feature points of the current frame image by comparing the first sub-pixel block and the second sub-pixel block to see whether they are similar. If so, it is determined that the feature points of the current frame image can be associated with the feature points in the point cloud library to obtain the third type of feature points. If not, it is determined that the feature points in the current frame image cannot be associated with the feature points in the point cloud library to obtain the fourth type of feature points. The first sub-pixel block is the pixel information of the feature point of the current frame image around the pixel point of the feature point, and the second sub-pixel block is the pixel information around the pixel point when the feature point in the point cloud library is first observed.

[0013] In one embodiment, the first sub-pixel block is compared with the second sub-pixel block to see whether they are close. If so, it is determined that the feature points of the current frame image can be associated with the feature points in the point cloud library to obtain the third type of feature points. If not, it is determined that the feature points in the current frame image cannot be associated with the feature points in the point cloud library to obtain the fourth type of feature points in the following manner:

[0014] Compare whether the pixel value of the first sub-pixel block is similar to the pixel value of the second sub-pixel block, and if not, determine that the feature point in the current frame image is not the same feature point as the feature point in the point cloud library, and the feature point corresponding to the first sub-pixel block is the fourth type of feature point;

[0015] If so, iterative tracking is performed to obtain the moving distance between the first sub-pixel block and the second sub-pixel block, and the moving distance is compared to see whether it is less than a preset distance. If so, it is determined that the feature point in the current frame image is the same feature point as the feature point in the point cloud library, and the feature point corresponding to the first sub-pixel block is the third type of feature point; if not, it is determined that the feature point in the current frame image is not the same feature point as the feature point in the point cloud library, and the feature point corresponding to the first sub-pixel block is the fourth type of feature point.

[0016] In one embodiment, the positioning method further includes: based on the state parameters and the characterization information of the feature points in the current frame image, using an optical flow prediction method to predict the predicted position of the feature points in the previous frame image in the current frame image; and using the predicted position as the initial value of the optical flow tracking.

[0017] In one embodiment, the positioning method further includes:

[0018] Based on the previous frame image and the current frame image, extracting a mask of a dynamic object in the current frame image;

[0019] According to the mask, the region of the dynamic object in the current frame image is identified to obtain the current frame static image with the mask.

[0020] In one embodiment, the positioning method also includes: dividing the static image of the current frame into grids, counting the number of feature points in the grids, and if the number of feature points in the grids is less than a preset number, extracting new feature points from the grids so that the grids contain no less than a preset number of feature points.

[0021] In one embodiment, updating the fourth category feature points to the point cloud library also includes: determining whether the fourth category feature points are external points, if so, discarding the fourth category feature points, if not, adding the fourth category feature points to the point cloud library.

[0022] In one embodiment, the method for determining whether the fourth category feature point is an external point is: obtaining the depth of the fourth category feature point in the current frame image, and determining whether the depth of the fourth category feature point exceeds a depth threshold; if so, determining that the fourth category feature point is an external point; if not, determining that the fourth category feature point is not an external point.

[0023] In a second aspect, the present application further provides a computer device for implementing the method described in any one of the embodiments of the first aspect above, wherein the device includes a positioning device, the positioning device including:

[0024] A data acquisition module, used to obtain a previous frame image, a current frame image and state parameters from the previous frame image to the current frame image captured by the device;

[0025] A first associating module is used to associate the feature points of the current frame image with the feature points of the previous frame image by optical flow tracking to obtain the first type of feature points, the second type of feature points and the current frame local map space of the current frame image; the first type of feature points refer to the feature points in the current frame image that can be associated with the feature points of the previous frame image; the second type of feature points refer to the feature points in the current frame image that cannot be associated with the feature points of the previous frame image;

[0026] A second associating module is used to associate the second type of feature points in the current frame image with the feature points in the pre-stored point cloud library to obtain the third type of feature points and / or the fourth type of feature points of the current frame image; the second type of feature points include the third type of feature points and / or the fourth type of feature points; the third type of feature points refer to feature points in the current frame image that cannot be associated with the feature points of the previous frame image but can be associated with the feature points in the pre-stored point cloud library; the fourth type of feature points refer to feature points in the current frame image that cannot be associated with the feature points of the previous frame image and cannot be associated with the feature points in the pre-stored point cloud library; the point cloud library stores all map points in the historical local map space;

[0027] A point cloud library updating module, used for updating the fourth type of feature points into the point cloud library;

[0028] A local map updating module, used to update the local map space of the current frame according to the association result of the feature points in the current frame image with the feature points in the previous frame image and the feature points in the point cloud library, so as to obtain an updated local map space of the current frame;

[0029] A calculation module is used to calculate the position and posture of the device in the updated current frame local map space based on the state parameters and the updated current frame local map space.

[0030] In a third aspect, the present application further provides a computer-readable storage medium, wherein a computer program is stored on the computer-readable storage medium, and when the processor executes the computer program, the method described in any one of the embodiments of the first aspect is implemented.

[0031] Compared with the prior art, the positioning method, computer device and medium of the device proposed in the present application firstly associate the feature points on the current frame image with the feature points on the previous frame image to obtain the first type of feature points and the second type of feature points, wherein the first type of feature points can be associated with the feature points of the previous frame image, and the second type of feature points cannot be associated with the feature points of the previous frame image, and the current frame image can be aligned with the previous frame image based on the first type of feature points to obtain the local map space of the current frame; secondly, by associating the second type of feature points with the point cloud library, the third type of feature points and / or the fourth type of feature points of the current frame image are obtained, the third type of feature points can be associated with the feature points in the pre-stored point cloud library, and the fourth type of feature points cannot be associated with the pre-stored feature points. The feature points in the stored point cloud library are associated. Since the point cloud library stores all the map points in the historical local map space, and the third type of feature points are missing in the previous frame image, the third type of feature points are associated with the map points in the point cloud library, and the feature points corresponding to the third type of feature points in the historical local map space can be retrieved, which avoids the situation where the optical flow cannot track the third type of feature points due to the lack of the third type of feature points in the previous frame image, and the same feature point is marked as two feature points, thereby improving the accuracy of the local map space. In the subsequent process of calculating the posture, the calculation is based on the state parameters and the updated local map space of the current frame, which also improves the accuracy of the posture calculation of the device in the local map space, that is, improves the positioning accuracy of the device in the local map space. The present application also updates the fourth type of feature points to the point cloud library to ensure that the point cloud library is the most accurate each time the point cloud library is called, and further enables each frame of image feature points to be compared and associated with the latest feature points in the point cloud library, that is, the updated local map space obtained in each frame is the most accurate state, thereby improving the accuracy and stability of the device positioning in the local map space. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 A diagram of an application environment of a positioning method for a device in an embodiment;

[0033] Figure 2 A schematic diagram of a flow chart of a method for positioning a device in an embodiment;

[0034] Figure 3 is a flow chart of a positioning method in an embodiment including a step of extracting a mask;

[0035] Figure 4 is a flow chart of a positioning method in an embodiment including a step of supplementing feature points;

[0036] Figure 5 is a flow chart of a positioning method in an embodiment including a depth estimation step;

[0037] Figure 6 is a flow chart of a method for positioning a device in another embodiment;

[0038] Figure 7 is a schematic structural diagram of a positioning device in an embodiment;

[0039] Figure 8 is a schematic structural diagram of a positioning device in another embodiment;

[0040] Fig. 9 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0042] The device positioning method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. Among them, the terminal 102 can be but is not limited to various personal computers, laptops, smart phones, tablets, Internet of Things devices and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The server 104 can be implemented with an independent server or a server cluster consisting of multiple servers.

[0043] In one embodiment, Figure 2 As shown, Figure 2 : is a flow chart of a method for positioning a device in an embodiment; this embodiment takes the method applied to a terminal as an example, and it can be understood that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0044] Step S11: obtaining the state parameters of the previous frame image, the current frame image, and the image from the previous frame image to the current frame image captured by the device.

[0045] Among them, the device is equipped with a camera and an inertial measurement unit. The camera is used to continuously capture monocular or binocular images or shoot videos, and the inertial measurement unit is used to measure state parameters. The device can be but is not limited to AR glasses, MR glasses, mobile robots, drones, self-driving cars, smart watches, somatosensory game controllers, smart devices with visual inertial odometers, etc., and no specific limitations are made here.

[0046] Furthermore, the state parameters are the position, velocity, and attitude of the device, and the state parameters are calculated using acceleration and angular velocity data obtained by the inertial odometer.

[0047] Step S12: using optical flow tracking to associate feature points of the current frame image with feature points of the previous frame image, obtaining first-category feature points and second-category feature points of the current frame image, and generating a local map space; wherein, the first-category feature points refer to feature points in the current frame image that can be associated with feature points of the previous frame image; the second-category feature points refer to feature points in the current frame image that cannot be associated with feature points of the previous frame image.

[0048] Optical flow tracking is a technique used in computer vision to estimate the motion of feature points between consecutive frame images. Optical flow describes the apparent motion of the surface of an object in a scene, that is, the change in the position of feature points in an image from one moment to another.

[0049] It should be noted that a feature point is a pixel point in an image whose representation information is significantly different from that of surrounding pixels. For example, in an image with a white wall and a black display in front of the white wall, the representation information of the vertices of the black display can be used as a feature point tracked by optical flow because the representation information of the vertices of the black display is significantly different from that of the surrounding white walls.

[0050] In one embodiment, the specific method of optical flow tracking can be performed in a reverse combination manner. When the device uses a monocular camera, the optical flow tracking can be performed by associating the feature points in the previous frame image with the feature points in the current frame image; when the device uses a binocular camera, the left eye image and the right eye image can be obtained at the same time at each moment. During the optical flow tracking, it is necessary not only to associate the feature points in the left eye image of the current frame with the feature points in the right eye image of the current frame, but also to associate the feature points in the left eye image of the previous frame with the feature points in the left eye image of the current frame, and to associate the feature points in the right eye image of the previous frame with the feature points in the right eye image of the current frame.

[0051] It can be understood that, taking a device using a monocular camera as an example, when the device captures the previous frame image and the current frame image, since the previous frame image and the current frame image need to be associated, there needs to be an overlapping area between the previous frame image and the current frame image, and the first type of feature points in the current frame image that can be associated with the feature points of the previous frame image all come from the above overlapping area. When the shooting position or angle of the monocular camera changes, there are still non-overlapping areas in the previous frame image and the current frame image, and the second type of feature points in the current frame image that cannot be associated with the feature points of the previous frame image come from the above non-overlapping area. In this way, when the previous frame image and the current frame image are used to generate a local map, the overlapping area can be used to determine the translation transformation relationship between the previous frame image and the current frame image, so as to associate the previous frame image and the current frame image, and the non-overlapping area can expand the scope of the local map.

[0052] Specifically, at least based on the previous frame image and the current frame image, the first type of feature points in the above two or more images can be identified and matched by the computer vision algorithm (such as SIFT, SURF, etc.) stored in the terminal to establish the correspondence between different images. And based on the matching associated first type of feature points, the three-dimensional coordinates of the objects in the scene are calculated using the multi-view geometry principle (such as structured light method, stereo vision, etc.) to generate a three-dimensional model. This process can be completed with the help of Agisoft Metashape, Pix4D, RealityCapture and other software. And the above three-dimensional model is converted into a specific geographic coordinate system to generate a local map space, and the position and direction of the camera in the device in the local map are inferred so that the device can update the positioning in time. Furthermore, the generated local map space and the positioning information of the device can also be integrated into other applications.

[0053] It should also be noted that, in fact, only one local map space is saved, but each time a latest current frame image is obtained, the method in this application will update the previous local map space to obtain the current frame local map space, and the local map space of the previous frame is not stored.

[0054] Step S13: Associating the second-category feature points in the current frame image with the feature points in the pre-stored point cloud library to obtain the third-category feature points and / or the fourth-category feature points of the current frame image; wherein the second-category feature points include the third-category feature points and the fourth-category feature points; the third-category feature points refer to feature points in the current frame image that cannot be associated with the feature points of the previous frame image but can be associated with the feature points in the pre-stored point cloud library; the fourth-category feature points refer to feature points in the current frame image that cannot be associated with the feature points of the previous frame image and cannot be associated with the feature points in the pre-stored point cloud library; the point cloud library stores all map points in the historical local map space.

[0055] It should be noted that before obtaining the current frame image and the previous frame image, at least two images at earlier times should be obtained, and the historical local map space is generated based on the two images at earlier times. All map points in the historical local map space are stored in the point cloud library with a suitable data structure to facilitate subsequent query and use, and the point cloud library is updated based on the feature points in the current frame image and the map points in the local map space. It should also be noted that the feature points in the current frame image are two-dimensional, while the map points in the local map space are three-dimensional. The map points can also be called landmark points and are stored in the point cloud library in the form of point clouds. When the local map space is generated based on the feature points in the image, the feature points can correspond to the map points in the local map space, and the feature points can correspond to the map points in the point cloud library.

[0056] It can be understood that by associating the second-category feature points in the current frame image with the feature points in the pre-stored point cloud library, the following three situations can be obtained: (1) only the third-category feature points of the current frame image are obtained; (2) only the fourth-category feature points of the current frame image are obtained; (3) the third-category feature points and / or the fourth-category feature points of the current frame image are obtained at the same time.

[0057] When the third type of feature points are obtained, it indicates that there is a frame break between the current frame image and the historical frame image that is earlier than the previous frame image, or that some objects in the previous frame image are blocked, causing objects that have appeared in the historical frame image to directly appear in the current frame image across the previous frame image.

[0058] In one embodiment, when the third type of feature points are obtained, since the third type of feature points are missing in the previous frame image but are recorded in the point cloud library, this embodiment associates the third type of feature points in the current frame image with the feature points in the pre-stored point cloud library, so that the third type of feature points in the current frame image can be associated and unified with the points in the point cloud library by retrieving the historically marked feature points, which is equivalent to using the point cloud library as a medium to connect the feature points in the current frame image with the feature points of the historical frame image. The historical frame image is obtained before the previous frame image, which solves the problem that the feature points cannot be associated after the image is interrupted, improves the continuity between images used to generate local maps, and associates the feature points newly extracted from the current frame image with the map points in the point cloud library, and then updates and optimizes the local map space, thereby improving the positioning accuracy and stability of the device.

[0059] For example, the historical frame image has a feature point of the first street lamp. The historical frame map space uses the first street lamp in the historical frame image as a map point and saves it in the point cloud library. Due to the device moving too fast, or the image is lost, or the first street lamp is blocked by the line of sight and other similar reasons, the feature point of the first street lamp does not appear in the previous frame image, but the current frame image captures the "second street lamp", but in fact, the "second street lamp" is the first street lamp. For the existing technology, the optical flow tracking method is used to associate the feature points of the current frame image with the feature points of the previous frame image, and the local map space obtained has the "second street lamp". The existing technology will mistakenly believe that there are the first street lamp and the "second street lamp". The positioning method in the present application can identify the "second street light" as a second type of feature point, and then by associating the second type of feature point with the feature points in the pre-stored point cloud library, it can identify that the "second street light" is the first street light in the point cloud library, and associate the "second street light" with the first street light, thereby retrieving the map points in the point cloud library, making the updated current frame local map more accurate.

[0060] Step S14: Update the fourth type of feature points to the point cloud library.

[0061] It is understandable that after associating the second type of feature points in the current frame image with the feature points in the pre-stored point cloud library, step S14 needs to be executed only when there are fourth type of feature points in the current frame image to timely update the point cloud library.

[0062] In one embodiment, the first type of feature points and the third type of feature points can also be updated to the point cloud library. It can be understood that no matter what the association result between the feature points in the current frame image and the feature points in the previous frame image and the feature points in the point cloud library is, the first type of feature points and the third type of feature points can be updated to the point cloud library to timely update the point cloud library. Even if the first type of feature points and the third type of feature points have been recorded in the point cloud library, the update of the first type of feature points and the third type of feature points in the point cloud library can ensure the accuracy and timeliness of the point cloud library.

[0063] Step S15: According to the association results of the feature points in the current frame image, the feature points in the previous frame image and the feature points in the point cloud library, the current frame local map space is updated to obtain an updated current frame local map space.

[0064] It can be understood that the above association result includes a first association result and a second association result, wherein the first association result refers to the result obtained by associating the feature points of the current frame image with the feature points of the previous frame image, and the second association result refers to the result obtained by associating the second type of feature points in the current frame image with the feature points in the pre-stored point cloud library. When only the fourth type of feature points are obtained in step S13, the updated local map space of the current frame is substantially the same as the local map space of the current frame before the update.

[0065] Step S16: Calculate the position and posture of the device in the updated local map space of the current frame based on the state parameters and the updated local map space of the current frame.

[0066] According to the use of the inertial measurement unit, the final output of this embodiment is a 6-DOF posture.

[0067] Illustratively, when the positioning method in this embodiment is applied in an autonomous driving car or navigation scenario, it is necessary to build a global map in advance. Step S16 obtains the position and posture of the device in the local map. Subsequently, the local map can be aligned with the global map to calculate the position and posture of the device in the global map, thereby navigating the autonomous driving car or the user.

[0068] In one embodiment, step S12 uses optical flow tracking to associate feature points of the current frame image with feature points of the previous frame image to obtain the first type of feature points, the second type of feature points of the current frame image, and the local map space of the current frame further includes:

[0069] Step S121: Based on the state parameters, an optical flow prediction method is used to predict the predicted position of the feature point in the previous frame image in the current frame image;

[0070] Step S122: Optical flow tracking is used for the feature points in the current frame image, and the predicted position is used as the initial value of the optical flow tracking to obtain the actual position of the feature points in the current frame image in the previous frame image to generate the local map space of the current frame.

[0071] In step S121, the image at frame k is taken as the previous frame, and the image at frame k+1 is taken as the current frame. First, based on the state parameters (speed, position, posture) between the previous frame and the current frame, a priori estimation of the posture of the device when shooting the current frame is obtained by pre-integration:

[0072] ;

[0073] Among them, b represents the IMU coordinate system, w represents the world coordinate system, is the pre-integrated quantity of the device's position, is the pre-integrated quantity of the device's speed, is the pre-integrated component of the device's posture, which is calculated as follows:

[0074] ;

[0075] is the position of the device's IMU in the world coordinate system at frame k+1, is the speed of the device's IMU in the world coordinate system at frame k+1, is the posture of the device's IMU in the world coordinate system at frame k+1, is the position of the device's IMU in the world coordinate system at frame k, is the speed of the device's IMU in the world coordinate system at frame k, is the posture of the device's IMU in the world coordinate system at frame k, Indicates the spatial transformation relationship between the world coordinate system and the device IMU posture at the kth frame. represents the time interval from frame k to frame k+1, represents the acceleration at a certain time t between frame k and frame k+1, b at Represents the acceleration bias at a certain time t between frame k and frame k+1, n a represents the noise of acceleration at a certain time t between frame k and frame k+1, represents the angular velocity at a certain time t between frame k and frame k+1, b wt Represents the bias of the angular velocity at a certain time t between frame k and frame k+1, n w represents the noise of the angular velocity at a certain time t between frame k and frame k+1, g w is the gravitational constant, Represents the rotation matrix in the IMU coordinate system at a certain time t between frame k and frame k+1, is the rotation matrix The quaternion of Used to represent spatial rotation relationships. Ω represents the left multiplication operator of a pure imaginary quaternion, and They are used together to represent a certain time t between frame k and frame k+1.

[0076] At the same time, the translation and rotation transformation relationship between the image at frame k and the image at frame k+1 is obtained by calculation , :

[0077] ;

[0078] Then, using the depth of each tracked feature point i in the local map and the predicted current pose, the position of the tracked feature point i in the current frame image is obtained by the following calculation:

[0079] Assumptions π is the projection model of the device's camera, π -1 is the back-projection model of the device's camera, is the image coordinate of the tracked feature point in the k-frame image, is the inverse depth of feature point i, then the position of feature point i in the local map space updated according to the k+1 frame image for:

[0080] ;

[0081] The possible position of the feature point i on the current frame image It can be calculated as follows:

[0082] ;

[0083] in, is the landmark point in the middle of the local map, a three-dimensional vector, Represents the values ​​of the first two dimensions of the three-dimensional vector. Represents the value of the third dimension of this three-dimensional vector.

[0084] In this embodiment, when a binocular camera is used in the device, it is necessary to associate the feature points of the left eye from the previous frame image to the current frame image, and also to associate the feature points of the left eye and the right eye from the current frame image. The optical flow prediction in this step provides an effective initial value for left eye tracking from the previous frame image to the current frame image, which can accelerate the speed and accuracy of iterative convergence.

[0085] Based on the above embodiment, in a positioning method disclosed in another embodiment, as Figure 3 As shown, step S21 is further added: extracting a mask (MASK), by adding a mask to the current frame image to mark the dynamic object in the current frame image. In this embodiment, step S21 includes:

[0086] Step 211: extracting a mask of a dynamic object in the current frame image based on the previous frame image and the current frame image;

[0087] Step 212: According to the mask, the region of the dynamic object in the current frame image is identified to obtain the current frame static image with the mask.

[0088] In this way, the local map space of the current frame generated based on the static image of the current frame can be free from interference from dynamic objects, so that the positioning method in this embodiment is not affected by dynamic scenes such as violent motion and obvious changes in illumination.

[0089] Furthermore, the step of extracting the mask (MASK) and the step of optical flow prediction can be performed synchronously to improve the efficiency of the method in calculating the pose. It can be understood that in the previous cycle, the mask has been extracted from the previous frame image to identify the dynamic objects in the previous frame image, thereby obtaining the previous frame image containing only static objects, that is, the previous frame image without dynamic objects used in this cycle has been obtained before entering this cycle; in this way, the results of the parallel calculation of the mask extraction and optical flow prediction in this cycle are used as the input of the next cycle, which can save the calculation time of the method in updating the local area space and updating the positioning of the device in the local map space, thereby ensuring the timeliness of the local map space update and the positioning update of the device in the local map space.

[0090] On the basis of adding the step of extracting the mask, correspondingly, the current frame static image is used to replace the current frame image used in other steps, specifically:

[0091] Step S22: using optical flow tracking to associate feature points of the current static image with feature points of the previous image to obtain first-category feature points and second-category feature points of the current static image;

[0092] Step S23: Associating the second type of feature points in the current frame static image with the feature points in the pre-stored point cloud library to obtain the third type of feature points and the fourth type of feature points of the current frame static image;

[0093] Step S24: updating the fourth type of feature points to the point cloud library;

[0094] Step S25: updating the local map space of the current frame according to the association result between the feature points in the static image of the current frame and the feature points in the previous frame image and the feature points in the point cloud library to obtain an updated local map space of the current frame;

[0095] Step S26: Calculate the position and posture of the device in the updated local map space of the current frame based on the state parameters and the updated local map space of the current frame.

[0096] In this embodiment, the contents repeated in other embodiments will not be described again.

[0097] It can be understood that by comparing the depth information and position of the feature points in the current frame image with the depth information and position of the previous frame image, the feature points with similar depth and little change in position are recorded as static objects or reference objects for subsequent optical flow tracking and as map points for building local map space, while objects with similar depth but large changes in distance from the reference object are dynamic objects, and masks are added to the areas of the dynamic objects. Schematically, the yolov6n network is used to identify all dynamic objects in the current frame image, or models such as EfficientDet, NanoDet, PP-YOLO-Tiny, etc. can be used to detect objects.

[0098] In this way, when using a static image of the current frame that does not contain dynamic objects to generate a local map space and the position of the computing device in the local map space, the interference of the movement of dynamic objects in the scene on the optical flow tracking can be filtered out, so that the positioning method proposed in this application also has higher robustness in complex scenes.

[0099] Based on the above embodiment, in a positioning method disclosed in another embodiment, as Figure 4 As shown, the difference between this embodiment and the above embodiment is that the positioning method also includes step S213 (supplementing feature points): dividing the current frame image into grids, counting the number of feature points in the grids, and if the number of feature points in the grids is less than a preset number, extracting new feature points from the grids so that the grids contain no less than a preset number of feature points.

[0100] It can be understood that when the number of feature points in the grid of the current frame image is less than a preset number, the present embodiment automatically extracts new feature points from the grid of the current frame image to supplement them. In this way, when the device obtains the next frame image to perform the next cycle of optical flow tracking and local map space update, it can avoid the failure of optical flow tracking caused by too few feature points, thereby enhancing the stability of optical flow tracking.

[0101] Furthermore, as mentioned above, the current frame image in step S213 can also be replaced with the current frame static image, then step S213 is: dividing the current frame static image into grids, counting the number of feature points in the grids, if the number of feature points in the grids is less than a preset number, extracting new feature points from the grids so that the grids contain no less than a preset number of feature points.

[0102] It can be understood that, as mentioned above, when the present embodiment automatically extracts new feature points from the grid of the static image of the current frame for supplementation, and avoids the area in the current frame image where the dynamic objects have been identified in step S212, the present embodiment avoids the failure of optical flow tracking due to too few feature points. At the same time, since the area marked with dynamic objects is avoided when supplementing the static image of the current frame with new feature points, the present embodiment can also eliminate the interference of dynamic objects on the optical flow prediction step and the step of updating the local map space of the current frame, thereby improving the robustness of the positioning method in the present embodiment in complex scenes.

[0103] Illustratively, in this embodiment, the FAST corner detection algorithm is used to extract and supplement new feature points for grids with fewer feature points than a preset number. In other embodiments, the Harris corner detection algorithm, the Shi-Tomasi corner detection algorithm, the SIFT feature point detection algorithm, the SURF feature point detection algorithm, the ORB feature point detection algorithm and other algorithms may also be used to extract and supplement new feature points, and the specific algorithm is not limited here.

[0104] Based on the above embodiment, in a positioning method disclosed in another embodiment, as Figure 5 As shown, the difference between this embodiment and the above embodiment is that: in step S14, updating the fourth type of feature points to the point cloud library also includes: determining whether the fourth type of feature points are external points, if so, discarding the fourth type of feature points, if not, adding the fourth type of feature points to the point cloud library. In this way, the introduction of erroneous information can be reduced and the accuracy of the local map space can be ensured.

[0105] Furthermore, the method for determining whether the fourth type of feature point is an external point is as follows:

[0106] Step S214 (depth estimation): obtaining the depth of the fourth type of feature points in the current frame image,

[0107] Step S141: Determine whether the depth of the fourth type of feature point exceeds the depth threshold. If so, determine that the fourth type of feature point is an outlier. If not, determine that the fourth type of feature point is not an outlier. In this way, the depth information is used to effectively filter feature points that exceed the depth threshold, reduce the introduction of erroneous information, and provide more accurate calculation constraints for local map space optimization and posture optimization, thereby ensuring the accuracy of map point and posture optimization during the local map space optimization process.

[0108] Specifically, for a feature point newly extracted from the current frame image, the depth d2 of the feature point is obtained when tracking the left and right gaze flows, and compared with the depth estimation result d1, if the depth d2 of the feature point exceeds the threshold, the feature point is discarded;

[0109] For feature points whose depths are restored through historical multi-frame triangulation, the depth d3 restored from the feature point is compared with the depth estimation result d1. If the depth d3 of the feature point exceeds the threshold, it is discarded.

[0110] Illustratively, this embodiment uses the Metric3Dv2 tool to obtain the depth of the feature point in the current frame static image. In other embodiments, the MiDaS depth estimation model, the Monodepth2 framework, etc. can also be used to obtain the depth of the feature point in the current frame static image.

[0111] Further, step S121: based on the state parameters, using the optical flow prediction method, predict the predicted position of the feature points in the previous frame image in the current frame image; step S211: based on the previous frame image and the current frame image, extract the mask of the dynamic object in the current frame image; step S214: obtain the depth of the fourth type of feature points in the current frame image in parallel. In the previous cycle, the previous frame image has been identified and the dynamic objects have been eliminated, so as to obtain the previous frame image containing only static objects, that is, the previous frame image without dynamic objects used in this cycle has been obtained before entering this cycle; in this way, the results of the parallel calculations of steps S121, S211, and S214 in this cycle are used as the input of the next cycle, which can save the calculation time of updating the local area space and updating the positioning of the device in the local map space of this method, thereby ensuring the timeliness of the local map space update and the positioning update of the device in the local map space. In other embodiments, steps S121, S211, and S214 can be performed sequentially or in any combination, which is not limited here.

[0112] In one embodiment, according to the state parameters and the updated local map space of the current frame, the posture of the device in the updated local map space of the current frame is calculated as follows: the posture of the device in the local map space of the current frame is calculated by optimizing the objective function, and a nonlinear optimization objective function is constructed to optimize the position, speed, IMU bias, position of all key frames in the local map, and position of map points of the device in the current frame image. The local map space is optimized by optimizing the objective function E, and the posture with the minimum value of the optimization objective function E is the optimized posture finally solved. In this way, by optimizing the local map space, the correct data after eliminating the outliers can be used to optimize the camera and IMU postures, thereby improving the positioning accuracy.

[0113] On the basis of the above embodiment, the device in this embodiment uses binocular cameras, one of which is used as a projection camera, and the image it captures is a projection image. The expression of the above optimization objective function E is as follows:

[0114] ;

[0115] Among them, x represents the index number in the observation set of all landmark points, yz represents the y-th key frame and the z-th key frame, r ix represents the reprojection error factor, represents the covariance of the state parameters, T represents the transposed calculation, r B Represents the pre-integration error factor.

[0116] Among them, the image reprojection error factor r ix The expression is as follows:

[0117] ;

[0118] in, r ix Represents the observation value of the device's target camera on the current frame image. π c(x) The projection model of the projection camera representing the target camera of the device, T wc(x) Represents the position of the target camera projected to the device in the world coordinate system. T h(i) Represents the coordinate transformation of the device's other cameras to the projection camera, p i(u,v,d) Represents the coordinates of feature point i in the world coordinate system.

[0119] IMU pre-integration error factor r B The expression is as follows:

[0120] ;

[0121] express 、 and A collection of represents the acceleration bias at time k, represents the acceleration bias at time k+1, represents the angular velocity bias at time k, represents the angular velocity bias at time k+1, represents the residual of the pre-integrated translation at time k and time k+1, represents the residual of the pre-integrated rotation amount at time k and time k+1, It represents the residual of the pre-integrated velocity at time k and time k+1, δb aIt represents the residual of the acceleration bias at time k and time k+1, δb g Represents the residual of the angular velocity bias at time k and time k+1.

[0122] The initial position of the device starts from the world coordinate origin. T wc(x) and The relationship is as follows:

[0123] ;

[0124] Used to form spatial coordinate transformation relationships , T wc(x) Represents the coordinate transformation relationship from the IMU coordinate system to the target projection camera.

[0125] In one embodiment, in step S13, the second type of feature points in the current frame image are associated with the feature points in the pre-stored point cloud library to obtain the third type of feature points and / or the fourth type of feature points of the current frame image by comparing whether the first sub-pixel block is similar to the second sub-pixel block. If so, it is determined that the feature points of the current frame image can be associated with the feature points in the point cloud library to obtain the third type of feature points. If not, it is determined that the feature points in the current frame image cannot be associated with the feature points in the point cloud library to obtain the fourth type of feature points.

[0126] Among them, the first sub-pixel block is the projection position of the feature point of the current frame image in the local map space of the current frame, and the second sub-pixel block is the projection position of the feature point in the point cloud library in the local map space of the current frame. The first sub-pixel block is the pixel information around the pixel point of the feature point of the current frame image. The second sub-pixel block is the first observed sub-pixel block recorded by the corresponding map point in the point cloud library.

[0127] It should be noted that in this embodiment, the pixel information of the second type of feature points is compared with the feature points in the point cloud library. In other embodiments, other characterization parameters of the second type of feature points and the feature points in the point cloud library may also be compared, such as brightness, depth, color value, etc., which are not limited here.

[0128] Further, the first sub-pixel block is compared with the second sub-pixel block to see whether they are similar. If so, it is determined that the feature points of the current frame image can be associated with the feature points in the point cloud library to obtain the third type of feature points. If not, it is determined that the feature points in the current frame image cannot be associated with the feature points in the point cloud library, and the fourth type of feature points are obtained as follows:

[0129] Step S131: comparing the pixel values ​​of the first sub-pixel block and the pixel values ​​of the second sub-pixel block to see whether they are similar. If not, determining that the feature point in the current frame image is not the same as the feature point in the point cloud library, the feature point corresponding to the first sub-pixel block is a fourth type of feature point;

[0130] If yes, execute step S132;

[0131] Step S132: Iteratively track the moving distance between the first sub-pixel block and the second sub-pixel block, and compare whether the moving distance is less than the preset distance. If so, it is determined that the feature point in the current frame image is the same feature point as the feature point in the point cloud library, and the feature point corresponding to the first sub-pixel block is a third-category feature point; if not, it is determined that the feature point in the current frame image is not the same feature point as the feature point in the point cloud library, and the feature point corresponding to the first sub-pixel block is a fourth-category feature point.

[0132] It can be understood that when generating the historical local map space, the positioning method in this embodiment has used at least one of steps S211 to S212, S131 to S132, and S14 to S16 to optimize the historical local map, so that the map points in the historical local map are more accurate, and the map points in the historical local map correspond to the feature points on the historical image. When the feature points in the historical local map do not appear in the previous frame of the image due to the interruption of the previous image frame, the object is blocked, etc., and directly appear in the current frame of the image frame across frames, this embodiment can solve the problem that the feature points cannot be associated after crossing frames by retrieving the historically marked map points, so that the positioning accuracy of the device in the local map space is higher and more stable.

[0133] In another embodiment, Figure 6 As shown, all the steps in the above embodiments are combined and executed, which are repeated and will not be described here.

[0134] In one embodiment, the internal and external parameters of the camera and IMU of the device need to be calibrated when inputting data. Specifically, the open source tool Kalibr can be used to calibrate the internal and external parameters of the camera and IMU. The camera model used in the above embodiment can be used as long as it can correctly calculate the projection and back-projection relationship of the pixel, and the specific camera model is not limited. When calibrating the internal parameters of the IMU, the bias of the accelerometer and gyroscope needs to be calibrated. In this way, the accuracy of the measured data and state parameters is guaranteed.

[0135] Based on the same inventive concept, the present application also provides a device positioning device, which is used to implement the above-mentioned embodiments and preferred implementation modes, and will not be repeated hereafter. The terms "module", "unit", "subunit", etc. used below may be a combination of software and / or hardware that implements predetermined functions. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0136] A computer device comprises a processor and a memory, wherein the memory stores program codes, and when the program codes are executed by the processor, the processor executes the steps of the method in any one of the above embodiments.

[0137] The computer equipment also includes positioning means, such as Figure 7 As shown, the positioning device includes a data acquisition module 100, a first association module 200, a second association module 300, a point cloud library update module 400, a local map update module 500 and a calculation module 600, wherein:

[0138] The data acquisition module 100 is used to obtain the state parameters of the previous frame image, the current frame image and the previous frame image to the current frame image captured by the device;

[0139] The first associating module 200 is used to associate the feature points of the current frame image with the feature points of the previous frame image by optical flow tracking to obtain the first type of feature points, the second type of feature points and the local map space of the current frame image; the first type of feature points refer to the feature points in the current frame image that can be associated with the feature points of the previous frame image; the second type of feature points refer to the feature points in the current frame image that cannot be associated with the feature points of the previous frame image;

[0140] The second association module 300 is used to associate the second type of feature points in the current frame image with the feature points in the pre-stored point cloud library to obtain the third type of feature points and / or the fourth type of feature points of the current frame image; the second type of feature points include the third type of feature points and the fourth type of feature points; the third type of feature points refer to feature points in the current frame image that cannot be associated with feature points in the previous frame image but can be associated with feature points in the pre-stored point cloud library; the fourth type of feature points refer to feature points in the current frame image that cannot be associated with feature points in the previous frame image and cannot be associated with feature points in the pre-stored point cloud library; the point cloud library stores all map points in the historical local map space;

[0141] The point cloud library updating module 400 is used to update the fourth type of feature points into the point cloud library;

[0142] The local map updating module 500 is used to update the local map space of the current frame according to the association result between the feature points in the current frame image and the feature points in the previous frame image and the feature points in the point cloud library to obtain the updated local map space of the current frame;

[0143] The calculation module 600 is used to calculate the position and posture of the device in the updated local map space of the current frame according to the state parameters and the updated local map space of the current frame.

[0144] In one embodiment, the second association module 300 is further used to: compare the pixel value of the first sub-pixel block with the pixel value of the second sub-pixel block, and if the pixel values ​​of the first sub-pixel block and the second sub-pixel block are similar, iteratively track the movement distance between the first sub-pixel block and the second sub-pixel block;

[0145] If the moving distance is less than the preset distance, the feature point is a map point, and the feature point is associated with the map point.

[0146] In one embodiment, Figure 8 As shown, the positioning device further includes at least one of an optical flow prediction module 210, a mask extraction module 220, and a depth estimation module 230, and the optical flow prediction module 210, the mask extraction module 220, and the depth estimation module 230 are in a parallel operation relationship, wherein:

[0147] The optical flow prediction module 210 is used to predict the predicted position of the feature point in the previous frame image in the current frame image based on the state parameters and the representation information of the feature point in the current frame image by using the optical flow prediction method; and use the predicted position as the initial value of the optical flow tracking;

[0148] The mask extraction module 220 is used to extract the mask of the dynamic object in the current frame image based on the previous frame image and the current frame image;

[0149] According to the mask, the region of the dynamic object in the current frame image is identified to obtain the static image of the current frame with the mask;

[0150] Eliminate the area of ​​dynamic objects in the current frame image to obtain a static image of the current frame;

[0151] The depth estimation module 230 is used to obtain the depth of the fourth type of feature points in the current frame image, and determine whether the depth of the fourth type of feature points exceeds a depth threshold.

[0152] In one embodiment, the positioning device also includes a feature point quantity judgment module, which is used to divide the static image of the current frame into grids and count the number of feature points in the grids. If the number of feature points in the grids is less than a preset number, new feature points are extracted from the grids so that the grids contain no less than a preset number of feature points.

[0153] In one embodiment, the positioning device further includes an external point removal module for determining whether the fourth type of feature point is an external point. If so, the fourth type of feature point is discarded; if not, the fourth type of feature point is added to the point cloud library.

[0154] In one embodiment, the positioning device further includes an optimization module, which optimizes the local map space by using the optimization objective function in the above formula (10).

[0155] The framework of the positioning device in the above example adopts a modular design, and the modules can be decoupled from each other, so that all modules can be replaced and optimized separately, so that the positioning device can adapt to different hardware platforms and application scenarios, thereby improving the adaptability and scalability of the positioning device. In addition, it has the following advantages:

[0156] 1) Continuously improve performance: Since each module can be optimized separately, when more advanced algorithms or technologies are introduced in the future, there is no need to change the framework structure of the entire positioning device, thereby continuously improving the overall performance of the positioning device.

[0157] 2) Reduce development costs and difficulty: Modular design simplifies the debugging and optimization process of different modules. Developers can focus on improving a certain module without affecting other modules, which improves development efficiency and reduces integration complexity.

[0158] 3) Customization capability: Under different application requirements, relevant modules can be flexibly replaced or optimized according to specific scenarios (such as indoor, outdoor, low light, dynamic scenes, etc.) to meet the customized requirements of specific applications and enhance the versatility of the system.

[0159] It should be noted that each module in the above-mentioned computer device can be a functional module or a program module, and can be implemented in whole or in part by software, hardware or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0160] In one embodiment, a computer device has a CPU and an NPU / GPU, a first association module is embedded in the CPU of the computer device, a mask extraction module is embedded in the NPU / GPU, and a depth estimation module is embedded in the NPU / GPU. In this way, when the first association module, the mask extraction module, and the depth estimation module operate in parallel, the computing power occupied by the computer device can be reduced.

[0161] Furthermore, the computer device may be a terminal, and its internal structure diagram may be as follows: Fig. 9As shown. The computer device also includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a positioning method of the device is implemented.

[0162] Those skilled in the art will understand that Fig. 9 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0163] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0164] Obtaining a previous frame image, a current frame image, and state parameters from the previous frame image to the current frame image captured by the device;

[0165] By adopting the optical flow tracking method, the feature points of the current frame image are associated with the feature points of the previous frame image to obtain the first type of feature points, the second type of feature points and the current frame local map space of the current frame image; the first type of feature points refer to the feature points in the current frame image that can be associated with the feature points of the previous frame image; the second type of feature points refer to the feature points in the current frame image that cannot be associated with the feature points of the previous frame image;

[0166] Associating the second-category feature points in the current frame image with the feature points in the pre-stored point cloud library to obtain the third-category feature points and / or the fourth-category feature points of the current frame image; the second-category feature points include the third-category feature points and the fourth-category feature points; the third-category feature points refer to the feature points in the current frame image that cannot be associated with the feature points in the previous frame image but can be associated with the feature points in the pre-stored point cloud library; the fourth-category feature points refer to the feature points in the current frame image that cannot be associated with the feature points in the previous frame image and cannot be associated with the feature points in the pre-stored point cloud library; the point cloud library stores all map points in the historical local map space;

[0167] Updating the fourth type of feature points into the point cloud library;

[0168] According to the association result of the feature points in the current frame image, the feature points in the previous frame image and the feature points in the point cloud library, the current frame local map space is updated to obtain an updated current frame local map space;

[0169] The position and posture of the device in the updated local map space of the current frame is calculated based on the state parameters and the updated local map space of the current frame.

[0170] In one embodiment, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the following steps:

[0171] Obtaining a previous frame image, a current frame image, and state parameters from the previous frame image to the current frame image captured by the device;

[0172] By adopting the optical flow tracking method, the feature points of the current frame image are associated with the feature points of the previous frame image to obtain the first type of feature points, the second type of feature points and the current frame local map space of the current frame image; the first type of feature points refer to the feature points in the current frame image that can be associated with the feature points of the previous frame image; the second type of feature points refer to the feature points in the current frame image that cannot be associated with the feature points of the previous frame image;

[0173] Associating the second-category feature points in the current frame image with the feature points in the pre-stored point cloud library to obtain the third-category feature points and / or the fourth-category feature points of the current frame image; the second-category feature points include the third-category feature points and the fourth-category feature points; the third-category feature points refer to the feature points in the current frame image that cannot be associated with the feature points in the previous frame image but can be associated with the feature points in the pre-stored point cloud library; the fourth-category feature points refer to the feature points in the current frame image that cannot be associated with the feature points in the previous frame image and cannot be associated with the feature points in the pre-stored point cloud library; the point cloud library stores all map points in the historical local map space;

[0174] Updating the fourth type of feature points into the point cloud library;

[0175] According to the association result of the feature points in the current frame image, the feature points in the previous frame image and the feature points in the point cloud library, the current frame local map space is updated to obtain an updated current frame local map space;

[0176] The position and posture of the device in the updated local map space of the current frame is calculated based on the state parameters and the updated local map space of the current frame.

[0177] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0178] In some possible implementations, various aspects of the positioning method provided in the present application may also be implemented in the form of a program product, which includes a computer program. When the program product is run on a computer device, the computer program is used to enable the computer device to execute the steps of the code stream fusion method according to various exemplary embodiments of the present application described above in this specification. For example, the computer device may execute the following steps: Figure 2 Follow the steps shown in .

[0179] The program product may adopt any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0180] The program product of the embodiment of the present application may adopt a portable compact disk read-only memory (CD-ROM) and include a computer program, and can be run on an electronic device. However, the program product of the present application is not limited thereto, and in this document, a readable storage medium may be any tangible medium containing or storing a program, which can be used by or in combination with a command execution system, apparatus, or device.

[0181] The readable storage medium may include a data signal propagated in baseband or as part of a carrier wave, wherein a readable computer program is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The readable signal medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with a command execution system, apparatus, or device.

[0182] The computer program embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0183] The computer program for performing the operation of the present application can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and also conventional procedural programming languages ​​such as "C" language or similar programming languages. The computer program can be executed entirely on the user's computer device, partially on the user's computer device, as a separate software package, partially on the user's computer device and partially on a remote computer device, or completely on the remote computer device. In the case of a remote computer device, the remote computer device can be connected to the user's computer device through any type of network including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer device (for example, using an Internet service provider to connect through the Internet).

[0184] In addition, although the operations of the method of the present application are described in a specific order in the drawings, this does not require or imply that the operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0185] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain a computer-usable computer program.

[0186] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program commands. These computer program commands can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the commands executed by the processor of the computer or other programmable data processing device generate commands for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1A device that provides the functions specified in a block or multiple blocks.

[0187] These computer program commands may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the commands stored in the computer readable memory produce an article of manufacture comprising a command device, the command device implementing the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0188] These computer program commands can also be loaded onto a computer or other programmable data processing device so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the commands executed on the computer or other programmable device provide the instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0189] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0190] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A method for positioning a device, characterized in that: include: Obtaining a previous frame image, a current frame image, and state parameters from the previous frame image to the current frame image captured by the device; By adopting the optical flow tracking method, the feature points of the current frame image are associated with the feature points of the previous frame image to obtain the first type of feature points, the second type of feature points and the current frame local map space of the current frame image; the first type of feature points refer to the feature points in the current frame image that can be associated with the feature points of the previous frame image; the second type of feature points refer to the feature points in the current frame image that cannot be associated with the feature points of the previous frame image; Associating the second-category feature points in the current frame image with the feature points in the pre-stored point cloud library to obtain the third-category feature points and / or the fourth-category feature points of the current frame image; the second-category feature points include the third-category feature points and the fourth-category feature points; the third-category feature points refer to the feature points in the current frame image that cannot be associated with the feature points in the previous frame image but can be associated with the feature points in the pre-stored point cloud library; the fourth-category feature points refer to the feature points in the current frame image that cannot be associated with the feature points in the previous frame image and cannot be associated with the feature points in the pre-stored point cloud library; the point cloud library stores all map points in the historical local map space; Updating the fourth type of feature points into the point cloud library; According to the association result of the feature points in the current frame image, the feature points in the previous frame image and the feature points in the point cloud library, the current frame local map space is updated to obtain an updated current frame local map space; The position and posture of the device in the updated local map space of the current frame is calculated based on the state parameters and the updated local map space of the current frame.

2. The device positioning method according to claim 1, characterized in that: The second type of feature points in the current frame image are associated with the feature points in the pre-stored point cloud library to obtain the third type of feature points and / or the fourth type of feature points of the current frame image: Comparing whether the first sub-pixel block is close to the second sub-pixel block, if so, determining that the feature points of the current frame image can be associated with the feature points in the point cloud library to obtain the third type of feature points, if not, determining that the feature points in the current frame image cannot be associated with the feature points in the point cloud library to obtain the fourth type of feature points; The first sub-pixel block is pixel information of a feature point of the current frame image around the pixel point of the feature point, and the second sub-pixel block is pixel information of a feature point in the point cloud library around the pixel point when the feature point is first observed.

3. The device positioning method according to claim 2, characterized in that: Compare whether the first sub-pixel block is close to the second sub-pixel block. If so, determine that the feature points of the current frame image can be associated with the feature points in the point cloud library to obtain the third type of feature points. If not, determine that the feature points in the current frame image cannot be associated with the feature points in the point cloud library. The fourth type of feature points are obtained in the following manner: Compare whether the pixel value of the first sub-pixel block is similar to the pixel value of the second sub-pixel block, and if not, determine that the feature point in the current frame image is not the same feature point as the feature point in the point cloud library, and the feature point corresponding to the first sub-pixel block is the fourth type of feature point; If so, iterative tracking is performed to obtain the moving distance between the first sub-pixel block and the second sub-pixel block, and the moving distance is compared to see whether it is less than a preset distance. If so, it is determined that the feature point in the current frame image is the same feature point as the feature point in the point cloud library, and the feature point corresponding to the first sub-pixel block is the third type of feature point; if not, it is determined that the feature point in the current frame image is not the same feature point as the feature point in the point cloud library, and the feature point corresponding to the first sub-pixel block is the fourth type of feature point.

4. The device positioning method according to claim 1, characterized in that: The positioning method further comprises: Based on the state parameters and the characterization information of the feature points in the current frame image, an optical flow prediction method is adopted to predict the predicted position of the feature points in the previous frame image in the current frame image; and the predicted position is used as the initial value of the optical flow tracking.

5. The device positioning method according to claim 4, characterized in that: The positioning method further comprises: Based on the previous frame image and the current frame image, extracting a mask of a dynamic object in the current frame image; According to the mask, the region of the dynamic object in the current frame image is identified to obtain the current frame static image with the mask.

6. The device positioning method according to claim 5, characterized in that: The positioning method also includes: dividing the current frame static image into grids, counting the number of feature points in the grids, and if the number of feature points in the grids is less than a preset number, extracting new feature points from the grids so that the grids contain no less than a preset number of feature points.

7. The device positioning method according to claim 5, characterized in that: The updating of the fourth type of feature points to the point cloud library further includes: It is determined whether the fourth type of feature point is an external point. If so, the fourth type of feature point is discarded; if not, the fourth type of feature point is added to the point cloud library.

8. The device positioning method according to claim 7, characterized in that: The method of determining whether the fourth type of feature point is an external point is: Obtain the depth of the fourth-category feature point in the current frame image, and determine whether the depth of the fourth-category feature point exceeds a depth threshold; if so, determine that the fourth-category feature point is an outlier; if not, determine that the fourth-category feature point is not an outlier.

9. A computer device, characterized in that: A method for implementing a positioning method of a device according to any one of claims 1 to 8, wherein the device comprises a positioning device, the positioning device comprising: A data acquisition module, used to obtain a previous frame image, a current frame image and state parameters from the previous frame image to the current frame image captured by the device; A first associating module is used to associate the feature points of the current frame image with the feature points of the previous frame image by optical flow tracking to obtain the first type of feature points, the second type of feature points and the current frame local map space of the current frame image; the first type of feature points refer to the feature points in the current frame image that can be associated with the feature points of the previous frame image; the second type of feature points refer to the feature points in the current frame image that cannot be associated with the feature points of the previous frame image; A second associating module is used to associate the second type of feature points in the current frame image with the feature points in the pre-stored point cloud library to obtain the third type of feature points and / or the fourth type of feature points of the current frame image; the second type of feature points include the third type of feature points and / or the fourth type of feature points; the third type of feature points refer to feature points in the current frame image that cannot be associated with the feature points of the previous frame image but can be associated with the feature points in the pre-stored point cloud library; the fourth type of feature points refer to feature points in the current frame image that cannot be associated with the feature points of the previous frame image and cannot be associated with the feature points in the pre-stored point cloud library; the point cloud library stores all map points in the historical local map space; A point cloud library updating module, used for updating the fourth type of feature points into the point cloud library; A local map updating module, used to update the local map space of the current frame according to the association result of the feature points in the current frame image with the feature points in the previous frame image and the feature points in the point cloud library, so as to obtain an updated local map space of the current frame; A calculation module is used to calculate the position and posture of the device in the updated current frame local map space based on the state parameters and the updated current frame local map space.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the positioning method of the device according to any one of claims 1 to 8 is implemented.

Images