Positioning method and device of self-moving device, self-moving device and storage medium

Abstract

The application provides a positioning method and device of a self-moving device, the self-moving device and a storage medium. The method comprises: acquiring an image frame at a current time, the image frame at the current time comprising a plurality of first feature points; converting the first feature points into second feature points in a top-view perspective coordinate system of the self-moving device according to a preset perspective conversion relationship; calculating a moving pixel distance of the matched second feature points and an angle change amount of the matched second feature points according to the second feature points at the current time and the second feature points acquired at a previous time; and determining the pose information of the self-moving device in a world coordinate system at the current time according to the moving pixel distance, the angle change amount and the pose information of the self-moving device in the world coordinate system at the previous time. The positioning complexity of the self-moving device is reduced while ensuring the positioning accuracy.

Description

TECHNICAL FIELD

[0001] The present application relates to the navigation technical field, and particularly to a positioning method and device of a self-moving device, the self-moving device and a storage medium. BACKGROUND

[0002] With the continuous development of visual odometry technology, the visual odometry technology has been widely applied in fields such as robots, wearable devices, augmented reality and automatic driving.

[0003] However, the common visual odometry algorithm at present needs to fuse the feature points extracted by the camera with the attitude information and speed information measured by the inertial measurement unit (IMU) to ensure the positioning accuracy. This is because when the feature points are collected by the camera, the moving speed of the feature points at a close distance is fast under the forward-looking view, and the moving speed of the feature points at a long distance is slow, so that the camera is difficult to accurately obtain the moving speed of the device, resulting in the phenomenon of positioning drift or failure. Although the fusion of the feature points extracted by the camera with the attitude information and speed information measured by the IMU can improve the positioning accuracy, there are problems of complex data processing process and high cost. SUMMARY

[0004] The present application provides a positioning method and device of a self-moving device, a self-moving device and a storage medium, which utilizes the principle that the moving distance of the feature points in the top view is relatively stable when the camera and the ground height and angle are approximately constant, converts the first feature points at the current time in the current time image frame into the second feature points at the current time in the top view coordinate system of the self-moving device, to calculate the relatively stable moving pixel distance of the second feature points, solves the problem that the moving speed of the feature points is fast in the close distance case of the forward-looking view, and the moving speed of the feature points is slow in the long distance case, so that the average moving distance of the feature points cannot be accurately obtained, and thus the pose information of the self-moving device is accurately determined according to the moving pixel distance of the second feature points, aiming to reduce the positioning complexity of the self-moving device while ensuring the positioning accuracy.

[0005] In a first aspect, the present application embodiment provides a positioning method of a self-moving device, comprising:

[0006] In each detection period, an image frame at a current time is obtained, and the image frame at the current time includes a plurality of first feature points at the current time;

[0007] According to a preset view conversion relationship, the first feature points at the current time are converted into second feature points at the current time in the top view coordinate system of the self-moving device;

[0008] The second feature points at the previous time are obtained.

[0009] According to the second feature point at the current moment and the second feature point at the last moment, a moving pixel distance of the matched second feature point and an angle change amount of the matched second feature point are calculated;

[0010] According to the moving pixel distance, the angle change amount and the acquired pose information of the mobile device in the world coordinate system at the last moment, the pose information of the mobile device in the world coordinate system at the current moment is determined.

[0011] In a second aspect, an embodiment of the present application provides a positioning device of a mobile device, comprising:

[0012] A first acquisition module is configured to acquire an image frame at a current moment in each detection period, and the image frame at the current moment comprises a plurality of first feature points at the current moment;

[0013] A perspective conversion module is configured to convert the first feature points into second feature points in a top-down perspective coordinate system of the mobile device according to a preset perspective conversion relationship;

[0014] A second acquisition module is configured to acquire second feature points at a last moment;

[0015] An angle calculation module is configured to calculate a moving pixel distance of matched second feature points and an angle change amount of the matched second feature points according to the second feature points at the current moment and the acquired second feature points at the last moment;

[0016] A pose determination module is configured to determine pose information of the mobile device in a world coordinate system at the current moment according to the moving pixel distance, the angle change amount and the acquired pose information of the mobile device in the world coordinate system at the last moment.

[0017] In a third aspect, an embodiment of the present application provides a mobile device, comprising a memory and a processor;

[0018] The memory is configured to store a computer program;

[0019] The processor is configured to execute the computer program and realize the steps of the positioning method of the mobile device according to the first aspect when executing the computer program.

[0020] In a fourth aspect, an embodiment of the present application further provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to make the processor realize the steps of the positioning method of the mobile device according to the first aspect.

[0021] The embodiment of the present application provides a positioning method and device of a self-moving device, the self-moving device and a storage medium. First, an image frame at a current moment is acquired, and a plurality of first feature points at the current moment included in the image frame at the current moment are converted into second feature points at the current moment in a top-view perspective coordinate system of the self-moving device according to a preset perspective conversion relationship. Then, a moving pixel distance of the matched second feature points is calculated according to the second feature points at the current moment and the second feature points at a previous moment, and an angle change amount of the matched second feature points is calculated. Finally, the pose information of the self-moving device at the current moment in the world coordinate system is determined according to the moving pixel distance of the matched second feature points, the angle change amount of the matched second feature points and the acquired pose information of the self-moving device at the previous moment in the world coordinate system.

[0022] Since the moving distance of the feature points in the top-view image is stable when the height and angle of the camera to the ground are approximately constant, and the height and angle of the camera to the ground are approximately constant during the movement of the self-moving device, the moving distance of the feature points is stable by converting the plurality of first feature points in the image frame at the current moment into the second feature points in the top-view perspective coordinate system of the self-moving device, and calculating the moving pixel distance of the second feature points. The problem that the average moving distance of the feature points cannot be accurately acquired can be solved, the pose information of the self-moving device can be accurately determined according to the moving pixel distance of the second feature points, the process of observing the triangulation calculation of the three-dimensional points is omitted, and the positioning complexity of the self-moving device can be reduced while ensuring the positioning accuracy.

[0023] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the disclosure of the embodiments of the present application. BRIEF DESCRIPTION OF DRAWINGS

[0024] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0025] Figure 1 A scene architecture diagram of a positioning method of a self-moving device provided by the embodiments of the present application;

[0026] Figure 2 A flowchart of a positioning method of a self-moving device provided by the embodiments of the present application;

[0027] Figure 3 A camera parameter and perspective principle schematic diagram provided by the embodiments of the present application;

[0028] Figure 4 a schematic diagram of a direction vector of an image block provided by an embodiment of the present application;

[0029] Figure 5 a schematic block diagram of a self-moving device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0030] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0031] The flowcharts shown in the drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor do they have to be executed in the described order. For example, some operations / steps can be further decomposed, combined or partially merged, so the actual execution order can be changed according to actual situations.

[0032] It should be noted that the positioning method and device of a self-moving device, the self-moving device and the storage medium provided by the embodiments of the present application can reduce the positioning complexity of the self-moving device while ensuring the positioning accuracy.

[0033] It can be understood that the positioning method of a self-moving device provided by the embodiments of the present application can be applied to any self-moving device with image processing capability or a control device for controlling a self-moving device. The self-moving device can be a device with a central processing unit (CPU) and / or a graphics processing unit (GPU). The self-moving device can be a device with self-moving assistance function, or a semi-self-moving device or a fully autonomous moving device. The self-moving assistance function can be implemented by a vehicle terminal, and the corresponding self-moving device can be a vehicle with the vehicle terminal. The self-moving device can also be a semi-self-moving device or a fully autonomous moving device. For example, but not limited to, a lawn mower, a sweeping robot, a garbage cleaning device, a fixed flying drone, a wheeled car, etc. The self-moving device can also have both CPU and GPU.

[0034] In actual applications, the positioning method of a self-moving device provided by the present application includes but is not limited to applications in the application environment as shown in Figure 1

[0035] As shown in Figure 1 ​As shown, the self-moving device 120 and the camera 140 are connected via a network. It should be noted that the camera 140 of the self-moving device 120 can be a front-view camera, a rear-view camera, or a side-view camera. For ease of explanation, this embodiment will be described based on a scenario where the camera 140 is a front-view camera. That is, the camera 140 is set at a preset fixed position in the front-view direction of the self-moving device 120, for example, mounted on a... Figure 1 The area above and in front of the self-moving device 120 shown is used for real-time acquisition of image information along the movement direction of the self-moving device 120. The real-time acquired image information consists of consecutive image frames, each including feature points corresponding to the target object at the location of the self-moving device 120 at a given time. The self-moving device 120 is equipped with a positioning device 1200, which can be implemented in hardware or software. When the positioning device 1200 is implemented in software, its functions can be logically divided into multiple modules, each with different functions. The functions of each module are implemented by the processor in the self-moving device 120 reading and executing instructions from memory.

[0036] For example, the positioning device 1200 of the self-moving device may include a first acquisition module 1201, a view conversion module 1202, a second acquisition module 1203, an angle calculation module 1204, and a pose determination module 1205. In one specific implementation, the positioning device 1200 of the self-moving device may perform the contents described in steps S201 to S204 below. It should be noted that the embodiments of this application only provide illustrative examples of the structure and functional modules of the positioning device 1200 of the self-moving device.

[0037] The system comprises the following modules: a first acquisition module 1201, which acquires an image frame captured by camera 140 at the current moment, wherein the image frame at the current moment includes multiple first feature points at the current moment; a viewpoint conversion module 1202, which converts the first feature points acquired by the first acquisition module 1201 at the current moment into second feature points at the current moment in the top-down viewpoint coordinate system of the mobile device according to a preset viewpoint conversion relationship; a second acquisition module 1203, which acquires the second feature points at the previous moment; an angle calculation module 1204, which calculates the moving pixel distance of the matching second feature points and the angle change of the matching second feature points based on the second feature points at the current moment and the second feature points at the previous moment; and a pose determination module 1205, which determines the pose information of the mobile device at the current moment in the world coordinate system based on the moving pixel distance of the matching second feature points, the angle change of the matching second feature points, and the pose information of the mobile device at the previous moment in the world coordinate system.

[0038] In addition, in some possible cases, some of the plurality of modules included in the positioning apparatus 1200 of the self-moving device described above can be combined into one module. For example, the first acquisition module 1201, the perspective conversion module 1202, and the second acquisition module 1203 described above can be combined into an image perspective conversion module, that is, the image perspective conversion module integrates the functions of the first acquisition module 1201, the conversion module 1202, and the second acquisition module 1203.

[0039] Please refer to Figure 2 as shown, Figure 2 A flowchart of a positioning method of a self-moving device provided by an embodiment of the present application. It should be noted that, Figure 2 from the perspective of the self-moving device, each step of the positioning method of the self-moving device provided by the embodiment of the present application is described in detail. By Figure 2 It can be known that the positioning method of the self-moving device provided by the embodiment of the present application includes S201 to S204. Details are as follows:

[0040] S201, in each detection period, an image frame at the current time is acquired, and the image frame at the current time includes a plurality of first feature points at the current time.

[0041] Each detection period can be set according to actual conditions, for example, each detection period can be 0.2 seconds, 0.5 seconds, 1 second, etc. The current time of the present application can be understood as the current detection period, for example, the image frame at the current time can be an image frame acquired when the current time reaches the current detection period of 0.2 seconds. The image frame is the smallest unit of a video, and the image frame refers to image information collected by a camera installed on the self-moving device at the current time.

[0042] In the embodiment of the present application, the plurality of first feature points at the current time in the image frame at the current time can be acquired by a pre-defined feature point extraction algorithm. Exemplarily, the pre-defined feature point extraction algorithm includes but is not limited to an angle point detection algorithm (for example, a FAST angle point detection algorithm, a harris angle point detection algorithm), a scale invariant feature transform (SIFT) algorithm, etc.

[0043] The first feature point at the current moment includes feature points of one or more objects detected in the current frame image. Specifically, the detected feature points are assigned feature point identification information and recorded to form a feature point sequence. Any feature point in this sequence is the first feature point. It should be noted that the feature point identification information is used to uniquely identify the first feature point. In some embodiments, the current frame image can be converted into a corresponding grayscale image using a preset feature point extraction algorithm. Points where the grayscale value changes drastically during the image conversion process, or points with large curvature on the image edges (i.e., the intersection of two edges), are extracted to obtain the first feature point.

[0044] S202, based on the preset view transformation relationship, convert the first feature point at the current moment into the second feature point at the current moment in the top-down view coordinate system of the mobile device.

[0045] It should be understood that the current image frame and the multiple current-time first feature points included in the current image frame are obtained from the camera's forward-looking view. Because feature points that are closer in the forward-looking view move faster than feature points that are farther away, it is impossible to accurately determine the actual movement distance of the feature points based on their movement distance in the image. Therefore, it is necessary to transform the feature points from the forward-looking view to the top-view view in order to accurately calculate the actual movement distance of the feature points.

[0046] It should be noted that, given the camera's tilt angle and height, the camera's extrinsic parameters can be obtained. For example, such as... Figure 3 As shown, Figure 3 This is a schematic diagram illustrating the camera parameters and viewing angle principle of an embodiment of this application. Figure 3 In this scenario, assuming the ground is a plane with the z-axis equal to 0, and the camera is currently positioned above the origin of the world coordinate system (i.e., on the z-axis), then the camera's extrinsic parameters can be obtained as the rotation matrix R and the translation matrix t. Specifically, the rotation matrix R of the camera's extrinsic parameters can be expressed as follows:

[0047]

[0048] Where θ is the angle between the camera's optical axis and the horizontal plane.

[0049] The translation matrix t of the camera's extrinsic parameters can be expressed as:

[0050]

[0051] Where h is the height of the camera above the horizontal plane.

[0052] After obtaining the rotation matrix and the translation matrix of the camera extrinsic parameters, the camera field of view observed when the camera is looking down can be obtained according to the pinhole camera model, the camera height (i.e. the distance d from the camera lens to the object), the camera focal length f, and the camera image sensor size w. Specifically, the camera field of view observed when the camera is looking down is equal to the product of the distance d from the camera lens to the object and the camera image sensor size w, and then divided by the camera focal length f.

[0053] According to the calculation process of the camera field of view, the pixel width to actual spatial distance scale ratio s can be determined, specifically, s = f / d. Thus, the mapping relationship between the pixel coordinates (u', v') of each point in the top view (i.e. the top view angle coordinate system) and the spatial position coordinates (X, Y, 0) of the corresponding world coordinate system on the actual scene ground in the top view angle can be obtained as follows: u' = X*s; v' = Y*s. It should be noted that since the ground is a plane with z axis equal to 0, Z = 0.

[0054] Further, the spatial position coordinates in the camera coordinate system can be obtained from the spatial position coordinates in the world coordinate system, denoted as (X, Y, 0). Specifically, the spatial position coordinates in the camera coordinate system are denoted as (x, y, z), and the conversion relationship between the spatial position coordinates in the camera coordinate system and the spatial position coordinates in the world coordinate system can be shown by the following formula (1):

[0055]

[0056] wherein, (x, y, z) is the spatial position coordinates of the spatial point P in the camera coordinate system, (X, Y, 0) is the spatial position coordinates of the spatial point P in the world coordinate system, R is the rotation matrix of the camera extrinsic parameters, and t is the translation matrix of the camera extrinsic parameters.

[0057] After obtaining the spatial position coordinates (x, y, z) in the camera coordinate system, the coordinates on the camera normalization plane can be further obtained, specifically, the coordinates on the camera normalization plane can be obtained by the following formula (2):

[0058] x' = x / z, y' = y / z (2)

[0059] wherein, (x', y') is the coordinates on the camera normalization plane.

[0060] After obtaining the coordinates on the camera normalization plane, the coordinates of the pixels after radial distortion and tangential distortion can be obtained by the following formula (3) and formula (4), specifically:

[0061] x" = x' * (1 + k1 * r 2 + k2 * r 4)+ 2 * p1 * x' * y' + p2 * (r 2 + 2x′ 2 )(3) y'' = y' * (1 + k1 * r2 + k2 * r 4 )+ 2 * p2 * x' * y' + p1 * (r2 + 2y′ 2 )(4)

[0062] Wherein, x" is the coordinate of the pixel after radial distortion, y" is the coordinate of the pixel after tangential distortion, r is the distance between the coordinate on the camera normalization plane and the coordinate origin, k1 and k2 are radial distortion coefficients, p1 and p2 are tangential distortion coefficients.

[0063] After obtaining the pixel distortion coordinates, the pixel coordinates under the front view angle can be further obtained by the following formula (5):

[0064] u = fx * x" + cx, v = fy * y" + cy (5)

[0065] Wherein, fx, fy, cx and cy are camera intrinsic parameters, which are obtained in advance through camera intrinsic parameter calibration.

[0066] The above process can obtain the corresponding conversion relationship between the pixel coordinates (u, v) under the front view angle coordinate system and the pixel coordinates (u', v') under the top view angle coordinate system, that is, the angle of view conversion relationship. Therefore, the first feature point at the current moment is substituted into the above formula for calculation, and the second feature point under the top view angle coordinate system is obtained. It should be noted that the pixel gray value under the front view angle coordinate is completely the same as the pixel gray value under the top view angle coordinate system.

[0067] It should be noted that since the image at a distance is sparse after stretching, the extracted second feature point will also be inaccurate, therefore, the present application embodiment only retains part of the second feature points in the field of view, and the size of the field of view can be set according to actual needs, which is not limited by the present application embodiment.

[0068] S203, obtaining the second feature point at the last moment.

[0069] Wherein, the second feature point at the last moment represents the feature point converted from the first feature point at the last moment to the top view angle coordinate system of the mobile device, and the specific implementation manner is the same as that of step S201, which will not be described herein. In addition, the last moment can be understood as the last detection period. Since the second feature point obtained by the first feature point angle conversion is saved in the preset database in each detection period, the second feature point at the last moment can be directly obtained from the database.

[0070] S204, calculate a moving pixel distance of the matched second feature point and an angle change amount of the matched second feature point according to the second feature point at the current moment and the second feature point at the last moment.

[0071] The moving pixel distance of the matched second feature point refers to the distance between pixels of the same feature point on different frames of images in a top-down perspective coordinate system of the mobile device. Specifically, the same feature points of the same object on different frames of images can be marked as pairs of feature points, and the moving pixel distance of the matched second feature point can be obtained by determining the average pixel distance of each pair of feature points on different frames of images.

[0072] In an embodiment, calculating the moving pixel distance of the matched second feature point comprises: determining the pixel distance of each pair of second feature points; calculating the average pixel distance according to the pixel distance of each pair of second feature points; and calculating the moving pixel distance of the matched second feature point according to the preset scale ratio and the calculated average pixel distance.

[0073] The pair of second feature points refers to a pair of feature points of the same feature point of the same object on two frames of images.

[0074] In an embodiment, the first feature points of the current time are matched with the first feature points obtained at the previous time to obtain matched first feature point pairs; and second feature point pairs corresponding to the first feature point pairs are determined, the second feature point pairs including the second feature points of the current time and the second feature points of the previous time. For example, the first frame image of the previous time and the second frame image of the current time are tracked by a preset feature point extraction algorithm to track the same first feature points of the same object on the first frame image and the second frame image, and the paired first feature points, i.e., the first feature point pairs, can be obtained. Since the first feature point pairs represent the first feature points of the current time and the first feature points of the previous time carrying the same feature point identification information, i.e., the first feature point pairs are obtained through the perspective transformation relationship to obtain the corresponding second feature point pairs. Or the second feature points of the previous time and the second feature points of the previous time carrying the same feature point identification information are matched as the second feature points, i.e., the second feature point pairs. The preset feature point extraction algorithm includes but is not limited to corner detection algorithms such as FAST corner detection algorithm, harris corner detection algorithm, scale invariant feature transform algorithm (SIFT), etc. Exemplarily, the preset feature point extraction algorithm is taken as the harris corner detection algorithm for illustration. Specifically, the first frame image (i.e., the image frame of the previous time) and the second frame image (i.e., the image frame of the current time) are respectively converted into gray images, and a local window is used to move on the first frame image and the second frame image respectively during the image conversion process, and the gray level changes of the second feature points of the previous time on the first frame image and the second feature points of the current time on the second frame image are respectively determined; the second feature points of the previous time and the second feature points of the current time with the gray level changes greater than a preset change threshold are extracted from the first frame image and the second frame image respectively; if there are second feature points with equal gray level changes on the first frame image and the second frame image, the second feature points of the current time and the second feature points of the previous time with equal gray level changes are extracted and taken as the paired second feature points.

[0075] In a specific implementation, according to the pixel distance of each paired second feature point, the process of calculating the average pixel distance includes: the pixel distance of each paired second feature point from the first image frame of the previous time to the second image frame of the current time is counted, and the average pixel distance of the paired second feature points is calculated by dividing the sum of the pixel distances of the paired second feature points by the number of the paired second feature points. Specifically, the pixel distance of each paired second feature point from the first image frame to the second image frame includes the pixel distance of each paired second feature point in the x direction and the pixel distance of each paired second feature point in the y direction.

[0076] Furthermore, after obtaining the average pixel distance, the calculated average pixel distance is substituted into the following formulas (6) and (7) to calculate the moving pixel distance of the second feature point.

[0077] △X = u' avg / s (6)

[0078] △Y = v' avg / s (7)

[0079] Where △X represents the pixel distance of the paired second feature points moving in the X direction, and △Y represents the pixel distance of the paired second feature points moving in the Y direction; u' avg v' represents the calculated average pixel distance of the second feature point in the X direction. avg represents the average pixel distance of the calculated second feature point in the Y direction; s represents the ratio of pixel width to the actual spatial distance scale.

[0080] In one embodiment, calculating the angular change of the matched second feature points includes: determining a first image region and a second image region in the top-view coordinate system of the self-moving device based on each pair of second feature points in the second feature point pair; wherein the first image region is determined by the second feature points in the second feature point pair at the current moment, and the second image region is determined by the second feature points in the second feature point pair at the previous moment; obtaining the first centroid position of the first image region based on the grayscale values ​​of the second feature points in the first image region at each current moment; obtaining the second centroid position of the second image region based on the grayscale values ​​of the second feature points in the second image region at each previous moment; and determining the angular change of the second feature point pair based on the first centroid position, the first geometric center position of the first image region, and the second centroid position and the second geometric center position of the second image region.

[0081] Specifically, determining the angular change of the second feature point pair based on the first centroid position and the first geometric center position of the first image region, as well as the second centroid position and the second geometric center position of the second image region, includes: obtaining a first direction vector based on the first geometric center position and the first centroid position; obtaining a second direction vector based on the second geometric center position and the second centroid position; and subtracting the angle of the first direction vector from the angle of the second direction vector to obtain the angular change of the second feature point pair.

[0082] It should be noted that the first center of mass position refers to the center with the gray value of the image frame at the current time as the weight, and the second center of mass position refers to the center with the gray value of the image frame at the last time as the weight. Specifically, the circumscribed rectangle of the second feature point region retained in both the image frame at the last time and the image frame at the current time can be selected respectively, an image block is taken in the circumscribed rectangle respectively, and the first center of mass position of the first image region and the second center of mass position of the second image region are determined by defining the moments of the image block. In this application, the moment of the image block can be represented by formula (8):

[0083] m pq =∑ x，y∈B x p x q I(x,y) (8)

[0084] Wherein, p and q = {0, 1}, I(x, y) is the gray value of the pixel coordinate position (x, y) in the image block, p and q are the orders of the image block respectively, B represents the image block, that is, the range of x and y should meet within the pixel row number and column number of the image block B, and m represents the moment of the image block.

[0085] The moments m 00 , m 01 , m 10 , m 11 can be calculated respectively by the above formula of the moment of the image block. Wherein, m 00 is the 0-order moment of the image block, which can be used to represent the area of the image block, m 01 and m 10 are the first-order moments of the image block, which are the moments of the image block about the x axis and the y axis, and can be used to determine the center of mass of the image block.

[0086] Specifically, the first center of mass position and the second center of mass position can be obtained by formula (9) respectively:

[0087]

[0088] Wherein, Q x represents the pixel coordinate of the center of mass of the image block in the x direction, and Q y represents the pixel coordinate of the center of mass of the image block in the y direction.

[0089] In addition, m 11 represents the inclination of the target region where the image block is located. In the embodiment of the application, the first direction vector can be obtained by connecting the geometric center of the image block of the first image region with the center of mass, the second direction vector can be obtained by connecting the geometric center of the image block of the second image region with the center of mass, and the angle change amount of the second feature point pair can be obtained by subtracting the angle of the first direction vector from the angle of the second direction vector. Exemplarily, as shown in Figure 4 ,Figure 4 This is a schematic diagram of the direction vector of an image patch. Figure 4 In the diagram, circle S1 represents the image block selected in the first image region for calculating the gray-scale centroid, and circle S2 represents the image block selected in the second image region for calculating the gray-scale centroid. The solid points in circles S1 and S2 are the extracted matching second feature points. G1 is the geometric center of the image block in the first image region, and Q1 is the centroid of the image block in the first image region; G2 is the geometric center of the image block in the second image region, and G2 is the centroid of the image block in the second image region. Connecting G1 and Q1 yields the first direction vector, and connecting G2 and Q2 yields the second direction vector. The angle α between the first direction vector and the second direction vector can be calculated by formula (10) respectively.

[0090] a = arctan(m) 01 / m 10 ) Formula (10)

[0091] Where, m 01 and m 10 Let be the first moments of the corresponding image patches, and be the moments of the corresponding image patches about the x-axis and y-axis, respectively. It should be understood that, according to the above formulas, the angle a1 of the first direction vector corresponding to the first image region at the current time and the angle a2 of the second direction vector corresponding to the second image region at the previous time can be calculated respectively. The corresponding angle change can then be denoted as ΔΦ = a2 - a1.

[0092] S205, based on the moving pixel distance, angle change, and the obtained pose information of the self-moving device in the world coordinate system at the previous moment, determine the pose information of the self-moving device in the world coordinate system at the current moment.

[0093] The pose information includes the attitude angle and position information of the self-moving device in the world coordinate system at the current moment. It should be noted that, except for the first frame image, the corresponding angle change ΔΦ is accumulated on the attitude angle Φ(t-1) in the world coordinate system at the previous moment for each frame image acquired, so as to obtain the attitude angle of the self-moving device in the world coordinate system at the current moment. Wherein, Φ(1) is the direction angle set when the first frame image is acquired. The attitude angle of the self-moving device in the world coordinate system at the current moment can be expressed as follows: Φ(t)=Φ(t-1)+ΔΦ;

[0094] Where Φ(t) is the attitude angle of the self-moving device in the world coordinate system at the current moment, and Φ(t-1) is the attitude angle of the self-moving device in the world coordinate system at the previous moment.

[0095] In a specific implementation, after obtaining the attitude angle of the mobile device in the world coordinate system at the current time, the mobile pixel distance is converted into the position change of the mobile device in the world coordinate system by converting the attitude angle into a rotation matrix, and the position information of the mobile device in the world coordinate system at the current time is further determined according to the position information of the mobile device at the previous time and the position change. Specifically, after converting the attitude angle Φ(t) of the mobile device in the world coordinate system at the current time into a rotation matrix, the obtained rotation matrix Rz can be expressed as:

[0096]

[0097] In addition, according to the above formula (6) and formula (7), the mobile pixel distance of the second feature point has been calculated as ΔX and ΔY, and it should be noted that the mobile pixel distance is the mobile pixel distance in the camera coordinate system, which can be converted into the world coordinate system according to the following formula (11) and formula (12). Specifically:

[0098] [△Xw,△Yw,△Zw] = [△X,△Y,0]*Rz formula (11)

[0099] Then, ΔXw and ΔYw are added to the point Pw in the world coordinate system, respectively.

[0100] Pw(t) = Pw(t-1) + [△Xw,△Yw,△Zw] formula (12)

[0101] Wherein, [△Xw,△Yw,△Zw] is the position change of the mobile device in the world coordinate system, Pw(t) is the position information of the mobile device in the world coordinate system at the current time, and Pw(t-1) is the position information of the mobile device in the world coordinate system at the previous time.

[0102] In each detection period, the image frames collected by the mobile device are subjected to feature point extraction, view angle conversion, and mobile pixel distance calculation according to the execution mode of steps S201 to S205, and then the pose information of the mobile device at the current time is determined. The present application can be applied to any inclination angle of the camera and the x-axis, y-axis, and z-axis directions of the world coordinate system, and only needs to fix the inclination angle of the camera unchanged, and there is an observation object in the field of view to provide feature points to realize the positioning of the mobile device. The application has a wide angle range and a simple calculation process, thereby improving the positioning efficiency of the mobile device.

[0103] As can be seen from the above analysis, the positioning method of the self-moving device provided in this application firstly acquires the image frame at the current moment, and converts multiple first feature points included in the image frame at the current moment into second feature points in the top-view coordinate system of the self-moving device according to a preset view transformation relationship; then, based on the second feature points at the current moment and the second feature points acquired at the previous moment, the moving pixel distance of the matching second feature points and the angle change of the matching second feature points are calculated; and finally, based on the moving pixel distance of the matching second feature points, the angle change of the matching second feature points, and the pose information of the self-moving device in the world coordinate system acquired at the previous moment, the pose information of the self-moving device in the world coordinate system at the current moment is determined. Since the movement distance of feature points in the top view is relatively stable when the height and angle between the camera and the ground are approximately constant, and the height and angle between the camera and the ground on the mobile device remain approximately constant during the movement of the mobile device, by converting multiple first feature points in the current image frame into second feature points in the top view coordinates of the mobile device, and calculating the movement pixel distance of the second feature points, a more stable feature point movement distance can be obtained. This can solve the problem of not being able to accurately obtain the average movement distance of feature points. Furthermore, the pose information of the mobile device can be accurately determined based on the movement pixel distance of the second feature points, while eliminating the process of triangulation calculation of observation points. This can reduce the complexity of the mobile device positioning while ensuring positioning accuracy.

[0104] The above text combines Figures 1 to 4 The present application provides a detailed description of the positioning method for the self-moving device. The positioning device and the self-moving device provided in this application will be described below with reference to the accompanying drawings.

[0105] See Figure 1 The system architecture diagram shown is a structural schematic of the positioning device of the self-moving device. The positioning device 1200 of the self-moving device includes:

[0106] The first acquisition module 1201 is used to acquire the image frame at the current time in each detection cycle. The image frame at the current time includes multiple first feature points at the current time.

[0107] The view conversion module 1202 is used to convert the first feature point at the current moment into the second feature point at the current moment in the top view coordinate system of the self-moving device according to the preset view conversion relationship.

[0108] The second acquisition module 1203 is used to acquire the second feature point of the previous time step;

[0109] The angle calculation module 1204 is configured to calculate a moving pixel distance of the matched second feature points and an angle change amount of the matched second feature points according to the second feature points at the current moment and the second feature points at the previous moment.

[0110] The pose determination module 1205 is configured to determine the pose information of the mobile device at the current moment in the world coordinate system according to the moving pixel distance, the angle change amount, and the pose information of the mobile device at the previous moment in the world coordinate system.

[0111] In an embodiment, the positioning apparatus 1200 of the mobile device further includes:

[0112] The first matching module is configured to match the first feature points at the current moment with the first feature points obtained at the previous moment to obtain a matched first feature point pair.

[0113] The second matching module is configured to determine a second feature point pair corresponding to the first feature point pair, the second feature point pair including the second feature points at the current moment and the second feature points at the previous moment.

[0114] In an embodiment, the angle calculation module 1204 includes:

[0115] The first determination unit is configured to determine a pixel distance of each of the paired second feature points in each second feature point pair.

[0116] The first calculation unit is configured to calculate an average pixel distance according to each of the pixel distances.

[0117] The second calculation unit is configured to calculate the moving pixel distance according to a preset scale ratio and the average pixel distance.

[0118] In an embodiment, the angle calculation module 1204 includes:

[0119] The second determination unit is configured to determine a first image region and a second image region in the top-view perspective coordinate system according to each of the paired second feature points in the second feature point pair, respectively; the first image region is determined by the second feature points at the current moment in the second feature point pair, and the second image region is determined by the second feature points obtained at the previous moment in the second feature point pair.

[0120] The first obtaining unit is configured to obtain a first centroid position of the first image region according to the gray values of the second feature points at the current moment in the first image region.

[0121] The second obtaining unit is configured to obtain a second centroid position of the second image region according to the gray values of the second feature points obtained at the previous moment in the second image region.

[0122] The third determining unit is configured to determine the angle change amount of the second feature point pair according to the first centroid position and the first geometric center position of the first image region, and the second centroid position and the second geometric center position of the second image region.

[0123] In an embodiment, the third determining unit comprises:

[0124] The first obtaining sub-unit is configured to obtain a first direction vector according to the first geometric center position and the first centroid position;

[0125] The second obtaining sub-unit is configured to obtain a second direction vector according to the second geometric center position and the second centroid position;

[0126] The third obtaining sub-unit is configured to subtract the angle of the first direction vector from the angle of the second direction vector to obtain the angle change amount of the second feature point pair.

[0127] In an embodiment, the pose information comprises a pose angle and position information of the mobile device, and the pose determining module 1205 comprises:

[0128] The obtaining unit is configured to obtain a pose angle of the mobile device at a previous time point in a world coordinate system, and position information at the previous time point;

[0129] The third obtaining unit is configured to obtain a pose angle of the mobile device at a current time point in the world coordinate system according to the pose angle at the previous time point and the angle change amount;

[0130] The first converting unit is configured to convert the pose angle into a rotation matrix;

[0131] The second converting unit is configured to convert the movement pixel distance into a position change amount of the mobile device in the world coordinate system according to the rotation matrix;

[0132] The fourth obtaining unit is configured to obtain the position information of the mobile device at the current time point in the world coordinate system according to the position information at the previous time point and the position change amount.

[0133] In an embodiment, the first matching module further comprises:

[0134] The matching unit is configured to perform optical flow tracking matching on the first feature points at the current time point and the first feature points obtained at the previous time point to obtain a matched first feature point pair.

[0135] The positioning apparatus of the mobile device according to the embodiments of the present application can correspond to performing the methods described in the embodiments of the present application, and the above and other operations and / or functions of each module in the positioning apparatus of the mobile device are respectively to realize the corresponding procedures of each method in the embodiments of the present application, and for brevity, will not be repeated here. Figure 2 ​

[0136] It should be noted that the above-described embodiments are merely illustrative, and the modules described as separate components can or can not be physically separated, and the components shown as modules can or can not be physical modules, i.e., can be located in one place or distributed to multiple network modules. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiments. In addition, the connection relationship between the modules in the device embodiment provided in the present application indicates that there is a communication connection between them, which can be implemented as one or more communication buses or signal lines.

[0137] Please refer to Figure 5 as shown, Figure 5 a schematic block diagram of the self-moving device provided in the embodiments of the present application.

[0138] As Figure 5 shown, the self-moving device 50 includes a processor 51, a memory 52, a communication interface 53 and a bus 54. Among them, the processor 51, the memory 52 and the communication interface 53 communicate through the bus 54, and can also realize communication through wireless transmission and other means. The memory 52 stores executable program codes, and the processor 51 can call the program codes stored in the memory 52 to execute the positioning method of the self-moving device in the foregoing method embodiments.

[0139] It should be understood that in the embodiments of the present application, the processor 51 can be a central processing unit CPU, and the processor 51 can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0140] The memory 52 can include read-only memory and random access memory, and provide instructions and data to the processor 51. The memory 52 can also include non-volatile random access memory. For example, the memory 52 can also store data sets.

[0141] The memory 52 can be a volatile memory or a nonvolatile memory, or can include both volatile and nonvolatile memory. Among them, the nonvolatile memory can be a read-only memory (ROM), a programmable ROM (PROM), an erasable PROM (EPROM), an electrically EPROM (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM) used as an external cache. By way of example, and not limitation, a number of forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchlink DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0142] The bus 54 can include a data bus in addition to, or instead of, the address and control buses shown in FIG. 1 for purposes of example, but can also include any of a wide variety of bus constructions, including, for example, 16-bit data buses or 32-bit data buses, with 8-bit, 16-bit, or 32-bit components.

[0143] It should be understood that the self-moving device 50 according to the embodiments of the present application can correspond to the target object collecting device in the embodiments of the present application, and can correspond to the corresponding subject performing the method according to the embodiments of the present application. Figure 2 The above and other operations and / or functions of the various devices in the self-moving device 50 are described respectively for realizing the corresponding processes of the various methods in the embodiments of the present application, and will not be described here for the sake of brevity. Figure 2 The above and other operations and / or functions of the various devices in the self-moving device 50 are described respectively for realizing the corresponding processes of the various methods in the embodiments of the present application, and will not be described here for the sake of brevity.

[0144] From the above description of the embodiments, those skilled in the art can clearly understand that the present application can be realized by means of software and necessary general hardware, and of course can also be realized by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. Generally, functions completed by computer programs can be easily realized by corresponding hardware, and the specific hardware structure for realizing the same function can also be various, such as analog circuits, digital circuits, or special circuits, etc.

[0145] However, for the present application, software program implementation is a better embodiment in more cases. Based on such understanding, the technical solution of the present application or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a magnetic disk or an optical disk of a computer, and includes a plurality of instructions to make a computer device (which can be a personal computer, a training device, or a network device, etc.) execute the method described in various embodiments of the present application.

[0146] In the above embodiments, all or part of them can be realized by software, hardware, firmware or any combination thereof. When realized by software, all or part of them can be realized in the form of a computer program product.

[0147] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices.

[0148] The computer instructions can be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another, for example, the computer instructions can be transmitted from one website, computer, training device or data center to another website, computer, training device or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) mode. The computer readable storage medium can be any available medium that can be stored by a computer or a data storage device such as a training device, a data center, etc. integrated with one or more available media sets. The available media can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid state disk (SSD)) and the like.

Claims

1. A positioning method of a self-moving device, characterized by, The method comprises: In each detection cycle, an image frame at a current time is acquired, the image frame at the current time comprising a plurality of first feature points at the current time; According to a preset perspective conversion relationship, the first feature points at the current time are converted into second feature points at the current time in a top-view perspective coordinate system of the self-moving device; Second feature points at a previous time are acquired; According to the second feature points at the current time and the second feature points at the previous time, the first feature points at the current time are matched with the first feature points acquired at the previous time to obtain a matched first feature point pair; a second feature point pair corresponding to the first feature point pair is determined, the second feature point pair comprising the second feature points at the current time and the second feature points at the previous time; a moving pixel distance of the matched second feature points is calculated, and an angle change amount of the matched second feature points is calculated, comprising: first and second image regions are respectively determined in the top-view perspective coordinate system according to each pair of second feature points in the second feature point pair; the first image region is determined by the second feature points at the current time in the second feature point pair, and the second image region is determined by the second feature points acquired at the previous time in the second feature point pair; a first centroid position of the first image region is obtained according to the gray values of the second feature points at the current time in the first image region; a second centroid position of the second image region is obtained according to the gray values of the second feature points acquired at the previous time in the second image region; the angle change amount of the second feature point pair is determined according to the first centroid position and a first geometric center position of the first image region, and the second centroid position and a second geometric center position of the second image region; According to the moving pixel distance, the angle change amount, and the acquired pose information of the self-moving device in a world coordinate system at the previous time, pose information of the self-moving device in the world coordinate system at the current time is determined.

2. The method of claim 1, wherein, The calculation of the moving pixel distance of the matched second feature points comprises: A pixel distance of each pair of second feature points in the second feature point pair is determined; An average pixel distance is calculated according to each of the pixel distances; The moving pixel distance is calculated according to a preset scale ratio and the average pixel distance.

3. The method of claim 1, wherein, The determination of the angle change amount of the second feature point pair according to the first centroid position and the first geometric center position of the first image region, and the second centroid position and the second geometric center position of the second image region comprises: A first direction vector is obtained according to the first geometric center position and the first centroid position; A second direction vector is obtained according to the second geometric center position and the second centroid position; The angle change amount of the second feature point pair is obtained by subtracting the angle of the second direction vector from the angle of the first direction vector.

4. The method of claim 3, wherein, The pose information includes a pose angle and position information of the self-moving device, and the pose information of the self-moving device at the current time in the world coordinate system is determined according to the moving pixel distance, the angle change amount and the obtained pose information of the self-moving device at the last time in the world coordinate system, and includes: obtaining the pose angle of the self-moving device at the last time in the world coordinate system and the position information at the last time; obtaining the pose angle in the world coordinate system at the current time according to the pose angle at the last time and the angle change amount; converting the pose angle into a rotation matrix; converting the moving pixel distance into the position change amount of the self-moving device in the world coordinate system according to the rotation matrix; obtaining the position information of the self-moving device in the world coordinate system at the current time according to the position information at the last time and the position change amount.

5. The method of claim 1, wherein, The matching of the first feature points at the current time and the first feature points obtained at the last time to obtain the matched first feature point pairs includes: performing optical flow tracking matching on the first feature points at the current time and the first feature points obtained at the last time to obtain the matched first feature point pairs.

6. A positioning apparatus of a self-moving device, characterized by, The device includes: a first obtaining module configured to obtain an image frame at a current time in each detection period, the image frame at the current time including a plurality of first feature points at the current time; a perspective conversion module configured to convert the first feature points at the current time into second feature points at the current time in a top-view perspective coordinate system of the self-moving device according to a preset perspective conversion relationship; a second obtaining module configured to obtain second feature points at a last time; an angle calculation module configured to calculate a moving pixel distance of the matched second feature points and an angle change amount of the matched second feature points according to the second feature points at the current time and the second feature points at the last time; the angle calculation module includes: a second determining unit configured to determine a first image region and a second image region in the top-view perspective coordinate system according to each pair of second feature points in the second feature point pairs, respectively; the first image region is determined by the second feature point at the current time in the pair of second feature points, and the second image region is determined by the second feature point obtained at the last time in the pair of second feature points; a first obtaining unit configured to obtain a first centroid position of the first image region according to the gray values of the second feature points at the current time in the first image region; a second obtaining unit configured to obtain a second centroid position of the second image region according to the gray values of the second feature points obtained at the last time in the second image region; and a third determining unit configured to determine the angle change amount of the pair of second feature points according to the first centroid position and a first geometric center position of the first image region, and the second centroid position and a second geometric center position of the second image region; a pose determination module configured to determine the pose information of the self-moving device at the current time in the world coordinate system according to the moving pixel distance, the angle change amount and the obtained pose information of the self-moving device at the last time in the world coordinate system; and further including: The first matching module is configured to match the first feature point at the current time with the first feature point obtained at the previous time to obtain a matched first feature point pair. The second matching module is configured to determine a second feature point pair corresponding to the first feature point pair, the second feature point pair including a second feature point at the current time and a second feature point at the previous time.

7. A self-moving device, characterized in that, The memory and the processor are included; The memory is configured to store a computer program; The processor is configured to execute the computer program and implement the steps of the positioning method of the self-moving device according to any one of claims 1 to 5 when executing the computer program.

8. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by the processor to enable the processor to implement the steps of the positioning method of the self-moving device according to any one of claims 1 to 5.

Images