Data Processing Method and Device

Abstract

The present application discloses a data processing method and apparatus, belonging to the field of simultaneous localization and mapping. The method includes: extracting a plurality of first feature point sets from video data including multiple frames of images; each first feature point set includes a plurality of feature points corresponding to the same feature object; wherein, each feature point is extracted from a corresponding frame of image; obtaining a descriptor for each feature point, and performing a merging process on the plurality of first feature point sets according to the obtained descriptors to obtain a plurality of second feature point sets; generating a corresponding map point according to each second feature point set, and determining at least one pair of map points to be judged among the generated plurality of map points; the distance between each pair of map points to be judged is less than a first distance threshold; performing a merging process on each pair of map points to be judged according to the reprojection error corresponding to each pair of map points to obtain a corresponding merged map point.

Description

Technical Field

[0001] This application belongs to the field of simultaneous localization and mapping, and particularly relates to a data processing method and apparatus. Background Art

[0002] SLAM (Simultaneous Localization and Mapping) refers to a robot performing self-localization while moving in an unknown environment and constructing an incremental map based on the collected environmental information. Visual SLAM, which acquires environmental information through a camera, has advantages such as low cost and rich image information, and this technology has received increasing attention.

[0003] However, with the movement of the camera, a certain feature object in the environmental information may disappear and then reappear from the camera's field of view, causing the feature object that reappears after disappearing in the environmental information collected by the camera to be regarded as a completely new feature object by the camera, thereby resulting in a large amount of redundant data in the environmental information collected by the camera. The redundant data may occupy a large amount of memory, leading to an increase in data processing time and a decrease in data processing effect. Summary of the Invention

[0004] The purpose of the embodiments of this application is to provide a data processing method and apparatus that can solve the problem of reducing redundant data in the collected environmental information.

[0005] In a first aspect, the embodiments of this application provide a data processing method, which includes:

[0006] Extracting a plurality of first feature point sets from video data including multiple frames of images; each of the first feature point sets includes a plurality of feature points corresponding to the same feature object; wherein, each of the feature points is extracted from a corresponding frame of image;

[0007] Obtaining a descriptor for each of the feature points, and performing a merging process on the plurality of first feature point sets according to the obtained descriptors to obtain a plurality of second feature point sets;

[0008] Generating a corresponding map point according to each of the second feature point sets, and determining at least one pair of map points to be judged among the generated map points; the distance between each pair of the map points to be judged is less than a first distance threshold;

[0009] Performing a merging process on each pair of the map points to be judged according to the reprojection error corresponding to each pair of the map points to be judged to obtain a corresponding merged map point.

[0010] In a second aspect, the embodiments of this application provide a data processing apparatus, which includes:

[0011] An extraction module, configured to extract a plurality of first feature point sets from video data including multiple frames of images; each of the first feature point sets includes a plurality of feature points corresponding to the same feature object; wherein, each of the feature points is extracted from a corresponding frame of image.

[0012] A first merging module, configured to obtain descriptors of each of the feature points, and perform a merging process on the plurality of first feature point sets according to the obtained descriptors to obtain a plurality of second feature point sets.

[0013] A determination module, configured to generate a corresponding map point according to each of the second feature point sets, and determine at least one pair of map points to be judged among the generated map points; the distance between each pair of the map points to be judged is less than a first distance threshold.

[0014] A second merging module, configured to perform a merging process on each pair of the map points to be judged according to the reprojection error corresponding to each pair of the map points to be judged to obtain a corresponding merged map point.

[0015] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor, a memory, and a program or instruction stored on the memory and executable on the processor. When the program or instruction is executed by the processor, the steps of the data processing method described in the first aspect are implemented.

[0016] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the data processing method described in the first aspect are implemented.

[0017] In a fifth aspect, an embodiment of the present application provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor, and the processor is configured to run a program or instruction to implement the data processing method described in the first aspect.

[0018] In the embodiments of the present application, multiple first feature point sets are extracted from video data including multiple frames of images; each first feature point set includes multiple feature points corresponding to the same feature object; wherein, each feature point is extracted from a corresponding frame of image; a descriptor of each feature point is obtained, and based on the obtained descriptors, the multiple first feature point sets are merged to obtain multiple second feature point sets; a corresponding map point is generated according to each second feature point set, and at least one pair of map points to be judged is determined among the generated multiple map points; the distance between each pair of map points to be judged is less than a first distance threshold; according to the reprojection error corresponding to each pair of map points to be judged, each pair of map points to be judged is merged to obtain the corresponding merged map point. Through the embodiments of the present application, when the number of observations of a feature object is relatively large, the multiple first feature point sets can be merged to obtain multiple second feature point sets, reducing redundant data caused by the same feature object being recognized as corresponding to multiple different first feature point sets. It is also possible to generate a map point corresponding to each second feature point set, and merge each pair of map points to be judged with a relatively short distance to reduce redundant data caused by the same feature object being recognized as corresponding to multiple different map points, thereby improving the data processing efficiency in the process of using environmental information for data processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 is a schematic flowchart of the first data processing method provided by the embodiments of the present application;

[0020] Figure 2 is a schematic flowchart of the second data processing method provided by the embodiments of the present application;

[0021] Figure 3 is a schematic flowchart of a process for merging first feature point sets provided by the embodiments of the present application;

[0022] Figure 4 is a schematic diagram of reprojection error provided by the embodiments of the present application;

[0023] Figure 5 is a schematic flowchart of a process for merging map points provided by the embodiments of the present application;

[0024] Figure 6 is a schematic block diagram of a data processing device provided by the embodiments of the present application;

[0025] Figure 7 is a schematic block diagram of an electronic device provided by the embodiments of the present application;

[0026] Figure 8 is a schematic hardware structure diagram of an electronic device provided by the embodiments of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0027] The technical solutions in the embodiments of the present application will be clearly described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present application belong to the scope of protection of the present application.

[0028] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, rather than to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are usually of the same type, and the number of objects is not limited. For example, the first object can be one or multiple. In addition, "and / or" in the specification and claims means at least one of the connected objects, and the character " / " generally indicates an "or" relationship between the associated objects before and after.

[0029] The data processing method and device provided in the embodiments of the present application will be described in detail below with reference to the accompanying drawings through specific embodiments and their application scenarios.

[0030] Figure 1 It is a schematic flowchart of the first data processing method provided in the embodiments of the present application.

[0031] Step 102, extract a plurality of first feature point sets from video data including multiple frames of images; each first feature point set includes a plurality of feature points corresponding to the same feature object; wherein, each feature point is extracted from a corresponding frame of image.

[0032] The video data can be obtained by shooting with a camera, and the camera can be a monocular camera, a binocular camera, or a depth camera. The embodiments of the present application do not impose special restrictions on the type of camera.

[0033] The camera can change its pose during the process of shooting the video data. The pose of the camera can include the position information of the camera and the shooting angle of the camera.

[0034] The feature object can be an object that has appeared in multiple frames of images included in the video data. For example, a tree, a stool, or a person. If a feature object has appeared in multiple frames of images, it can be considered that the feature object has been observed multiple times by the camera that shoots the video data. Then, in a period of video data, the number of frames of images containing the same feature object can be regarded as the number of observations of the feature object.

[0035] Each first set of feature points includes multiple feature points corresponding to the same feature object, and each feature point is extracted from a corresponding frame of image. For example, a video data includes 60 frames of images. Among them, from the 11th frame of image to the 20th frame of image, a stool is included in these 10 frames of images. And since the pose of the camera changes over time, the position of the stool in these 10 frames of images is different. The feature points corresponding to the stool are extracted from the 11th frame of image, the feature points corresponding to the stool are extracted from the 12th frame of image... the feature points corresponding to the stool are extracted from the 20th frame of image. Then the 10 feature points extracted form a first set of feature points. These 10 feature points correspond to the same stool, and each feature point is extracted from a corresponding frame of image.

[0036] To extract a first set of feature points from the video data, it can be to extract at least one feature point from each frame of the video data. For each feature point, feature matching or feature tracking is performed between two adjacent frames of images, so as to determine a first set of feature points corresponding to the same feature object.

[0037] To extract at least one feature point from each frame of the video data, it can be to extract ORB (Oriented Fast and Rotated Brief) feature points, SIFT (Scale-invariant feature transform) feature points, SURF (speed up robust feature) feature points, etc. from a frame of image.

[0038] Feature matching can be descriptor matching. Feature tracking can be optical flow tracking.

[0039] A descriptor can be understood as that for each feature point, the feature information of a circle of pixel points around the feature point can be described by a group of binary numbers. The feature information includes but is not limited to brightness information, color information, etc.

[0040] In specific implementation, before step 102 is executed, each frame of image in the video data can be preprocessed. The image preprocessing method can be to perform distortion correction on each frame of image according to the calibration parameters of the camera. The calibration parameters include but are not limited to the camera focal length, camera offset, and camera distortion parameters. The image preprocessing method can also be to perform an operation on adjusting the brightness of the image, and can also be to perform an operation on motion blur processing of the image, etc. By preprocessing the image, an image with better quality is obtained, so as to reduce the feature extraction error in the process of extracting feature points.

[0041] Step 104, obtain the descriptor of each feature point, and perform a merging process on multiple first sets of feature points according to the obtained descriptors to obtain multiple second sets of feature points.

[0042] After feature extraction and feature matching are completed, similarity judgment can be performed on the first set of feature points. In the embodiments of the present application, descriptor distance can be used to determine the similarity degree of the first set of feature points.

[0043] Optionally, according to the obtained descriptors, multiple first sets of feature points are merged to obtain multiple second sets of feature points, including: calculating the descriptor distance between any two feature points in each first set of feature points according to the descriptor of each feature point; according to the calculated descriptor distance between any two feature points, counting the descriptor distance set corresponding to each feature point; the descriptor distance set includes the descriptor distances of other feature points that belong to the same first set of feature points as the corresponding feature point; determining the target descriptor corresponding to each first set of feature points according to the descriptor distance set corresponding to each feature point; and merging multiple first sets of feature points according to the target descriptor corresponding to each first set of feature points to obtain multiple second sets of feature points.

[0044] Calculating the descriptor distance between any two feature points in each first set of feature points according to the descriptor of each feature point; counting the descriptor distance set corresponding to each feature point according to the calculated descriptor distance between any two feature points; the descriptor distance set includes the descriptor distances of other feature points that belong to the same first set of feature points as the corresponding feature point.

[0045] For example, the first set of feature points 1 includes 3 feature points, namely feature point 1, feature point 2, and feature point 3. For feature point 1, feature point 2 and feature point 3 are other feature points that belong to the same first set of feature points as this feature point 1. For feature point 2, feature point 1 and feature point 3 are other feature points that belong to the same first set of feature points as this feature point 1. For feature point 3, feature point 1 and feature point 2 are other feature points that belong to the same first set of feature points as this feature point 1.

[0046] The descriptor distance between any two feature points in the first set of feature points 1 can be calculated: the descriptor distance r1 between the descriptor of feature point 1 and the descriptor of feature point 2, the descriptor distance r2 between the descriptor of feature point 1 and the descriptor of feature point 3, and calculate the descriptor distance r3 between the descriptor of feature point 2 and the descriptor of feature point 3.

[0047] Then, the descriptor distance set corresponding to feature point 1 includes: r1, r2; the descriptor distance set corresponding to feature point 2 includes: r1, r3; the descriptor distance set corresponding to feature point 3 includes: r2, r3.

[0048] Optionally, determining a target descriptor corresponding to each first feature point set according to the descriptor distance sets corresponding to each feature point includes: averaging the descriptor distances in the descriptor distance sets corresponding to each feature point to obtain an average descriptor distance corresponding to each feature point; in each first feature point set, determining the descriptor of the feature point with the smallest corresponding average descriptor distance as the target descriptor corresponding to each first feature point set.

[0049] Averaging the descriptor distances in the descriptor distance sets corresponding to each feature point to obtain an average descriptor distance corresponding to each feature point. For example, the first feature point set 1 includes 3 feature points, namely feature point 1, feature point 2, and feature point 3. The descriptor distance set corresponding to feature point 1 includes: r1, r2; the descriptor distance set corresponding to feature point 2 includes: r1, r3; the descriptor distance set corresponding to feature point 3 includes: r2, r3. Then, averaging the descriptor distances in the descriptor distance set corresponding to feature point 1 to obtain the average descriptor distance corresponding to feature point 1, that is, 1 / 2(r1 + r2); averaging the descriptor distances in the descriptor distance set corresponding to feature point 2 to obtain the average descriptor distance corresponding to feature point 2, that is, 1 / 2(r1 + r3); averaging the descriptor distances in the descriptor distance set corresponding to feature point 3 to obtain the average descriptor distance corresponding to feature point 3, that is, 1 / 2(r2 + r3).

[0050] In each first feature point set, determining the descriptor of the feature point with the smallest corresponding average descriptor distance as the target descriptor corresponding to each first feature point set. For example, in the aforementioned first feature point set 1, the average descriptor distance 1 / 2(r1 + r2) corresponding to feature point 1 is the smallest, the average descriptor distance 1 / 2(r1 + r3) corresponding to feature point 2 is the largest, and the average descriptor distance 1 / 2(r2 + r3) corresponding to feature point 3 is between the two. Then, the descriptor of feature point 1 can be determined as the target descriptor corresponding to the first feature point set 1.

[0051] Optionally, the multiple second feature point sets include at least one merged feature point set and at least one non-merged feature point set; merging the multiple first feature point sets according to the target descriptors corresponding to each first feature point set to obtain multiple second feature point sets, including: determining at least one pair of feature point sets to be judged in the multiple first feature point sets; calculating the descriptor distance between each pair of feature point sets to be judged according to the target descriptors corresponding to each first feature point set to obtain a target descriptor distance; merging each pair of feature point sets to be judged with a target descriptor distance less than the second distance threshold to obtain a corresponding merged feature point set.

[0052] The second set of feature points can be a corresponding merged set of feature points obtained by merging at least two first sets of feature points, or a non-merged set of feature points corresponding to the first set of feature points. Among multiple first sets of feature points, some first sets of feature points can be merged, while some first sets of feature points cannot be merged.

[0053] Two first sets of feature points that can be merged can be regarded as two first sets of feature points corresponding to the same feature object. For example, in a 60-second video data, a first set of feature points 1 corresponding to stone A is extracted from the 10th frame to the 20th frame, and a first set of feature points 2 corresponding to stone B is extracted from the 45th frame to the 52nd frame. If the first set of feature points 1 and the first set of feature points 2 can be merged, then it can be regarded that stone A and stone B are the same stone.

[0054] Determine at least one pair of feature point sets to be judged among multiple first sets of feature points. For example, multiple first sets of feature points include a first set of feature points 1 corresponding to stone A and a first set of feature points 2 corresponding to stone B. When the similarity between stone A and stone B is greater than a preset similarity threshold, the first set of feature points 1 and the first set of feature points 2 are determined as a pair of feature point sets to be judged.

[0055] According to the target descriptors corresponding to each first set of feature points, calculate the descriptor distance of each pair of feature point sets to be judged to obtain the target descriptor distance. For example, a pair of feature point sets to be judged includes a first set of feature points 1 and a first set of feature points 2. The first set of feature points 1 includes feature point 1, feature point 2, and feature point 3, and the first set of feature points 2 includes feature point 4, feature point 5, feature point 6, and feature point 7. The target descriptor corresponding to the first set of feature points 1 is the descriptor of feature point 1, and the target descriptor corresponding to the first set of feature points 2 is the descriptor of feature point 6. Then calculate the descriptor distance between the descriptor of feature point 1 and the descriptor of feature point 6 as the descriptor distance of this pair of feature point sets to be judged, that is, the target descriptor distance.

[0056] Merge each pair of feature point sets to be judged with a target descriptor distance less than the second distance threshold to obtain a corresponding merged set of feature points.

[0057] The second distance threshold can be a pre-set distance threshold for descriptor distance.

[0058] When the target descriptor distance of any pair of feature point sets to be judged is less than the second distance threshold, merge this pair of feature point sets to be judged to obtain a merged set of feature points corresponding to this pair of feature point sets to be judged.

[0059] When the target descriptor distance between any pair of sets of feature points to be judged is greater than the second distance threshold, it is determined that the pair of sets of feature points to be judged does not need to be merged.

[0060] The following may be combined with Figure 2 to specifically illustrate step 104. Figure 2 It is a schematic flow chart of a process for merging a first set of feature points provided by an embodiment of the present application.

[0061] In specific implementation, according to the similarity judgment result of the feature objects corresponding to the first set of feature points, the first set of feature points 1 and the first set of feature points 2 to be compared can be determined from multiple first sets of feature points, or any two first sets of feature points in the multiple first sets of feature points can be used as the first set of feature points 1 and the first set of feature points 2 to be compared.

[0062] For example Figure 2 As shown, in step 202, calculate the descriptor distance of the first set of feature points 1.

[0063] In step 204, calculate the descriptor distance of the first set of feature points 2.

[0064] Step 202 and step 204 can be executed simultaneously, or step 202 can be executed first and then step 204, or vice versa.

[0065] In step 206, obtain the target descriptor of the first set of feature points 1.

[0066] After step 202 is executed, step 206 is executed.

[0067] In step 208, obtain the target descriptor of the first set of feature points 2.

[0068] After step 204 is executed, step 208 is executed.

[0069] In step 210, calculate the descriptor distance between the target descriptor of the first set of feature points 1 and the target descriptor of the first set of feature points 2.

[0070] In step 212, determine whether the descriptor distance is less than the second distance threshold.

[0071] If so, execute step 214.

[0072] In step 214, merge the first set of feature points 1 and the first set of feature points 2 to obtain the second set of feature points 1.

[0073] In step 106, generate a corresponding map point according to each second set of feature points, and determine at least one pair of map points to be judged among the generated multiple map points; the distance between each pair of map points to be judged is less than the first distance threshold.

[0074] The map points are used to reflect the specific position information of the feature points in the three-dimensional space on each frame of image. For a monocular camera, the triangulation method is generally used to obtain the depth information corresponding to the feature points; for a stereo camera, the depth information corresponding to the feature points can be obtained by calculating the left-right visual disparity; for a depth camera, the depth information corresponding to the feature points can be directly obtained.

[0075] To determine at least one pair of map points to be judged among the generated multiple map points, it can be to calculate the map point distance between two adjacent map points, and then judge whether the map point distance is less than the first distance threshold. If so, these two adjacent map points are determined as a pair of map points to be judged.

[0076] The first distance threshold can be a preset map point distance threshold.

[0077] Optionally, before generating a corresponding map point according to each second feature point set, the data processing method further includes: obtaining the target camera pose of the camera that collects the video data; generating a corresponding map point according to each second feature point set, including: determining the coordinate information and target depth information of the initial map point corresponding to each second feature point set according to the coordinate information and depth information of each feature point in each second feature point set; correcting the coordinate information of the initial map point according to the target camera pose and the target depth information to obtain the target map point corresponding to each second feature point set.

[0078] The target camera pose can be the position information and attitude information of the camera that collects the video data.

[0079] The visual slam system can use the reprojection error to construct a least squares problem to optimize the results of the system, such as pose and depth.

[0080] Figure 3 It is a schematic diagram of the reprojection error provided by an embodiment of the present application. As Figure 3 shown, assume that the coordinates of a certain spatial point are and its projected pixel coordinates are . The corresponding relationship between the pixel position and the spatial position is as follows:

[0081] (1)

[0082] Written in matrix form as:

[0083] su = KTP (2)

[0084] Where s is the depth information corresponding to the feature point in the camera coordinate system, K is the camera internal parameter, and T is the camera pose.

[0085] As Figure 3 shown, point P can be a spatial point, p1 can be the projection of P when the camera is in pose 1, and p2 can be the projection of P when the camera is in pose 2. However, in fact, the spatial point P calculated from the coordinate information, depth information of p1, and camera pose 1 has an inaccurate projection coordinate at pose 2, such as p2'.

[0086] That is, due to the existence of system noise in the visual slam system, there is an error in the above formula. Summing the errors to construct a least squares problem to determine the target camera pose and depth, making the value of the following formula (3) the smallest.

[0087] (3)

[0088] According to the target camera pose and target depth information, correct the coordinate information of the initial map points to obtain the target map points corresponding to each second feature point set. It can be understood that substituting the target camera pose and target depth information into formula (1) can determine the coordinate information of the target map points corresponding to each second feature point set.

[0089] After the feature points are observed by the camera multiple times, it is necessary to construct constraints to participate in the coordinate correction of the map points to obtain the optimal value of the system. When the feature point sets are merged, the map points corresponding to the feature points already have depth values. Therefore, in addition to merging the self-attributes of the feature points, it is also necessary to merge the map points corresponding to the feature points. In this paper, the number of feature points is used as the weight value to update the depth. For example, the first feature point set 1 has w1 feature points with a depth of d1, and the first feature point set 2 has w2 feature points with a depth of d2. The merged depth d is shown in the following formula (4).

[0090] (4)

[0091] It can be understood that in the case of merging the first feature point set, the more feature points included in the first feature point set, the higher the weight of the target depth information corresponding to the first feature point set with more feature points.

[0092] In the technical solutions of observing feature points through a monocular camera or through a binocular camera, a large amount of relatively accurate depth information and a small amount of less accurate depth information may be obtained. In this case, after merging the depths of the feature point sets in the above manner, the credibility of the depth information is improved. Furthermore, it is not necessary to re-obtain the depth information through other acquisition means. In addition, in the subsequent step 108, it is necessary to merge the map points using the depth information of the feature points. By merging the depth information to reduce the amount of depth information, the calculation amount can be effectively reduced during the process of merging the map points, and the influence of sensor noise on the system accuracy can also be avoided.

[0093] Step 108: Based on the reprojection error corresponding to each pair of map points to be judged, perform a merging process on each pair of map points to be judged to obtain the corresponding merged map points.

[0094] Optionally, based on the reprojection error corresponding to each pair of map points to be judged, performing a merging process on each pair of map points to be judged to obtain the corresponding merged map points includes: in each pair of map points to be judged, according to the depth information and coordinate information of each feature point in the second feature point set corresponding to each map point to be judged, calculate the reprojection error of each feature point to obtain the first reprojection error of each feature point; determine the target feature point corresponding to each map point to be judged according to the median of the first reprojection errors of each feature point; according to the depth information and coordinate information of the target feature point corresponding to each map point to be judged, cross-calculate the reprojection error of each target feature point to obtain the second reprojection error of each target feature point; perform a merging process on each pair of map points to be judged whose sum of the two second reprojection errors is less than the preset error threshold to obtain a corresponding merged map point.

[0095] In each pair of map points to be judged, according to the depth information and coordinate information of each feature point in the second feature point set corresponding to each map point to be judged, calculate the reprojection error of each feature point to obtain the first reprojection error R1 of each feature point. For example, a pair of map points to be judged includes map point 1 and map point 2. The calculation formula of the reprojection error can refer to the aforementioned formula (3), that is

[0096] R1 = (5)

[0097] The second feature point set corresponding to map point 1 includes feature point 1, feature point 2, and feature point 3; the second feature point set corresponding to map point 2 includes feature point 4, feature point 5, and feature point 6. The reprojection errors of feature points 1 to 6 can be calculated respectively to obtain six first reprojection errors R1.

[0098] Determine the target feature point corresponding to each map point to be judged according to the median of the first reprojection errors of each feature point. For example, for map point 1, the R1 value of feature point 1 is the smallest, the R1 value of feature point 2 is slightly larger, and the R1 value of feature point 3 is the largest, then the target feature point corresponding to map point 1 is feature point 2.

[0099] According to the depth information and coordinate information of the target feature point corresponding to each map point to be judged, cross-calculate the reprojection error of each target feature point to obtain the second reprojection error of each target feature point.

[0100] For example, cross-calculate the reprojection error between the target feature point of map point 1 and map point 2 as the second reprojection error R2 of the target feature point of map point 1; cross-calculate the reprojection error between the target feature point of map point 2 and map point 1 as the second reprojection error R2' of the target feature point of map point 2.

[0101] Perform a merging process on each pair of map points to be determined whose sum of the two second reprojection errors is less than a preset error threshold, and obtain a corresponding merged map point.

[0102] The preset error threshold can be a preset reprojection error threshold r. For example, when R2 + R2' is less than r, map point 1 and map point 2 can be merged to obtain a corresponding merged map point; when R2 + R2' is greater than or equal to r, it can be determined that map point 1 and map point 2 do not need to be merged.

[0103] Figure 4 It is a schematic flow diagram of a method for merging map points provided by an embodiment of the present application.

[0104] As Figure 4 shown, in step 402, calculate the reprojection error of the associated feature points of map point 1.

[0105] In step 404, calculate the reprojection error of the associated feature points of map point 2.

[0106] In step 406, obtain the target feature point corresponding to map point 1.

[0107] In step 408, obtain the target feature point corresponding to map point 2.

[0108] In step 410, cross-calculate the reprojection error with map point 2 to obtain r_1.

[0109] In step 412, cross-calculate the reprojection error with map point 1 to obtain r_2.

[0110] In step 414, determine whether the sum of r_1 and r_2 is less than the preset error threshold.

[0111] If so, execute step 416.

[0112] In step 416, merge map point 1 and map point 2.

[0113] Optionally, after performing merging processing on each pair of map points to be judged according to the reprojection error corresponding to each pair of map points to obtain the corresponding merged map points, the method further includes: determining whether the number of feature points corresponding to each merged map point is greater than a preset number threshold; if the number of feature points corresponding to each merged map point is greater than the preset number threshold, determining the feature points corresponding to the merged map point as the feature points to be screened, and calculating the corresponding association score according to the first reprojection error, average descriptor distance, and the number of times the corresponding feature object is observed for each feature point to be screened; screening the feature points to be screened corresponding to the merged map point according to the association score corresponding to each feature point to be screened to obtain at least one representative feature point corresponding to the merged map point.

[0114] After step 108 is executed, it is also possible to analyze the number of feature points associated with the map point, determine whether it is greater than the set number threshold, and screen out the best set of feature points. The number of feature points associated with the map point can be the number of feature points included in the second set of feature points corresponding to the map point.

[0115] This solution calculates the association score of the feature points associated with the map point through the following formula (6), sorts according to the association score, and determines at least one representative feature point corresponding to the merged map point according to the sorting result.

[0116] (6)

[0117] where s is the association score, E rep is the reprojection error, a is the coefficient corresponding to the reprojection error, D avg is the average descriptor distance, b is the coefficient of the average descriptor distance, N ob is the number of times the feature point is observed, and c is the coefficient of the number of feature points.

[0118] In Figure 1In the illustrated embodiment, a plurality of first feature point sets are extracted from video data including multiple frames of images; each first feature point set includes a plurality of feature points corresponding to the same feature object; wherein, each feature point is extracted from a corresponding frame of image; a descriptor of each feature point is obtained, and based on the obtained descriptors, the plurality of first feature point sets are merged to obtain a plurality of second feature point sets; according to each second feature point set, a corresponding map point is generated, and at least one pair of map points to be judged is determined among the generated plurality of map points; the distance between each pair of map points to be judged is less than a first distance threshold; according to the reprojection error corresponding to each pair of map points to be judged, each pair of map points to be judged is merged to obtain the corresponding merged map point. Through the embodiments of the present application, when the number of observations of the feature object is relatively large, the plurality of first feature point sets can be merged to obtain a plurality of second feature point sets, reducing redundant data caused by the same feature object being recognized as corresponding to multiple different first feature point sets. Moreover, map points corresponding to each second feature point set can be generated, and each pair of map points to be judged with a relatively close distance can be merged to reduce redundant data caused by the same feature object being recognized as corresponding to multiple different map points, thereby improving the data processing efficiency in the process of using environmental information for data processing.

[0119] Figure 5 It is a schematic flowchart of a second data processing method provided by an embodiment of the present application.

[0120] Step 502, preprocess the input image data.

[0121] Step 504, extract and match image features.

[0122] Step 506, fuse similar feature points.

[0123] Step 508, generate map points.

[0124] Step 510, correct and update map points.

[0125] Step 512, fuse map points.

[0126] For the above embodiments of the data processing method, since it is basically similar to the foregoing embodiments of each data processing method, the description is relatively simple. For the relevant parts, refer to the partial descriptions of the foregoing embodiments of each data processing method.

[0127] It should be noted that for the data processing method provided by the embodiments of the present application, the execution subject can be a data processing device, or a control module in the data processing device for executing the data processing method. In the embodiments of the present application, taking the data processing device as an example of executing the data processing method, the data processing device provided by the embodiments of the present application is described.

[0128] Figure 6 It is a schematic block diagram of a data processing device provided by an embodiment of the present application.

[0129] As Figure 6 shown, the data processing device includes:

[0130] An extraction module 601, configured to extract a plurality of first feature point sets from video data including multiple frames of images; each first feature point set includes a plurality of feature points corresponding to the same feature object; wherein, each feature point is extracted from a corresponding frame of image;

[0131] A first merging module 602, configured to obtain a descriptor of each feature point, and perform a merging process on the plurality of first feature point sets according to the obtained descriptors to obtain a plurality of second feature point sets;

[0132] A first determination module 603, configured to generate a corresponding map point according to each second feature point set, and determine at least one pair of map points to be judged among the generated map points; the distance between each pair of map points to be judged is less than a first distance threshold;

[0133] A second merging module 604, configured to perform a merging process on each pair of map points to be judged according to the reprojection error corresponding to each pair of map points to be judged to obtain a corresponding merged map point.

[0134] Optionally, the first merging module 602 includes:

[0135] A calculation unit, configured to calculate the descriptor distance between any two feature points in each first feature point set according to the descriptor of each feature point;

[0136] A statistics unit, configured to statistically obtain a descriptor distance set corresponding to each feature point according to the calculated descriptor distance between any two feature points; the descriptor distance set includes the descriptor distances of other feature points belonging to the same first feature point set as the corresponding feature point;

[0137] A determination unit, configured to determine a target descriptor corresponding to each first feature point set according to the descriptor distance set corresponding to each feature point;

[0138] A merging unit, configured to perform a merging process on the plurality of first feature point sets according to the target descriptor corresponding to each first feature point set to obtain a plurality of second feature point sets.

[0139] Optionally, the determination unit is specifically configured to:

[0140] Perform an averaging process on the descriptor distances in the descriptor distance set corresponding to each feature point to obtain an average descriptor distance corresponding to each feature point;

[0141] In each set of first feature points, the descriptor of the feature point with the smallest corresponding average descriptor distance is determined as the target descriptor corresponding to each set of first feature points.

[0142] Optionally, the multiple sets of second feature points include at least one merged feature point set and at least one unmerged feature point set; the merging unit is specifically configured to:

[0143] Determine at least one pair of feature point sets to be judged in the multiple sets of first feature points;

[0144] Calculate the descriptor distance of each pair of feature point sets to be judged according to the target descriptor corresponding to each set of first feature points, and obtain the target descriptor distance;

[0145] Perform a merging process on each pair of feature point sets to be judged with a target descriptor distance less than the second distance threshold, and obtain a corresponding merged feature point set.

[0146] Optionally, the data processing device further includes:

[0147] An acquisition module, configured to acquire the target camera pose of the camera that acquires video data;

[0148] The first determination module 603 is specifically configured to:

[0149] According to the coordinate information and depth information of each feature point in each set of second feature points, determine the coordinate information and target depth information of the initial map point corresponding to each set of second feature points;

[0150] Correct the coordinate information of the initial map point according to the target camera pose and the target depth information, and obtain the target map point corresponding to each set of second feature points.

[0151] Optionally, the second merging module 604 is specifically configured to:

[0152] In each pair of map points to be judged, calculate the reprojection error of each feature point according to the depth information and coordinate information of each feature point in the set of second feature points corresponding to each map point to be judged, and obtain the first reprojection error of each feature point;

[0153] Determine the target feature point corresponding to each map point to be judged according to the median of the first reprojection errors of each feature point;

[0154] Cross-calculate the reprojection error of each target feature point according to the depth information and coordinate information of the target feature point corresponding to each map point to be judged, and obtain the second reprojection error of each target feature point;

[0155] For each pair of to-be-determined map points whose sum of two second reprojection errors is less than a preset error threshold, perform a merging process to obtain a corresponding merged map point.

[0156] Optionally, the data processing device further includes:

[0157] A second determination module, configured to determine whether the number of feature points corresponding to each merged map point is greater than a preset number threshold;

[0158] If the number of feature points corresponding to each merged map point is greater than the preset number threshold, then run a calculation module. The calculation module is configured to determine the feature points corresponding to the merged map point as to-be-screened feature points, and calculate corresponding association scores according to the first reprojection error, average descriptor distance, and the number of times the corresponding feature object is observed for each to-be-screened feature point.

[0159] A screening module, configured to screen the to-be-screened feature points corresponding to the merged map point according to the association scores corresponding to each to-be-screened feature point, to obtain at least one representative feature point corresponding to the merged map point.

[0160] In the embodiments of the present application, multiple first feature point sets are extracted from video data including multiple frames of images; each first feature point set includes multiple feature points corresponding to the same feature object; wherein, each feature point is extracted from a corresponding frame of image; obtain the descriptor of each feature point, and perform a merging process on the multiple first feature point sets according to the obtained descriptors to obtain multiple second feature point sets; generate a corresponding map point according to each second feature point set, and determine at least one pair of to-be-determined map points among the generated multiple map points; the distance between each pair of to-be-determined map points is less than a first distance threshold; perform a merging process on each pair of to-be-determined map points according to the reprojection error corresponding to each pair of to-be-determined map points to obtain a corresponding merged map point. Through the embodiments of the present application, it is possible to perform a merging process on multiple first feature point sets when the number of times the feature object is observed is relatively large, to obtain multiple second feature point sets, reduce redundant data caused by the same feature object being recognized as corresponding to multiple different first feature point sets, and also be able to generate map points corresponding to each second feature point set, and perform a merging process on each pair of to-be-determined map points with a relatively close distance to reduce redundant data caused by the same feature object being recognized as corresponding to multiple different map points, thereby improving the data processing efficiency in the process of using environmental information for data processing.

[0161] The data processing device in the embodiments of the present application can be a device, or a component, an integrated circuit, or a chip in a terminal. The device can be a mobile electronic device or a non-mobile electronic device. Exemplarily, the mobile electronic device can be a mobile phone, a tablet computer, a laptop computer, a handheld computer, a vehicle-mounted electronic device, a wearable device, an ultra-mobile personal computer (UMPC), a netbook, or a personal digital assistant (PDA), etc., and the non-mobile electronic device can be a server, a Network Attached Storage (NAS), a personal computer (PC), a television (TV), a teller machine, or a self-service machine, etc. The embodiments of the present application do not make specific limitations.

[0162] The data processing device in the embodiments of the present application can be a device with an operating system. The operating system can be the Android operating system, the iOS operating system, or other possible operating systems. The embodiments of the present application do not make specific limitations.

[0163] The data processing device provided in the embodiments of the present application can implement Figures 1 to 5 each process implemented by the method embodiments. To avoid repetition, details are not described herein again.

[0164] Figure 7 is a schematic block diagram of an electronic device provided in the embodiments of the present application. Optionally, as Figure 7 shown, the embodiments of the present application further provide an electronic device 700, including a processor 701, a memory 702, a program or instruction stored in the memory 702 and executable on the processor 701. When the program or instruction is executed by the processor 701, it implements each process of the above data processing method embodiments and can achieve the same technical effects. To avoid repetition, details are not described herein again.

[0165] It should be noted that the electronic device in the embodiments of the present application includes the above-mentioned mobile electronic devices and non-mobile electronic devices.

[0166] Figure 8 is a schematic hardware structure diagram of an electronic device provided in the embodiments of the present application.

[0167] The electronic device 800 includes, but is not limited to: a radio frequency unit 801, a network module 802, an audio output unit 803, an input unit 804, a sensor 805, a display unit 806, a user input unit 807, an interface unit 808, a memory 809, and a processor 810, etc.

[0168] Those skilled in the art can understand that the electronic device 800 may further include a power source (such as a battery) for powering each component. The power source can be logically connected to the processor 810 through a power management system, so as to implement functions such as management of charging, discharging, and power consumption management through the power management system. Figure 8 The structure of the electronic device shown does not limit the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here.

[0169] Among them, the processor 810 is configured to extract a plurality of first feature point sets from video data including multiple frames of images; each of the first feature point sets includes a plurality of feature points corresponding to the same feature object; wherein, each of the feature points is extracted from a corresponding frame of image;

[0170] Obtain a descriptor for each of the feature points, and perform a merging process on the plurality of first feature point sets according to the obtained descriptors to obtain a plurality of second feature point sets;

[0171] Generate a corresponding map point according to each of the second feature point sets, and determine at least one pair of map points to be judged among the generated map points; the distance between each pair of the map points to be judged is less than a first distance threshold;

[0172] Perform a merging process on each pair of the map points to be judged according to the reprojection error corresponding to each pair of the map points to be judged to obtain a corresponding merged map point.

[0173] In an embodiment of the present application, a plurality of first feature point sets are extracted from video data including multiple frames of images; each first feature point set includes multiple feature points corresponding to the same feature object; wherein each feature point is extracted from a corresponding frame of image; a descriptor of each feature point is obtained, and based on the obtained descriptors, the plurality of first feature point sets are merged to obtain a plurality of second feature point sets; according to each second feature point set, a corresponding map point is generated, and at least one pair of map points to be judged is determined among the generated multiple map points; the distance between each pair of map points to be judged is less than a first distance threshold; according to the reprojection error corresponding to each pair of map points to be judged, each pair of map points to be judged is merged to obtain a corresponding merged map point. Through the embodiment of the present application, when the number of observations of a feature object is relatively large, the plurality of first feature point sets can be merged to obtain a plurality of second feature point sets, reducing redundant data caused by the same feature object being recognized as corresponding to multiple different first feature point sets. It is also possible to generate map points corresponding to each second feature point set, and merge each pair of map points to be judged with a relatively short distance to reduce redundant data caused by the same feature object being recognized as corresponding to multiple different map points, thereby improving the data processing efficiency in the process of using environmental information for data processing.

[0174] Optionally, the processor 110 is further configured to: based on the obtained descriptors, merge the plurality of first feature point sets to obtain a plurality of second feature point sets, including:

[0175] According to the descriptors of each feature point, calculate the descriptor distance between any two feature points in each first feature point set;

[0176] Based on the calculated descriptor distances between any two feature points, count the descriptor distance set corresponding to each feature point; the descriptor distance set includes the descriptor distances of other feature points that belong to the same first feature point set as the corresponding feature point;

[0177] Based on the descriptor distance set corresponding to each feature point, determine the target descriptor corresponding to each first feature point set;

[0178] Based on the target descriptor corresponding to each first feature point set, merge the plurality of first feature point sets to obtain a plurality of second feature point sets.

[0179] Optionally, the processor 110 is further configured to: based on the descriptor distance set corresponding to each feature point, determine the target descriptor corresponding to each first feature point set, including:

[0180] Perform an averaging process on the descriptor distances in the descriptor distance set corresponding to each feature point to obtain the average descriptor distance corresponding to each feature point;

[0181] In each set of first feature points, determine the descriptor of the feature point with the smallest corresponding average descriptor distance as the target descriptor corresponding to each set of first feature points.

[0182] Optionally, the multiple sets of second feature points include at least one merged feature point set and at least one unmerged feature point set; the processor 110 is further configured to: perform a merging process on the multiple sets of first feature points according to the target descriptor corresponding to each set of first feature points, to obtain multiple sets of second feature points, including:

[0183] Determine at least one pair of feature point sets to be judged in the multiple sets of first feature points;

[0184] Calculate the descriptor distance of each pair of feature point sets to be judged according to the target descriptor corresponding to each set of first feature points, to obtain the target descriptor distance;

[0185] Perform a merging process on each pair of feature point sets to be judged with a target descriptor distance less than the second distance threshold, to obtain a corresponding merged feature point set.

[0186] Optionally, the processor 110 is further configured to:

[0187] Before generating a corresponding map point according to each set of second feature points, obtain the target camera pose of the camera that collects video data;

[0188] Generate a corresponding map point according to each set of second feature points, including:

[0189] Determine the coordinate information and target depth information of the initial map point corresponding to each set of second feature points according to the coordinate information and depth information of each feature point in each set of second feature points;

[0190] Correct the coordinate information of the initial map point according to the target camera pose and the target depth information, to obtain the target map point corresponding to each set of second feature points.

[0191] Optionally, the processor 110 is further configured to: perform a merging process on each pair of map points to be judged according to the reprojection error corresponding to each pair of map points to be judged, to obtain the corresponding merged map points, including:

[0192] In each pair of map points to be judged, calculate the reprojection error of each feature point according to the depth information and coordinate information of each feature point in the second feature point set corresponding to each map point to be judged, to obtain the first reprojection error of each feature point;

[0193] Determine the target feature point corresponding to each map point to be judged according to the median of the first reprojection errors of each feature point;

[0194] According to the depth information and coordinate information of the target feature points corresponding to each map point to be determined, the reprojection error of each target feature point is cross-calculated to obtain the second reprojection error of each target feature point;

[0195] For each pair of map points to be determined whose sum of the two second reprojection errors is less than a preset error threshold, a merging process is performed to obtain a corresponding merged map point.

[0196] Optionally, the processor 110 is further configured to: after performing a merging process on each pair of map points to be determined according to the reprojection error corresponding to each pair of map points to be determined to obtain the corresponding merged map point, further include:

[0197] Determine whether the number of feature points corresponding to each merged map point is greater than a preset number threshold;

[0198] If the number of feature points corresponding to each merged map point is greater than the preset number threshold, the feature points corresponding to the merged map point are determined as feature points to be filtered, and the corresponding association scores are calculated according to the first reprojection error, average descriptor distance, and the number of times the corresponding feature object is observed of each feature point to be filtered;

[0199] According to the association scores corresponding to each feature point to be filtered, the feature points to be filtered corresponding to the merged map point are filtered to obtain at least one representative feature point corresponding to the merged map point.

[0200] Through the embodiments of the present application, it is possible to merge the first set of feature points, merge the map points to be determined, reduce redundant data from the environmental information collected by the camera, improve the data processing efficiency, and also reduce the number of feature points corresponding to each map point.

[0201] It should be understood that in the embodiments of the present application, the input unit 804 may include a Graphics Processing Unit (GPU) 8041 and a microphone 8042. The graphics processor 8041 processes the image data of static pictures or videos obtained by an image capture device (such as a camera) in the video capture mode or the image capture mode. The display unit 806 may include a display panel 8061, and the display panel 8061 may be configured in the form of a liquid crystal display, an organic light-emitting diode, etc. The user input unit 807 includes a touch panel 8071 and other input devices 8072. The touch panel 8071 is also known as a touch screen. The touch panel 8071 may include two parts: a touch detection device and a touch controller. The other input devices 8072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be elaborated here. The memory 809 can be used to store software programs and various data, including but not limited to application programs and operating systems. The processor 810 may integrate an application processor and a modem processor. Among them, the application processor mainly processes the operating system, user interface, application programs, etc., and the modem processor mainly processes wireless communication. It can be understood that the above-mentioned modem processor may not be integrated into the processor 810.

[0202] The embodiments of the present application further provide a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, it implements each process of the above-mentioned data processing method embodiments and can achieve the same technical effects. To avoid repetition, it will not be elaborated here.

[0203] Among them, the processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes a computer-readable storage medium, such as a computer Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disc, etc.

[0204] The embodiments of the present application further provide a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run a program or instruction to implement each process of the above-mentioned data processing method embodiments and can achieve the same technical effects. To avoid repetition, it will not be elaborated here.

[0205] It should be understood that the chip mentioned in the embodiments of the present application may also be referred to as a system-on-chip, system chip, chip system, or system-on-chip, etc.

[0206] It should be noted that in this text, the term "including", "comprising" or any other variant thereof is intended to cover non-exclusive inclusion, such that a process, method, article or device comprising a series of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "comprising an..." does not exclude the presence of additional identical elements in the process, method, article or device comprising that element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in a reverse order according to the functions involved. For example, the described methods may be performed in an order different from that described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0207] Through the description of the above embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus a necessary general hardware platform. Of course, it can also be implemented by hardware, but in many cases the former is a better implementation. Based on this understanding, the technical solution of the present application, in essence or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disc) and includes several instructions to enable a terminal (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in various embodiments of the present application.

[0208] The embodiments of the present application have been described above with reference to the accompanying drawings. However, the present application is not limited to the above specific embodiments. The above specific embodiments are merely illustrative and not restrictive. Under the inspiration of the present application, those of ordinary skill in the art can also make many forms without departing from the purpose of the present application and the scope protected by the claims, and all of them fall within the protection scope of the present application.

Claims

1. A data processing method, characterized in that, Including: Extracting a plurality of first feature point sets from video data including multiple frames of images; each of the first feature point sets includes a plurality of feature points corresponding to the same feature object; wherein, each of the feature points is extracted from a corresponding frame of image; Obtaining a descriptor for each of the feature points, and performing a merging process on the plurality of first feature point sets according to the obtained descriptors to obtain a plurality of second feature point sets; Generating a corresponding map point for each of the second feature point sets, and determining at least one pair of map points to be judged among the generated plurality of map points; the distance between each pair of the map points to be judged is less than a first distance threshold; Performing a merging process on each pair of the map points to be judged according to the reprojection error corresponding to each pair of the map points to be judged to obtain a corresponding merged map point; The performing a merging process on the plurality of first feature point sets according to the obtained descriptors to obtain a plurality of second feature point sets includes: Calculating the descriptor distance between any two feature points in each of the first feature point sets according to the descriptor of each of the feature points; According to the calculated descriptor distances between any two feature points, statistically obtaining a descriptor distance set corresponding to each of the feature points; the descriptor distance set includes the descriptor distances of other feature points that belong to the same first feature point set as the corresponding feature point; Determining a target descriptor corresponding to each of the first feature point sets according to the descriptor distance set corresponding to each of the feature points; Performing a merging process on the plurality of first feature point sets according to the target descriptor corresponding to each of the first feature point sets to obtain a plurality of second feature point sets.

2. The method according to claim 1, characterized in that The determining a target descriptor corresponding to each of the first feature point sets according to the descriptor distance set corresponding to each of the feature points includes: Performing an averaging process on the descriptor distances in the descriptor distance set corresponding to each of the feature points to obtain an average descriptor distance corresponding to each of the feature points; In each of the first feature point sets, determining the descriptor of the feature point with the smallest average descriptor distance as the target descriptor corresponding to each of the first feature point sets.

3. The method according to claim 1, characterized in that The plurality of second feature point sets include at least one merged feature point set and at least one non-merged feature point set; the performing a merging process on the plurality of first feature point sets according to the target descriptor corresponding to each of the first feature point sets to obtain a plurality of second feature point sets includes: Determining at least one pair of feature point sets to be judged among the plurality of first feature point sets; Calculating the descriptor distance between each pair of the feature point sets to be judged according to the target descriptor corresponding to each of the first feature point sets to obtain a target descriptor distance; Performing a merging process on each pair of the feature point sets to be judged with a target descriptor distance less than a second distance threshold to obtain a corresponding merged feature point set.

4. The method according to claim 1, wherein Before generating a corresponding map point for each of the second feature point sets, further including: Obtaining the target camera pose of the camera that collected the video data; Generating a corresponding map point according to each of the second feature point sets includes: Determining coordinate information and target depth information of an initial map point corresponding to each of the second feature point sets according to the coordinate information and depth information of each feature point in each of the second feature point sets; Correcting the coordinate information of the initial map point according to the target camera pose and the target depth information to obtain a target map point corresponding to each of the second feature point sets.

5. The method according to claim 1, characterized in that Performing a merging process on each pair of the to-be-judged map points according to the reprojection error corresponding to each pair of the to-be-judged map points to obtain a corresponding merged map point, including: In each pair of the to-be-judged map points, calculating the reprojection error of each feature point according to the depth information and coordinate information of each feature point in the second feature point set corresponding to each to-be-judged map point to obtain a first reprojection error of each feature point; Determining a target feature point corresponding to each to-be-judged map point according to the median of the first reprojection errors of each feature point; Cross-calculating the reprojection error of each target feature point according to the depth information and coordinate information of the target feature point corresponding to each to-be-judged map point to obtain a second reprojection error of each target feature point; Performing a merging process on each pair of the to-be-judged map points for which the sum of the two second reprojection errors is less than a preset error threshold to obtain a corresponding merged map point.

6. The method according to claim 1, wherein After performing a merging process on each pair of the to-be-judged map points according to the reprojection error corresponding to each pair of the to-be-judged map points to obtain a corresponding merged map point, it further includes: Determining whether the number of feature points corresponding to each merged map point is greater than a preset number threshold; If the number of feature points corresponding to each merged map point is greater than the preset number threshold, determining the feature points corresponding to the merged map point as to-be-screened feature points, and calculating a corresponding association score according to the first reprojection error, average descriptor distance, and the number of times the corresponding feature object is observed of each to-be-screened feature point; Screening the to-be-screened feature points corresponding to the merged map point according to the association score corresponding to each to-be-screened feature point to obtain at least one representative feature point corresponding to the merged map point.

7. A data processing device, characterized in that, Including: An extraction module, configured to extract a plurality of first feature point sets from video data including multiple frames of images; each of the first feature point sets includes a plurality of feature points corresponding to the same feature object; wherein, each feature point is extracted from a corresponding frame of image; A first merging module, configured to obtain a descriptor of each feature point, and perform a merging process on the plurality of first feature point sets according to the obtained descriptors to obtain a plurality of second feature point sets; A first determination module, configured to generate a corresponding map point according to each of the second feature point sets, and determine at least one pair of to-be-judged map points among the generated multiple map points; the distance between each pair of the to-be-judged map points is less than a first distance threshold; A second merging module, configured to perform a merging process on each pair of the to-be-determined map points according to the reprojection error corresponding to each pair of the to-be-determined map points, so as to obtain corresponding merged map points; The first merging module includes: A calculation unit, configured to calculate the descriptor distance between any two feature points in each of the first feature point sets according to the descriptors of each of the feature points; A statistics unit, configured to statistically obtain a descriptor distance set corresponding to each of the feature points according to the calculated descriptor distances between any two feature points; the descriptor distance set includes the descriptor distances of other feature points that belong to the same first feature point set as the corresponding feature point; A determination unit, configured to determine a target descriptor corresponding to each of the first feature point sets according to the descriptor distance set corresponding to each of the feature points; A merging unit, configured to perform a merging process on the multiple first feature point sets according to the target descriptor corresponding to each of the first feature point sets, so as to obtain multiple second feature point sets.

8. The device according to claim 7, characterized in that, The determination unit is specifically configured to: Perform an averaging process on the descriptor distances in the descriptor distance set corresponding to each of the feature points, so as to obtain an average descriptor distance corresponding to each of the feature points; In each of the first feature point sets, determine the descriptor of the feature point with the smallest corresponding average descriptor distance as the target descriptor corresponding to each of the first feature point sets.

9. The device according to claim 7, wherein The multiple second feature point sets include at least one merged feature point set and at least one non-merged feature point set; the merging unit is specifically configured to: Determine at least one pair of to-be-determined feature point sets in the multiple first feature point sets; Calculate the descriptor distance between each pair of the to-be-determined feature point sets according to the target descriptor corresponding to each of the first feature point sets, so as to obtain a target descriptor distance; Perform a merging process on each pair of the to-be-determined feature point sets with the target descriptor distance less than a second distance threshold, so as to obtain a corresponding merged feature point set.

10. The device according to claim 7, characterized in that, It further includes: An acquisition module, configured to acquire the target camera pose of the camera that acquires the video data; The first determination module is specifically configured to: Determine the coordinate information and the target depth information of the initial map point corresponding to each of the second feature point sets according to the coordinate information and the depth information of each of the feature points in each of the second feature point sets; Correct the coordinate information of the initial map point according to the target camera pose and the target depth information, so as to obtain the target map point corresponding to each of the second feature point sets.

11. The device according to claim 7, characterized in that The second merging module is specifically configured to: In each pair of the to-be-determined map points, calculate the reprojection error of each of the feature points according to the depth information and the coordinate information of each of the feature points in the second feature point set corresponding to each of the to-be-determined map points, so as to obtain a first reprojection error of each of the feature points; Determine the target feature point corresponding to each of the to-be-determined map points according to the median of the first reprojection errors of each of the feature points; According to the depth information and coordinate information of the target feature points corresponding to each map point to be judged, the reprojection error of each target feature point is cross-calculated to obtain the second reprojection error of each target feature point; For each pair of map points to be judged whose sum of the two second reprojection errors is less than a preset error threshold, a merging process is performed to obtain a corresponding merged map point.

12. The device according to claim 7, characterized in that, It further includes: A second determination module, configured to determine whether the number of feature points corresponding to each merged map point is greater than a preset number threshold; If the number of feature points corresponding to each merged map point is greater than the preset number threshold, the calculation module is run. The calculation module is configured to determine the feature points corresponding to the merged map point as the feature points to be screened, and calculate the corresponding association scores according to the first reprojection error, average descriptor distance, and the number of times the corresponding feature object is observed for each feature point to be screened; A screening module, configured to screen the feature points to be screened corresponding to the merged map point according to the association scores corresponding to each feature point to be screened, to obtain at least one representative feature point corresponding to the merged map point.

Images