Visual data compression method and device, equipment and storage medium

Abstract

The invention discloses a visual data compression method and device, equipment and a storage medium, and relates to the technical field of computers, and the visual data compression method comprises the steps: carrying out the clustering according to the three-dimensional space distribution of each visual feature point, and obtaining a visual noise point, at least one feature point cluster, and a visual core point of the feature point cluster; compressing the plurality of visual feature points according to the visual noisy points and the visual core points of the feature point clusters to obtain a plurality of compressed feature points and a plurality of reserved feature points; and obtaining a data compression result of the plurality of visual feature points according to the plurality of compressed feature points and the plurality of reserved feature points. Through the mode, the number of feature points is greatly reduced, the scale of optimization problems corresponding to subsequent visual tasks is reduced, the solving speed of the optimization problems is improved, and meanwhile it is guaranteed that the optimization problems have enough generality and problem solving reliability and precision.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a visual data compression method, device, equipment and storage medium. Background Art

[0002] Visual localization is an important problem in computer vision. Its core lies in estimating the position and posture of the camera using visual feature points. Theoretically, a small number of visual feature point pairs can solve the relative position relationship between two cameras. However, in practical applications, due to the existence of observation noise, it is often difficult to obtain stable and accurate results simply by relying on a small number of feature points. Therefore, optimization and filtering methods have become the two mainstream solutions to solve the problem of visual localization. The filtering method is fast, but it generally lags behind the optimization method in terms of accuracy and robustness; although the optimization method performs well in terms of accuracy and robustness, its large amount of calculation leads to a slow solution process, which limits its ability to be applied in real time.

[0003] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are prior art. Summary of the invention

[0004] The main purpose of this application is to provide a visual data compression method, device, equipment and storage medium, aiming to solve the technical problems of low computational efficiency, insufficient robustness and a large amount of redundant information affecting positioning accuracy of the existing visual positioning algorithm when processing data sets.

[0005] To achieve the above purpose, the present application proposes a visual data compression method, the method comprising:

[0006] Clustering is performed according to the three-dimensional spatial distribution of each visual feature point to obtain a visual noise point, at least one feature point cluster, and a visual core point of the feature point cluster;

[0007] Compressing multiple visual feature points according to the visual noise points and the visual core points of each feature point cluster to obtain multiple compressed feature points and multiple retained feature points;

[0008] Data compression results of multiple visual feature points are obtained according to the multiple compressed feature points and the multiple retained feature points.

[0009] In one embodiment, before the step of clustering the three-dimensional spatial distribution of each visual feature point to obtain visual noise points, at least one feature point cluster and the visual core point of the feature point cluster, the step further includes:

[0010] Perform depth information detection on multiple two-dimensional feature points observed visually;

[0011] When each two-dimensional feature point has depth information, obtaining the number of continuous tracking times of each two-dimensional feature point;

[0012] When the number of continuous tracking of each two-dimensional feature point reaches a preset number, each two-dimensional feature point is used as a visual feature point to be compressed.

[0013] In one embodiment, after the step of performing depth information detection on the plurality of visually observed two-dimensional feature points, the method further includes:

[0014] When there are two-dimensional feature points but no depth information, obtaining a target image pair including the two-dimensional feature points and a corresponding relationship between the feature points;

[0015] constructing a projection matrix based on image parameters of the image pair;

[0016] When the corresponding relationship of the feature points satisfies the geometric constraint condition, a triangulation calculation is performed according to the projection matrix to determine the depth information of the two-dimensional feature points.

[0017] In one embodiment, the step of clustering the three-dimensional spatial distribution of each visual feature point to obtain visual noise points, at least one feature point cluster, and a visual core point of the feature point cluster comprises:

[0018] Determine the three-dimensional spatial distribution of each visual feature point according to the depth information of each visual feature point;

[0019] Clustering is performed according to a target clustering algorithm, a target search radius, and a three-dimensional spatial distribution of each visual feature point to determine at least one feature point cluster and a visual noise point;

[0020] A visual core point is determined from a plurality of visual feature points in the feature point cluster according to the cluster internal density of the feature point cluster.

[0021] In one embodiment, the step of compressing multiple visual feature points according to the visual noise points and the visual core points of the feature point clusters to obtain multiple compressed feature points and multiple retained feature points includes:

[0022] Retaining feature points of the visual noise points and the visual core points of each feature point cluster, determining a plurality of first feature points, and determining a plurality of second feature points based on the plurality of first feature points and the plurality of visual feature points;

[0023] When the number of retained feature points corresponding to the plurality of first feature points is less than a preset number, performing a difference calculation based on the number of retained feature points and the preset number to determine the number of feature points to be supplemented;

[0024] Supplement feature points in the plurality of second feature points according to the feature point stability ranking information of each feature point cluster and the number of feature points to be supplemented, so as to obtain a plurality of third feature points and a plurality of compressed feature points;

[0025] A plurality of retained feature points are obtained according to the plurality of first feature points and the plurality of third feature points.

[0026] In one embodiment, before the step of supplementing feature points from a plurality of second feature points according to the feature point stability ranking information of each feature point cluster and the number of feature points to be supplemented to obtain a plurality of third feature points and a plurality of compressed feature points, the step further includes:

[0027] Get the tracking times of each visual feature point;

[0028] Sorting the tracking times of each visual feature point in the feature point cluster to determine the stability ranking order of each visual feature point in the feature point cluster;

[0029] According to the stability ranking order of each visual feature point in the feature point cluster, feature point stability ranking information of the feature point cluster is determined.

[0030] In one embodiment, after the step of obtaining data compression results of multiple visual feature points according to the multiple compressed feature points and the multiple retained feature points, the method further includes:

[0031] Perform reprojection calculation on each retained feature point to determine the reprojected two-dimensional coordinates of each retained feature point;

[0032] The error is calculated based on the reprojected two-dimensional coordinates of each retained feature point and the original two-dimensional coordinates of each retained feature point to determine the reprojection error of each retained feature point;

[0033] When the reprojection error of each retained feature point is less than the error threshold, data of each retained feature point is retained.

[0034] In addition, to achieve the above purpose, the present application also proposes a visual data compression device, the visual data compression device comprising: a clustering module, used to cluster the three-dimensional spatial distribution of each visual feature point to obtain a visual noise point, at least one feature point cluster and a visual core point of the feature point cluster;

[0035] A compression module, used for compressing a plurality of visual feature points according to the visual noise points and the visual core points of the feature point clusters to obtain a plurality of compressed feature points and a plurality of retained feature points;

[0036] The processing module is used to obtain data compression results of multiple visual feature points according to multiple compressed feature points and multiple retained feature points.

[0037] In addition, to achieve the above-mentioned purpose, the present application also proposes a visual data compression device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the visual data compression method described above.

[0038] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the visual data compression method described above are implemented.

[0039] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the visual data compression method described above are implemented.

[0040] The present application provides a visual data compression method, which performs clustering according to the three-dimensional spatial distribution of each visual feature point to obtain visual noise points, at least one feature point cluster and the visual core points of the feature point clusters; compresses multiple visual feature points according to the visual noise points and the visual core points of each feature point cluster to obtain multiple compressed feature points and multiple retained feature points; and obtains data compression results of multiple visual feature points according to the multiple compressed feature points and the multiple retained feature points. In the above manner, clustering is performed according to the three-dimensional spatial distribution of each visual feature point to obtain a clustering result, and multiple visual feature points are compressed according to the clustering result. The compression of multiple visual feature points is completed by obtaining multiple compressed feature points and multiple retained feature points, which greatly reduces the number of feature points, reduces the scale of optimization problems corresponding to subsequent visual tasks, and improves the speed of solving optimization problems. At the same time, it ensures that the optimization problem has sufficient generalization, the reliability of problem solving and the accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0042] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0043] Figure 1 A flowchart of the first embodiment of the visual data compression method of the present application is provided;

[0044] Figure 2A clustering diagram provided in Example 1 of the present application;

[0045] Figure 3 A flowchart diagram of Embodiment 2 of the visual data compression method of the present application is provided;

[0046] Figure 4 A flowchart of Embodiment 3 of the visual data compression method of the present application is provided;

[0047] Figure 5 A brief flowchart of the third embodiment of the present invention is provided.

[0048] Figure 6 This is a schematic diagram of the module structure of the visual data compression device according to an embodiment of the present application;

[0049] Figure 7 Schematic diagram of the device structure of the hardware operating environment involved in the visual data compression method in the embodiment of the present application.

[0050] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0051] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0052] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0053] The main solution of the embodiment of the present application is: clustering according to the three-dimensional spatial distribution of each visual feature point to obtain visual noise points, at least one feature point cluster and the visual core point of the feature point cluster; compressing multiple visual feature points according to the visual noise points and the visual core points of each feature point cluster to obtain multiple compressed feature points and multiple retained feature points; obtaining data compression results of multiple visual feature points based on the multiple compressed feature points and the multiple retained feature points.

[0054] The core of visual positioning problem construction is visual feature points, which can be understood as visual observation. Theoretically, eight pairs of visual feature points can solve the relative position relationship between two cameras. However, in order to deal with observation noise, optimization and filtering methods came into being. These are also the two mainstream solutions of visual positioning algorithms. The filtering method is fast, but generally lags behind the optimization method in terms of accuracy and robustness; the optimization method is subject to large computing power consumption, and has higher hardware costs than the filtering method on embedded platforms. The core of computing power consumption lies in the large scale of the constructed optimization problem, which leads to a slow solution process.

[0055] This application uses a geometric space clustering algorithm to cluster the distribution of feature points to ensure that the feature points are distributed as evenly as possible in the observation space. On this basis, the observation quality of the feature points is sorted and screened based on prior information to control the number of feature points. The feature points that successfully pass the screening are used as visual observations for optimization, thereby achieving controllable scale of the optimization problem. In most cases, spatial clustering and prior screening can ensure that the quality is high while the number of feature point observations is limited, and the optimization problem is still sufficiently general.

[0056] The present application clusters the three-dimensional spatial distribution of each visual feature point to obtain a clustering result, and compresses multiple visual feature points through the clustering result, and completes the compression of multiple visual feature points through the obtained multiple compressed feature points and multiple retained feature points, thereby greatly reducing the number of feature points, reducing the scale of optimization problems corresponding to subsequent visual tasks, and improving the speed of solving the optimization problem, while ensuring that the optimization problem has sufficient generalization, reliability and accuracy of problem solving.

[0057] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, a visual data compression device, etc. The following takes the visual data compression device as an example to illustrate this embodiment and the following embodiments.

[0058] Based on this, the present application embodiment provides a visual data compression method, referring to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the visual data compression method of the present application.

[0059] In this embodiment, the visual data compression method includes steps S10 to S30:

[0060] Step S10 , clustering is performed according to the three-dimensional spatial distribution of each visual feature point to obtain visual noise points, at least one feature point cluster and a visual core point of the feature point cluster.

[0061] It should be noted that visual feature points refer to two-dimensional feature points that are detected in visual tasks and have depth information and are tracked continuously for a preset number of times. Three-dimensional spatial distribution refers to the position and arrangement of visual feature points in a three-dimensional coordinate system to represent the specific layout of visual feature points in three-dimensional space. In this embodiment, two-dimensional feature points refer to points with significant visual characteristics identified in an image by a feature detection algorithm (such as SIFT, SURF, ORB, etc.).

[0062] It can be understood that visual noise refers to visual feature points that do not belong to any feature point cluster. The geometric position of the visual noise is isolated and cannot be classified into any existing feature point cluster. Therefore, it is marked as a noise point by the clustering algorithm. In visual positioning, visual noise is needed to cover the observed scene as much as possible; feature point clustering refers to a cluster containing multiple visual feature points with similar three-dimensional spatial distribution, revealing the structural information in the scene; visual core point refers to the most representative visual feature point in the feature point cluster, which is usually the most stable and reliable point in the cluster and is used for subsequent processing and analysis.

[0063] In a specific implementation, clustering is performed according to the three-dimensional spatial distribution of multiple visual feature points through a clustering algorithm (such as DBSCAN, K-means, etc.), so that visual feature points with close geometric space distances are divided into the same feature point cluster, and at least one visual core point in the feature point cluster is determined by the internal density of the cluster. After obtaining the feature point clusters corresponding to all visual feature points, the visual feature points independent of the feature point clusters are regarded as visual noise points. In this embodiment, the DBSCAN clustering algorithm is used for feature point clustering.

[0064] In a feasible implementation, step S10 may include steps A11 to A13:

[0065] Step A11, determining the three-dimensional spatial distribution of each visual feature point according to the depth information of each visual feature point.

[0066] It should be noted that the depth information of a visual feature point refers to the distance from the camera that observes the visual feature point to the visual feature point in three-dimensional space. By collecting the depth information of all visual feature points, a corresponding three-dimensional point cloud can be formed, which can further clarify the position and arrangement of each visual feature point in the three-dimensional coordinate system and obtain the three-dimensional spatial distribution of each visual feature point.

[0067] Step A12, clustering is performed according to the target clustering algorithm, the target search radius, and the three-dimensional spatial distribution of each visual feature point to determine at least one feature point cluster and a visual noise point.

[0068] It should be noted that in this embodiment, the target clustering algorithm adopts the DBSCAN clustering algorithm, and other clustering algorithms may also be adopted, which is not limited in this embodiment; the target search radius refers to the search radius of the target clustering algorithm. Based on the minimum number of points in the defined field, the target search radius, and the three-dimensional spatial distribution of each visual feature point, clustering is performed by the target clustering algorithm to obtain at least one feature point cluster and visual noise points that do not belong to any feature point cluster.

[0069] Step A13: determining a visual core point from a plurality of visual feature points in the feature point cluster according to the cluster internal density of the feature point cluster.

[0070] It should be noted that for each feature point cluster, the internal density of each visual feature point in its domain is calculated, and the density can be measured by the number of points in the domain. In this embodiment, visual feature points whose internal density in the cluster is greater than a preset density can be selected as core points, or visual core points can be determined from multiple visual feature points in the feature point cluster by other methods, which is not limited in this embodiment. In this embodiment, each feature point cluster can have one or more visual core points. Figure 2 As shown, after clustering multiple visual feature points using the target search radius and the DBSCAN clustering algorithm, a feature point cluster and visual noise points 4, 5, and 8 are obtained. There are multiple visual core points in the feature point cluster, which are 4, 6, and 7 respectively.

[0071] Step S20 , compressing a plurality of visual feature points according to the visual noise points and the visual core points of each feature point cluster to obtain a plurality of compressed feature points and a plurality of retained feature points.

[0072] It should be noted that compressed feature points refer to compressed visual feature points, which will not flow into the subsequent optimization process; retained feature points refer to selected visual feature points, which will be used in the subsequent optimization process. In this embodiment, priority is given to ensuring that the visual core points of each feature point cluster and the visual noise points outside the feature point cluster are selected, thereby obtaining multiple retained feature points. Among the multiple visual feature points, the visual feature points except the retained feature points are all compressed feature points. During the compression process, a preset number needs to be set to control the overall number of multiple retained feature points.

[0073] In this embodiment, a flag variable is set for each compressed feature point to indicate that the feature point is a compressed point. The compressed feature point will not be discarded but will be placed in a reserve pool.

[0074] Step S30, obtaining data compression results of a plurality of visual feature points according to the plurality of compressed feature points and the plurality of retained feature points.

[0075] It should be noted that after obtaining a plurality of compressed feature points and a plurality of retained feature points, feature point compression of a plurality of visual feature points is completed, thereby obtaining data compression results of a plurality of visual feature points.

[0076] In a feasible implementation manner, after step S30, steps B11 to B13 may also be included:

[0077] Step B11, performing reprojection calculation on each retained feature point to determine the reprojected two-dimensional coordinates of each retained feature point.

[0078] It should be noted that the projection formula is used to perform reprojection calculation on each retained feature point, and the three-dimensional coordinates of each retained feature point are projected back to the two-dimensional image plane, thereby obtaining the reprojection coordinates of each retained feature point on the two-dimensional image plane. In this embodiment, the reprojected two-dimensional coordinates refer to the reprojection coordinates of the retained feature point on the two-dimensional image plane.

[0079] Step B12, performing error calculation based on the reprojected two-dimensional coordinates of each retained feature point and the original two-dimensional coordinates of each retained feature point to determine the reprojection error of each retained feature point.

[0080] It should be noted that the original two-dimensional coordinates refer to the coordinates on the two-dimensional image plane corresponding to each retained feature point originally detected. The reprojection error of each retained feature point is calculated by comparing the reprojected two-dimensional coordinates of each retained feature point with the original two-dimensional coordinates.

[0081] Step B13: when the reprojection error of each retained feature point is less than the error threshold, data of each retained feature point is retained.

[0082] It should be noted that the error threshold is set according to application requirements and system accuracy requirements. For each retained feature point, it is determined whether its reprojection error is less than the error threshold. If the reprojection error is less than the error threshold, the retained feature point is deemed to be reliable and can be used in subsequent optimization processes, and the retained feature point is retained; if the reprojection error is greater than or equal to the error threshold, the retained feature point is deemed to be unreliable, marked as an outlier, and deleted from the point manager.

[0083] It can be understood that in addition to the above-mentioned method of determining whether the retained feature points are reliable through reprojection errors, it is also possible to determine through the depth value of the retained feature points. When the depth value of the retained feature points is not negative, the retained feature points are deemed to be reliable and can be used in subsequent optimization processes, and the retained feature points are retained; when the depth value of the retained feature points is negative, the retained feature points are retained, and the retained feature points are deemed to be unreliable, marked as external points, and deleted from the point manager.

[0084] The present embodiment provides a visual data compression method, which performs clustering according to the three-dimensional spatial distribution of each visual feature point to obtain visual noise points, at least one feature point cluster, and visual core points of the feature point clusters; compresses multiple visual feature points according to the visual noise points and the visual core points of each feature point cluster to obtain multiple compressed feature points and multiple retained feature points; and obtains data compression results of multiple visual feature points according to the multiple compressed feature points and the multiple retained feature points. In the above manner, clustering is performed according to the three-dimensional spatial distribution of each visual feature point to obtain a clustering result, and multiple visual feature points are compressed according to the clustering result. The compression of multiple visual feature points is completed by obtaining multiple compressed feature points and multiple retained feature points, which greatly reduces the number of feature points, reduces the scale of the optimization problem corresponding to the subsequent visual task, and improves the speed of solving the optimization problem. At the same time, it ensures that the optimization problem has sufficient generalization, the reliability of problem solving, and the accuracy.

[0085] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction, and will not be repeated in the following. Figure 3 Before step S10, the visual data compression method further includes steps S01 to S03:

[0086] Step S01: performing depth information detection on a plurality of two-dimensional feature points observed visually.

[0087] It should be noted that all visually observed (ie, detected and tracked) two-dimensional feature points are obtained, and it is determined one by one whether each two-dimensional feature point has depth information.

[0088] In a feasible implementation manner, after step S01, steps C11 to C13 may also be included:

[0089] Step C11: when there are two-dimensional feature points but no depth information, obtaining a target image pair including the two-dimensional feature points and a corresponding relationship between the feature points.

[0090] Step C12: constructing a projection matrix according to the image parameters of the image pair.

[0091] Step C13, when the corresponding relationship of the feature points meets the geometric constraint condition, perform triangulation calculation according to the projection matrix to determine the depth information of the two-dimensional feature points.

[0092] It should be noted that when there is a two-dimensional feature point but no depth information, it means that the two-dimensional feature point is a newly added two-dimensional feature point. At this time, triangulation is required to calculate the depth value corresponding to the two-dimensional feature point so that all two-dimensional feature points have corresponding depth information.

[0093] It can be understood that the specific process of triangulation is: obtain a target image pair of two or more different perspectives containing a two-dimensional feature point without depth information, and find the feature points under other perspectives corresponding to the two-dimensional feature point in the target image pair, and establish a feature point correspondence relationship. Obtain the internal parameter matrix of each image, including focal length, principal point offset, etc., and obtain the external parameters of each image, including the rotation matrix and translation vector, and combine the internal parameters and external parameters to construct the projection matrix of each image. Detect whether the feature point correspondence relationship satisfies the geometric constraints, such as the polar line constraint. When the feature point correspondence relationship satisfies the geometric constraints, a triangulation method is used to calculate the three-dimensional coordinates of the two-dimensional feature points from the two-dimensional coordinates of multiple perspectives based on the projection matrix, and extract the depth information from the calculated three-dimensional coordinates. In this embodiment, the image parameters include external parameters and internal parameters of the image.

[0094] Step S02: when each two-dimensional feature point has depth information, obtain the number of consecutive tracking times of each two-dimensional feature point.

[0095] It should be noted that when each two-dimensional feature point has depth information, it means that the two-dimensional feature point is not a newly added two-dimensional feature point, but a point retained after processing. At this time, the number of continuous tracking times of each two-dimensional feature point being continuously tracked and detected is obtained. In this embodiment, it is necessary to ensure that all two-dimensional feature points have three-dimensional depth information.

[0096] Step S03: when the number of consecutive tracking of each two-dimensional feature point reaches a preset number, each two-dimensional feature point is used as a visual feature point to be compressed.

[0097] It should be noted that when the number of continuous tracking times of a two-dimensional feature point reaches a preset number of times, it means that the two-dimensional feature point is stable and will be used in the subsequent compression process. At this time, the two-dimensional feature point is used as the visual feature point to be compressed; when the number of continuous tracking times of a two-dimensional feature point does not reach the preset number of times, it means that the two-dimensional feature point is unstable and will not be used in the subsequent compression process. At this time, the two-dimensional feature point is marked as "temporarily" unavailable. Since the two-dimensional feature point may still be tracked and detected, the two-dimensional feature point will not be eliminated.

[0098] This embodiment provides a visual data compression method, which detects depth information of multiple two-dimensional feature points observed visually; when each two-dimensional feature point has depth information, obtains the number of continuous tracking of each two-dimensional feature point; when the number of continuous tracking of each two-dimensional feature point reaches a preset number, each two-dimensional feature point is used as a visual feature point to be compressed. In the above manner, the number of feature points in the compression process is further reduced, and the stability of the feature points is guaranteed.

[0099] Based on the first embodiment and / or the second embodiment of the present application, in the third embodiment of the present application, the same or similar contents as those of the above-mentioned first and second embodiments can refer to the above introduction, and will not be repeated later. Figure 4 , step S20, the visual data compression method further includes steps S21 to S24:

[0100] Step S21, retaining feature points of the visual noise points and the visual core points of each feature point cluster, determining a plurality of first feature points, and determining a plurality of second feature points based on the plurality of first feature points and the plurality of visual feature points.

[0101] It should be noted that when compressing feature points, priority is given to ensuring that the visual core points and visual noise points of each feature point cluster are retained, thereby obtaining multiple first feature points. During the compression process, the retained visual feature points are controlled by a preset number, and the selection is stopped when the preset number is reached. The process of selecting the retained feature points can be performed randomly on multiple visual core points and multiple visual noise points; the ratio of the number of visual core points and visual noise points can also be set so that the retained visual feature points are as close to the ratio as possible; it can also be performed based on the number of times each visual core point and visual noise point is tracked, and feature points with more tracked times are selected first.

[0102] In this embodiment, the second feature point is a visual feature point that needs to be compressed. Among the multiple visual feature points, all visual feature points except the first feature point are second feature points, and the second feature points are put into the reserve pool.

[0103] Step S22: When the number of retained feature points corresponding to the plurality of first feature points is less than a preset number, a difference calculation is performed based on the number of retained feature points and the preset number to determine the number of feature points to be supplemented.

[0104] It should be noted that the preset number is the number of visual feature points set for subsequent optimization, that is, the upper limit of the visual feature points retained during the feature point compression process. When the number of retained feature points corresponding to all feature points is less than the preset number, the difference between the number of retained feature points and the preset number is calculated to determine the number of feature points to be supplemented.

[0105] Step S23 , supplementing feature points in the plurality of second feature points according to the feature point stability ranking information of each feature point cluster and the number of feature points to be supplemented, to obtain a plurality of third feature points and a plurality of compressed feature points.

[0106] It should be noted that the feature point stability ranking information refers to the ranking order of each visual feature point in the feature point cluster. In each feature point cluster, the ranking order of each visual feature point is obtained by sorting the prior information of each visual feature point. It can be sorted in descending order or ascending order, which is not limited in this embodiment. In this embodiment, the prior information includes but is not limited to the number of times the visual feature point is tracked by optical flow.

[0107] It can be understood that based on the feature point stability ranking information of each feature point cluster, the reserve pool is screened again to select the second feature point with a high number of tracking times. The second feature point will be retained for the subsequent optimization process. The number of selected features is the number of feature points to be supplemented. The final number of feature points to be supplemented + the number of retained feature points corresponding to multiple first feature points = the preset number. In this embodiment, the third feature point refers to the second feature point retained after the second screening; the remaining unselected visual feature points in the reserve pool are all used as compressed feature points and do not participate in the subsequent optimization process.

[0108] In a feasible implementation manner, before step S23, steps D11 to D13 may also be included:

[0109] Step D11, obtaining the tracking times of each visual feature point.

[0110] It should be noted that the tracking times of optical flow tracking of each visual feature point are obtained. In this embodiment, optical flow tracking is a technology used to track the movement of feature points in a video sequence. If a feature point is successfully matched between the current frame and the next frame, the tracking times of the feature point are increased.

[0111] Step D12, sorting the tracking times of each visual feature point in the feature point cluster to determine the stability sorting order of each visual feature point in the feature point cluster.

[0112] Step D13, determining the stability ranking information of the feature point cluster according to the stability ranking order of each visual feature point in the feature point cluster.

[0113] It should be noted that for each feature point cluster: the tracking times corresponding to the existing visual feature points are sorted, and the sorting can be selected in descending order or ascending order, and this embodiment does not limit this, so as to obtain the stability sorting order of each visual feature point. In this embodiment, the more tracking times, the longer the tracking time, which proves that the observation of the feature point is more stable.

[0114] It can be understood that, for each feature point cluster: after determining the stability ranking order of multiple existing visual feature points, the feature point stability ranking information of the feature point cluster can be obtained to reflect the stability of the visual feature points existing in the feature point cluster.

[0115] Step S24, obtaining a plurality of retained feature points according to the plurality of first feature points and the plurality of third feature points.

[0116] It should be noted that all the first feature points and the third feature points are summarized and used as reserved feature points for the subsequent optimization process.

[0117] The present embodiment provides a visual data compression method, which determines a plurality of first feature points by retaining feature points of the visual noise points and the visual core points of each feature point cluster, and determines a plurality of second feature points based on the plurality of first feature points and the plurality of visual feature points; when the number of retained feature points corresponding to the plurality of first feature points is less than a preset number, a difference calculation is performed based on the number of retained feature points and the preset number to determine the number of feature points to be supplemented; feature points are supplemented in the plurality of second feature points based on the feature point stability ranking information of each feature point cluster and the number of feature points to be supplemented to obtain a plurality of third feature points and a plurality of compressed feature points; and a plurality of retained feature points are obtained based on the plurality of first feature points and the plurality of third feature points. In the above manner, while ensuring that the number of retained feature points is controllable, feature points with poor observation quality are filtered out, thereby increasing the robustness of the algorithm.

[0118] For example, to help understand the implementation process of the visual data compression method obtained by combining the present embodiment with the above-mentioned embodiment 1 / 2, please refer to Figure 5 , Figure 5A brief flow chart of a visual data compression method is provided. Specifically, all detected and tracked two-dimensional feature points are traversed as input, and depth information is first determined point by point. If there is, it means that it is not a newly added point but a point that has been processed by the subsequent system and retained; if there is no depth information, it means that the newly added two-dimensional point needs to be triangulated to calculate the initial depth value, so that all points have three-dimensional depth information. If a point has not been tracked continuously for a certain number of times, it means that the point is unstable and will not be flowed into the subsequent system, and will be marked as "temporarily" unavailable, but will not be removed because the point may still be tracked. Then the feature points that meet the conditions are compressed, and the compressed points will be marked with a flag (i.e., a flag variable is set) to indicate that they are compressed points. These points will not be discarded immediately, but will be placed in a reserve pool. However, when the number of feature points flowing into the back-end system is insufficient, the number of points will be supplemented from the reserve pool to reach the preset number, such as 70 points. After processing, the points that flow into the backend optimization will be reprojected and judged. Points with too large reprojections will be marked as outliers and deleted from the point manager (this is the only deletion action). If the reprojection meets the requirements, these points will continue to be retained. The specific compression process is: 1. Geometric space clustering (based on DBSCAN): Clustering is performed according to the three-dimensional spatial distribution of feature points, so that points with close geometric space distances are divided into the same cluster. 2. Sorting of prior information of the same cluster: Sort by the number of optical flow tracking. The longer the tracking time, the more stable the observation of the feature point. 3. Feature point screening: Prioritize the selection of the core points of each cluster and the "noise points" outside the cluster. In visual positioning, these "noise points" are needed to cover the observed scene as much as possible. If there are still vacancies in feature point candidates, they are selected in descending order of the number of tracking times in each cluster.

[0119] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the visual data compression method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0120] This application also provides a visual data compression device, please refer to Figure 6 , the visual data compression device comprises:

[0121] The clustering module 10 is used to perform clustering according to the three-dimensional spatial distribution of each visual feature point to obtain visual noise points, at least one feature point cluster and a visual core point of the feature point cluster.

[0122] The compression module 20 is used to compress multiple visual feature points according to the visual noise points and the visual core points of each feature point cluster to obtain multiple compressed feature points and multiple retained feature points.

[0123] The processing module 30 is used to obtain data compression results of multiple visual feature points according to the multiple compressed feature points and the multiple retained feature points.

[0124] Optionally, the clustering module 10 is further used for:

[0125] Depth information is detected for multiple two-dimensional feature points observed visually; when depth information exists for each two-dimensional feature point, the number of continuous tracking of each two-dimensional feature point is obtained; when the number of continuous tracking of each two-dimensional feature point reaches a preset number, each two-dimensional feature point is used as a visual feature point to be compressed.

[0126] Optionally, the clustering module 10 is further used for:

[0127] When there are two-dimensional feature points but no depth information, a target image pair containing the two-dimensional feature points and a correspondence relationship between the feature points are obtained; a projection matrix is ​​constructed according to image parameters of the image pair; and when the correspondence relationship between the feature points satisfies geometric constraints, a triangulation calculation is performed according to the projection matrix to determine the depth information of the two-dimensional feature points.

[0128] Optionally, the clustering module 10 is further used for:

[0129] The three-dimensional spatial distribution of each visual feature point is determined according to the depth information of each visual feature point; clustering is performed according to the target clustering algorithm, the target search radius and the three-dimensional spatial distribution of each visual feature point to determine at least one feature point cluster and a visual noise point; and a visual core point is determined among multiple visual feature points existing in the feature point cluster according to the cluster internal density of the feature point cluster.

[0130] Optionally, the compression module 20 is further used for:

[0131] Feature points are retained for the visual noise points and the visual core points of each feature point cluster to determine a plurality of first feature points, and a plurality of second feature points are determined based on the plurality of first feature points and the plurality of visual feature points; when the number of retained feature points corresponding to the plurality of first feature points is less than a preset number, a difference calculation is performed based on the number of retained feature points and the preset number to determine the number of feature points to be supplemented; feature points are supplemented in the plurality of second feature points based on the feature point stability ranking information of each feature point cluster and the number of feature points to be supplemented to obtain a plurality of third feature points and a plurality of compressed feature points; a plurality of retained feature points are obtained based on the plurality of first feature points and the plurality of third feature points.

[0132] Optionally, the compression module 20 is further used for:

[0133] Obtain the tracking times of each visual feature point; sort the tracking times of each visual feature point in the feature point cluster to determine the stability sorting order of each visual feature point in the feature point cluster; determine the feature point stability sorting information of the feature point cluster according to the stability sorting order of each visual feature point in the feature point cluster.

[0134] Optionally, the processing module 30 is further used for:

[0135] Reprojection calculation is performed on each retained feature point to determine the reprojected two-dimensional coordinates of each retained feature point; error calculation is performed based on the reprojected two-dimensional coordinates of each retained feature point and the original two-dimensional coordinates of each retained feature point to determine the reprojection error of each retained feature point; when the reprojection error of each retained feature point is less than the error threshold, data of each retained feature point is retained.

[0136] The visual data compression device provided by the present application adopts the visual data compression method in the above embodiment, which can solve the technical problems that the visual positioning algorithm has low computational efficiency and insufficient robustness when processing data sets, and there is a large amount of redundant information that affects the positioning accuracy. Compared with the prior art, the beneficial effects of the visual data compression device provided by the present application are the same as the beneficial effects of the visual data compression method provided by the above embodiment, and the other technical features in the visual data compression device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0137] The present application provides a visual data compression device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the visual data compression method in the above-mentioned embodiment one.

[0138] Reference below Figure 7 , which shows a schematic diagram of the structure of a visual data compression device suitable for implementing the embodiment of the present application. The visual data compression device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 7 The visual data compression device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0139] like Figure 7 As shown, the visual data compression device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the visual data compression device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the visual data compression device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows a visual data compression device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0140] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0141] The visual data compression device provided by the present application adopts the visual data compression method in the above embodiment, which can solve the technical problems that the visual positioning algorithm has low computational efficiency and insufficient robustness when processing data sets, and there is a large amount of redundant information that affects the positioning accuracy. Compared with the prior art, the beneficial effects of the visual data compression device provided by the present application are the same as the beneficial effects of the visual data compression method provided by the above embodiment, and the other technical features in the visual data compression device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0142] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0143] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0144] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the visual data compression method in the above-mentioned embodiment.

[0145] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0146] The computer-readable storage medium may be included in the visual data compression device; or may exist independently without being assembled into the visual data compression device.

[0147] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by the visual data compression device, the visual data compression device: clusters each visual feature point according to its three-dimensional spatial distribution to obtain visual noise points, at least one feature point cluster and visual core points of the feature point clusters; compresses multiple visual feature points according to the visual noise points and the visual core points of each feature point cluster to obtain multiple compressed feature points and multiple retained feature points; obtains data compression results of multiple visual feature points based on the multiple compressed feature points and the multiple retained feature points.

[0148] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0149] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0150] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0151] The readable storage medium provided in this application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned visual data compression method, and can solve the technical problems that the visual positioning algorithm has low computational efficiency and insufficient robustness when processing data sets, and there is a large amount of redundant information affecting the positioning accuracy. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as the beneficial effects of the visual data compression method provided in the above-mentioned embodiment, and will not be repeated here.

[0152] The present application also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the steps of the visual data compression method as described above are implemented.

[0153] The computer program product provided by the present application can solve the technical problems that the visual positioning algorithm has low computational efficiency and insufficient robustness when processing data sets, and there is a large amount of redundant information that affects the positioning accuracy. Compared with the prior art, the beneficial effects of the computer program product provided by the present application are the same as the beneficial effects of the visual data compression method provided by the above embodiment, and will not be repeated here.

[0154] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A visual data compression method, characterized in that: The visual data compression method comprises: Clustering is performed according to the three-dimensional spatial distribution of each visual feature point to obtain visual noise points, at least one feature point cluster, and a visual core point of the feature point cluster; Compressing multiple visual feature points according to the visual noise points and the visual core points of each feature point cluster to obtain multiple compressed feature points and multiple retained feature points; Data compression results of multiple visual feature points are obtained according to the multiple compressed feature points and the multiple retained feature points.

2. The method according to claim 1, characterized in that Before the step of clustering the three-dimensional spatial distribution of each visual feature point to obtain visual noise points, at least one feature point cluster and the visual core point of the feature point cluster, the step further includes: Perform depth information detection on multiple two-dimensional feature points observed visually; When each two-dimensional feature point has depth information, obtaining the number of continuous tracking times of each two-dimensional feature point; When the number of continuous tracking of each two-dimensional feature point reaches a preset number, each two-dimensional feature point is used as a visual feature point to be compressed.

3. The method according to claim 2, characterized in that After the step of detecting depth information of the plurality of two-dimensional feature points observed visually, the method further includes: When there are two-dimensional feature points but no depth information, obtaining a target image pair including the two-dimensional feature points and a corresponding relationship between the feature points; constructing a projection matrix according to image parameters of the image pair; When the corresponding relationship of the feature points satisfies the geometric constraint condition, a triangulation calculation is performed according to the projection matrix to determine the depth information of the two-dimensional feature points.

4. The method according to claim 1, characterized in that The step of clustering the three-dimensional spatial distribution of each visual feature point to obtain visual noise points, at least one feature point cluster and a visual core point of the feature point cluster comprises: Determine the three-dimensional spatial distribution of each visual feature point according to the depth information of each visual feature point; Clustering is performed according to a target clustering algorithm, a target search radius, and a three-dimensional spatial distribution of each visual feature point to determine at least one feature point cluster and a visual noise point; A visual core point is determined from a plurality of visual feature points in the feature point cluster according to the cluster internal density of the feature point cluster.

5. The method according to claim 1, characterized in that The step of compressing a plurality of visual feature points according to the visual noise points and the visual core points of the feature point clusters to obtain a plurality of compressed feature points and a plurality of retained feature points comprises: Retaining feature points of the visual noise points and the visual core points of each feature point cluster, determining a plurality of first feature points, and determining a plurality of second feature points based on the plurality of first feature points and the plurality of visual feature points; When the number of retained feature points corresponding to the plurality of first feature points is less than a preset number, performing a difference calculation based on the number of retained feature points and the preset number to determine the number of feature points to be supplemented; Supplement feature points in the plurality of second feature points according to the feature point stability ranking information of each feature point cluster and the number of feature points to be supplemented, so as to obtain a plurality of third feature points and a plurality of compressed feature points; A plurality of retained feature points are obtained according to the plurality of first feature points and the plurality of third feature points.

6. The method according to claim 5, characterized in that Before the step of supplementing feature points from a plurality of second feature points according to the feature point stability ranking information of each feature point cluster and the number of feature points to be supplemented to obtain a plurality of third feature points and a plurality of compressed feature points, the method further includes: Get the tracking times of each visual feature point; Sorting the tracking times of each visual feature point in the feature point cluster to determine the stability ranking order of each visual feature point in the feature point cluster; According to the stability ranking order of each visual feature point in the feature point cluster, feature point stability ranking information of the feature point cluster is determined.

7. The method according to any one of claims 1 to 6, characterized in that After the step of obtaining data compression results of multiple visual feature points according to the multiple compressed feature points and the multiple retained feature points, the method further includes: Perform reprojection calculation on each retained feature point to determine the reprojected two-dimensional coordinates of each retained feature point; The error is calculated based on the reprojected two-dimensional coordinates of each retained feature point and the original two-dimensional coordinates of each retained feature point to determine the reprojection error of each retained feature point; When the reprojection error of each retained feature point is less than the error threshold, data of each retained feature point is retained.

8. A visual data compression device, characterized in that: The visual data compression device comprises: A clustering module, used for clustering the visual feature points according to their three-dimensional spatial distribution to obtain visual noise points, at least one feature point cluster and a visual core point of the feature point cluster; A compression module, used for compressing a plurality of visual feature points according to the visual noise points and the visual core points of the feature point clusters to obtain a plurality of compressed feature points and a plurality of retained feature points; The processing module is used to obtain data compression results of multiple visual feature points according to multiple compressed feature points and multiple retained feature points.

9. A visual data compression device, characterized in that: The visual data compression device comprises: a memory, a processor, and a visual data compression program stored in the memory and executable on the processor, wherein the visual data compression program is configured to implement the visual data compression method according to any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium stores a visual data compression program, and when the visual data compression program is executed by the processor, the visual data compression method according to any one of claims 1 to 7 is implemented.

Images