Aggregation method, apparatus and electronic device for feature descriptors of consecutive frame images

Abstract

The application provides a feature descriptor aggregation method, device and electronic equipment for continuous frame images. In the aggregation method, the feature descriptors of adjacent frame images of a target reference frame image are fused into the feature descriptor of the target reference frame image, so that an aggregated feature descriptor of the target reference frame image is obtained. The aggregated feature descriptor contains time sequence information and information of adjacent frame images. When loop detection is performed through the aggregated feature descriptors of each reference frame image, false positive isolated group matching reference frame images with high similarity scores can be effectively avoided, the accuracy of loop detection is improved, only one image matching is needed, time consumption is reduced, the determination process of the aggregated feature descriptor is simple, the calculation amount is small, and the technical problem that the prior art cannot improve the accuracy of loop detection while ensuring low calculation amount and low time consumption is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to a feature descriptor aggregation method and device for continuous frame images and an electronic device. BACKGROUND

[0002] Simultaneous Localization and Mapping (SLAM) is usually applied in the field of autonomous driving and robots, and loop closure detection is an important part of the SLAM system. On the one hand, it is related to the correctness of the estimated trajectory and map over a long period of time. On the other hand, since loop closure detection provides the relevance of the current data and the historical data, it can also be used for relocalization. One of the most critical tasks in this process is to correctly identify the previously visited scene. In appearance-based loop closure detection algorithms, the core problem is how to calculate the similarity between images. Visual Place Recognition (VPR) is a component of this task, which retrieves the image highly matched with the current frame image from the stored reference frame image according to the visual data.

[0003] Feature-based VPR methods store the feature points and their corresponding descriptors of the image. In the retrieval process, the similarity between images can be calculated by matching the feature descriptors of the images, so as to determine whether the scene has been visited. In the loop closure detection algorithm, the accuracy and recall are usually used to describe the performance of the loop closure detection algorithm. The accuracy describes the probability that the algorithm detects all the loops that are real loops; the recall refers to the probability that all real loops are correctly detected. In the SLAM algorithm, the accuracy requirement is high, while the recall requirement is relatively loose, because false positive loops (i.e. the detected loop is not a real loop) will cause the entire SLAM system to be completely wrong. The so-called false positive loop refers to that when image matching is performed, the Mth reference frame stored in the scene database is the matching frame image of the current frame image, but in the actual matching process, the feature descriptor of the current frame image has the highest similarity with the feature descriptor of the Nth reference frame in the scene database, so the matching frame image of the current frame image obtained is the Nth reference frame, which causes a matching error.

[0004] In order to avoid the false positive loop and improve the accuracy of loop detection, the prior art generally adopts the following two schemes: one is to match the current frame image with each reference frame of the scene database one by one, obtain a plurality of matching scores, and then calculate the matching score sum of the current frame image and each reference frame according to the plurality of matching scores, and take the reference frame with the highest matching score sum as the matching frame image of the current frame image, so that the matching error caused by the high matching score of a single data frame can be avoided. However, this scheme needs to be matched multiple times, and the time consumption of image matching is long; the other is to obtain the feature descriptor of each frame of the continuous frame of the current frame image, and then perform serialization processing on all the feature descriptors of the continuous frame to obtain the serialized feature descriptor of the current frame image, and similarly, the feature descriptors of the continuous frame of each reference frame in the scene database are serialized to obtain the serialized feature descriptors of each reference frame in the scene database, and then the matching frame image of the current frame image is determined through the serialized feature descriptor matching. Although this scheme only needs to be matched once, the serialization process is complex, the calculation amount is large, and the serialized feature descriptor occupies a large space memory.

[0005] In summary, how to improve the accuracy of loop detection under the premise of low calculation amount and low time consumption has become a technical problem to be solved at present. SUMMARY

[0006] Therefore, the purpose of the present application is to provide a continuous frame image feature descriptor aggregation method and device and electronic equipment to alleviate the technical problem that the prior art cannot improve the accuracy of loop detection under the premise of low calculation amount and low time consumption.

[0007] In a first aspect, an embodiment of the present application provides a continuous frame image feature descriptor aggregation method, comprising:

[0008] Obtaining the feature descriptor of the continuous frame image of the target reference frame image in the scene database, wherein the target reference frame image is any reference frame image in the scene database, and the continuous frame image includes the target reference frame image, k frame images before the target reference frame image, and k frame images after the target reference frame image;

[0009] Summing the feature descriptors of the continuous frame image, and normalizing the summed feature descriptors to obtain the aggregated feature descriptor of the target reference frame image, and further obtaining the aggregated feature descriptors of each reference frame image in the scene database, so as to use the aggregated feature descriptors of each reference frame image in the scene database for loop detection.

[0010] Further, the feature descriptor of the continuous frame image of the target reference frame image in the scene database is acquired, comprising:

[0011] The feature descriptor of each frame image in the continuous frame image is acquired by the following method:

[0012] The target image is subjected to semantic segmentation, and the segmented regions in the semantic segmentation result are subjected to hierarchical processing according to each category information, to obtain a binary image corresponding to each category information, wherein the target image is any frame image in the continuous frame image;

[0013] The skeleton corresponding to the category information is extracted from the binary image corresponding to the category information, and the key point and the center point corresponding to the category information are determined according to the extracted skeleton;

[0014] The shape context algorithm is used to encode the key point and the center point corresponding to each category information, to obtain the feature descriptor of the key point and the feature descriptor of the center point corresponding to each category information;

[0015] The feature descriptor of the key point and the feature descriptor of the center point corresponding to each category information are subjected to spatial aggregation, to obtain the feature descriptor corresponding to each category information;

[0016] The feature descriptors corresponding to various category information are straightened and normalized, to obtain the feature descriptor of the target image.

[0017] Further, the segmented regions in the semantic segmentation result are subjected to hierarchical processing according to each category information, comprising:

[0018] The segmented region of the dynamic category information in the semantic segmentation result is set as a background region, to obtain a first preprocessed semantic segmentation result;

[0019] The segmented regions of different category information in the first preprocessed semantic segmentation result are merged according to a preset category information merging strategy, to obtain a second preprocessed semantic segmentation result;

[0020] In the second preprocessed semantic segmentation result, the segmented region of each category information is taken as a foreground, and the remaining segmented regions are taken as a background, to construct an initial binary image corresponding to each category information;

[0021] The initial binary image corresponding to each category information is subjected to image morphological processing, to obtain an intermediate binary image corresponding to each category information;

[0022] In the intermediate binary image corresponding to each category information, the independent segmented regions in the closed segmented region are filled according to a preset filling strategy, to obtain the binary image corresponding to each category information.

[0023] Further, the skeleton corresponding to the category information is extracted from the binary image corresponding to the category information, and the skeleton extraction method comprises:

[0024] The skeleton corresponding to the category information is extracted from the binary image corresponding to the category information by using an OpenCV skeleton thinning algorithm.

[0025] Further, the key point and the center point corresponding to the category information are determined according to the extracted skeleton, and the key point and the center point determination method comprises:

[0026] The end point and the intersection point of the skeleton are taken as the key point corresponding to the category information.

[0027] The center position coordinates of the key point are calculated according to the position coordinates of the key point, so as to obtain the center point.

[0028] Further, the feature descriptor of the key point and the feature descriptor of the center point corresponding to each category information are spatially aggregated, and the aggregation method comprises:

[0029] The feature descriptor of each key point and the feature descriptor of the center point are differentially calculated, so as to obtain a plurality of differential results.

[0030] The plurality of differential results are summed, so as to obtain a sum result.

[0031] The sum result is normalized, so as to obtain the feature descriptor corresponding to each category information.

[0032] Further, the method further comprises:

[0033] The feature descriptor of the current frame image is obtained.

[0034] The inner product between the feature descriptor of the current frame image and the aggregated feature descriptor of each reference frame image in the scene database is calculated.

[0035] The maximum inner product is determined in the inner product, and the reference frame image corresponding to the maximum inner product is taken as a matching reference frame image matched with the current frame image.

[0036] The position coordinates of the matching reference frame image are taken as the position coordinates corresponding to the current frame image.

[0037] In a second aspect, an embodiment of the present application further provides an aggregation device for feature descriptors of continuous frame images, comprising:

[0038] An acquisition unit is configured to acquire feature descriptors of continuous frame images of a target reference frame image in a scene database, wherein the target reference frame image is any reference frame image in the scene database, and the continuous frame images include the target reference frame image, k frame images before the target reference frame image, and k frame images after the target reference frame image.

[0039] An aggregation unit is configured to sum the feature descriptors of the continuous frame images, normalize the summed feature descriptors, obtain an aggregated feature descriptor of the target reference frame image, and further obtain aggregated feature descriptors of each reference frame image in the scene database, so as to use the aggregated feature descriptors of each reference frame image in the scene database for loop detection.

[0040] In a third aspect, an electronic device is provided, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor executes the computer program to implement the steps of the method in any one of the first aspect.

[0041] In a fourth aspect, a computer readable storage medium is provided, which stores machine executable instructions, and the machine executable instructions, when invoked and executed by a processor, cause the processor to execute the method in any one of the first aspect.

[0042] In the embodiment of the present application, a method for aggregating feature descriptors of continuous frame images is provided, comprising: obtaining feature descriptors of continuous frame images of a target reference frame image in a scene database, wherein the target reference frame image is any reference frame image in the scene database, and the continuous frame images comprise the target reference frame image, k frame images before the target reference frame image, and k frame images after the target reference frame image; summing the feature descriptors of the continuous frame images, and performing normalization processing on the summed feature descriptors to obtain an aggregated feature descriptor of the target reference frame image, and further obtaining aggregated feature descriptors of each reference frame image in the scene database, so as to use the aggregated feature descriptors of each reference frame image in the scene database for loop detection. As can be seen from the above description, in the method for aggregating feature descriptors of continuous frame images, the feature descriptors of adjacent frame images of the target reference frame image are fused into the feature descriptors of the target reference frame image, so as to obtain the aggregated feature descriptor of the target reference frame image, which contains time sequence information and information of adjacent frame images. When loop detection is performed by using the aggregated feature descriptors of each reference frame image, false positive single outlier matching reference frame images with high similarity scores can be effectively avoided, the accuracy of loop detection is improved, only one image matching needs to be performed, time consumption is reduced, in addition, the determination process of the aggregated feature descriptor is simple, the calculation amount is small, and the technical problem that the prior art cannot improve the accuracy of loop detection under the condition of low calculation amount and low time consumption is solved. BRIEF DESCRIPTION OF DRAWINGS

[0043] In order to more clearly illustrate the specific embodiments of the present application or the technical solutions in the prior art, the drawings needed in the specific embodiments or the prior art description will be briefly introduced below. Obviously, the drawings in the following description are some embodiments of the present application, and those skilled in the art can also obtain other drawings according to these drawings without creative labor.

[0044] Figure 1 A flowchart of a method for aggregating feature descriptors of continuous frame images provided in the embodiment of the present application;

[0045] Figure 2 A method flowchart for obtaining feature descriptors of continuous frame images of a target reference frame image in a scene database provided in the embodiment of the present application;

[0046] Figure 3 A schematic diagram of an aggregation device for feature descriptors of continuous frame images provided in the embodiment of the present application;

[0047] Figure 4 A schematic diagram of an electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0048] The technical solutions of the present application will be described clearly and completely in combination with the embodiments. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all the other embodiments obtained by those skilled in the art without creative labor fall within the protection scope of the present application.

[0049] The prior art cannot improve the accuracy of loop detection under the condition of ensuring low calculation amount and low time consumption.

[0050] Therefore, in the aggregation method of the feature descriptors of the continuous frame images, the feature descriptors of the adjacent frame images of the target reference frame image are fused into the feature descriptors of the target reference frame image, so that the aggregated feature descriptors of the target reference frame image are obtained, the aggregated feature descriptors of the target reference frame image contain the time sequence information and the information of the adjacent frame images, the false positive single outlier matching reference frame image with high similarity score can be effectively avoided when the loop detection is performed by using the aggregated feature descriptors of the target reference frame image, the accuracy of the loop detection is improved, only one image matching is needed, the time consumption is reduced, and the determination process of the aggregated feature descriptors is simple and the calculation amount is small.

[0051] In order to facilitate the understanding of the present embodiment, first, the aggregation method of the feature descriptors of the continuous frame images disclosed in the present embodiment is introduced in detail.

[0052] Embodiment one:

[0053] According to the embodiment of the present application, an embodiment of the aggregation method of the feature descriptors of the continuous frame images is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a group of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0054] Figure 1 The flowchart of the aggregation method of the feature descriptors of the continuous frame images according to the embodiment of the present application is shown in FIG. 1, which comprises the following steps: Figure 1

[0055] In step S102, the feature descriptors of the continuous frame images of the target reference frame image in the scene database are acquired, wherein the target reference frame image is any reference frame image in the scene database, and the continuous frame images include the target reference frame image, k frame images before the target reference frame image and k frame images after the target reference frame image.

[0056] In the embodiment of the present application, k is a positive integer greater than or equal to 1.​

[0057] The feature descriptor of each frame image of the continuous frame images can be appearance feature information of the image or spatial distribution feature information of the image, and embodiments of the present application do not make specific limitations thereto.

[0058] In step S104, the feature descriptors of the continuous frame images are summed, and the summed feature descriptors are normalized to obtain the aggregated feature descriptor of the target reference frame image, and further to obtain the aggregated feature descriptors of the reference frame images in the scene database, so as to use the aggregated feature descriptors of the reference frame images in the scene database for loop detection.

[0059] Specifically, the elements of the corresponding dimensions of the feature descriptors of each frame image in the continuous frame images are summed, and the summed feature descriptors are normalized, so as to obtain the aggregated feature descriptor of the target reference frame image, and further to obtain the aggregated feature descriptors of the reference frame images in the scene database. The aggregated feature descriptors of the reference frame images in the scene database contain image feature information of time and space.

[0060] The inventor considers that if there is a false matching relationship between the current frame image and a reference frame image in the scene database, then the similarity between the current frame image and the image adjacent to the reference frame image is low, and if there is a true matching relationship between the current frame image and a reference frame image in the scene database, then the similarity between the current frame image and the image adjacent to the matching reference frame image is also high. Therefore, the inventor designs the aggregation method of the feature descriptors of the continuous frame images, fuses the feature descriptors of the images adjacent to the reference frame image into the feature descriptor of the reference frame image, thereby obtaining the aggregated feature descriptor, and further performs loop detection through the aggregated feature descriptor, effectively avoids the false positive isolated group matching reference frame image with a high similarity score, and improves the accuracy of loop detection.

[0061] When subsequent matching is performed through the aggregated feature descriptor, since the spatial and temporal information is contained, only one matching between the feature descriptor of the current frame image and the aggregated feature descriptor of the reference frame image is needed, and it is not necessary to perform one-by-one matching between the feature descriptor of the current frame image and the feature descriptors of the continuous frame images of the reference frame image, thereby reducing the time consumption and improving the matching efficiency.

[0062] As known from the process of step S104, the calculation process of the aggregated feature descriptor is simple, the calculation amount is small, and the dimension of the obtained aggregated feature descriptor is equal to that of the original feature descriptor, that is, the memory occupation space of the aggregated feature descriptor does not increase, and compared with the traditional serialized feature descriptor, the aggregated feature descriptor is simpler.

[0063] In the embodiment of the present application, a feature descriptor aggregation method of continuous frame images is provided, comprising: obtaining feature descriptors of continuous frame images of a target reference frame image in a scene database, wherein the target reference frame image is any reference frame image in the scene database, and the continuous frame images comprise the target reference frame image, k frame images before the target reference frame image, and k frame images after the target reference frame image; summing the feature descriptors of the continuous frame images, and performing normalization processing on the summed feature descriptors to obtain an aggregated feature descriptor of the target reference frame image, and further obtaining aggregated feature descriptors of each reference frame image in the scene database, so as to use the aggregated feature descriptors of each reference frame image in the scene database for loop detection. As can be seen from the above description, in the feature descriptor aggregation method of continuous frame images of the present application, the feature descriptors of the adjacent frame images of the target reference frame image are fused into the feature descriptor of the target reference frame image, thereby obtaining the aggregated feature descriptor of the target reference frame image. The aggregated feature descriptor contains time sequence information and information of adjacent frame images. When loop detection is performed by using the aggregated feature descriptors of each reference frame image, false positive single outlier matching reference frame images with high similarity scores can be effectively avoided, the accuracy of loop detection is improved, and only one image matching is required, thereby reducing time consumption. In addition, the determination process of the aggregated feature descriptor is simple and the calculation amount is small, thereby solving the technical problem that the prior art cannot improve the accuracy of loop detection under the condition of low calculation amount and low time consumption.

[0064] The above describes the feature descriptor aggregation method of continuous frame images of the present application briefly, and the specific contents involved therein are described in detail below.

[0065] In an optional embodiment of the present application, the reference Figure 2 The step S102 of obtaining the feature descriptors of the continuous frame images of the target reference frame image in the scene database specifically comprises the following steps:

[0066] The feature descriptors of each frame image in the continuous frame images are obtained by the following method:

[0067] In step S201, the target image is subjected to semantic segmentation, and the segmentation regions in the semantic segmentation result are subjected to hierarchical processing according to each type of information to obtain a binary image corresponding to each type of information, wherein the target image is any frame image in the continuous frame images.

[0068] Specifically, the target image is subjected to semantic segmentation by using an open-source CNN model, and the semantic segmentation result is obtained. The semantic segmentation result includes at least one segmentation region and type information corresponding to each segmentation region.

[0069] The hierarchical processing of the segmentation region in the semantic segmentation result according to each category information refers to that pixel blocks with the same category label are divided into the same layer, the value of the pixel points thereof is set to 1, and the value of the remaining pixel points is set to 0, so that a binary graph corresponding to each category information is obtained.

[0070] The process of the hierarchical processing will be described below in detail, and will not be described here again.

[0071] In step S202, the skeleton corresponding to the category information is extracted from the binary graph corresponding to the category information, and the key point and the center point corresponding to the category information are determined according to the extracted skeleton;

[0072] The extracted skeleton can adapt to the dramatic change of the appearance, that is, when the appearance changes dramatically, the extracted skeleton is also stable and accurate, and then the key point and the center point corresponding to the category information determined according to the extracted skeleton are also stable and robust.

[0073] In step S203, the shape context algorithm is used to encode the key point and the center point corresponding to each category information, to obtain the feature descriptor of the key point and the feature descriptor of the center point corresponding to each category information;

[0074] Specifically, taking a key point pi as a reference point, N equidistant concentric circles are drawn in a local region with the key point pi as the center and a radius R (for example, 10 equidistant concentric circles are drawn with a radius of 10, that is, 10 concentric circles are drawn with radii of 1, 2, 3, …, 10, and the distance between adjacent two concentric circles is 1, which is equidistant), and then the concentric circles are equally divided into M parts to form a target-shaped template. Each block region of the target-shaped template is numbered, and there are MxN block regions. The number of key points and center points appearing in the MxN block regions is counted, that is, each element in the feature descriptor of the key point pi is obtained. In this way, the feature descriptor of the key point pi is obtained, and the feature descriptor of the center point is obtained in the same way.

[0075] In step S204, the feature descriptor of the key point and the feature descriptor of the center point corresponding to each category information are spatially aggregated to obtain the feature descriptor corresponding to each category information.

[0076] Specifically, referring to the idea of VLAD (Vector of Locally Aggregated Descriptor), the feature descriptors of all key points and the feature descriptors of all center points corresponding to one category information are aggregated into the feature descriptor corresponding to the category information. When spatial aggregation is performed, the center point corresponding to the category information is used for aggregation, and the aggregation process will be described in detail below.

[0077] In step S205, the feature descriptors corresponding to various types of information are straightened and normalized to obtain the feature descriptor of the target image.

[0078] Specifically, the feature descriptors corresponding to all types of information are straightened and normalized to obtain the final feature descriptor of the target image, which is simpler.

[0079] The above feature descriptor acquisition process is based on the semantic segmentation result to obtain a binary image corresponding to each type of information, and then the skeleton corresponding to the type of information is extracted, and the key points and the center point corresponding to the type of information are determined based on the skeleton, and the feature descriptor of the key points and the feature descriptor of the center point are obtained by encoding, and finally, the feature descriptors of the key points and the center points are spatially aggregated to obtain the feature descriptor corresponding to the type of information, and the feature descriptors corresponding to various types of information are straightened and normalized to obtain the feature descriptor of the target image. In the above process, even when the appearance changes dramatically (for example, the tree has leaves in summer, and there are no leaves on the tree in winter, which is a case of dramatic change in the appearance of the tree), the skeleton corresponding to each type of information extracted based on the semantic segmentation result is stable and accurate, so the feature descriptor of the key points and the feature descriptor of the center point corresponding to each type of information obtained subsequently are stable and accurate, and the final feature descriptor of the target image is also stable and robust. In addition, the above feature descriptor of the target image is obtained by spatially aggregating the feature descriptors of the key points and the center points, and then straightening and normalizing the feature descriptors corresponding to various types of information, that is, the above feature descriptor of the target image is simpler, which greatly reduces the calculation amount and speeds up the image matching speed and improves the image matching efficiency during subsequent image matching.

[0080] In an optional embodiment of the present application, step S201 processes the segmentation regions in the semantic segmentation result according to each type of information, specifically including the following steps:

[0081] (1) The segmentation region of dynamic type information in the semantic segmentation result is set as a background region to obtain a first preprocessed semantic segmentation result;

[0082] Specifically, in order to minimize the impact of missegmentation, a series of refinement processing needs to be performed on the semantic segmentation result. In the semantic segmentation result, there are some segmented regions of dynamic category information that cannot be used as scene features, such as pedestrian or car segmented regions. These segmented regions cannot be used as stable scene features (not moving over time, such as trees, grass, buildings, roads, etc.), so the segmented regions of dynamic category information need to be ignored, that is, set as background regions, and then a first preprocessed semantic segmentation result is obtained.

[0083] (2) merging different category information segmented regions in the first preprocessed semantic segmentation result according to a preset category information merging strategy, to obtain a second preprocessed semantic segmentation result;

[0084] Specifically, the preset category information merging strategy is that some segmented regions of often confused category information should be merged, for example, the segmented regions of "wall" and "fence" should be merged, and the second preprocessed semantic segmentation result is obtained after merging.

[0085] (3) in the second preprocessed semantic segmentation result, taking each category information segmented region as foreground and the rest as background to construct an initial binary graph corresponding to each category information;

[0086] (4) performing image morphological processing on the initial binary graph corresponding to each category information to obtain an intermediate binary graph corresponding to each category information;

[0087] Specifically, in order to eliminate noise and connect adjacent regions, image morphological processing such as dilation and erosion is performed on the initial binary graph corresponding to each category information, so as to obtain the intermediate binary graph corresponding to each category information.

[0088] (5) in the intermediate binary graph corresponding to each category information, filling independent segmented regions in the closed segmented region according to a preset filling strategy, to obtain a binary graph corresponding to each category information.

[0089] Specifically, in order to further reduce the impact of missegmentation on subsequent skeleton extraction, some small "holes" contained in the closed segmented region are filled, and some small segmented regions that are still independent of the whole after the previous processing are removed.

[0090] In an optional embodiment of the present application, the skeleton corresponding to each category information is extracted from the binary graph corresponding to the category information, specifically including:

[0091] The skeleton corresponding to each category information is extracted from the binary graph corresponding to the category information by using the OpenCV skeleton thinning algorithm.

[0092] In an optional embodiment of the present invention, determining the key points and center points corresponding to this type of category information based on the extracted skeleton specifically includes the following steps:

[0093] (1) The endpoints and intersections of the skeleton are taken as the key points corresponding to this type of information;

[0094] (2) Calculate the center coordinates of the key points based on their position coordinates to obtain the center point.

[0095] Specifically, if the key points corresponding to a category of information are P = {p1, p2, ..., p...} |key-points| In this context, |key-points| represents the total number of keypoints corresponding to this type of information. Furthermore, the center point corresponding to this type of information is the coordinate of the center position of the aforementioned keypoints. For example, if there are 10 keypoints, the x-coordinates of these 10 keypoints are summed and averaged to obtain the x-coordinate of the center point, and the y-coordinates of these 10 keypoints are summed and averaged to obtain the y-coordinate of the center point. Thus, the center point is calculated.

[0096] In an optional embodiment of the present invention, step S204 above, which involves spatial aggregation of the feature descriptors of key points and center points corresponding to each type of information, specifically includes the following steps:

[0097] (1) Calculate the difference between the feature descriptor of each key point and the feature descriptor of the center point to obtain multiple difference results;

[0098] (2) Summ the multiple difference results to obtain the summation result;

[0099] (3) Normalize the summation result to obtain the feature descriptor corresponding to each category of information.

[0100] The above process can be represented as: Where V(k) represents the feature descriptor corresponding to k categories of information, k represents a category of information, and d i d represents the feature descriptor of the i-th key point. c The feature descriptor representing the center point, |key-points| k This represents the total number of key points corresponding to category k information.

[0101] The feature descriptor corresponding to each category of information obtained above is an encoding of the two-dimensional spatial layout relationship of the objects of each category of information, and is a local descriptor.

[0102] In order to encode the relationship between objects of different categories, the feature descriptors corresponding to various category information need to be processed. In an optional embodiment of the present application, the feature descriptors corresponding to various category information are straightened and normalized, specifically including the following steps:

[0103] (1) The feature descriptors corresponding to various category information are spliced to obtain spliced feature descriptors;

[0104] (2) The spliced feature descriptors are normalized to obtain the feature descriptors of the target image.

[0105] For example, the feature descriptor corresponding to one category information is a 256-dimensional vector, and after splicing the feature descriptors corresponding to five categories of information, the spliced feature descriptor is a 256*5-dimensional vector. In this way, the final feature descriptor of the target image is a fixed-dimensional multi-dimensional descriptor vector, which is a global descriptor.

[0106] In the process of obtaining the feature descriptor of the present application, the final feature descriptor of the target image describes the spatial distribution relationship between the same category semantic objects in the scene and the spatial distribution relationship between different category semantic objects. Unlike other methods, in the method of the present application, the spatial distribution refers to the distribution of pixels with the same semantic category in the image. The present application encodes the above spatial distribution relationship, extracts the key points of the semantic skeleton of each category as a local descriptor of an image, and then, by referring to the idea of local aggregation vector descriptor, aggregates the local semantic skeleton representation of each category into a fixed-dimensional global descriptor to represent the entire image.

[0107] In an optional embodiment of the present application, the method further includes the following steps:

[0108] (1) Obtaining the feature descriptor of the current frame image;

[0109] Specifically, the feature descriptor of the current frame image can be obtained by referring to the process of steps S201 to S205, which will not be described here.

[0110] (2) Calculating the inner product between the feature descriptor of the current frame image and the aggregated feature descriptor of each reference frame image in the scene database;

[0111] Specifically, the scene database contains a plurality of reference frame images, the corresponding relationship between the feature descriptor of each reference frame image and the position coordinates of each reference frame image.

[0112] (3) Determining the maximum inner product in the inner product, and taking the reference frame image corresponding to the maximum inner product as the matching reference frame image matched with the current frame image;

[0113] (4) taking the position coordinates of the matched reference frame image as the position coordinates corresponding to the current frame image.

[0114] By encoding the feature descriptor information of the adjacent frame image into the feature descriptor of the target reference frame image, the robustness and accuracy of the aggregated feature descriptor can be enhanced in a long-term scene. Considering the semantic consistency of the adjacent frames, the occurrence of isolated matched reference frame images can be effectively avoided when retrieving images from a scene database composed of time sequences and spatial sequences, and this method only needs one frame matching instead of complex frame sequence matching, so that the entire operation process does not increase too much computational load when matching images.

[0115] Embodiment two:

[0116] The embodiment of the application further provides a feature descriptor aggregation device of continuous frame images, which is mainly used for executing the feature descriptor aggregation method of continuous frame images provided in the embodiment one of the application. The feature descriptor aggregation device of continuous frame images provided in the embodiment of the application is specifically introduced as follows.

[0117] Figure 3 is a schematic diagram of a feature descriptor aggregation device of continuous frame images according to the embodiment of the application. The device mainly comprises an acquisition unit 10 and an aggregation unit 20, wherein:

[0118] The acquisition unit is configured to acquire the feature descriptors of the continuous frame images of the target reference frame image in the scene database, wherein the target reference frame image is any reference frame image in the scene database, and the continuous frame images comprise the target reference frame image, k frame images before the target reference frame image, and k frame images after the target reference frame image.

[0119] The aggregation unit is configured to sum the feature descriptors of the continuous frame images, normalize the summed feature descriptors, obtain the aggregated feature descriptor of the target reference frame image, and further obtain the aggregated feature descriptors of the reference frame images in the scene database, so as to use the aggregated feature descriptors of the reference frame images in the scene database for loop detection.

[0120] In the embodiment of the present application, an aggregation device of feature descriptors of continuous frame images is provided, comprising: obtaining the feature descriptors of continuous frame images of a target reference frame image in a scene database, wherein the target reference frame image is any reference frame image in the scene database, and the continuous frame images comprise the target reference frame image, k frame images before the target reference frame image, and k frame images after the target reference frame image; summing the feature descriptors of the continuous frame images, and performing normalization processing on the summed feature descriptors to obtain an aggregated feature descriptor of the target reference frame image, and further obtaining aggregated feature descriptors of each reference frame image in the scene database, so as to use the aggregated feature descriptors of each reference frame image in the scene database for loop detection. As can be seen from the above description, in the aggregation device of feature descriptors of continuous frame images of the present application, the feature descriptors of adjacent frame images of the target reference frame image are fused into the feature descriptor of the target reference frame image, so that the aggregated feature descriptor of the target reference frame image is obtained, which contains time sequence information and information of adjacent frame images. When loop detection is performed by using the aggregated feature descriptors of each reference frame image, false positive single outlier matching reference frame images with high similarity scores can be effectively avoided, the accuracy of loop detection is improved, and only one image matching is required, so that the time consumption is reduced. In addition, the determination process of the aggregated feature descriptor is simple, the calculation amount is small, and the technical problem that the prior art cannot improve the accuracy of loop detection under the condition of low calculation amount and low time consumption is solved.

[0121] Optionally, the obtaining unit is further configured to obtain the feature descriptor of each frame image in the continuous frame images by the following method: performing semantic segmentation on a target image, and performing hierarchical processing on the segmentation regions in the semantic segmentation result according to each type of information to obtain a binary image corresponding to each type of information, wherein the target image is any frame image in the continuous frame images; extracting a skeleton corresponding to each type of information from the binary image corresponding to the type of information, and determining a key point and a center point corresponding to the type of information according to the extracted skeleton; encoding the key point and the center point corresponding to each type of information by using a shape context algorithm to obtain a feature descriptor of the key point and a feature descriptor of the center point corresponding to each type of information; performing spatial aggregation on the feature descriptors of the key point and the center point corresponding to each type of information to obtain a feature descriptor corresponding to each type of information; and performing straightening and normalization on the feature descriptors corresponding to various types of information to obtain the feature descriptor of the target image.

[0122] Optionally, the obtaining unit is further configured to: set a segmentation region of the dynamic class information in the semantic segmentation result as a background region to obtain a first preprocessed semantic segmentation result; merge segmentation regions of different class information in the first preprocessed semantic segmentation result according to a preset class information merging strategy to obtain a second preprocessed semantic segmentation result; construct an initial binary graph corresponding to each class information in the second preprocessed semantic segmentation result, taking a segmentation region of each class information as a foreground and taking the rest segmentation regions as a background; perform image morphological processing on the initial binary graph corresponding to each class information to obtain an intermediate binary graph corresponding to each class information; and fill independent segmentation regions in a closed segmentation region in the intermediate binary graph corresponding to each class information according to a preset filling strategy to obtain a binary graph corresponding to each class information.

[0123] Optionally, the obtaining unit is further configured to: extract a skeleton corresponding to the class information from the binary graph corresponding to the class information by using an OpenCV skeleton thinning algorithm.

[0124] Optionally, the obtaining unit is further configured to: take an end point and an intersection point of the skeleton as a key point corresponding to the class information; calculate a center position coordinate of the key point according to a position coordinate of the key point to obtain a center point.

[0125] Optionally, the obtaining unit is further configured to: perform difference calculation on a feature descriptor of each key point and a feature descriptor of the center point to obtain a plurality of difference results; sum the plurality of difference results to obtain a sum result; and perform normalization processing on the sum result to obtain a feature descriptor corresponding to each class information.

[0126] Optionally, the apparatus is further configured to: obtain a feature descriptor of a current frame image; calculate an inner product between the feature descriptor of the current frame image and an aggregate feature descriptor of each reference frame image in the scene database; determine a maximum inner product in the inner product, and take a reference frame image corresponding to the maximum inner product as a matching reference frame image matched with the current frame image; and take a position coordinate of the matching reference frame image as a position coordinate corresponding to the current frame image.

[0127] The apparatus provided in the embodiments of the present application has the same implementation principles and technical effects as the foregoing method embodiments. For brevity, the part not mentioned in the apparatus embodiment part can be referred to the corresponding content in the foregoing method embodiments.

[0128] As Figure 4As shown, the electronic device 600 provided by the embodiment of the present application comprises a processor 601, a memory 602 and a bus. The memory 602 stores machine readable instructions executable by the processor 601. When the electronic device is running, the processor 601 and the memory 602 communicate through the bus. The processor 601 executes the machine readable instructions to perform the steps of the method for aggregating feature descriptors of consecutive frame images.

[0129] Specifically, the memory 602 and the processor 601 can be general memory and processor, which are not specifically limited herein. When the processor 601 runs the computer program stored in the memory 602, the method for aggregating feature descriptors of consecutive frame images can be executed.

[0130] The processor 601 can be an integrated circuit chip with signal processing capability. In the implementation process, each step of the above method can be completed by the integrated logic circuit of hardware in the processor 601 or the instruction in the form of software. The processor 601 can be a general processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. Each method, step and logic block diagram disclosed in the embodiment of the present application can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in combination with the embodiment of the present application can be directly embodied as hardware code processor for execution, or executed by the combination of hardware and software modules in the code processor. The software module can be located in the random access memory, the flash memory, the read only memory, the programmable read only memory or the electrically erasable programmable memory, the register or other mature storage medium in the art. The storage medium is located in the memory 602, and the processor 601 reads the information in the memory 602 and combines the hardware to complete the steps of the above method.

[0131] Corresponding to the aggregation method of the feature descriptors of the continuous frame images, the embodiment of the present application further provides a computer readable storage medium, the computer readable storage medium stores computer executable instructions, when the computer executable instructions are invoked and run by a processor, the computer executable instructions cause the processor to run the steps of the aggregation method of the feature descriptors of the continuous frame images.

[0132] The aggregation device of the feature descriptors of the continuous frame images provided by the embodiment of the present application can be specific hardware on a device or software or firmware installed on the device, etc. The device provided by the embodiment of the present application has the same implementation principle and technical effects as the foregoing method embodiments, and for brief description, the part not mentioned in the device embodiment can be referred to the corresponding content in the foregoing method embodiments. Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system, device and unit described above can be referred to the corresponding process in the foregoing method embodiments, which will not be described here.

[0133] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented by other ways. The device embodiments described above are only schematic, for example, the division of the units is only a logical function division, and there can be another division manner in actual implementation, and for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some communication interfaces, devices or units, and can be electrical, mechanical or other forms.

[0134] For another example, the flowcharts and block diagrams in the drawings show the possible implementation architecture, function and operation of the devices, methods and computer program products according to the embodiments of the present application. In this regard, each block in the flowcharts or block diagrams can represent a module, a program segment or a part of code, which contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur in different order from that shown in the drawings. For example, two consecutive blocks can actually be executed substantially in parallel, and sometimes they can be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and the combination of blocks in the block diagrams and / or flowcharts, can be implemented by a dedicated hardware-based system for executing the specified functions or actions, or can be implemented by a combination of special-purpose hardware and computer instructions.

[0135] The units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, i.e. may be located in one place, or may be distributed on multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0136] In addition, the functional units in the embodiments provided in the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit.

[0137] If the functions are realized in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application can be embodied in the form of software product, and the computer software product is stored in a storage medium, including a plurality of instructions for making an electronic device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the vehicle marking method described in the embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, and various media that can store program codes.

[0138] It should be noted that: similar reference numbers and letters represent similar items in the following drawings, therefore, once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings, in addition, the terms "first", "second", "third" and the like are only used to distinguish description, and cannot be understood as indicating or implying relative importance.

[0139] Finally, it should be noted that: the above-described embodiments are only specific embodiments of the present application, used to illustrate the technical solutions of the present application, and not to limit them, the protection scope of the present application is not limited thereto, although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any skilled person in the art can modify or easily think of changes to the technical solutions recorded in the foregoing embodiments within the technical range disclosed by the present application, or make equivalent replacement to part of the technical features; and these modifications, changes or replacements do not make the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application. All should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for aggregating feature descriptors of consecutive frame images, characterized in that, include: Obtain feature descriptors of consecutive frame images of a target reference frame image in a scene database, wherein the target reference frame image is any reference frame image in the scene database, and the consecutive frame images include: the target reference frame image, k frame images before the target reference frame image, and k frame images after the target reference frame image; The feature descriptors of the consecutive frame images are summed, and the summed feature descriptors are normalized to obtain the aggregated feature descriptor of the target reference frame image, and then the aggregated feature descriptor of each reference frame image in the scene database is obtained, so as to use the aggregated feature descriptor of each reference frame image in the scene database for loop closure detection. Among them, obtaining feature descriptors of consecutive frame images of the target reference frame image in the scene database includes: The feature descriptor for each frame in the continuous frame images is obtained using the following method: The target image is semantically segmented, and the segmented regions in the semantic segmentation result are layered according to each category information to obtain a binary image corresponding to each category information. The target image is any frame image in the continuous frame images. Extract the skeleton corresponding to each type of information from the binary image, and determine the key points and center points corresponding to each type of information based on the extracted skeleton; The shape context algorithm is used to encode the key points and center points corresponding to each type of information, so as to obtain the feature descriptors of the key points and the center points corresponding to each type of information; Spatial aggregation is performed on the feature descriptors of key points and center points corresponding to each type of information to obtain the feature descriptors corresponding to each type of information; The feature descriptors corresponding to various categories of information are straightened and normalized to obtain the feature descriptors of the target image.

2. The method according to claim 1, characterized in that, The segmented regions in the semantic segmentation results are processed hierarchically according to each category of information, including: The segmented regions of dynamic category information in the semantic segmentation results are set as background regions to obtain the first preprocessed semantic segmentation results; According to the preset category information merging strategy, the segmented regions of different category information in the first preprocessed semantic segmentation result are merged to obtain the second preprocessed semantic segmentation result. In the semantic segmentation result after the second preprocessing, an initial binary image corresponding to each category of information is constructed by using the segmented region of each category of information as the foreground and the remaining segmented regions as the background. Image morphology processing is performed on the initial binary image corresponding to each category of information to obtain the intermediate binary image corresponding to each category of information. In the intermediate binary image corresponding to each category of information, the independent segmented regions within the closed segmented regions are filled according to a preset filling strategy to obtain the binary image corresponding to each category of information.

3. The method according to claim 1, characterized in that, Extract the skeleton corresponding to each type of category information from the binary image, including: The OpenCV skeleton refinement algorithm is used to extract the skeleton corresponding to each type of information from the binary image.

4. The method according to claim 1, characterized in that, Based on the extracted skeleton, the key points and center points corresponding to this category of information are determined, including: The endpoints and intersections of the skeleton are used as key points corresponding to this type of category information; The center coordinates of the key points are calculated based on their position coordinates to obtain the center point.

5. The method according to claim 1, characterized in that, Spatial aggregation is performed on the feature descriptors of key points and center points corresponding to each category of information, including: The feature descriptor of each key point is compared with the feature descriptor of the center point to obtain multiple difference results. The summation results are obtained by summing the multiple difference results. The summation result is normalized to obtain the feature descriptor corresponding to each category of information.

6. The method according to claim 1, characterized in that, The method further includes: Obtain the feature descriptor of the current frame image; Calculate the inner product between the feature descriptor of the current frame image and the aggregated feature descriptor of each reference frame image in the scene database; The largest inner product is determined among the inner products, and the reference frame image corresponding to the largest inner product is used as the matching reference frame image to match the current frame image; The position coordinates of the matching reference frame image are used as the position coordinates of the current frame image.

7. An apparatus for aggregating feature descriptors of consecutive frame images, characterized in that, include: The acquisition unit is used to acquire feature descriptors of consecutive frame images of a target reference frame image in a scene database, wherein the target reference frame image is any reference frame image in the scene database, and the consecutive frame images include: the target reference frame image, k frame images before the target reference frame image, and k frame images after the target reference frame image; An aggregation unit is used to sum the feature descriptors of the consecutive frame images and normalize the summed feature descriptors to obtain the aggregated feature descriptor of the target reference frame image, and then obtain the aggregated feature descriptor of each reference frame image in the scene database, so as to use the aggregated feature descriptor of each reference frame image in the scene database for loop closure detection. The acquisition unit is further configured to: acquire the feature descriptor of each frame in the continuous frame images according to the following method: perform semantic segmentation on the target image, and perform layered processing on the segmented regions in the semantic segmentation result according to each category information to obtain a binary image corresponding to each category information, wherein the target image is any frame image in the continuous frame images; extract the skeleton corresponding to each category information from the binary image corresponding to each category information, and determine the key points and center points corresponding to each category information based on the extracted skeleton; encode the key points and center points corresponding to each category information using a shape context algorithm to obtain the feature descriptor of the key points and the feature descriptor of the center points corresponding to each category information; perform spatial aggregation on the feature descriptor of the key points and the feature descriptor of the center points corresponding to each category information to obtain the feature descriptor corresponding to each category information; straighten and normalize the feature descriptors corresponding to various categories information to obtain the feature descriptor of the target image.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores machine-executable instructions that, when invoked and executed by a processor, cause the processor to perform the method according to any one of claims 1 to 6.

Images