Image feature extraction method, device and electronic equipment
By performing semantic segmentation and hierarchical processing on images, extracting skeletons and encoding feature descriptors, the problem of unstable image features in scenes with drastic appearance changes is solved, and efficient image matching is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- NEUSOFT REACH AUTOMOBILE TECH (SHENYANG) CO LTD
- Filing Date
- 2022-07-21
- Publication Date
- 2026-07-10
Smart Images
Figure CN115205550B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image processing technology, and in particular to a method, apparatus and electronic device for extracting image features. Background Technology
[0002] Autonomous driving technology is currently a hot topic, and Simultaneous Localization and Mapping (SLAM) technology can build a model of the surrounding environment while an autonomous vehicle is driving, using sensors without prior environmental information, and simultaneously calculate the vehicle's own pose. When returning to a previously visited environment, the vehicle can use the previously stored pose data to adjust its subsequent pose in that environment. One of the most critical tasks in this process is correctly identifying whether the vehicle is currently in a previously visited scene, and Visual Place Recognition (VPR) is a component of achieving this task.
[0003] Visual Retrieval Programming (VPR) retrieves images that highly match the current scene from a stored collection of visual data. Feature-based VPR stores image feature points and their corresponding descriptors. During retrieval, the similarity between images is calculated by matching their descriptors, thus determining whether the user has visited the scene before. A descriptor describes the key features of an image in a deliberately designed way. For traditional appearance-based VPR methods, several local descriptors have been proposed, such as SIFT, SURF, BRIEF, and BRISK. Although these descriptors are rotation-, illumination-, and scale-invariant, they are unstable when handling features with appearance variations. Furthermore, the computation of key points corresponding to descriptors significantly impacts matching efficiency and accuracy during image retrieval. Therefore, a tree-based search algorithm based on keypoint descriptors such as the bag-of-words algorithm is proposed. When constructing the search tree, the bag-of-words algorithm clusters all given descriptors using k-means, and repeats k-means multiple times to form a tree. For the query image, all its descriptors are grouped into vectors according to the pre-trained bag-of-words algorithm. Another method to speed up image retrieval is to aggregate local descriptors of an image into a fixed-dimensional global descriptor, such as the Vector of Locally Aggregated Descriptor (VLAD). With VLAD, all descriptors of an image are aggregated into a global descriptor by summing the descriptor residuals. The similarity between two images can then be easily calculated by comparing their global descriptors. While the bag-of-words approach only considers the coordinate distribution of descriptors, VLAD considers the spatial relationships between them.
[0004] In recent years, several descriptors that fuse semantic and scene appearance information have been proposed to address scenarios with changing appearances. However, these methods may still fail when dealing with scenes experiencing drastic changes in appearance. These methods also neglect the shape and spatial layout of semantic objects, as well as the relationships between elements of the same category and between elements of different categories. When used for drastically changing scenes, such as different seasons, these descriptors are prone to failure. Several methods also utilize the edges of semantically segmented regions to describe images, but these methods all rely on good and stable semantic segmentation results.
[0005] In summary, existing technologies cannot extract stable and robust image features in scenes with drastic changes in appearance, and the extracted image features are computationally intensive, slow, and inefficient when used for subsequent image matching. Summary of the Invention
[0006] In view of this, the purpose of the present invention is to provide an image feature extraction method, apparatus and electronic device to alleviate the technical problems of existing technologies being unable to extract stable and robust image features in scenarios with drastic changes in appearance, and the high computational load, slow speed and low efficiency of the extracted image features in subsequent image matching.
[0007] In a first aspect, embodiments of the present invention provide a method for extracting image features, comprising:
[0008] Semantic segmentation is performed on the keyframes to be processed captured by the target camera, and the segmented regions in the semantic segmentation results are processed in layers according to each category information to obtain the binary map corresponding to each category information.
[0009] Extract the skeleton corresponding to each type of information from the binary image, and determine the key points and center points corresponding to each type of information based on the extracted skeleton;
[0010] The shape context algorithm is used to encode the key points and center points corresponding to each type of information, so as to obtain the feature descriptors of the key points and the center points corresponding to each type of information;
[0011] Spatial aggregation is performed on the feature descriptors of key points and center points corresponding to each type of information to obtain the feature descriptors corresponding to each type of information;
[0012] The feature descriptors corresponding to various categories of information are straightened and normalized to obtain the feature descriptors of the keyframe to be processed.
[0013] Furthermore, the segmented regions in the semantic segmentation results are processed hierarchically according to each category of information, including:
[0014] The segmented regions of dynamic category information in the semantic segmentation results are set as background regions to obtain the first preprocessed semantic segmentation results;
[0015] According to the preset category information merging strategy, the segmented regions of different category information in the first preprocessed semantic segmentation result are merged to obtain the second preprocessed semantic segmentation result.
[0016] In the semantic segmentation result after the second preprocessing, an initial binary image corresponding to each category of information is constructed by using the segmented region of each category of information as the foreground and the remaining segmented regions as the background.
[0017] Image morphology processing is performed on the initial binary image corresponding to each category of information to obtain the intermediate binary image corresponding to each category of information.
[0018] In the intermediate binary image corresponding to each category of information, the independent segmented regions within the closed segmented regions are filled according to a preset filling strategy to obtain the binary image corresponding to each category of information.
[0019] Furthermore, the skeleton corresponding to each type of category information is extracted from the binary image corresponding to that category information, including:
[0020] The OpenCV skeleton refinement algorithm is used to extract the skeleton corresponding to each type of information from the binary image.
[0021] Furthermore, based on the extracted skeleton, the key points and center points corresponding to this category of information are determined, including:
[0022] The endpoints and intersections of the skeleton are used as key points corresponding to this type of category information;
[0023] The center coordinates of the key points are calculated based on their position coordinates to obtain the center point.
[0024] Furthermore, spatial aggregation is performed on the feature descriptors of the key points and the feature descriptors of the center points corresponding to each category of information, including:
[0025] The feature descriptor of each key point is compared with the feature descriptor of the center point to obtain multiple difference results.
[0026] The summation results are obtained by summing the multiple difference results.
[0027] The summation result is normalized to obtain the feature descriptor corresponding to each category of information.
[0028] Furthermore, the feature descriptors corresponding to various categories of information are straightened and normalized, including:
[0029] The feature descriptors corresponding to the various categories of information are concatenated to obtain the concatenated feature descriptors;
[0030] The spliced feature descriptors are normalized to obtain the feature descriptors of the keyframe to be processed.
[0031] Furthermore, the method also includes:
[0032] Calculate the inner product between the feature descriptor of the keyframe to be processed and the feature descriptor of each reference frame in the scene database;
[0033] The maximum inner product is determined from the inner products, and the reference frame corresponding to the maximum inner product is used as the target reference frame to match the key frame to be processed.
[0034] The position coordinates of the target reference frame are used as the position coordinates of the key frame to be processed.
[0035] Secondly, embodiments of the present invention also provide an image feature extraction apparatus, comprising:
[0036] The semantic segmentation and hierarchical processing unit is used to perform semantic segmentation on the keyframes to be processed captured by the target camera, and to perform hierarchical processing on the segmented regions in the semantic segmentation results according to each category information to obtain the binary map corresponding to each category information.
[0037] The extraction and determination unit is used to extract the skeleton corresponding to each type of information from the binary image corresponding to each type of information, and determine the key points and center points corresponding to each type of information based on the extracted skeleton;
[0038] The encoding unit is used to encode the key points and center points corresponding to each type of information using the shape context algorithm, so as to obtain the feature descriptors of the key points and the center points corresponding to each type of information;
[0039] The spatial aggregation unit is used to spatially aggregate the feature descriptors of key points and center points corresponding to each type of information to obtain the feature descriptors corresponding to each type of information.
[0040] The straightening and normalization unit is used to straighten and normalize the feature descriptors corresponding to various categories of information to obtain the feature descriptors of the keyframe to be processed.
[0041] Thirdly, embodiments of the present invention also provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method described in any of the first aspects above.
[0042] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing machine-executable instructions, which, when invoked and executed by a processor, cause the processor to perform the method described in any of the first aspects above.
[0043] In this embodiment of the invention, an image feature extraction method is provided, comprising: performing semantic segmentation on keyframes to be processed captured by a target camera, and performing layered processing on the segmented regions in the semantic segmentation results according to each category of information to obtain a binary image corresponding to each category of information; extracting the skeleton corresponding to each category of information from the binary image corresponding to each category of information, and determining the key points and center points corresponding to each category of information based on the extracted skeleton; encoding the key points and center points corresponding to each category of information using a shape context algorithm to obtain feature descriptors of the key points and center points corresponding to each category of information; spatially aggregating the feature descriptors of the key points and center points corresponding to each category of information to obtain feature descriptors corresponding to each category of information; and straightening and normalizing the feature descriptors corresponding to various categories of information to obtain feature descriptors of the keyframes to be processed. As described above, the image feature extraction method of the present invention obtains a binary image corresponding to each category of information based on the semantic segmentation result, and then extracts the skeleton corresponding to that category of information from it. Subsequently, the key points and center points corresponding to that category of information are determined based on the skeleton, and the feature descriptors of the key points and center points corresponding to that category of information are encoded. Finally, the feature descriptors of the key points and center points are spatially aggregated to obtain the feature descriptors corresponding to that category of information. Then, the feature descriptors corresponding to various categories of information are straightened and normalized to obtain the feature descriptors of the keyframe to be processed. In the above image feature extraction process, even when the appearance changes drastically, the skeleton corresponding to each category of information extracted based on the semantic segmentation results remains stable and accurate. Therefore, the feature descriptors of key points and center points corresponding to each category of information obtained subsequently are stable and accurate. Consequently, the feature descriptors of the final keyframe to be processed are also stable and robust. In addition, the feature descriptors of the keyframe to be processed are obtained by spatially aggregating the feature descriptors of key points and center points, and then straightening and normalizing the feature descriptors corresponding to various categories of information after spatial aggregation. That is, the feature descriptors of the keyframe to be processed are simpler. In subsequent image matching, the computational load is greatly reduced, the speed of image matching is accelerated, and the efficiency of image matching is improved. This alleviates the technical problem that existing technologies cannot extract stable and robust image features in scenarios with drastic appearance changes, and that the extracted image features are computationally intensive, slow, and inefficient in subsequent image matching. Attached Figure Description
[0044] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.
[0045] Figure 1 A flowchart illustrating an image feature extraction method provided in an embodiment of the present invention;
[0046] Figure 2 This is a schematic diagram illustrating the process of obtaining a binary image corresponding to each category of information from a keyframe to be processed, as provided in an embodiment of the present invention.
[0047] Figure 3 A flowchart of a method for spatially aggregating feature descriptors of key points and center points corresponding to each type of information, provided in an embodiment of the present invention;
[0048] Figure 4 A schematic diagram of an image feature extraction device provided in an embodiment of the present invention;
[0049] Figure 5 This is a schematic diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation
[0050] The technical solution of the present invention will be clearly and completely described below with reference to the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0051] Existing technologies cannot extract stable and robust image features in scenes with drastic changes in appearance, and the extracted image features are computationally intensive, slow, and inefficient when used for subsequent image matching.
[0052] Based on this, the image feature extraction method of the present invention obtains a binary image corresponding to each category of information by layering based on the semantic segmentation result, and then extracts the skeleton corresponding to the category of information from it. Subsequently, the key points and center points corresponding to the category of information are determined based on the skeleton, and the feature descriptors of the key points and center points corresponding to the category of information are encoded. Finally, the feature descriptors of the key points and center points are spatially aggregated to obtain the feature descriptors corresponding to the category of information. Then, the feature descriptors corresponding to various categories of information are straightened and normalized to obtain the feature descriptors of the key frames to be processed. During the image feature extraction process described above, even when the appearance changes drastically, the skeleton corresponding to each category of information extracted based on the semantic segmentation results remains stable and accurate. Therefore, the feature descriptors of key points and center points corresponding to each category of information obtained subsequently are stable and accurate. Consequently, the feature descriptors of the final keyframe to be processed are also stable and robust. Furthermore, the feature descriptors of the keyframe to be processed are obtained by spatially aggregating the feature descriptors of key points and center points, and then straightening and normalizing the feature descriptors corresponding to various categories of information after spatial aggregation. That is, the feature descriptors of the keyframe to be processed are simpler, which greatly reduces the amount of computation, speeds up the image matching process, and improves the efficiency of image matching.
[0053] To facilitate understanding of this embodiment, a method for extracting image features disclosed in this embodiment of the invention will first be described in detail.
[0054] Example 1:
[0055] According to an embodiment of the present invention, an embodiment of an image feature extraction method is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.
[0056] Figure 1 This is a flowchart of an image feature extraction method according to an embodiment of the present invention, such as... Figure 1 As shown, the method includes the following steps:
[0057] Step S102: Semantic segmentation is performed on the keyframes to be processed captured by the target camera, and the segmented regions in the semantic segmentation results are processed in layers according to each category information to obtain the binary map corresponding to each category information.
[0058] In this embodiment of the invention, the target camera can be a camera on an autonomous vehicle, and the key frame to be processed can be an image frame obtained by sampling from the image captured by the target camera at a preset sampling frequency.
[0059] like Figure 2 As shown, this illustrates the process of obtaining a binary map corresponding to each category of information from the keyframes to be processed.
[0060] Specifically, by using an open-source CNN model to perform semantic segmentation on the keyframes to be processed, a semantic segmentation result is obtained, which includes at least one segmentation region and the category information corresponding to each segmentation region.
[0061] The above-mentioned layered processing of segmented regions in semantic segmentation results according to each category information refers to dividing pixel blocks with the same category label into the same layer, setting the value of their pixel points to 1, and setting the value of the remaining pixel points to 0, thus obtaining the binary image corresponding to each category information.
[0062] The process of layered processing will be described in detail below, and will not be repeated here.
[0063] Step S104: Extract the skeleton corresponding to each type of information from the binary image corresponding to each type of information, and determine the key points and center points corresponding to each type of information based on the extracted skeleton.
[0064] The extracted skeleton can adapt to drastic changes in appearance. That is, even when the appearance changes drastically, the extracted skeleton remains stable and accurate. Furthermore, the key points and center points corresponding to this category information are also stable and robust when determined based on the extracted skeleton.
[0065] Step S106: Use the shape context algorithm to encode the key points and center points corresponding to each type of information to obtain the feature descriptors of the key points and center points corresponding to each type of information.
[0066] Specifically, taking a keypoint pi as a reference point, within a local area, draw N equidistant concentric circles with radius R centered on keypoint pi (e.g., draw 10 equidistant concentric circles with radius 10, i.e., draw 10 concentric circles with radii of 1, 2, 3, ..., 10 respectively, with a distance of 1 between adjacent concentric circles). Divide the concentric circles into M equal parts to form a target-shaped template. Number each block of the target-shaped template, resulting in a total of M×N blocks. Count the number of keypoints and center points appearing within the M×N blocks, which are the elements in the feature descriptor of keypoint pi. This yields the feature descriptor of keypoint pi. The feature descriptor of the center point is obtained using the same method.
[0067] Step S108: Spatial aggregation is performed on the feature descriptors of key points and center points corresponding to each type of information to obtain the feature descriptors corresponding to each type of information;
[0068] Specifically, referencing the VLAD concept, the feature descriptors of all keypoints and the feature descriptors of the centroids corresponding to a certain category of information are aggregated into a single feature descriptor for that category. During spatial aggregation, the centroids corresponding to that category are used for aggregation. The aggregation process will be described in detail below.
[0069] Step S110: Straighten and normalize the feature descriptors corresponding to various categories of information to obtain the feature descriptors of the keyframe to be processed.
[0070] Specifically, by straightening and normalizing the feature descriptors corresponding to all categories of information, we obtain the final feature descriptor of the keyframe to be processed. This feature descriptor of the keyframe to be processed is simpler, which reduces the computational load of subsequent image matching and improves the speed and efficiency of image matching.
[0071] In this embodiment of the invention, an image feature extraction method is provided, comprising: performing semantic segmentation on keyframes to be processed captured by a target camera, and performing layered processing on the segmented regions in the semantic segmentation results according to each category of information to obtain a binary image corresponding to each category of information; extracting the skeleton corresponding to each category of information from the binary image corresponding to each category of information, and determining the key points and center points corresponding to each category of information based on the extracted skeleton; encoding the key points and center points corresponding to each category of information using a shape context algorithm to obtain feature descriptors of the key points and center points corresponding to each category of information; spatially aggregating the feature descriptors of the key points and center points corresponding to each category of information to obtain feature descriptors corresponding to each category of information; and straightening and normalizing the feature descriptors corresponding to various categories of information to obtain feature descriptors of the keyframes to be processed. As described above, the image feature extraction method of the present invention obtains a binary image corresponding to each category of information based on the semantic segmentation result, and then extracts the skeleton corresponding to that category of information from it. Subsequently, the key points and center points corresponding to that category of information are determined based on the skeleton, and the feature descriptors of the key points and center points corresponding to that category of information are encoded. Finally, the feature descriptors of the key points and center points are spatially aggregated to obtain the feature descriptors corresponding to that category of information. Then, the feature descriptors corresponding to various categories of information are straightened and normalized to obtain the feature descriptors of the keyframe to be processed. In the above image feature extraction process, even when the appearance changes drastically, the skeleton corresponding to each category of information extracted based on the semantic segmentation results remains stable and accurate. Therefore, the feature descriptors of key points and center points corresponding to each category of information obtained subsequently are stable and accurate. Consequently, the feature descriptors of the final keyframe to be processed are also stable and robust. In addition, the feature descriptors of the keyframe to be processed are obtained by spatially aggregating the feature descriptors of key points and center points, and then straightening and normalizing the feature descriptors corresponding to various categories of information after spatial aggregation. That is, the feature descriptors of the keyframe to be processed are simpler. In subsequent image matching, the computational load is greatly reduced, the speed of image matching is accelerated, and the efficiency of image matching is improved. This alleviates the technical problem that existing technologies cannot extract stable and robust image features in scenarios with drastic appearance changes, and that the extracted image features are computationally intensive, slow, and inefficient in subsequent image matching.
[0072] The above provides a brief overview of the image feature extraction method of the present invention. The specific details involved are described in detail below.
[0073] In an optional embodiment of the present invention, step S102 above, which involves performing hierarchical processing on the segmented regions in the semantic segmentation result according to each type of information, specifically includes the following steps:
[0074] (1) Set the segmented region of dynamic category information in the semantic segmentation result as the background region to obtain the semantic segmentation result after the first preprocessing;
[0075] Specifically, to minimize the impact of missegmentation, a series of refinements are needed to the semantic segmentation results. The semantic segmentation results inevitably contain some dynamic category information segmentation regions that cannot be used as scene features, such as pedestrian or vehicle segmentation regions. These segmentation regions cannot serve as stable scene features (they do not move over time, such as trees, grass, buildings, roads, etc.), so these dynamic category information segmentation regions need to be ignored, i.e., set as background regions, thus obtaining the first preprocessed semantic segmentation result.
[0076] (2) Merge the segmented regions of different category information in the first preprocessed semantic segmentation result according to the preset category information merging strategy to obtain the second preprocessed semantic segmentation result;
[0077] Specifically, the aforementioned pre-defined category information merging strategy requires that segmentation regions of frequently confused category information be merged. For example, the segmentation regions of "wall" and "fence" should be merged, and the merged regions yield the second pre-processed semantic segmentation result.
[0078] (3) In the semantic segmentation result after the second preprocessing, the segmentation region of each category information is used as the foreground and the remaining segmentation regions are used as the background to construct the initial binary image corresponding to each category information.
[0079] like Figure 2 As shown, a binary image corresponding to the road category information is constructed by using the segmented region of road category information as the foreground and the remaining segmented regions as the background; a binary image corresponding to the building category information is constructed by using the segmented region of building category information as the foreground and the remaining segmented regions as the background; a binary image corresponding to the plant category information is constructed by using the segmented region of plant category information as the foreground and the remaining segmented regions as the background. Thus, we obtain... Figure 2 The initial binary graphs corresponding to the three categories of information shown in the figure.
[0080] (4) Perform image morphology processing on the initial binary image corresponding to each category of information to obtain the intermediate binary image corresponding to each category of information;
[0081] Specifically, in order to eliminate noise and connect adjacent regions, image morphology processing, such as dilation and erosion, is performed on the initial binary image corresponding to each type of information to obtain the intermediate binary image corresponding to each type of information.
[0082] (5) In the intermediate binary image corresponding to each category of information, fill the independent segmented regions in the closed segmented regions according to the preset filling strategy to obtain the binary image corresponding to each category of information.
[0083] Specifically, to further reduce the impact of missegmentation on subsequent skeleton extraction, small "holes" within closed segmented regions are filled, and some small segmented regions that remain independent of the whole after the previous processing are removed. For example... Figure 2 As shown, the top row of the binary image is an enlarged view of the boxed portion of the corresponding binary image below it. The circled areas are filled (blank areas) or removed (background areas).
[0084] In an optional embodiment of the present invention, extracting the skeleton corresponding to each type of category information from the binary image corresponding to that type of category information specifically includes:
[0085] The OpenCV skeleton refinement algorithm is used to extract the skeleton corresponding to each category of information from the binary image.
[0086] In an optional embodiment of the present invention, determining the key points and center points corresponding to this type of category information based on the extracted skeleton specifically includes the following steps:
[0087] (1) The endpoints and intersections of the skeleton are taken as the key points corresponding to this type of information;
[0088] (2) Calculate the center coordinates of the key points based on their position coordinates to obtain the center point.
[0089] Specifically, if the key points corresponding to a category of information are P = {p1, p2, ..., p...} |key-points| In this context, |key-points| represents the total number of keypoints corresponding to this type of information. Furthermore, the center point corresponding to this type of information is the coordinate of the center position of the aforementioned keypoints. For example, if there are 10 keypoints, the x-coordinates of these 10 keypoints are summed and averaged to obtain the x-coordinate of the center point, and the y-coordinates of these 10 keypoints are summed and averaged to obtain the y-coordinate of the center point. Thus, the center point is calculated.
[0090] In an alternative embodiment of the present invention, reference is made to... Figure 3Step S108 above involves spatial aggregation of the feature descriptors of key points and center points corresponding to each category of information, specifically including the following steps:
[0091] Step S301: Calculate the difference between the feature descriptor of each key point and the feature descriptor of the center point to obtain multiple difference results;
[0092] Step S302: Summate the multiple difference results to obtain the summation result;
[0093] Step S303: Normalize the summation result to obtain the feature descriptor corresponding to each category of information.
[0094] The above process can be represented as: Where V(k) represents the feature descriptor corresponding to k categories of information, k represents a category of information, and d i d represents the feature descriptor of the i-th key point. c The feature descriptor representing the center point, |key-points| k This represents the total number of key points corresponding to category k information.
[0095] The feature descriptor corresponding to each category of information obtained above is an encoding of the two-dimensional spatial layout relationship of the objects of each category of information, and is a local descriptor.
[0096] To encode the relationships between objects of different categories, it is necessary to process the feature descriptors corresponding to various categories. In an optional embodiment of the present invention, the feature descriptors corresponding to various categories are straightened and normalized, specifically including the following steps:
[0097] (1) Concatenate the feature descriptors corresponding to various categories of information to obtain the concatenated feature descriptors;
[0098] (2) Normalize the spliced feature descriptors to obtain the feature descriptors of the keyframes to be processed.
[0099] For example, if the feature descriptor corresponding to one category of information is a 256-dimensional vector, then concatenating the feature descriptors corresponding to the five categories will result in a 256*5-dimensional vector. Thus, the final feature descriptor of the keyframe to be processed is a fixed-dimensional multi-dimensional descriptor vector, which is the global descriptor.
[0100] In the image feature extraction method of this invention, the feature descriptors of the final keyframes to be processed describe the spatial distribution relationships between semantic objects of the same category and between semantic objects of different categories in the scene. Unlike other methods, in the method of this invention, spatial distribution refers to the distribution of pixels with the same semantic category in the image. This invention encodes the above spatial distribution relationships, extracts the key points of the semantic skeleton of each category into a local descriptor of an image, and then, drawing on the idea of local aggregated vector descriptors, aggregates the local semantic skeleton representations of each category into a fixed-dimensional global descriptor to represent the entire image.
[0101] In an optional embodiment of the present invention, the method further includes the following steps:
[0102] (1) Calculate the inner product between the feature descriptor of the keyframe to be processed and the feature descriptor of each reference frame in the scene database;
[0103] Specifically, the scene database contains multiple reference frames, feature descriptors for each reference frame, and the correspondence between the position coordinates of each reference frame.
[0104] (2) Determine the maximum inner product in the inner product and use the reference frame corresponding to the maximum inner product as the target reference frame to be matched with the key frame to be processed;
[0105] (3) Use the position coordinates of the target reference frame as the position coordinates of the key frame to be processed.
[0106] In the above image matching process, the time complexity of matching an image is fixed and will not change with the environment. The process is fast and can meet the real-time requirements.
[0107] The method of this invention was compared with three publicly available urban datasets: SYNTHIA, Oxford Robot Car, and Extended-CMU Seasons. These datasets all have the common feature of having multiple sets of data on the same scene with changes in season, light, and weather.
[0108] Experimental results show that the method of this invention performs excellently on three datasets with transformed appearances, especially achieving an AUC of 0.99 (out of a maximum of 1) on the SYNTHIA dataset. This indicates that the method of this invention (which can be called "Semantic Skeleton Representation Vector of Locally Aggregated Descriptor", or SSR-VLAD for short) can balance accuracy and recall in VPR tasks. Furthermore, for datasets with poor semantic segmentation performance, performing hierarchical processing on the segmented regions in the semantic segmentation results before extracting the skeleton can effectively improve the robustness of this invention. In addition, experiments show that the extracted skeleton has a certain resistance to segmentation noise even with added noise.
[0109] Regarding computational overhead, the computational overhead of the method of the present invention was statistically analyzed. Experimental results show that the method of the present invention only requires about 150ms to encode an image. When performing matching, more than 100 pairs of images can be matched in 1ms, and the time consumption is very stable and does not change with the scene.
[0110] Example 2:
[0111] This invention also provides an image feature extraction device, which is mainly used to perform the image feature extraction method provided in Embodiment 1 of this invention. The image feature extraction device provided in this invention will be described in detail below.
[0112] Figure 4 This is a schematic diagram of an image feature extraction device according to an embodiment of the present invention. The device mainly includes: a semantic segmentation and hierarchical processing unit 10, an extraction and determination unit 20, an encoding unit 30, a spatial aggregation unit 40, and a straightening and normalization unit 50, wherein:
[0113] The semantic segmentation and hierarchical processing unit is used to perform semantic segmentation on the keyframes to be processed captured by the target camera, and to perform hierarchical processing on the segmented regions in the semantic segmentation results according to each category information to obtain the binary map corresponding to each category information.
[0114] The extraction and determination unit is used to extract the skeleton corresponding to each type of information from the binary image corresponding to each type of information, and determine the key points and center points corresponding to each type of information based on the extracted skeleton;
[0115] The encoding unit is used to encode the key points and center points corresponding to each type of information using the shape context algorithm, so as to obtain the feature descriptors of the key points and the center points corresponding to each type of information;
[0116] The spatial aggregation unit is used to spatially aggregate the feature descriptors of key points and center points corresponding to each type of information to obtain the feature descriptors corresponding to each type of information.
[0117] The straightening and normalization unit is used to straighten and normalize the feature descriptors corresponding to various categories of information to obtain the feature descriptors of the keyframe to be processed.
[0118] In this embodiment of the invention, an image feature extraction device is provided, comprising: performing semantic segmentation on a keyframe to be processed captured by a target camera, and performing layered processing on the segmented regions in the semantic segmentation result according to each type of information to obtain a binary image corresponding to each type of information; extracting the skeleton corresponding to each type of information from the binary image corresponding to each type of information, and determining the key points and center points corresponding to each type of information based on the extracted skeleton; encoding the key points and center points corresponding to each type of information using a shape context algorithm to obtain feature descriptors of the key points and center points corresponding to each type of information; spatially aggregating the feature descriptors of the key points and center points corresponding to each type of information to obtain feature descriptors corresponding to each type of information; and straightening and normalizing the feature descriptors corresponding to various types of information to obtain the feature descriptors of the keyframe to be processed. As described above, the image feature extraction device of the present invention obtains a binary image corresponding to each category of information based on the semantic segmentation result, and then extracts the skeleton corresponding to that category of information from it. Subsequently, the key points and center points corresponding to that category of information are determined based on the skeleton, and the feature descriptors of the key points and center points corresponding to that category of information are encoded. Finally, the feature descriptors of the key points and center points are spatially aggregated to obtain the feature descriptors corresponding to that category of information. Then, the feature descriptors corresponding to various categories of information are straightened and normalized to obtain the feature descriptors of the keyframe to be processed. In the above image feature extraction process, even when the appearance changes drastically, the skeleton corresponding to each category of information extracted based on the semantic segmentation results remains stable and accurate. Therefore, the feature descriptors of key points and center points corresponding to each category of information obtained subsequently are stable and accurate. Consequently, the feature descriptors of the final keyframe to be processed are also stable and robust. In addition, the feature descriptors of the keyframe to be processed are obtained by spatially aggregating the feature descriptors of key points and center points, and then straightening and normalizing the feature descriptors corresponding to various categories of information after spatial aggregation. That is, the feature descriptors of the keyframe to be processed are simpler. In subsequent image matching, the computational load is greatly reduced, the speed of image matching is accelerated, and the efficiency of image matching is improved. This alleviates the technical problem that existing technologies cannot extract stable and robust image features in scenarios with drastic appearance changes, and that the extracted image features are computationally intensive, slow, and inefficient in subsequent image matching.
[0119] Optionally, the semantic segmentation and hierarchical processing unit is further configured to: set the segmented regions of dynamic category information in the semantic segmentation result as background regions to obtain a first preprocessed semantic segmentation result; merge the segmented regions of different category information in the first preprocessed semantic segmentation result according to a preset category information merging strategy to obtain a second preprocessed semantic segmentation result; in the second preprocessed semantic segmentation result, construct an initial binary image corresponding to each category information with the segmented regions of each category information as foreground and the remaining segmented regions as background; perform image morphological processing on the initial binary image corresponding to each category information to obtain an intermediate binary image corresponding to each category information; and fill the independent segmented regions in the closed segmented regions in the intermediate binary image corresponding to each category information according to a preset filling strategy to obtain a binary image corresponding to each category information.
[0120] Optionally, the extraction and determination unit is also used to: extract the skeleton corresponding to each type of information from the binary image corresponding to each type of information using the OpenCV skeleton refinement algorithm.
[0121] Optionally, the extraction and determination unit is also used to: take the endpoints and intersections of the skeleton as key points corresponding to this type of information; calculate the center position coordinates of the key points based on the position coordinates of the key points to obtain the center point.
[0122] Optionally, the spatial aggregation unit is also used to: calculate the difference between the feature descriptor of each key point and the feature descriptor of the center point to obtain multiple difference results; sum the multiple difference results to obtain a summation result; and normalize the summation result to obtain the feature descriptor corresponding to each type of information.
[0123] Optionally, the straightening and normalization unit is also used to: concatenate the feature descriptors corresponding to various categories of information to obtain concatenated feature descriptors; and normalize the concatenated feature descriptors to obtain feature descriptors of the keyframe to be processed.
[0124] Optionally, the device is further configured to: calculate the inner product between the feature descriptor of the keyframe to be processed and the feature descriptor of each reference frame in the scene database; determine the maximum inner product in the inner product, and use the reference frame corresponding to the maximum inner product as the target reference frame that matches the keyframe to be processed; and use the position coordinates of the target reference frame as the position coordinates of the keyframe to be processed.
[0125] The device provided in this embodiment of the invention has the same implementation principle and technical effect as the aforementioned method embodiment. For the sake of brevity, any parts not mentioned in the device embodiment can be referred to the corresponding content in the aforementioned method embodiment.
[0126] like Figure 5As shown in the figure, an electronic device 600 provided in this application includes: a processor 601, a memory 602 and a bus. The memory 602 stores machine-readable instructions that can be executed by the processor 601. When the electronic device is running, the processor 601 communicates with the memory 602 through the bus. The processor 601 executes the machine-readable instructions to perform the steps of the image feature extraction method described above.
[0127] Specifically, the memory 602 and processor 601 mentioned above can be general-purpose memory and processor, without any specific limitations. When the processor 601 runs the computer program stored in the memory 602, it can execute the above-mentioned image feature extraction method.
[0128] The processor 601 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of the processor 601 or by instructions in software form. The processor 601 may be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it may also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly manifested as execution by a hardware decoding processor, or execution by a combination of hardware and software modules in the decoding processor. The software module can reside in a mature storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory 602, and processor 601 reads the information from memory 602 and, in conjunction with its hardware, completes the steps of the above method.
[0129] Corresponding to the above-described image feature extraction method, this application embodiment also provides a computer-readable storage medium storing machine-executable instructions. When the machine-executable instructions are invoked and executed by a processor, the machine-executable instructions cause the processor to perform the steps of the above-described image feature extraction method.
[0130] The image feature extraction device provided in this application embodiment can be specific hardware on a device or software or firmware installed on the device. The implementation principle and technical effects of the device provided in this application embodiment are the same as those in the foregoing method embodiments. For the sake of brevity, any parts not mentioned in the device embodiment can be referred to the corresponding content in the foregoing method embodiments. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can all be referred to the corresponding processes in the above method embodiments, and will not be repeated here.
[0131] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.
[0132] For example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.
[0133] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0134] In addition, the functional units in the embodiments provided in this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.
[0135] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause an electronic device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the vehicle marking method described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0136] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In addition, the terms "first", "second", "third", etc. are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.
[0137] Finally, it should be noted that the above-described embodiments are merely specific implementations of this application, used to illustrate the technical solutions of this application, and not to limit them. The protection scope of this application is not limited thereto. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this application; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application. All should be covered within the protection scope of this application. Therefore, the protection scope of this application should be determined by the protection scope of the claims.
Claims
1. A method for extracting image features, characterized in that, include: Semantic segmentation is performed on the keyframes to be processed captured by the target camera, and the segmented regions in the semantic segmentation results are processed in layers according to each category information to obtain the binary map corresponding to each category information. Extract the skeleton corresponding to each type of information from the binary image, and determine the key points and center points corresponding to each type of information based on the extracted skeleton; The shape context algorithm is used to encode the key points and center points corresponding to each type of information, so as to obtain the feature descriptors of the key points and the center points corresponding to each type of information; Spatial aggregation is performed on the feature descriptors of key points and center points corresponding to each type of information to obtain the feature descriptors corresponding to each type of information; The feature descriptors corresponding to various categories of information are straightened and normalized to obtain the feature descriptors of the key frame to be processed. Specifically, the segmented regions in the semantic segmentation results are processed hierarchically according to each category of information, including: The segmented regions of dynamic category information in the semantic segmentation results are set as background regions to obtain the first preprocessed semantic segmentation results; According to the preset category information merging strategy, the segmented regions of different category information in the first preprocessed semantic segmentation result are merged to obtain the second preprocessed semantic segmentation result. In the semantic segmentation result after the second preprocessing, an initial binary image corresponding to each category of information is constructed by using the segmented region of each category of information as the foreground and the remaining segmented regions as the background. Image morphology processing is performed on the initial binary image corresponding to each category of information to obtain the intermediate binary image corresponding to each category of information. In the intermediate binary image corresponding to each category of information, the independent segmented regions within the closed segmented regions are filled according to a preset filling strategy to obtain the binary image corresponding to each category of information.
2. The method according to claim 1, characterized in that, Extract the skeleton corresponding to each type of category information from the binary image, including: The OpenCV skeleton refinement algorithm is used to extract the skeleton corresponding to each type of information from the binary image.
3. The method according to claim 1, characterized in that, Based on the extracted skeleton, the key points and center points corresponding to this category of information are determined, including: The endpoints and intersections of the skeleton are used as key points corresponding to this type of category information; The center coordinates of the key points are calculated based on their position coordinates to obtain the center point.
4. The method according to claim 1, characterized in that, Spatial aggregation is performed on the feature descriptors of key points and center points corresponding to each category of information, including: The feature descriptor of each key point is compared with the feature descriptor of the center point to obtain multiple difference results. The summation results are obtained by summing the multiple difference results. The summation result is normalized to obtain the feature descriptor corresponding to each category of information.
5. The method according to claim 1, characterized in that, The feature descriptors corresponding to various categories of information are straightened and normalized, including: The feature descriptors corresponding to the various categories of information are concatenated to obtain the concatenated feature descriptors; The spliced feature descriptors are normalized to obtain the feature descriptors of the keyframe to be processed.
6. The method according to claim 1, characterized in that, The method further includes: Calculate the inner product between the feature descriptor of the keyframe to be processed and the feature descriptor of each reference frame in the scene database; The maximum inner product is determined from the inner products, and the reference frame corresponding to the maximum inner product is used as the target reference frame to match the key frame to be processed. The position coordinates of the target reference frame are used as the position coordinates of the key frame to be processed.
7. An image feature extraction device, characterized in that, include: The semantic segmentation and hierarchical processing unit is used to perform semantic segmentation on the keyframes to be processed captured by the target camera, and to perform hierarchical processing on the segmented regions in the semantic segmentation results according to each category information to obtain the binary map corresponding to each category information. The extraction and determination unit is used to extract the skeleton corresponding to each type of information from the binary image corresponding to each type of information, and determine the key points and center points corresponding to each type of information based on the extracted skeleton; The encoding unit is used to encode the key points and center points corresponding to each type of information using the shape context algorithm, so as to obtain the feature descriptors of the key points and the center points corresponding to each type of information; The spatial aggregation unit is used to spatially aggregate the feature descriptors of key points and center points corresponding to each type of information to obtain the feature descriptors corresponding to each type of information. The straightening and normalization unit is used to straighten and normalize the feature descriptors corresponding to various categories of information to obtain the feature descriptors of the key frame to be processed. The semantic segmentation and hierarchical processing unit is further configured to: set the segmented regions of dynamic category information in the semantic segmentation result as background regions to obtain a first preprocessed semantic segmentation result; merge the segmented regions of different category information in the first preprocessed semantic segmentation result according to a preset category information merging strategy to obtain a second preprocessed semantic segmentation result; in the second preprocessed semantic segmentation result, construct an initial binary image corresponding to each category information with the segmented regions of each category information as foreground and the remaining segmented regions as background; perform image morphological processing on the initial binary image corresponding to each category information to obtain an intermediate binary image corresponding to each category information; and fill the independent segmented regions in the closed segmented regions of the intermediate binary image corresponding to each category information according to a preset filling strategy to obtain the binary image corresponding to each category information.
8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.
9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores machine-executable instructions that, when invoked and executed by a processor, cause the processor to perform the method according to any one of claims 1 to 6.