Image feature extraction method, device and electronic equipment

Abstract

The application provides an image feature extraction method, device and electronic equipment, the image feature extraction method is based on the semantic segmentation result to construct the binary graph corresponding to each static category information, and then the skeleton corresponding to the category information is extracted from the binary graph, then the skeleton topology graph corresponding to the category information is constructed based on the skeleton, finally, the description of the node and the description of the edge in the skeleton topology graph are extracted, and the image feature corresponding to the category information in the key frame to be processed is obtained. In the above process, even when the appearance changes dramatically, the skeleton corresponding to each category information extracted based on the semantic segmentation result is stable and accurate, therefore, the description of the node and the description of the edge in the skeleton topology graph obtained subsequently are stable and accurate, that is, the method of the application can also extract stable and robust image features when the appearance changes dramatically, and the stability and accuracy of the subsequent loop detection are improved.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method, apparatus and electronic device for extracting image features. Background Technology

[0002] Simultaneous Localization and Mapping (SLAM) technology provides real-time positioning information and a map of the surrounding environment for autonomous vehicle navigation systems. Loop closure detection (also known as scene recognition) is a crucial component of SLAM. It eliminates accumulated errors and accurately locates the current position using current observation data and a previous scene database (i.e., a map), allowing the autonomous vehicle's navigation system to restart at any previously visited location. Traditional loop closure detection methods establish a correspondence between encoded feature descriptors of the retrieved frame and pre-stored feature descriptors in the scene database. These feature descriptors are typically generated in binary encoding, and the distance between the retrieved frame's feature descriptors and the pre-stored feature descriptors in the scene database is calculated as a similarity score. For example, the bag-of-words model is most commonly used in SLAM systems. However, drastic changes in the environment can lead to inaccurate encoded feature descriptors of the retrieved frame. Consequently, when matching these descriptors with the pre-stored feature descriptors in the scene database (which stores the correspondence between feature descriptors and position coordinates in the global coordinate system), accurate matching fails, thus hindering accurate positioning. Therefore, stable and accurate loop closure detection is the core issue in loop closure detection across various scenarios. Autonomous vehicles should operate 24 / 7 in all scenarios and should not be limited by changes in appearance.

[0003] The core of the aforementioned problem lies in how to extract stable and accurate features as descriptors for scene description, thus transforming this problem into an image retrieval problem. Graph-based methods can encode the relationships between landmarks, and by matching feature descriptors, the relationships between the retrieved image and reference images in the scene database can be obtained. Integrating scene-specific information into the graph provides robustness and accuracy when dealing with the same scene with different appearances. With the continuous advancement of deep learning technology, some deep neural networks have been developed for loop closure detection. Semantic segmentation-based methods heavily rely on the quality of semantic segmentation, and the segmentation boundaries are uncertain (e.g., the boundary of the same tree differs between winter and summer), which affects the stability and accuracy of loop closure detection.

[0004] In summary, for loop closure detection, how to extract stable and accurate image features has become a pressing technical problem that needs to be solved. Summary of the Invention

[0005] In view of this, the purpose of the present invention is to provide a method, apparatus and electronic device for extracting image features, so as to alleviate the technical problem that the prior art cannot extract stable and accurate image features.

[0006] In a first aspect, embodiments of the present invention provide a method for extracting image features, comprising:

[0007] Acquire keyframes to be processed captured by the target camera, and perform semantic segmentation on the keyframes to be processed to obtain semantic segmentation results, wherein the semantic segmentation results include at least one segmentation region and category information corresponding to each segmentation region;

[0008] In the semantic segmentation result, a binary image corresponding to each static category information is constructed by using the segmented region of each static category information as the foreground and the remaining segmented regions as the background.

[0009] Extract the skeleton corresponding to each type of static category information from the binary graph corresponding to that category information, and construct the skeleton topology graph corresponding to that category information based on the skeleton;

[0010] The nodes of the skeleton topology graph are encoded using a shape context algorithm to obtain node descriptors, and the length and orientation angle of the edges in the skeleton topology graph are used as descriptors of the edges in the skeleton topology graph.

[0011] The descriptors of the nodes and the descriptors of the edges are used as image features corresponding to this type of information in the keyframe to be processed, thereby obtaining the image features of the keyframe to be processed.

[0012] Furthermore, extracting the skeleton corresponding to each type of static category information from the binary image includes:

[0013] The OpenCV skeleton refinement algorithm is used to extract the skeleton corresponding to each type of static category information from the binary image.

[0014] Furthermore, constructing a skeleton topology graph corresponding to this type of category information based on the skeleton includes:

[0015] The endpoints and intersections of the skeleton are used as nodes in the skeleton topology graph;

[0016] The nodes of the skeleton topology graph are triangulated to obtain the edges of the skeleton topology graph, and thus the skeleton topology graph is obtained.

[0017] Furthermore, the method also includes:

[0018] A graph matching algorithm based on random walks is used to match the image features corresponding to each category of information in the keyframe to be processed with the image features of the corresponding category of information in each reference frame in the scene database, so as to obtain the matching score of each category of information in the keyframe to be processed with the corresponding category of information in each reference frame.

[0019] Calculate the average of the matching scores between the various categories of information in the key frame to be processed and the corresponding category information of each reference frame, and then obtain the matching score between the key frame to be processed and each reference frame.

[0020] The target reference frame that matches the key frame to be processed is determined based on the matching score between the key frame to be processed and each reference frame.

[0021] The position coordinates of the target reference frame are used as the position coordinates of the key frame to be processed.

[0022] Further, determining the target reference frame that matches the keyframe to be processed based on the matching score between the keyframe to be processed and each reference frame includes:

[0023] The highest matching score is determined from the matching scores;

[0024] The reference frame corresponding to the highest matching score is used as the target reference frame.

[0025] Furthermore, before performing the matching calculation of the image features corresponding to each category information in the keyframe to be processed and the image features of the corresponding category information in each reference frame in the scene database using the graph matching algorithm based on random walk, the method further includes:

[0026] In each reference frame in the scene database, a candidate reference frame is determined;

[0027] The graph matching algorithm based on random walk performs matching calculations on the image features corresponding to each category information in the keyframe to be processed and the image features of the corresponding category information in each reference frame in the scene database. This includes: performing matching calculations on the image features corresponding to each category information in the keyframe to be processed and the image features of the corresponding category information in each candidate reference frame to obtain matching scores between the various category information in the keyframe to be processed and the corresponding category information in each candidate reference frame.

[0028] Furthermore, semantic segmentation is performed on the keyframes to be processed, including:

[0029] The keyframe to be processed is semantically segmented using a semantic segmentation model to obtain at least one segmented region, and the at least one segmented region is used as the semantic segmentation result, wherein each segmented region corresponds to a target object in the keyframe to be processed.

[0030] Secondly, embodiments of the present invention also provide an image feature extraction apparatus, comprising:

[0031] The acquisition and semantic segmentation unit is used to acquire key frames to be processed captured by the target camera, and to perform semantic segmentation on the key frames to be processed to obtain semantic segmentation results, wherein the semantic segmentation results include at least one segmentation region and category information corresponding to each segmentation region.

[0032] Binary graph construction unit is used to construct a binary graph corresponding to each static category information in the semantic segmentation result, with the segmented region of each static category information as the foreground and the remaining segmented regions as the background.

[0033] The skeleton extraction unit is used to extract the skeleton corresponding to each type of static category information from the binary graph corresponding to that type of category information, and to construct the skeleton topology graph corresponding to that type of category information based on the skeleton.

[0034] The encoding unit is used to encode the nodes of the skeleton topology graph using a shape context algorithm to obtain the descriptors of the nodes, and to use the length and orientation angle of the edges in the skeleton topology graph as the descriptors of the edges in the skeleton topology graph.

[0035] The setting unit is used to use the descriptors of the nodes and the descriptors of the edges as image features corresponding to the category information in the key frame to be processed, thereby obtaining the image features of the key frame to be processed.

[0036] Thirdly, embodiments of the present invention also provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method described in any of the first aspects above.

[0037] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing machine-executable instructions, which, when invoked and executed by a processor, cause the processor to perform the method described in any of the first aspects above.

[0038] In this embodiment of the invention, an image feature extraction method is provided, comprising: acquiring a keyframe to be processed captured by a target camera, and performing semantic segmentation on the keyframe to be processed to obtain a semantic segmentation result, wherein the semantic segmentation result includes at least one segmentation region and category information corresponding to each segmentation region; in the semantic segmentation result, constructing a binary map corresponding to each static category information with the segmentation region of each static category information as the foreground and the remaining segmentation regions as the background; extracting the skeleton corresponding to each category information from the binary map corresponding to each static category information, and constructing a skeleton topology map corresponding to each category information based on the skeleton; encoding the nodes of the skeleton topology map using a shape context algorithm to obtain the node descriptors, and using the length and orientation angle of the edges in the skeleton topology map as the edge descriptors in the skeleton topology map; using the node descriptors and the edge descriptors as the image features corresponding to the category information in the keyframe to be processed, thereby obtaining the image features of the keyframe to be processed. As described above, the image feature extraction method of this invention constructs a binary image corresponding to each static category information based on the semantic segmentation results, then extracts the skeleton corresponding to that category information from it, subsequently constructs a skeleton topology graph corresponding to that category information based on the skeleton, and finally extracts the descriptors of the nodes and edges in the skeleton topology graph to obtain the image features corresponding to that category information in the keyframe to be processed. During the above image feature extraction process, even when the appearance changes drastically, the skeleton corresponding to each category information extracted based on the semantic segmentation results remains stable and accurate. Therefore, the descriptors of the nodes and edges in the subsequently obtained skeleton topology graph are stable and accurate. That is, the image feature extraction method of this invention can extract stable and robust image features even when the appearance changes drastically, improving the stability and accuracy of subsequent loop closure detection and alleviating the technical problem that existing technologies cannot extract stable and accurate image features. Attached Figure Description

[0039] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0040] Figure 1 A flowchart illustrating an image feature extraction method provided in an embodiment of the present invention;

[0041] Figure 2 A schematic diagram of a keyframe to be processed provided in an embodiment of the present invention;

[0042] Figure 3 Provided for embodiments of the present invention Figure 2 A schematic diagram of the semantic segmentation results corresponding to the keyframes to be processed in the image;

[0043] Figure 4 This is a schematic diagram of a binary image corresponding to the tree category information provided in an embodiment of the present invention;

[0044] Figure 5 A schematic diagram of the skeleton topology map corresponding to the tree category information provided in the embodiments of the present invention;

[0045] Figure 6 A flowchart of a graph matching method provided in an embodiment of the present invention;

[0046] Figure 7 A schematic diagram of an image feature extraction device provided in an embodiment of the present invention;

[0047] Figure 8 This is a schematic diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0048] The technical solution of the present invention will be clearly and completely described below with reference to the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0049] In existing technologies, when extracting features from images of the same scene before and after a drastic change in appearance, the extracted image features are different. In other words, existing technologies cannot extract stable and accurate image features from the same scene with a drastic change in appearance.

[0050] Based on this, the image feature extraction method of the present invention constructs a binary map corresponding to each static category information based on the semantic segmentation results, then extracts the skeleton corresponding to that category information from it, subsequently constructs a skeleton topology map corresponding to that category information based on the skeleton, and finally extracts the descriptors of the nodes and edges in the skeleton topology map to obtain the image features corresponding to that category information in the keyframe to be processed. In the above image feature extraction process, even when the appearance changes drastically, the skeleton corresponding to each category information extracted based on the semantic segmentation results remains stable and accurate. Therefore, the descriptors of the nodes and edges in the subsequently obtained skeleton topology map are stable and accurate. That is, the image feature extraction method of the present invention can extract stable and robust image features even when the appearance changes drastically, improving the stability and accuracy of subsequent loop closure detection.

[0051] To facilitate understanding of this embodiment, a method for extracting image features disclosed in this embodiment of the invention will first be described in detail.

[0052] Example 1:

[0053] According to an embodiment of the present invention, an embodiment of an image feature extraction method is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0054] Figure 1 This is a flowchart of an image feature extraction method according to an embodiment of the present invention, such as... Figure 1 As shown, the method includes the following steps:

[0055] Step S102: Obtain the key frame to be processed captured by the target camera, and perform semantic segmentation on the key frame to be processed to obtain the semantic segmentation result. The semantic segmentation result includes at least one segmentation region and category information corresponding to each segmentation region.

[0056] In this embodiment of the invention, the target camera can be a camera on an autonomous vehicle, and the key frame to be processed can be an image frame obtained by sampling from the image captured by the target camera at a preset sampling frequency.

[0057] like Figure 2 As shown, a schematic diagram of the keyframe to be processed is presented. Figure 3 It shows Figure 2 The semantic segmentation results corresponding to the keyframes to be processed in the image.

[0058] Specifically, the semantic segmentation model is used to perform semantic segmentation on the keyframe to be processed, resulting in at least one segmented region. This at least one segmented region is then used as the semantic segmentation result, where each segmented region corresponds to a target object in the keyframe to be processed.

[0059] Step S104: In the semantic segmentation result, a binary image corresponding to each static category information is constructed using the segmented region of each static category information as the foreground and the remaining segmented regions as the background.

[0060] The static category information mentioned above can be understood as stable category information that does not move over time, such as trees, grass, buildings, and roads. However, some moving category information (e.g., pedestrians, vehicles) is ignored and treated as background. Figure 3The semantic segmentation result shown uses the segmented regions containing tree category information as the foreground and the remaining segmented regions as the background to construct a binary map corresponding to the tree category information, as follows. Figure 4 As shown. Of course, we can also construct binary maps corresponding to other categories of information, thus obtaining the binary map corresponding to each category of information in the keyframe to be processed.

[0061] Step S106: Extract the skeleton corresponding to each type of static category information from the binary graph corresponding to that type of category information, and construct the skeleton topology graph corresponding to that type of category information based on the skeleton.

[0062] Specifically, such as Figure 4 As shown, the skeleton corresponding to the tree category information is extracted from the binary graph corresponding to the tree category information, and then a skeleton topology graph corresponding to the tree category information is constructed based on the extracted skeleton, as shown in the figure. Figure 5 As shown.

[0063] The extracted skeleton can adapt to drastic changes in appearance. That is, even when the appearance changes drastically, the extracted skeleton remains stable and accurate, and consequently, the image features extracted based on the skeleton are also stable and robust.

[0064] Step S108: The shape context algorithm is used to encode the nodes of the skeleton topology graph to obtain the node descriptors, and the length and orientation angle of the edges in the skeleton topology graph are used as the descriptors of the edges in the skeleton topology graph.

[0065] The following is based on Image features representing tree category information, where, and Represents a matrix containing nodes and edges; This represents the association matrix between nodes. When node i and node j are connected by edge e, a ie =a je =1, This represents the descriptor matrix of nodes generated by the shape context algorithm. The node descriptor describes how the other nodes are distributed around the current node. Specifically, in log-polar coordinates, N equidistant concentric circles are drawn with the current node as the center and radius R (e.g., 10 equidistant concentric circles with radius 10, i.e., 10 concentric circles with radii of 1, 2, 3, ..., 10 respectively, with a distance of 1 between adjacent concentric circles). These concentric circles are then divided into M equal parts, and the M×N regions are encoded. The number of nodes appearing in each of the M×N regions is the element in the current node's descriptor. After obtaining the descriptors of all nodes, they are normalized to obtain the final node descriptor. The length and orientation angle of the edges in the skeleton topology graph are used as the descriptors of the edges in the skeleton topology graph.

[0066] Step S110: The descriptors of the nodes and the descriptors of the edges are used as the image features corresponding to the category information in the key frame to be processed, thereby obtaining the image features of the key frame to be processed.

[0067] The image features of the keyframe to be processed are the image features corresponding to various categories of information in the keyframe to be processed, namely the descriptors of nodes and edges corresponding to various categories of information.

[0068] As can be seen, this invention treats each category as a separate binary image, and then extracts the image features corresponding to that category from each binary image. Essentially, the image features extracted from each binary image describe the feature description information of objects of that category in the scene corresponding to the keyframe to be processed.

[0069] In this embodiment of the invention, an image feature extraction method is provided, comprising: acquiring a keyframe to be processed captured by a target camera, and performing semantic segmentation on the keyframe to be processed to obtain a semantic segmentation result, wherein the semantic segmentation result includes at least one segmentation region and category information corresponding to each segmentation region; in the semantic segmentation result, constructing a binary map corresponding to each static category information with the segmentation region of each static category information as the foreground and the remaining segmentation regions as the background; extracting the skeleton corresponding to each category information from the binary map corresponding to each static category information, and constructing a skeleton topology map corresponding to each category information based on the skeleton; encoding the nodes of the skeleton topology map using a shape context algorithm to obtain the node descriptors, and using the length and orientation angle of the edges in the skeleton topology map as the edge descriptors in the skeleton topology map; using the node descriptors and the edge descriptors as the image features corresponding to the category information in the keyframe to be processed, thereby obtaining the image features of the keyframe to be processed. As described above, the image feature extraction method of this invention constructs a binary image corresponding to each static category information based on the semantic segmentation results, then extracts the skeleton corresponding to that category information from it, subsequently constructs a skeleton topology graph corresponding to that category information based on the skeleton, and finally extracts the descriptors of the nodes and edges in the skeleton topology graph to obtain the image features corresponding to that category information in the keyframe to be processed. During the above image feature extraction process, even when the appearance changes drastically, the skeleton corresponding to each category information extracted based on the semantic segmentation results remains stable and accurate. Therefore, the descriptors of the nodes and edges in the subsequently obtained skeleton topology graph are stable and accurate. That is, the image feature extraction method of this invention can extract stable and robust image features even when the appearance changes drastically, improving the stability and accuracy of subsequent loop closure detection and alleviating the technical problem that existing technologies cannot extract stable and accurate image features.

[0070] The above provides a brief overview of the image feature extraction method of the present invention. The specific details involved are described in detail below.

[0071] In an optional embodiment of the present invention, extracting the skeleton corresponding to each type of static category information from the binary image specifically includes:

[0072] The OpenCV skeleton refinement algorithm is used to extract the skeleton corresponding to each type of static category information from the binary image.

[0073] In an optional embodiment of the present invention, constructing a skeleton topology graph corresponding to this type of category information based on the skeleton specifically includes the following steps:

[0074] (1) Use the endpoints and intersections of the skeleton as nodes of the skeleton topology graph;

[0075] (2) Triangulate the nodes of the skeleton topology graph to obtain the edges of the skeleton topology graph, and then obtain the skeleton topology graph.

[0076] The image feature extraction method of this invention is based on semantic skeletons, which can stably describe image features despite changes in environmental conditions such as season and lighting. Unlike other semantic-based methods, this invention proposes a feature extraction method based on semantic skeleton graph features. This approach benefits from a novel hierarchical image description method (binary graphs corresponding to various categories of information) and a unique feature extraction method.

[0077] In an optional embodiment of the present invention, such as Figure 6 As shown, the method also includes:

[0078] Step S601: The graph matching algorithm based on random walk is used to match the image features corresponding to each category of information in the key frame to be processed with the image features of the corresponding category of information in each reference frame in the scene database to obtain the matching score of each category of information in the key frame to be processed with the corresponding category of information in each reference frame.

[0079] Specifically, the scene database contains the image features of each reference frame, the information of each category of each reference frame, and the correspondence between the position coordinates of each reference frame.

[0080] Step S602: Calculate the average of the matching scores between the various categories of information in the key frame to be processed and the corresponding category information of each reference frame, and then obtain the matching score between the key frame to be processed and each reference frame.

[0081] To facilitate a deeper understanding of this process, a specific example will be used below:

[0082] Assuming the scene database contains 10 reference frames, each with 5 categories: category A, category B, category C, category D, and category E, and the keyframe to be processed also has 5 categories: category A, category B, category C, category D, and category E, then a random walk graph matching algorithm is used to match the image features corresponding to category A information in the keyframe to be processed with the image features of category A information in the first reference frame in the scene database. This yields the matching score between category A information in the keyframe to be processed and category A information in the first reference frame. Similarly, the matching score for category A information in the keyframe to be processed can be obtained. The matching scores are calculated as follows: the matching score of category B information with category B information of the first reference frame; the matching score of category C information of the keyframe to be processed with category C information of the first reference frame; the matching score of category D information of the keyframe to be processed with category D information of the first reference frame; and the matching score of category E information of the keyframe to be processed with category E information of the first reference frame. Then, the average of the matching scores of category A, B, C, D, and E information of the keyframe to be processed with the corresponding category information of each reference frame is calculated. This yields the matching score between the keyframe to be processed and the first reference frame.

[0083] For subsequent reference frames, the same calculation process is performed to obtain the matching score between the key frame to be processed and each reference frame.

[0084] Step S603: Determine the target reference frame that matches the key frame to be processed based on the matching score between the key frame to be processed and each reference frame.

[0085] Specifically, the process of determining the target reference frame that matches the key frame to be processed based on the matching score between the key frame to be processed and each reference frame includes: determining the highest matching score in the matching scores; and taking the reference frame corresponding to the highest matching score as the target reference frame.

[0086] Step S604: Use the position coordinates of the target reference frame as the position coordinates of the key frame to be processed.

[0087] In an optional embodiment of the present invention, before the graph matching algorithm based on random walk performs matching calculations on the image features corresponding to each category information in the keyframe to be processed and the image features of the corresponding category information of each reference frame in the scene database, the method further includes:

[0088] In each reference frame in the scene database, a candidate reference frame is determined;

[0089] The graph matching algorithm based on random walks performs matching calculations between the image features corresponding to each category of information in the keyframe to be processed and the image features corresponding to the category of information in each reference frame in the scene database. This includes: matching the image features corresponding to each category of information in the keyframe to be processed with the image features corresponding to the category of information in each candidate reference frame, obtaining matching scores between the various categories of information in the keyframe to be processed and the corresponding categories of information in each candidate reference frame; then, calculating the average of the matching scores between the various categories of information in the keyframe to be processed and the corresponding categories of information in each candidate reference frame, thus obtaining the matching score between the keyframe to be processed and each candidate reference frame; determining the target candidate reference frame that matches the keyframe to be processed based on the matching scores between the keyframe to be processed and each candidate reference frame; and using the position coordinates of the target candidate reference frame as the position coordinates of the keyframe to be processed.

[0090] This process only performs matching calculations on candidate reference frames, without having to perform matching calculations on every reference frame, which greatly reduces the computational load of image matching.

[0091] The feature extraction and graph matching process of this invention (i.e., the loop closure detection process) can continue to work in scenarios with drastic changes in appearance, even when the traditional bag-of-words model fails. Furthermore, it can maintain a very high precision rate (the loop closure frames found are very accurate) with a 100% recall rate, demonstrating that the loop closure detection of this invention has the ability to resist changes in environmental appearance.

[0092] Example 2:

[0093] This invention also provides an image feature extraction device, which is mainly used to perform the image feature extraction method provided in Embodiment 1 of this invention. The image feature extraction device provided in this invention will be described in detail below.

[0094] Figure 7 This is a schematic diagram of an image feature extraction device according to an embodiment of the present invention. The device mainly includes: an acquisition and semantic segmentation unit 10, a binary image construction unit 20, a skeleton extraction unit 30, an encoding unit 40, and a setting unit 50, wherein:

[0095] The acquisition and semantic segmentation unit is used to acquire the key frame to be processed captured by the target camera and perform semantic segmentation on the key frame to be processed to obtain the semantic segmentation result. The semantic segmentation result includes at least one segmentation region and category information corresponding to each segmentation region.

[0096] Binary graph construction unit is used to construct a binary graph corresponding to each static category information in the semantic segmentation result, with the segmented region of each static category information as the foreground and the remaining segmented regions as the background.

[0097] The skeleton extraction unit is used to extract the skeleton corresponding to each type of static category information from the binary graph corresponding to that category information, and to construct the skeleton topology graph corresponding to that category information based on the skeleton.

[0098] The encoding unit is used to encode the nodes of the skeleton topology graph using the shape context algorithm to obtain the descriptors of the nodes, and uses the length and orientation angle of the edges in the skeleton topology graph as the descriptors of the edges in the skeleton topology graph.

[0099] The setting unit is used to take the descriptors of nodes and edges as image features corresponding to the type of information in the key frame to be processed, and thus obtain the image features of the key frame to be processed.

[0100] In this embodiment of the invention, an image feature extraction device is provided, comprising: acquiring a keyframe to be processed captured by a target camera, and performing semantic segmentation on the keyframe to be processed to obtain a semantic segmentation result, wherein the semantic segmentation result includes at least one segmentation region and category information corresponding to each segmentation region; in the semantic segmentation result, constructing a binary map corresponding to each static category information with the segmentation region of each static category information as the foreground and the remaining segmentation regions as the background; extracting the skeleton corresponding to each category information from the binary map corresponding to each static category information, and constructing a skeleton topology map corresponding to the category information based on the skeleton; encoding the nodes of the skeleton topology map using a shape context algorithm to obtain the node descriptors, and using the length and orientation angle of the edges in the skeleton topology map as the edge descriptors in the skeleton topology map; using the node descriptors and the edge descriptors as the image features corresponding to the category information in the keyframe to be processed, thereby obtaining the image features of the keyframe to be processed. As described above, the image feature extraction device of the present invention constructs a binary map corresponding to each static category information based on the semantic segmentation results, then extracts the skeleton corresponding to that category information from it, subsequently constructs a skeleton topology map corresponding to that category information based on the skeleton, and finally extracts the descriptors of the nodes and edges in the skeleton topology map to obtain the image features corresponding to that category information in the keyframe to be processed. During the above image feature extraction process, even when the appearance changes drastically, the skeleton corresponding to each category information extracted based on the semantic segmentation results remains stable and accurate. Therefore, the descriptors of the nodes and edges in the subsequently obtained skeleton topology map are stable and accurate. That is, the image feature extraction method of the present invention can extract stable and robust image features even when the appearance changes drastically, improving the stability and accuracy of subsequent loop closure detection and alleviating the technical problem that existing technologies cannot extract stable and accurate image features.

[0101] Optionally, the skeleton extraction unit is also used to: extract the skeleton corresponding to each type of category information from the binary image corresponding to each type of static category information using the OpenCV skeleton refinement algorithm.

[0102] Optionally, the skeleton extraction unit is also used to: use the endpoints and intersections of the skeleton as nodes of the skeleton topology graph; perform triangulation on the nodes of the skeleton topology graph to obtain the edges of the skeleton topology graph, and thus obtain the skeleton topology graph.

[0103] Optionally, the device is further configured to: perform matching calculations on the image features corresponding to each category of information in the keyframe to be processed and the image features of the corresponding category of information in each reference frame in the scene database using a graph matching algorithm based on random walks, to obtain matching scores between the various categories of information in the keyframe to be processed and the corresponding category of information in each reference frame; calculate the average value of the matching scores between the various categories of information in the keyframe to be processed and the corresponding category of information in each reference frame, thereby obtaining the matching score between the keyframe to be processed and each reference frame; determine the target reference frame that matches the keyframe to be processed based on the matching scores between the keyframe to be processed and each reference frame; and use the position coordinates of the target reference frame as the position coordinates of the keyframe to be processed.

[0104] Optionally, the device is also used to: determine the highest matching score in the matching scores; and use the reference frame corresponding to the highest matching score as the target reference frame.

[0105] Optionally, the device is further configured to: determine candidate reference frames in each reference frame in the scene database; and perform matching calculations between the image features corresponding to each category of information in the key frame to be processed and the image features of the corresponding category of information in each reference frame in the scene database based on a graph matching algorithm of random walk, including: performing matching calculations between the image features corresponding to each category of information in the key frame to be processed and the image features of the corresponding category of information in each candidate reference frame based on a graph matching algorithm of random walk, to obtain matching scores between the various category information in the key frame to be processed and the corresponding category information in each candidate reference frame.

[0106] Optionally, the acquisition and semantic segmentation unit is further configured to: perform semantic segmentation on the keyframe to be processed using a semantic segmentation model to obtain at least one segmented region, and use the at least one segmented region as the semantic segmentation result, wherein each segmented region corresponds to a target object in the keyframe to be processed.

[0107] The device provided in this embodiment of the invention has the same implementation principle and technical effect as the aforementioned method embodiment. For the sake of brevity, any parts not mentioned in the device embodiment can be referred to the corresponding content in the aforementioned method embodiment.

[0108] like Figure 8As shown in the figure, an electronic device 600 provided in this application includes: a processor 601, a memory 602 and a bus. The memory 602 stores machine-readable instructions that can be executed by the processor 601. When the electronic device is running, the processor 601 communicates with the memory 602 through the bus. The processor 601 executes the machine-readable instructions to perform the steps of the image feature extraction method described above.

[0109] Specifically, the memory 602 and processor 601 mentioned above can be general-purpose memory and processor, without any specific limitations. When the processor 601 runs the computer program stored in the memory 602, it can execute the above-mentioned image feature extraction method.

[0110] The processor 601 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of the processor 601 or by instructions in software form. The processor 601 may be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it may also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly manifested as execution by a hardware decoding processor, or execution by a combination of hardware and software modules in the decoding processor. The software module can reside in a mature storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory 602, and processor 601 reads the information from memory 602 and, in conjunction with its hardware, completes the steps of the above method.

[0111] Corresponding to the above-described image feature extraction method, this application embodiment also provides a computer-readable storage medium storing machine-executable instructions. When the machine-executable instructions are invoked and executed by a processor, the machine-executable instructions cause the processor to perform the steps of the above-described image feature extraction method.

[0112] The image feature extraction device provided in this application embodiment can be specific hardware on a device or software or firmware installed on the device. The implementation principle and technical effects of the device provided in this application embodiment are the same as those in the foregoing method embodiments. For the sake of brevity, any parts not mentioned in the device embodiment can be referred to the corresponding content in the foregoing method embodiments. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can all be referred to the corresponding processes in the above method embodiments, and will not be repeated here.

[0113] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0114] For example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0115] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0116] In addition, the functional units in the embodiments provided in this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0117] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause an electronic device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the vehicle marking method described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0118] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In addition, the terms "first", "second", "third", etc. are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0119] Finally, it should be noted that the above-described embodiments are merely specific implementations of this application, used to illustrate the technical solutions of this application, and not to limit them. The protection scope of this application is not limited thereto. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this application; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application. All should be covered within the protection scope of this application. Therefore, the protection scope of this application should be determined by the protection scope of the claims.

Claims

1. A method for extracting image features, characterized in that, include: Acquire keyframes to be processed captured by the target camera, and perform semantic segmentation on the keyframes to be processed to obtain semantic segmentation results, wherein the semantic segmentation results include at least one segmentation region and category information corresponding to each segmentation region; In the semantic segmentation result, a binary image corresponding to each static category information is constructed by using the segmented region of each static category information as the foreground and the remaining segmented regions as the background. Extract the skeleton corresponding to each type of static category information from the binary graph corresponding to that category information, and construct the skeleton topology graph corresponding to that category information based on the skeleton; The nodes of the skeleton topology graph are encoded using a shape context algorithm to obtain node descriptors, and the length and orientation angle of the edges in the skeleton topology graph are used as descriptors of the edges in the skeleton topology graph. The descriptors of the nodes and the descriptors of the edges are used as image features corresponding to this type of information in the keyframe to be processed, thereby obtaining the image features of the keyframe to be processed.

2. The method according to claim 1, characterized in that, Extracting the skeleton corresponding to each type of static category information from the binary image includes: The OpenCV skeleton refinement algorithm is used to extract the skeleton corresponding to each type of static category information from the binary image.

3. The method according to claim 1, characterized in that, Based on the skeleton, construct the skeleton topology graph corresponding to this type of category information, including: The endpoints and intersections of the skeleton are used as nodes in the skeleton topology graph; The nodes of the skeleton topology graph are triangulated to obtain the edges of the skeleton topology graph, and thus the skeleton topology graph is obtained.

4. The method according to claim 1, characterized in that, After obtaining the image features of the keyframe to be processed, the method further includes: A graph matching algorithm based on random walks is used to match the image features corresponding to each category of information in the keyframe to be processed with the image features of the corresponding category of information in each reference frame in the scene database, so as to obtain the matching score of each category of information in the keyframe to be processed with the corresponding category of information in each reference frame. Calculate the average of the matching scores between the various categories of information in the key frame to be processed and the corresponding category information of each reference frame, and then obtain the matching score between the key frame to be processed and each reference frame. The target reference frame that matches the key frame to be processed is determined based on the matching score between the key frame to be processed and each reference frame. The position coordinates of the target reference frame are used as the position coordinates of the key frame to be processed.

5. The method according to claim 4, characterized in that, Determining the target reference frame that matches the keyframe to be processed based on the matching score between the keyframe to be processed and each reference frame includes: The highest matching score is determined from the matching scores; The reference frame corresponding to the highest matching score is used as the target reference frame.

6. The method according to claim 4, characterized in that, Before performing matching calculations on the image features corresponding to each category information in the keyframe to be processed and the image features of the corresponding category information in each reference frame in the scene database using a graph matching algorithm based on random walk, the method further includes: In each reference frame in the scene database, a candidate reference frame is determined; The graph matching algorithm based on random walk performs matching calculations on the image features corresponding to each category information in the keyframe to be processed and the image features of the corresponding category information in each reference frame in the scene database. This includes: performing matching calculations on the image features corresponding to each category information in the keyframe to be processed and the image features of the corresponding category information in each candidate reference frame to obtain matching scores between the various category information in the keyframe to be processed and the corresponding category information in each candidate reference frame.

7. The method according to claim 1, characterized in that, Semantic segmentation of the keyframes to be processed includes: The keyframe to be processed is semantically segmented using a semantic segmentation model to obtain at least one segmented region, and the at least one segmented region is used as the semantic segmentation result, wherein each segmented region corresponds to a target object in the keyframe to be processed.

8. An image feature extraction device, characterized in that, include: The acquisition and semantic segmentation unit is used to acquire key frames to be processed captured by the target camera, and to perform semantic segmentation on the key frames to be processed to obtain semantic segmentation results, wherein the semantic segmentation results include at least one segmentation region and category information corresponding to each segmentation region. Binary graph construction unit is used to construct a binary graph corresponding to each static category information in the semantic segmentation result, with the segmented region of each static category information as the foreground and the remaining segmented regions as the background. The skeleton extraction unit is used to extract the skeleton corresponding to each type of static category information from the binary graph corresponding to that type of category information, and to construct the skeleton topology graph corresponding to that type of category information based on the skeleton. The encoding unit is used to encode the nodes of the skeleton topology graph using a shape context algorithm to obtain the descriptors of the nodes, and to use the length and orientation angle of the edges in the skeleton topology graph as the descriptors of the edges in the skeleton topology graph. The setting unit is used to use the descriptors of the nodes and the descriptors of the edges as image features corresponding to the category information in the key frame to be processed, thereby obtaining the image features of the key frame to be processed.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores machine-executable instructions that, when invoked and executed by a processor, cause the processor to perform the method according to any one of claims 1 to 7.

Images