Feature matching method and device, electronic equipment, vehicle and storage medium

Abstract

The invention discloses a feature matching method and device, electronic equipment, a vehicle and a storage medium. The method comprises the steps of obtaining current image frame data of a current scene where the vehicle is located; inputting the current image frame data into a feature point detection model for feature detection to obtain first feature point information of the current image frame data; if the frame type of the current image frame data is a key frame type, determining a feature matching strategy as a first feature matching strategy, and calling the first feature matching strategy to match the first feature point information with second feature point information of the interval image frame data to obtain a first matching result; if the frame type of the current image frame data is a non-key frame type, determining the feature matching strategy as a second feature matching strategy, and calling the second feature matching strategy to match the first feature point information with third feature point information of the previous image frame data to obtain a second matching result, the problem that the accuracy of feature matching of image frame data of a scene is low is solved.

Description

Technical Field

[0001] This application relates to the field of vehicles, and more specifically, to a feature matching method, apparatus, electronic device, vehicle, and storage medium. Background Technology

[0002] Currently, when performing feature matching on continuous image frame data of a vehicle scene, traditional feature matching models (e.g., feature matching models built on neural networks) are often used to perform feature matching on the aforementioned continuous image frame data.

[0003] However, the aforementioned feature matching methods are ill-suited for continuous image frame data in complex scenes. Furthermore, using these methods to extract features from continuous image frame data lacking texture also fails to yield effective scene features, resulting in low accuracy in feature matching of scene image frame data.

[0004] There is currently no effective solution to the technical problem of low accuracy in feature matching of image frame data in the above scenarios. Summary of the Invention

[0005] This invention provides a feature matching method, apparatus, electronic device, vehicle, and storage medium to at least address the technical problem of low accuracy in feature matching of image frame data in a scene.

[0006] According to one aspect of the present invention, a feature matching method is provided. The method includes: acquiring current image frame data of the current scene in which a vehicle is located; inputting the current image frame data into a feature point detection model for feature detection to obtain first feature point information of the current image frame data, wherein the first feature point information is used to represent the image corner feature of the current scene at the current moment under the corresponding scene category; if the frame type of the current image frame data is a keyframe type, determining the feature matching strategy as a first feature matching strategy, and calling the first feature matching strategy to match the first feature point information and second feature point information of the interval image frame data to obtain a first matching result, wherein the feature matching strategy is used to represent the feature matching rule to be executed on the first feature point information, the first feature matching strategy is used to represent the feature matching rule to be executed on the first feature point information and the second feature point information, and the historical moment corresponding to the interval image frame data is earlier than the current moment. In the previous moment, the second feature point information is used to represent the current scene at the historical moment, and the image corner features under the corresponding scene category. The first matching result is used to represent the feature matching degree between the first feature point information and the second feature point information. If the frame type of the current image frame data is a non-key frame type, then the feature matching strategy is determined to be the second feature matching strategy, and the second feature matching strategy is called to match the first feature point information and the third feature point information of the previous image frame data to obtain the second matching result. The second feature matching strategy is used to represent the feature matching rule to be executed on the first feature point information and the third feature point information. The previous moment corresponding to the previous image frame data is later than the historical moment and earlier than the current moment. The third feature point information is used to represent the current scene at the previous moment, and the image corner features under the corresponding scene category. The second matching result is used to represent the feature matching degree between the first feature point information and the third feature point information.

[0007] Based on the first feature point information representing the corner features of the image, the feature matching strategy corresponding to the frame type is called according to the different frame types of the current image frame data to match the first feature point information with the second feature point information of the interval image frame data, or to match the first feature point information with the third feature point information of the previous image frame data. This achieves the goal of extracting a sufficient number of effective feature points in textureless or weakly textured environments, thereby improving the accuracy of feature matching of image frame data in the scene.

[0008] Optionally, the current image frame data is input into a feature point detection model for feature detection to obtain the first feature point information of the current image frame data, including: inputting the current image frame data into the scale prediction layer of the feature point detection model for scale prediction to obtain scale prediction results corresponding to different scene categories, and inputting the current image frame data into the feature encoding layer of the feature point detection model for encoding to obtain the feature encoding result of the current image frame data; and fusing the scale prediction result and the feature encoding result to obtain the first feature point information.

[0009] By performing scale prediction and feature encoding on the current image frame data respectively, we can obtain scale prediction results and feature encoding results. Then, we can fuse the above scale prediction results and feature encoding results into image corner features under the corresponding scene category. This achieves the goal of obtaining the first feature point information of the current image frame data, thereby improving the technical effect of the first feature point information.

[0010] Optionally, a first feature matching strategy is invoked to match the first feature point information and the second feature point information of the interval image frame data to obtain a first matching result, including: performing equalization processing on the first feature point information; invoking the first feature matching strategy to match the processed first feature point information and the second feature point information in the bag-of-words database to obtain a first matching result.

[0011] By using a bag-of-words-based fast matching method to match the equalized first and second feature point information, the degree of feature matching between the first and second feature point information can be obtained, thereby achieving the goal of obtaining the first matching result and thus realizing the technical effect of improving the accuracy of the first matching result.

[0012] Optionally, the method further includes: inputting the scale prediction result and feature encoding result into the descriptor generator of the feature point detection model to generate a first descriptor information of the first feature point information, wherein the first descriptor information is used to describe the first feature point information; invoking a second feature matching strategy to match the first feature point information with the third feature point information of the previous image frame data to obtain a second matching result, including: invoking the second feature matching strategy, using projection matching combined with optical flow tracking to project the feature point corresponding to the third feature point information into the current image frame data to obtain a projection result, wherein the projection result is used to indicate whether the feature point corresponding to the third feature point information has been successfully projected into the current image frame data; in response to the projection result indicating the first feature point information, the method further includes: invoking the second feature matching strategy, using projection matching combined with optical flow tracking to project the feature point corresponding to the third feature point information into the current image frame data to obtain a projection result, wherein the projection result is used to indicate whether the feature point corresponding to the third feature point information has been successfully projected into the current image frame data; and in response to the projection result indicating the first feature point information, the method further includes: invoking the second feature matching strategy to generate a first descriptor information of the first feature point information, wherein the first descriptor information is used to describe the first feature point information ... second descriptor information of the first feature point information; invoking the second feature matching strategy to generate a second descriptor information of the first feature point information; and in response to the projection result indicating the The feature points corresponding to the three feature point information have been successfully projected into the current image frame data. The projection position of the feature point corresponding to the third feature point information in the current image frame data is determined. Within the current area where the projection position is located, based on the distance between the first descriptor information and the second descriptor information of the third feature point information, the target feature point is determined from multiple candidate feature points corresponding to the third feature point information. The feature matching degree between the fourth feature point information of the target feature point and the first feature point information is determined as the second matching result. The distance between the first descriptor information and the descriptor information of the target feature point is less than the distance between the first descriptor information and the descriptor information of any candidate feature point other than the target feature point among the multiple candidate feature points.

[0013] By using projection matching combined with optical flow tracking, the first feature point information and the third feature point information of the previous image frame data are matched to obtain the feature matching degree between the first feature point information and the third feature point information. This achieves the goal of obtaining the second matching result and thus improves the accuracy of the second matching result.

[0014] Optionally, the method further includes: in response to the failure to determine the target feature point from the feature points after optical flow tracking among multiple candidate feature points corresponding to the third feature point information, extracting the target feature point.

[0015] Since no map point corresponding to the previous frame is found in the current frame, the corresponding point is extracted from the points after optical flow tracking to supplement it, thereby achieving the purpose of determining the target feature point and thus realizing the technical effect of improving the accuracy of the target feature point.

[0016] Optionally, the method further includes: acquiring current image frame data samples of the current scene where the vehicle is located; using the current image frame data samples, performing scale distribution training on the initial feature point detection model, and reconstructing the initial loss function of the initial feature point detection model to obtain a feature point detection model, wherein the feature point detection model is used to perform feature detection on the current image frame data to obtain first feature point information.

[0017] By using the current image frame data samples to train the initial feature point detection model with scale distribution and reconstructing the initial loss function of the initial feature point detection model, and then using the trained and reconstructed initial feature point detection model as the final feature point detection model for feature detection of the current image frame data, the goal of training the initial feature point detection model and obtaining the feature point detection model is achieved, thereby realizing the technical effect of improving the robustness of the feature point detection model.

[0018] According to one aspect of the present invention, a feature matching apparatus is provided, the apparatus comprising: a first acquisition unit, configured to acquire current image frame data of the current scene in which the vehicle is located; a detection unit, configured to input the current image frame data into a feature point detection model for feature detection to obtain first feature point information of the current image frame data, wherein the first feature point information is used to represent the image corner point features of the current scene at the current moment under the corresponding scene category; and a first matching unit, configured to determine the feature matching strategy as a first feature matching strategy if the frame type of the current image frame data is a keyframe type, and to invoke the first feature matching strategy to match the first feature point information and second feature point information of the interval image frame data to obtain a first matching result, wherein the feature matching strategy is used to represent the feature matching rule to be executed on the first feature point information, and the first feature matching strategy is used to represent the feature matching rule to be executed on the first feature point information and the second feature point information, and the interval image frame data is used to represent the feature matching rule to be executed on the second feature point information. The historical moment is earlier than the current moment. The second feature point information is used to represent the current scene at the historical moment, and the image corner features under the corresponding scene category. The first matching result is used to represent the feature matching degree between the first feature point information and the second feature point information. The second matching unit is used to determine the feature matching strategy as the second feature matching strategy if the frame type of the current image frame data is a non-key frame type, and to call the second feature matching strategy to match the first feature point information and the third feature point information of the previous image frame data to obtain the second matching result. The second feature matching strategy is used to represent the feature matching rule to be executed on the first feature point information and the third feature point information. The previous moment corresponding to the previous image frame data is later than the historical moment but earlier than the current moment. The third feature point information is used to represent the current scene at the previous moment, and the image corner features under the corresponding scene category. The second matching result is used to represent the feature matching degree between the first feature point information and the third feature point information.

[0019] According to another aspect of the present invention, a processor is also provided. The processor is used to run a program, wherein the program, when run by the processor, performs the feature matching method of the present invention.

[0020] According to another aspect of the embodiments of the present invention, an electronic device is also provided, comprising: a memory storing a computer program; and a processor for running the computer program, wherein the computer program executes the vehicle feature matching method of various embodiments of the present invention when it runs.

[0021] According to another aspect of the embodiments of the present invention, a vehicle is also provided, which includes the electronic equipment of the present invention.

[0022] According to another aspect of the present invention, a computer-readable storage medium is also provided. The computer-readable storage medium includes a stored computer program that, when executed by a processor, implements the feature matching method of the present invention.

[0023] According to another aspect of the present invention, a computer program product is also provided, the computer program product including a computer program, wherein the computer program, when executed by a processor, implements the feature matching method of the present invention.

[0024] According to another aspect of the present invention, a computer program product is also provided, including a non-volatile computer-readable storage medium for storing a computer program, which, when executed by a processor, implements the feature matching method of the present invention.

[0025] According to another aspect of the embodiments of the present invention, the embodiments of this application also provide a computer program that, when executed by a processor, implements the feature matching method described in the embodiments of the present invention.

[0026] In this embodiment of the invention, during the feature matching process, the current image frame data of the current scene where the vehicle is located can be obtained; the current image frame data is input into a feature point detection model for feature detection to obtain the first feature point information of the current image frame data; if the frame type of the current image frame data is a key frame type, then the feature matching strategy is determined to be the first feature matching strategy, and the first feature matching strategy is invoked to match the first feature point information and the second feature point information of the interval image frame data to obtain the first matching result; if the frame type of the current image frame data is a non-key frame type, then the feature matching strategy is determined to be the second feature matching strategy, and the second feature matching strategy is invoked to match the first feature point information and the third feature point information of the previous image frame data to obtain the second matching result. Since the embodiments of the present invention, based on the first feature point information representing the corner features of the image, call the feature matching strategy corresponding to the frame type according to the different frame types of the current image frame data, to match the first feature point information with the second feature point information of the interval image frame data, or to match the first feature point information with the third feature point information of the previous image frame data, the purpose of extracting a sufficient number of effective feature points in a textureless or weakly textured environment is achieved. This solves the technical problem of low accuracy of feature matching of image frame data in the scene, and thus achieves the technical effect of improving the accuracy of feature matching of image frame data in the scene. Attached Figure Description

[0027] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0028] Figure 1 This is a flowchart of a feature matching method according to an embodiment of the present invention;

[0029] Figure 2(a) is a flowchart of a dynamic scaling prediction method according to an embodiment of the present invention;

[0030] Figure 2(b) is a schematic diagram of an improved SuperPoint network key module according to an embodiment of the present invention;

[0031] Figure 2(c) is a flowchart of a feature matching method based on SuperPoint features according to an embodiment of the present invention;

[0032] Figure 3 This is a schematic diagram of a feature matching device according to an embodiment of the present invention;

[0033] Figure 4 This is a schematic diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0034] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0035] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0036] According to an embodiment of the present invention, a feature matching method is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0037] Figure 1 This is a flowchart of a feature matching method according to an embodiment of the present invention, such as... Figure 1 As shown, the method may include the following steps:

[0038] Step S101: Obtain the current image frame data of the current scene where the vehicle is located.

[0039] In the technical solution provided by step S101 of the present invention, the display content of the current image frame data may include scene features of the current scene. For example, the current image frame data may be an image sequence, and the image sequence may include image frames; the current scene may include: highway scene, town street scene, and internal road scene, etc., which are only examples and are not specifically limited.

[0040] In this embodiment, current image frame data of the current scene where the vehicle is located is acquired. Optionally, this embodiment utilizes a camera deployed on the vehicle to continuously capture images of the current scene where the vehicle is located, thereby obtaining current image frame data and achieving the purpose of acquiring current image frame data of the current scene where the vehicle is located. The aforementioned camera may include at least one of the following devices: a camera, a video camera, and an infrared imaging device, etc.

[0041] Optionally, if the vehicle is in motion, the current scene where the vehicle is in motion is continuously captured to obtain the current image frame data when the vehicle is in motion; if the vehicle is in parked state, the current scene where the vehicle is in parked state is captured to obtain the current image frame data when the vehicle is parked.

[0042] Step S102: Input the current image frame data into the feature point detection model for feature detection to obtain the first feature point information of the current image frame data. The first feature point information is used to represent the image corner features of the current scene at the current moment under the corresponding scene category.

[0043] In the technical solution provided by step S102 of the present invention, the first feature point information can be used to represent the image corner features of the current scene at the current moment under the corresponding scene category. For example, the image corner features represented by the first feature point information can be dynamic scale features, and the scene category can also be simply referred to as scene type, which can be divided into: close-up category, medium-range category, and long-range category.

[0044] In this embodiment, after obtaining the current image frame data of the current scene where the vehicle is located, the current image frame data is input into the feature point detection model for feature detection to obtain the first feature point information of the current image frame data.

[0045] Optionally, this embodiment, based on the acquired current image frame data, inputs the current frame image data into the scale prediction layer of the feature point detection model for scale prediction, thereby obtaining scale prediction results corresponding to different scene categories. It also inputs the current frame image data into the feature encoding layer of the feature point detection model for encoding, thereby obtaining the feature encoding results of the current image frame data. Then, based on the scale prediction results obtained from the scale prediction and the feature encoding results obtained from the encoding, the first feature point information of the current image frame data can be determined, thus achieving the purpose of determining the image corner features of the current scene at the current moment under the corresponding scene category.

[0046] Step S103: If the frame type of the current image frame data is a keyframe type, then the feature matching strategy is determined to be the first feature matching strategy, and the first feature matching strategy is invoked to match the first feature point information and the second feature point information of the interval image frame data to obtain the first matching result. Here, the feature matching strategy is used to represent the feature matching rule to be executed on the first feature point information, the first feature matching strategy is used to represent the feature matching rule to be executed on the first feature point information and the second feature point information, the historical time corresponding to the interval image frame data is earlier than the current time, the second feature point information is used to represent the current scene at the historical time, the image corner feature under the corresponding scene category, and the first matching result is used to represent the feature matching degree between the first feature point information and the second feature point information.

[0047] In the technical solution provided by step S103 of the present invention, the above feature matching strategy can be used to represent the feature matching rule to be executed on the first feature point information.

[0048] In this embodiment, the first feature matching strategy described above can be used to represent the feature matching rule to be executed on the first feature point information and the second feature point information. For example, the first feature matching strategy described above can be a bag-of-words-based fast matching strategy.

[0049] In this embodiment, the historical time corresponding to the aforementioned interval image frame data is earlier than the current time, and the interval between the historical time and the current time is a preset time. That is, the aforementioned interval image frame data is image frame data that is spaced apart from the current image frame data by a preset time.

[0050] In this embodiment, the aforementioned second feature point information can be used to represent the image corner features of the current scene at a historical moment, under the corresponding scene category. For example, the image corner features represented by the aforementioned second feature point information can be dynamic scale features.

[0051] In this embodiment, the first matching result can be used to represent the degree of feature matching between the first feature point information and the second feature point information. For example, the first matching result can be the degree of feature matching between keyframe features and image corner features of interval image frame data.

[0052] In this embodiment, after inputting the current image frame data into the feature point detection model for feature detection and obtaining the first feature point information of the current image frame data, if the frame type of the current image frame data is a key frame type, then the feature matching strategy is determined to be the first feature matching strategy, and the first feature matching strategy is invoked to match the first feature point information and the second feature point information of the interval image frame data to obtain the first matching result.

[0053] Optionally, based on the first feature point information, this embodiment determines whether the frame type of the current image frame data is a key frame type or a non-key frame type. If it is determined that the frame type of the current image frame data is a key frame type, then the feature matching strategy is determined to be the first feature matching strategy, and the first feature matching strategy is invoked to match the first feature point information and the second feature point information of the interval image frame data to obtain the first matching result, that is, the feature matching degree between the first feature point information and the second feature point information can be obtained.

[0054] Step S104: If the frame type of the current image frame data is a non-key frame type, then the feature matching strategy is determined to be the second feature matching strategy, and the second feature matching strategy is invoked to match the first feature point information and the third feature point information of the previous image frame data to obtain the second matching result. The second feature matching strategy is used to represent the feature matching rule to be executed on the first feature point information and the third feature point information. The previous time corresponding to the previous image frame data is later than the historical time and earlier than the current time. The third feature point information is used to represent the image corner feature of the current scene at the previous time under the corresponding scene category. The second matching result is used to represent the feature matching degree between the first feature point information and the third feature point information.

[0055] In the technical solution provided by step S104 of the present invention, the second feature matching strategy can be used to represent the feature matching rules to be executed on the first feature point information and the third feature point information. For example, the second feature matching strategy can be a matching strategy based on projection matching combined with optical flow tracking.

[0056] In this embodiment, the previous time corresponding to the aforementioned previous image frame data is later than the historical time but earlier than the current time. That is, the aforementioned previous image frame data is the frame data preceding the current image frame data.

[0057] In this embodiment, the aforementioned third feature point information can be used to represent the image corner features of the current scene at the previous moment, under the corresponding scene category. For example, the image corner features represented by the aforementioned third feature point information can be dynamic scale features.

[0058] In this embodiment, the second matching result can be used to represent the degree of feature matching between the first feature point information and the third feature point information. For example, the second matching result can be the degree of feature matching between non-keyframe features and image corner features of the previous image frame data.

[0059] In this embodiment, after inputting the current image frame data into the feature point detection model for feature detection and obtaining the first feature point information of the current image frame data, if the frame type of the current image frame data is a non-key frame type, then the feature matching strategy is determined to be the second feature matching strategy, and the second feature matching strategy is invoked to match the first feature point information and the third feature point information of the previous image frame data to obtain the second matching result.

[0060] Optionally, based on the first feature point information, this embodiment determines whether the frame type of the current image frame data is a key frame type or a non-key frame type. If it is determined that the frame type of the current image frame data is a non-key frame type, the feature matching strategy is determined to be the second feature matching strategy, and the second feature matching strategy is called to match the first feature point information and the third feature point information of the previous image frame data to obtain the second matching result. That is, the feature matching degree between the first feature point information and the third feature point information can be obtained.

[0061] Optionally, the second feature matching strategy described above can be invoked to match the first feature point information with the third feature point information of the previous image frame data to obtain a second matching result. For example, by invoking the second feature matching strategy to project the feature point corresponding to the third feature point information into the current image frame data, a projection result can be obtained. If the obtained projection result indicates that the feature point corresponding to the third feature point information has been successfully projected into the current image frame data, then the target feature point is determined from the multiple candidate feature points corresponding to the third feature point information, and then the first feature point information and the fourth feature point information of the target feature point are matched to obtain a second matching result. Alternatively, if the obtained projection result indicates that the feature point corresponding to the third feature point information has been successfully projected into the current image frame data, then the target feature point is extracted from the feature points after optical flow tracking, and then the first feature point information and the fourth feature point information of the target feature point are matched to obtain a second matching result.

[0062] In steps S101 to S104 of this application, during the feature matching process, the current image frame data of the current scene where the vehicle is located can be obtained; the current image frame data is input into a feature point detection model for feature detection to obtain the first feature point information of the current image frame data; if the frame type of the current image frame data is a key frame type, the feature matching strategy is determined to be the first feature matching strategy, and the first feature matching strategy is invoked to match the first feature point information and the second feature point information of the interval image frame data to obtain the first matching result; if the frame type of the current image frame data is a non-key frame type, the feature matching strategy is determined to be the second feature matching strategy, and the second feature matching strategy is invoked to match the first feature point information and the third feature point information of the previous image frame data to obtain the second matching result. Since the embodiments of the present invention, based on the first feature point information representing the corner features of the image, call the feature matching strategy corresponding to the frame type according to the different frame types of the current image frame data, to match the first feature point information with the second feature point information of the interval image frame data, or to match the first feature point information with the third feature point information of the previous image frame data, the purpose of extracting a sufficient number of effective feature points in a textureless or weakly textured environment is achieved. This solves the technical problem of low accuracy of feature matching of image frame data in the scene, and thus achieves the technical effect of improving the accuracy of feature matching of image frame data in the scene.

[0063] The method described below for inputting the current image frame data into a feature point detection model to perform feature detection and obtain the first feature point information of the current image frame data will be further described below.

[0064] As an optional embodiment, step S102, inputting the current image frame data into the feature point detection model for feature detection to obtain the first feature point information of the current image frame data, includes: inputting the current image frame data into the scale prediction layer of the feature point detection model for scale prediction to obtain scale prediction results corresponding to different scene categories, and inputting the current image frame data into the feature encoding layer of the feature point detection model for encoding to obtain the feature encoding result of the current image frame data; fusing the scale prediction result and the feature encoding result to obtain the first feature point information.

[0065] In this embodiment, the feature point prediction model described above can be an improved SuperPoint network model.

[0066] In this embodiment, the scale prediction layer can be a dynamic scale prediction layer.

[0067] In this embodiment, the above scale prediction result can be a multi-scale feature corresponding to the scene category.

[0068] In this embodiment, the feature encoding layer described above may be equipped with a feature encoder.

[0069] In this embodiment, the above-mentioned fusion operation can also be referred to as a merge operation.

[0070] In this embodiment, after acquiring the current image frame data of the current scene where the vehicle is located, the current image frame data is input into the scale prediction layer of the feature point detection model for scale prediction to obtain the scale prediction results corresponding to different scene categories, and the current image frame data is input into the feature encoding layer of the feature point detection model for encoding to obtain the feature encoding results of the current image frame data.

[0071] In this embodiment, after obtaining the scale prediction results corresponding to different scene categories and the feature encoding results of the current image frame data, the scale prediction results and the feature encoding results are fused to obtain the first feature point information.

[0072] Optionally, in this embodiment, based on the acquired current image frame data, the current frame image data is input into the dynamic scale prediction layer of the feature point detection model for scale prediction, thereby obtaining scale prediction results corresponding to different scene categories. The current frame image data is also input into the feature encoder of the feature point detection model for encoding, thereby obtaining the feature encoding result of the current image frame data. Then, the scale prediction result obtained from scale prediction and the feature encoding result obtained from encoding are fused to obtain the first feature point information of the current image frame data.

[0073] By performing scale prediction and feature encoding on the current image frame data respectively, we can obtain scale prediction results and feature encoding results. Then, we can fuse the above scale prediction results and feature encoding results into image corner features under the corresponding scene category. This achieves the goal of obtaining the first feature point information of the current image frame data, thereby improving the technical effect of the first feature point information.

[0074] For example, the improved SuperPoint network model described above can be represented by the following equation:

[0075] (1)

[0076] (2)

[0077] in, λ 1 can be used to represent the weight of the feature point residual. λ 2 can be used to represent the weight of the descriptor residual. λ 3 can be used to represent the weights of the feature scale prediction residuals. Lp It can be used to represent feature point residuals. L m It can be used to represent descriptor residuals. It can be used to represent the predicted value of the feature scale. S k It can be used to represent the truth value of descriptor residuals. k It can be used to represent the k-th feature point. L scale It can be used for the scale of feature points in a table.

[0078] The method described below for matching the first feature point information and the second feature point information of the interval image frame data using the first feature matching strategy described above in this embodiment to obtain the first matching result will be further described.

[0079] As an optional embodiment, step S103 involves invoking a first feature matching strategy to match the first feature point information and the second feature point information of the interval image frame data to obtain a first matching result, including: performing equalization processing on the first feature point information; and invoking the first feature matching strategy to match the processed first feature point information and the second feature point information in the bag-of-words database to obtain a first matching result.

[0080] In this embodiment, the above-mentioned equalization process can be a quadtree-based equalization process. For example, when a node splits for the first time, a root node is split into 4 nodes, and the node splitting is iterated sequentially until the number of nodes is greater than or equal to the expected number of nodes. Finally, from each node, the feature point with the highest SuperPoint feature score is selected as the unique feature point.

[0081] In this embodiment, after inputting the current image frame data into the feature point detection model for feature detection to obtain the first feature point information of the current image frame data, the first feature point information is subjected to equalization processing; the first feature matching strategy is invoked to match the processed first feature point information and the second feature point information in the bag-of-words database to obtain the first matching result.

[0082] Optionally, based on the obtained first feature point information, this embodiment determines whether the frame type of the current image frame data is a keyframe type or a non-keyframe type. If it is determined that the frame type of the current image frame data is a keyframe type, the aforementioned first feature matching strategy is invoked to match the first feature point information and the second feature point information of the interval image frame data, thereby obtaining a first matching result, that is, the feature matching degree between the first feature point information and the second feature point information. Since the first feature point information and the second feature point information after equalization are matched using a bag-of-words-based fast matching method, the feature matching degree between the first feature point information and the second feature point information can be obtained, thus achieving the goal of obtaining the first matching result, and thereby achieving the technical effect of improving the accuracy of the first matching result.

[0083] The following section further describes the method of using the second feature matching strategy described above in this embodiment to match the first feature point information and the third feature point information of the previous image frame data to obtain the second matching result.

[0084] As an optional embodiment, the method further includes: inputting the scale prediction result and the feature encoding result into the descriptor generator of the feature point detection model to generate a first descriptor information of the first feature point information, wherein the first descriptor information is used to describe the first feature point information; step S104, calling a second feature matching strategy to match the first feature point information with the third feature point information of the previous image frame data to obtain a second matching result, including: calling the second feature matching strategy, using projection matching combined with optical flow tracking to project the feature points corresponding to the third feature point information into the current image frame data to obtain a projection result, wherein the projection result is used to indicate whether the feature points corresponding to the third feature point information have been successfully projected into the current image frame data; response The projection result indicates that the feature point corresponding to the third feature point information has been successfully projected into the current image frame data. The projection position of the feature point corresponding to the third feature point information in the current image frame data is determined. Within the current area where the projection position is located, based on the distance between the first descriptor information and the second descriptor information of the third feature point information, the target feature point is determined from multiple candidate feature points corresponding to the third feature point information. The feature matching degree between the fourth feature point information of the target feature point and the first feature point information is determined as the second matching result. The distance between the first descriptor information and the descriptor information of the target feature point is less than the distance between the first descriptor information and the descriptor information of any candidate feature point other than the target feature point among the multiple candidate feature points.

[0085] In this embodiment, the first descriptor information described above can be used to describe the first feature point information.

[0086] In this embodiment, the projection result can be used to indicate whether the feature point corresponding to the third feature point information has been successfully projected into the current image frame data. That is, the projection result can be used to indicate that the feature point corresponding to the third feature point information has been successfully projected into the current image frame data, or the projection result can also be used to indicate that the feature point corresponding to the third feature point information has not been successfully projected into the current image frame data. For example, the projection result can be represented by the projection point where the feature point has been successfully projected into the current image frame data.

[0087] In this embodiment, the distance between the first descriptor information and the descriptor information of the target feature point is less than the distance between the first descriptor information and the descriptor information of any candidate feature point other than the target feature point among the multiple candidate feature points.

[0088] In this embodiment, the current region can be a circular region near the projection point.

[0089] In this embodiment, the aforementioned candidate feature points can be feature points within a circular region near the projection point.

[0090] In this embodiment, the aforementioned target feature point is the optimal matching point.

[0091] In this embodiment, after inputting the current image frame data into the feature point detection model for feature detection to obtain the first feature point information of the current image frame data, the scale prediction result and feature encoding result are input into the descriptor generator of the feature point detection model for generation to obtain the first descriptor information of the first feature point information.

[0092] In this embodiment, after generating the first descriptive information of the first feature point information, the second feature matching strategy is invoked, and projection matching combined with optical flow tracking is used to project the feature point corresponding to the third feature point information into the current image frame data to obtain the projection result; in response to the projection result indicating that the feature point corresponding to the third feature point information has been successfully projected into the current image frame data, the projection position of the feature point corresponding to the third feature point information in the current image frame data is determined; within the current area where the projection position is located, based on the distance between the first descriptive information and the second descriptive information of the third feature point information, the target feature point is determined from multiple candidate feature points corresponding to the third feature point information, and the feature matching degree between the fourth feature point information of the target feature point and the first feature point information is determined as the second matching result.

[0093] Optionally, this embodiment, based on the obtained first feature point information, determines whether the frame type of the current image frame data is a keyframe type or a non-keyframe type. If the frame type of the current image frame data is determined to be a non-keyframe type, the determined second feature matching strategy is invoked to project the feature point corresponding to the third feature point information into the current image frame data, thereby obtaining a projection result. Then, the obtained projection result is parsed. If the parsed projection result indicates that the feature point corresponding to the third feature point information has been successfully projected into the current image frame data, the feature point corresponding to the third feature point information is projected into the current image frame data, thereby obtaining the projection position. Next, within the current area where the above projection position is located, based on the distance between the first descriptor information and the second descriptor information of the third feature point information, a target feature point is determined from multiple candidate feature points corresponding to the third feature point information. Furthermore, feature matching is performed between the fourth feature point information of the determined target feature point and the first feature point information to obtain the feature matching degree between the fourth feature point information and the first feature point information. This feature matching degree is then determined as the second matching result.

[0094] By using projection matching combined with optical flow tracking, the first feature point information and the third feature point information of the previous image frame data are matched to obtain the feature matching degree between the first feature point information and the third feature point information. This achieves the goal of obtaining the second matching result and thus improves the accuracy of the second matching result.

[0095] The method for determining target feature points described in this embodiment will be further described below.

[0096] As an optional embodiment, the method further includes: in response to the fact that no target feature point is determined from multiple candidate feature points corresponding to the third feature point information, extracting the target feature point from the feature points after optical flow tracking.

[0097] In this embodiment, in response to the failure to determine a target feature point from multiple candidate feature points corresponding to the third feature point information, the target feature point is extracted from the feature points after optical flow tracking. Optionally, in the case where the target feature point is not determined from multiple candidate feature points corresponding to the third feature point information, that is, when no map point corresponding to the previous image frame data is found in the current image frame data, the corresponding feature point is extracted from the feature points after optical flow tracking and used as the target feature point. Since the corresponding point is extracted from the points after optical flow tracking to supplement the map point when no map point corresponding to the previous frame is found in the current frame, the purpose of determining the target feature point is achieved, thereby realizing the technical effect of improving the accuracy of the target feature point.

[0098] The method for obtaining the feature point detection model described above in this embodiment will be further described below.

[0099] As an optional embodiment, the method further includes: acquiring current image frame data samples of the current scene where the vehicle is located; using the current image frame data samples, performing scale distribution training on the initial feature point detection model, and reconstructing the initial loss function of the initial feature point detection model to obtain a feature point detection model, wherein the feature point detection model is used to perform feature detection on the current image frame data to obtain first feature point information.

[0100] In this embodiment, the above-mentioned feature point detection model can be used to perform feature detection on the current image frame data to obtain the first feature point information.

[0101] In this embodiment, after obtaining the current image frame data sample of the current scene where the vehicle is located, the initial feature point detection model is trained using the current image frame data sample, and the initial loss function of the initial feature point detection model is reconstructed to obtain the feature point detection model.

[0102] Optionally, this embodiment downloads current image frame data samples from the scene image frame database, uses the downloaded current image frame data samples to train the initial feature point detection model on scale distribution, and redesigns the initial loss function of the initial feature point detection model, thereby completing the construction of the feature point detection model.

[0103] By using the current image frame data samples to train the initial feature point detection model with scale distribution and reconstructing the initial loss function of the initial feature point detection model, and then using the trained and reconstructed initial feature point detection model as the final feature point detection model for feature detection of the current image frame data, the goal of training the initial feature point detection model and obtaining the feature point detection model is achieved, thereby realizing the technical effect of improving the robustness of the feature point detection model.

[0104] In this embodiment of the invention, during the feature matching process, the current image frame data of the current scene where the vehicle is located can be obtained; the current image frame data is input into a feature point detection model for feature detection to obtain the first feature point information of the current image frame data; if the frame type of the current image frame data is a key frame type, then the feature matching strategy is determined to be the first feature matching strategy, and the first feature matching strategy is invoked to match the first feature point information and the second feature point information of the interval image frame data to obtain the first matching result; if the frame type of the current image frame data is a non-key frame type, then the feature matching strategy is determined to be the second feature matching strategy, and the second feature matching strategy is invoked to match the first feature point information and the third feature point information of the previous image frame data to obtain the second matching result. Since the embodiments of the present invention, based on the first feature point information representing the corner features of the image, call the feature matching strategy corresponding to the frame type according to the different frame types of the current image frame data, to match the first feature point information with the second feature point information of the interval image frame data, or to match the first feature point information with the third feature point information of the previous image frame data, the purpose of extracting a sufficient number of effective feature points in a textureless or weakly textured environment is achieved. This solves the technical problem of low accuracy of feature matching of image frame data in the scene, and thus achieves the technical effect of improving the accuracy of feature matching of image frame data in the scene.

[0105] The technical solutions of the embodiments of the present invention will be illustrated below with reference to preferred embodiments.

[0106] Currently, when performing feature matching on continuous image frame data of a vehicle scene, traditional feature matching models (e.g., feature matching models built on neural networks) are often used to perform feature matching on the aforementioned continuous image frame data.

[0107] However, the aforementioned feature matching methods are ill-suited for continuous image frame data in complex scenes. Furthermore, using these methods to extract features from continuous image frame data lacking texture also fails to yield effective scene features, resulting in low accuracy in feature matching of scene image frame data.

[0108] To address the aforementioned technical problems, this invention proposes a feature matching method. Based on obtaining first feature point information representing image corner features, and according to different frame types of the current image frame data, a feature matching strategy corresponding to that frame type is invoked to match the first feature point information with second feature point information of the interval image frame data, or to match the first feature point information with third feature point information of the previous image frame data. This achieves the goal of extracting a sufficient number of effective feature points in textureless or weakly textured environments, thereby solving the technical problem of low accuracy in feature matching of scene image frame data and ultimately improving the accuracy of feature matching of scene image frame data.

[0109] In this embodiment, the current image frame data can be normalized by performing a dynamic scale prediction method. For example, Figure 2(a) is a flowchart of a dynamic scale prediction method according to an embodiment of the present invention. As shown in Figure 2(a), the method may include the following steps:

[0110] Step S201: Input the current image frame data into the convolutional layer for processing.

[0111] In the technical solution provided by step S201 of the present invention, the number of input channels of the above convolutional layer (Conv) is 3, the number of output channels is 16, the kernel size is 3x3, the stride is 1, and the padding is 1.

[0112] After the current image frame data is input into the convolutional layer for processing, step S202 is performed, where the processed current image frame data is input into the activation function layer for calculation.

[0113] In the technical solution provided by step S202 of the present invention, the activation function layer (ReLU) can use the ReLU activation function.

[0114] After the processed current image frame data is input into the activation function layer for calculation, step S203 is entered, where the calculated current image frame data is input into the adaptive average pooling layer for pooling.

[0115] In the technical solution provided by step S203 of the present invention, the adaptive average pooling layer is a two-dimensional adaptive average pooling (AdaptiveAvgPool2D) layer.

[0116] After inputting the calculated current image frame data into the adaptive average pooling layer for pooling, step S204 is entered, where the pooled scene image data is input into the fully connected layer for conversion.

[0117] In the technical solution provided in step S204 of the present invention, the input dimension of the fully connected layer (Linear) is 16 and the output dimension is 3.

[0118] After the pooled current image frame data is input into the fully connected layer for transformation, step S205 is performed, where the transformed current image frame data is input into the maximal function layer for normalization.

[0119] In the technical solution provided by step S205 of the present invention, the above-mentioned maximum function layer (Softmax) can be used to output a normalized probability distribution.

[0120] For example, the aforementioned feature point detection model may include an improved SuperPoint network key module as shown in Figure 2(b). Figure 2(b) is a schematic diagram of an improved SuperPoint network key module according to an embodiment of the present invention. As shown in Figure 2(b), the module 210 may include: a dynamic scale prediction branch 2101, a feature encoder 2102, a dynamic pyramid module 2103, a multi-scale feature set 2104, a feature detector 2105, and a descriptor generator 2106. Specifically, the dynamic scale prediction branch 2101 can be used to perform dynamic scale prediction on the input current image frame data, the feature encoder 2102 can be used to perform feature encoding on the input current image frame data, and the dynamic scale prediction branch 2101 and the feature encoder 2102 can be used to output the basic feature map obtained by dynamic scale prediction and feature encoding; the dynamic pyramid module 2103 can be used to generate multi-scale features from the basic feature map, and to merge the multi-scale features; the multi-scale feature set 2104, the feature detector 2105 and the descriptor generator 2106 can be used to detect the merged multi-scale features respectively to obtain dynamic scale feature points and feature descriptors.

[0121] For example, based on the scale probability, the number of pyramid levels for each sample can be determined. Then, for each sample, a multi-scale feature layer is dynamically generated. Finally, the multi-scale features are merged, and the merged multi-scale features are output to the SuperPoint-based feature detector, descriptor detector, and newly added multi-scale feature set.

[0122] In this embodiment, by executing a feature matching method based on SuperPoint features, the first feature point information can be matched with different feature point information to obtain the corresponding matching results. For example, Figure 2(c) is a flowchart of a feature matching method based on SuperPoint features according to an embodiment of the present invention. As shown in Figure 2(c), the method may include the following steps:

[0123] Step S221: Obtain the current image frame data sample.

[0124] After acquiring scene image data samples, proceed to step S222, where the improved SuperPoint network model is trained using the current image frame data samples.

[0125] In the technical solution provided by step S222 of the present invention, the loss function corresponding to the improved SuperPoint network model can be composed of the original SuperPoint loss (corner point, feature descriptor) and scale prediction loss. For example, the loss function can be as shown in (1) above.

[0126] After training the improved SuperPoint network model using the current image frame data samples, proceed to step S223, where the trained improved SuperPoint network model is used to extract multi-scale features from the current image frame data.

[0127] In the technical solution provided in step S223 of the present invention, the scene scale statistics (depth mean and variance) are calculated by combining the point cloud depth map output by radar (LiDAR), and three scale labels are assigned: "near view (depth < 2m), mid view (2-10m), and far view (> 10m)". At the same time, data augmentation is performed by randomly adding objects of different scales to enhance the network's learning of scale distribution.

[0128] After using the trained improved SuperPoint network model to extract multi-scale features from the current image frame data, the process proceeds to steps S224 and S225 to train a bag-of-words model based on SuperPoint features and to equalize the extracted SuperPoint features.

[0129] In the technical solution provided by step S225 of the present invention, when a node splits for the first time, a root node splits into 4 nodes, and the nodes are split in turn until the number of nodes is greater than or equal to the expected number of nodes. Finally, the feature point with the highest SuperPoint feature score is selected from each node as the unique feature point.

[0130] After equalizing the extracted SuperPoint features, proceed to step S226 to determine whether the SuperPoint features are keyframe features.

[0131] If the SuperPoint feature is a keyframe feature, then step S227 is executed, using a bag-of-words model based on the SuperPoint feature to match the keyframe feature and the image corner features of the interval image frame data.

[0132] In the technical solution provided in step S227 of the present invention, for image sequences (e.g., keyframes) that are far apart from each other, a fast matching method based on the bag-of-words model can be used to directly index all feature points of the same node in the bag-of-words database, thereby accelerating feature matching between different images.

[0133] If the SuperPoint feature is not a keyframe feature, then step S228 is executed, and the non-keyframe features and the image corner features of the previous image frame data are matched using a projection-based matching method.

[0134] In the technical solution provided in step S228 of the present invention, projection matching combined with optical flow tracking is used between consecutive ordinary frames. Map points in the previous consecutive frame are projected into the current frame. Matching points are searched within a circular area near the projected points, and the optimal matching point can be determined based on the distance of the descriptor. For points that fail to match (i.e., cannot find a map point in the current frame that corresponds to a point in the previous frame), feature points are extracted from the points tracked by optical flow to supplement them.

[0135] After matching keyframe features and image corner features of interval image frame data using a bag-of-words model based on SuperPoint features, or matching non-keyframe features and image corner features of the previous image frame data using a projection-based matching method, proceed to step S229 to determine whether the feature points are successfully matched.

[0136] If the feature point matching is successful, proceed to step S230 to perform matching between consecutive frames.

[0137] If the feature points are not matched successfully, step S231 is executed to supplement the unmatched feature points using the results of optical flow tracing.

[0138] After supplementing the unmatched feature points using the results of optical flow tracking, the process proceeds to step S230 to perform matching between consecutive frames.

[0139] In this embodiment, during feature matching, the current image frame data of the current scene where the vehicle is located can be obtained; the current image frame data is input into a feature point detection model for feature detection to obtain the first feature point information of the current image frame data; if the frame type of the current image frame data is a key frame type, the feature matching strategy is determined to be the first feature matching strategy, and the first feature matching strategy is invoked to match the first feature point information with the second feature point information of the interval image frame data to obtain the first matching result; if the frame type of the current image frame data is a non-key frame type, the feature matching strategy is determined to be the second feature matching strategy, and the second feature matching strategy is invoked to match the first feature point information with the third feature point information of the previous image frame data to obtain the second matching result. Since the embodiments of the present invention, based on the first feature point information representing the corner features of the image, call the feature matching strategy corresponding to the frame type according to the different frame types of the current image frame data, to match the first feature point information with the second feature point information of the interval image frame data, or to match the first feature point information with the third feature point information of the previous image frame data, the purpose of extracting a sufficient number of effective feature points in a textureless or weakly textured environment is achieved. This solves the technical problem of low accuracy of feature matching of image frame data in the scene, and thus achieves the technical effect of improving the accuracy of feature matching of image frame data in the scene.

[0140] According to embodiments of the present invention, a feature matching apparatus is also provided. It should be noted that this feature matching apparatus can be used to execute one of the feature matching methods described in the embodiments.

[0141] Figure 3 This is a schematic diagram of a feature matching device according to an embodiment of the present invention. Figure 3 As shown, the feature matching device 300 may include: a first acquisition unit 301, a detection unit 302, a first matching unit 303, and a second matching unit 304.

[0142] The first acquisition unit 301 is used to acquire the current image frame data of the current scene where the vehicle is located.

[0143] The detection unit 302 is used to input the current image frame data into the feature point detection model for feature detection to obtain the first feature point information of the current image frame data. The first feature point information is used to represent the image corner features of the current scene at the current moment under the corresponding scene category.

[0144] The first matching unit 303 is used to determine the feature matching strategy as the first feature matching strategy if the frame type of the current image frame data is a key frame type, and to call the first feature matching strategy to match the first feature point information and the second feature point information of the interval image frame data to obtain the first matching result. The feature matching strategy is used to represent the feature matching rule to be executed on the first feature point information. The first feature matching strategy is used to represent the feature matching rule to be executed on the first feature point information and the second feature point information. The historical time corresponding to the interval image frame data is earlier than the current time. The second feature point information is used to represent the current scene at the historical time and the image corner feature under the corresponding scene category. The first matching result is used to represent the feature matching degree between the first feature point information and the second feature point information.

[0145] The second matching unit 304 is used to determine the feature matching strategy as the second feature matching strategy if the frame type of the current image frame data is a non-key frame type, and to call the second feature matching strategy to match the first feature point information and the third feature point information of the previous image frame data to obtain the second matching result. The second feature matching strategy is used to represent the feature matching rule to be executed on the first feature point information and the third feature point information. The previous time corresponding to the previous image frame data is later than the historical time and earlier than the current time. The third feature point information is used to represent the image corner feature of the current scene at the previous time under the corresponding scene category. The second matching result is used to represent the feature matching degree between the first feature point information and the third feature point information.

[0146] Optionally, the detection unit 302 may include: a prediction and encoding module, used to input the current image frame data into the scale prediction layer of the feature point detection model for scale prediction to obtain scale prediction results corresponding to different scene categories, and to input the current image frame data into the feature encoding layer of the feature point detection model for encoding to obtain the feature encoding result of the current image frame data; and a fusion module, used to fuse the scale prediction result and the feature encoding result to obtain the first feature point information.

[0147] Optionally, the first matching unit 303 may include: an equalization module for equalizing the first feature point information; and a matching module for invoking a first feature matching strategy to match the processed first feature point information and second feature point information in the bag-of-words database to obtain a first matching result.

[0148] Optionally, the feature matching device 300 may include: a generation unit, used to input the scale prediction result and feature encoding result into the descriptor generator of the feature point detection model for generation, to obtain first descriptor information of the first feature point information, wherein the first descriptor information is used to describe the first feature point information; the second matching unit 304 may include: a projection module, used to call a second feature matching strategy, using projection matching combined with optical flow tracking, to project the feature points corresponding to the third feature point information onto the current image frame data, to obtain a projection result, wherein the projection result is used to indicate whether the feature points corresponding to the third feature point information have been successfully projected onto the current image frame data; and a first determination module, used to determine whether the feature points corresponding to the third feature point information have been successfully projected onto the current image frame data in response to the projection result indicating that the feature points corresponding to the third feature point information... The feature points have been successfully projected into the current image frame data, and the projection position of the feature point corresponding to the third feature point information in the current image frame data is determined; the second determining module is used to determine the target feature point from multiple candidate feature points corresponding to the third feature point information within the current area where the projection position is located, based on the distance between the first descriptor information and the second descriptor information of the third feature point information, and to determine the feature matching degree between the fourth feature point information of the target feature point and the first feature point information as the second matching result, wherein the distance between the first descriptor information and the descriptor information of the target feature point is less than the distance between the first descriptor information and the descriptor information of any candidate feature point other than the target feature point among the multiple candidate feature points.

[0149] Optionally, the feature matching device 300 may include an extraction unit, configured to extract a target feature point from the feature points after optical flow tracking in response to the absence of a target feature point from a plurality of candidate feature points corresponding to the third feature point information.

[0150] Optionally, the feature matching device 300 may include: a second acquisition unit, used to acquire current image frame data samples of the current scene where the vehicle is located; and a training and reconstruction unit, used to use the current image frame data samples to train the initial feature point detection model on a scale distribution and to reconstruct the initial loss function of the initial feature point detection model to obtain a feature point detection model, wherein the feature point detection model is used to perform feature detection on the current image frame data to obtain first feature point information.

[0151] In this embodiment, a feature matching device is provided, which may include: a first acquisition unit, configured to acquire current image frame data of the current scene in which the vehicle is located; a detection unit, configured to input the current image frame data into a feature point detection model for feature detection to obtain first feature point information of the current image frame data, wherein the first feature point information is used to represent the image corner feature of the current scene at the current moment under the corresponding scene category; and a first matching unit, configured to determine the feature matching strategy as a first feature matching strategy if the frame type of the current image frame data is a keyframe type, and to call the first feature matching strategy to match the first feature point information and the second feature point information of the interval image frame data to obtain a first matching result, wherein the feature matching strategy is used to represent the feature matching rule to be executed on the first feature point information, the first feature matching strategy is used to represent the feature matching rule to be executed on the first feature point information and the second feature point information, the historical moment corresponding to the interval image frame data is earlier than the current moment, and the second feature point information is used to represent the image corner feature of the current scene at the historical moment under the corresponding scene category. The first matching unit is used to represent the degree of feature matching between the first feature point information and the second feature point information. The second matching unit is used to determine the feature matching strategy as the second feature matching strategy if the frame type of the current image frame data is a non-key frame type, and to call the second feature matching strategy to match the first feature point information and the third feature point information of the previous image frame data to obtain the second matching result. The second feature matching strategy is used to represent the feature matching rule to be executed on the first feature point information and the third feature point information. The previous time corresponding to the previous image frame data is later than the historical time and earlier than the current time. The third feature point information is used to represent the image corner feature of the current scene at the previous time under the corresponding scene category. The second matching result is used to represent the degree of feature matching between the first feature point information and the third feature point information. This achieves the goal of extracting a sufficient number of effective feature points in a textureless or weakly textured environment, thereby solving the technical problem of low accuracy of feature matching of scene image frame data, and thus achieving the technical effect of improving the accuracy of feature matching of scene image frame data.

[0152] According to embodiments of the present invention, an electronic device is also provided. Figure 4 This is a schematic diagram of an electronic device according to an embodiment of the present invention, such as... Figure 4 As shown, the electronic device 400 may include a memory 410 and a processor 420, wherein the memory 410 is used to store computer programs; and the processor 420 is used to run the programs stored in the memory 410 to implement the feature matching method of this application.

[0153] In this application, "multiple" refers to two or more.

[0154] In this application, unless otherwise expressly defined, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection between two components. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances.

[0155] The terms “first,” “second,” “third,” “fourth,” etc., in this application (if present) are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0156] In this application, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, in this application, the character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0157] Unless otherwise specified, all steps of this application may be performed sequentially or randomly. For example, the feature matching method of this application may include steps S101 and S102, meaning that the feature matching method of this application may include steps S101 and S102 performed sequentially, or it may include steps S102 and S101 performed sequentially. For example, the feature matching method of this application may also include step S103, meaning that step S103 may be added to the method in any order. For example, the feature matching method of this application may include steps S101, S102, and S103, or it may include steps S101, S103, and S102, or it may include steps S103, S101, and S102, etc.

[0158] According to an embodiment of the present invention, a processor is also provided for running a program, wherein the program is executed by the processor to perform the feature matching method in the embodiment.

[0159] According to another aspect of the embodiments of the present invention, a vehicle is also provided, which includes the electronic equipment of the present invention.

[0160] According to another aspect of the present invention, a computer-readable storage medium is also provided. The computer-readable storage medium includes a stored computer program that, when executed by a processor, implements the feature matching method of the present invention.

[0161] According to an embodiment of the present invention, a computer program product is also provided, the computer program product including a computer program, wherein the computer program, when executed by a processor, implements the feature matching method of the embodiment.

[0162] According to an embodiment of the present invention, a computer program product is also provided, including a non-volatile computer-readable storage medium for storing a computer program, which, when executed by a processor, implements the feature matching method in the embodiment.

[0163] According to an embodiment of the present invention, a computer program is also provided, which, when executed by a processor, implements the feature matching method of the embodiment.

[0164] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0165] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0166] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some interfaces; indirect couplings or communication connections between units or modules may be electrical or other forms.

[0167] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0168] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0169] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, in essence, or the part that contributes to related technologies, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0170] The above are merely preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A feature matching method, characterized in that, include: Obtain the current image frame data of the current scene where the vehicle is located; The current image frame data is input into a feature point detection model for feature detection to obtain the first feature point information of the current image frame data. The first feature point information is used to represent the image corner features of the current scene at the current moment under the corresponding scene category. If the frame type of the current image frame data is a keyframe type, then the feature matching strategy is determined to be the first feature matching strategy, and the first feature matching strategy is invoked to match the first feature point information and the second feature point information of the interval image frame data to obtain a first matching result. The feature matching strategy represents the feature matching rule to be executed on the first feature point information. The first feature matching strategy represents the feature matching rule to be executed on the first feature point information and the second feature point information. The historical time corresponding to the interval image frame data is earlier than the current time. The second feature point information represents the image corner feature of the current scene at the historical time under the corresponding scene category. The first matching result represents the degree of feature matching between the first feature point information and the second feature point information. If the frame type of the current image frame data is a non-critical frame type, then the feature matching strategy is determined to be the second feature matching strategy, and the second feature matching strategy is invoked to match the first feature point information and the third feature point information of the previous image frame data to obtain a second matching result. The second feature matching strategy is used to represent the feature matching rule to be executed on the first feature point information and the third feature point information. The previous time corresponding to the previous image frame data is later than the historical time and earlier than the current time. The third feature point information is used to represent the image corner feature of the current scene at the previous time under the corresponding scene category. The second matching result is used to represent the feature matching degree between the first feature point information and the third feature point information.

2. The method according to claim 1, characterized in that, The current image frame data is input into a feature point detection model for feature detection to obtain the first feature point information of the current image frame data, including: The current image frame data is input into the scale prediction layer of the feature point detection model for scale prediction to obtain scale prediction results corresponding to different scene categories, and the current image frame data is input into the feature encoding layer of the feature point detection model for encoding to obtain the feature encoding results of the current image frame data. The scale prediction result and the feature encoding result are fused to obtain the first feature point information.

3. The method according to claim 1, characterized in that, The first feature matching strategy is invoked to match the first feature point information and the second feature point information of the interval image frame data to obtain a first matching result, including: The first feature point information is subjected to equalization processing; The first feature matching strategy is invoked to match the processed first feature point information and the second feature point information in the bag-of-words database to obtain the first matching result.

4. The method according to claim 2, characterized in that, The method further includes: The scale prediction result and the feature encoding result are input into the descriptor generator of the feature point detection model to generate a first descriptor information of the first feature point information, wherein the first descriptor information is used to describe the first feature point information; The second feature matching strategy is invoked to match the first feature point information and the third feature point information of the previous image frame data to obtain a second matching result. This includes: invoking the second feature matching strategy, using projection matching combined with optical flow tracking, to project the feature point corresponding to the third feature point information into the current image frame data to obtain a projection result. The projection result is used to indicate whether the feature point corresponding to the third feature point information has been successfully projected into the current image frame data. In response to the projection result indicating that the feature point corresponding to the third feature point information has been successfully projected into the current image frame data, the projection position of the feature point corresponding to the third feature point information in the current image frame data is determined. Within the current area where the projection position is located, based on the distance between the first descriptor information and the second descriptor information of the third feature point information, a target feature point is determined from multiple candidate feature points corresponding to the third feature point information, and the feature matching degree between the fourth feature point information of the target feature point and the first feature point information is determined as the second matching result, wherein the distance between the first descriptor information and the descriptor information of the target feature point is less than the distance between the first descriptor information and the descriptor information of any candidate feature point other than the target feature point among the multiple candidate feature points.

5. The method according to claim 4, characterized in that, The method further includes: In response to the failure to determine the target feature point from among the multiple candidate feature points corresponding to the third feature point information, the target feature point is extracted from the feature points after optical flow tracking.

6. The method according to any one of claims 1 to 5, characterized in that, include: Obtain the current image frame data sample of the current scene where the vehicle is located; Using the current image frame data samples, the initial feature point detection model is trained with scale distribution, and the initial loss function of the initial feature point detection model is reconstructed to obtain the feature point detection model. The feature point detection model is used to perform feature detection on the current image frame data to obtain the first feature point information.

7. A feature matching device, characterized in that, include: The first acquisition unit is used to acquire the current image frame data of the current scene where the vehicle is located; The detection unit is used to input the current image frame data into the feature point detection model for feature detection to obtain the first feature point information of the current image frame data, wherein the first feature point information is used to represent the image corner feature of the current scene at the current time under the corresponding scene category; The first matching unit is configured to determine the feature matching strategy as the first feature matching strategy if the frame type of the current image frame data is a keyframe type, and to call the first feature matching strategy to match the first feature point information and the second feature point information of the interval image frame data to obtain a first matching result. The feature matching strategy represents the feature matching rule to be executed on the first feature point information, and the first feature matching strategy represents the feature matching rule to be executed on the first feature point information and the second feature point information. The historical time corresponding to the interval image frame data is earlier than the current time. The second feature point information represents the image corner feature of the current scene at the historical time under the corresponding scene category. The first matching result represents the feature matching degree between the first feature point information and the second feature point information. The second matching unit is configured to determine the feature matching strategy as the second feature matching strategy if the frame type of the current image frame data is a non-key frame type, and to call the second feature matching strategy to match the first feature point information and the third feature point information of the previous image frame data to obtain a second matching result. The second feature matching strategy is used to represent the feature matching rule to be executed on the first feature point information and the third feature point information. The previous time corresponding to the previous image frame data is later than the historical time and earlier than the current time. The third feature point information is used to represent the image corner feature of the current scene at the previous time under the corresponding scene category. The second matching result is used to represent the feature matching degree between the first feature point information and the third feature point information.

8. An electronic device, characterized in that, include: Memory, which stores computer programs; A processor for running the program, wherein the program executes the feature matching method according to any one of claims 1 to 6 when it runs.

9. A vehicle, characterized in that, It includes the electronic device as described in claim 8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method described in any one of claims 1 to 6.

Images