Feature extraction and matching method, system, electronic device and medium

Abstract

The application provides a feature extraction and matching method, system, electronic device and medium. The method fuses vehicle point cloud information and multiple frames of environment images, pre-processes to generate a point cloud set and a first semantic feature of a feature region containing a semantic boundary pixel set, calculates a geometric error by projecting the point cloud to a semantic segmentation image, optimizes the semantic region in combination with a preset geometric prior, corrects boundary disconnection and sawtooth noise, enhances the geometric constraint and semantic stability of the semantic feature, and solves the problem of difficult differentiation of geometrically similar structures. Meanwhile, the same semantic features of multiple frames are optimized across frames, single-frame noise is weakened through tracking filtering and time sequence smoothing, time sequence continuous features are generated, and the robustness of complex scenes is improved. Descriptors are generated based on the optimized features, the complexity of feature description is compressed, the distinguishability is improved, and fast and accurate inter-frame matching is achieved, thereby effectively improving the feature extraction quality and matching efficiency.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a feature extraction and matching method, system, electronic device and medium. Background Technology

[0002] In current autonomous driving technologies, vehicles need to extract and match features from images of their surrounding environment to perform pose calculations and autonomous driving control. However, current feature extraction and matching schemes typically rely on semantic segmentation techniques to identify different feature regions in the vehicle's environment images and perform feature matching. However, feature recognition based on semantic segmentation cannot distinguish between geometrically similar structures and background interference, and directly using semantic information can lead to significant noise in the extracted features, resulting in low feature extraction quality and inefficient feature matching. Summary of the Invention

[0003] Based on this, in order to improve the quality of the extracted features and ensure the efficiency of feature matching, embodiments of this application provide a feature extraction and matching method, system, electronic device, and medium.

[0004] The embodiments of this application disclose the following technical solutions:

[0005] In a first aspect, embodiments of this application provide a feature extraction and matching method applied to vehicles, the method comprising:

[0006] Based on the point cloud information of the vehicle within a preset body area and the multiple frames of vehicle environment images continuously acquired within a preset time period, data preprocessing is performed to determine the point cloud set for the vehicle environment and the first semantic feature for each frame of the vehicle environment image; the first semantic feature includes: a set of semantic boundary pixels and a first semantic feature region.

[0007] For each frame of the vehicle environment image, the point cloud set is subjected to coordinate projection transformation based on the first semantic feature region to generate a first projection pixel set; the first projection pixel set is divided into regions based on the semantic type of each projection point.

[0008] Based on the semantic type of each projection point in the first projection pixel set and the semantic boundary pixel set, the geometric error between each projection point and the corresponding semantic boundary is calculated to generate an observation error set; the observation error set consists of a list of observation errors corresponding to multiple different semantic types.

[0009] The first semantic feature region is optimized using the observation error set and the preset semantic geometric prior to generate a second semantic feature region with geometric constraints. The second semantic feature region and the set of semantic boundary pixels are then defined as the second semantic feature. The preset semantic geometric prior is used to characterize the geometric rules set for different semantic features.

[0010] Based on the second semantic features of the same semantic type in multiple frames of the vehicle environment image, cross-frame semantic feature optimization is performed to generate temporally continuous third semantic features;

[0011] Based on the third semantic features in each frame of the vehicle environment image, a descriptor for each vehicle environment image is generated, and inter-frame feature matching is achieved through the descriptors of each vehicle environment image.

[0012] In one possible implementation, the step of optimizing the first semantic feature region using the observation error set and a preset semantic geometric prior to generate a second semantic feature region with geometric constraints includes:

[0013] For all feature points within the first semantic feature region, noise points are removed according to the observation error list corresponding to each feature point, so as to remove feature points with errors greater than a preset first threshold, thereby obtaining a feature point set.

[0014] Based on the spatial clustering algorithm, feature points in the feature point set whose pixel spacing is not greater than a preset second threshold are merged to generate multiple semantic feature candidate regions.

[0015] The second semantic feature region is generated by optimizing the regional features of each of the semantic feature candidate regions using the preset semantic geometry prior.

[0016] In one possible implementation, the step of performing cross-frame semantic feature optimization based on the second semantic features of the same semantic type in multiple frames of the vehicle environment images to generate temporally continuous third semantic features includes:

[0017] The second semantic features of the same semantic type in multiple frames of the vehicle environment image are subjected to tracking filtering to generate cross-frame feature trajectories and current frame features;

[0018] Based on the preset semantic geometric prior and the cross-frame feature trajectory, feature constraint processing is performed on the current frame features to generate a set of current frame feature boundary points that conform to the preset cross-frame rules;

[0019] The current frame feature boundary point set that conforms to the preset cross-frame rules is subjected to temporal smoothing processing to generate the third semantic feature.

[0020] In one possible implementation, generating a descriptor for each vehicle environment image based on the third semantic feature in each frame of the vehicle environment image includes:

[0021] Extract the pixel gradient information and associated environment information of the third semantic feature in each frame of the vehicle environment image, and divide the pixel gradient information into multiple gradient strips through the associated environment information;

[0022] For each frame of the vehicle environment image, the descriptor is generated based on the multiple gradient strips, the boundary attributes of the third semantic feature, and the vehicle environment information represented in the vehicle environment image.

[0023] In one possible implementation, the vehicle includes multiple onboard cameras; the step of performing coordinate projection transformation on the point cloud set based on the first semantic feature region to generate a first projected pixel set includes:

[0024] Based on the intrinsic and extrinsic parameters of the vehicle-mounted camera, the point cloud set is projected onto a two-dimensional pixel plane to generate a second projected pixel set.

[0025] For each frame of the vehicle environment image, the second projection pixel set is divided into the semantic boundary pixel set and the semantic feature region to generate the first projection pixel set.

[0026] In one possible implementation, after generating descriptors for each of the vehicle environment images, the method further includes:

[0027] The descriptor is then binary-coded.

[0028] In one possible implementation, the semantic type includes: static target, dynamic target, and environmental background; the method of region division based on the semantic type of each projection point in the first projection pixel set includes:

[0029] The region is divided into regions that contain only projection points whose semantic type is the static target or the environmental background.

[0030] Secondly, embodiments of this application provide a feature extraction and matching system applied to vehicles, the system comprising:

[0031] The data preprocessing module is used to perform data preprocessing based on the point cloud information of the vehicle within a preset body range and multiple frames of vehicle environment images continuously acquired within a preset time period, so as to determine the point cloud set for the vehicle environment and the first semantic feature for each frame of the vehicle environment image; the first semantic feature includes: a set of semantic boundary pixels and a first semantic feature region.

[0032] The projection module is used to perform coordinate projection transformation on the point cloud set based on the first semantic feature region for each frame of the vehicle environment image to generate a first projection pixel set; the first projection pixel set is divided into regions based on the semantic type of each projection point.

[0033] The error observation module is used to calculate the geometric error between each projection point and the corresponding semantic boundary based on the semantic type of each projection point in the first projection pixel set and the semantic boundary pixel set, so as to generate an observation error set; the observation error set is composed of observation error lists corresponding to multiple different semantic types.

[0034] The first feature optimization module is used to optimize the region features of the first semantic feature region by using the observation error set and the preset semantic geometric prior, so as to generate a second semantic feature region with geometric constraints, and to determine the second semantic feature region and the set of semantic boundary pixels as the second semantic feature; the preset semantic geometric prior is used to characterize the geometric rules set for different semantic features.

[0035] The second feature optimization module is used to perform cross-frame semantic feature optimization based on the second semantic features of the same semantic type in multiple frames of the vehicle environment image, so as to generate a temporally continuous third semantic feature.

[0036] The feature matching module is used to generate descriptors for each vehicle environment image based on the third semantic features in each frame of the vehicle environment image, and to achieve inter-frame feature matching through the descriptors of each vehicle environment image.

[0037] Thirdly, embodiments of this application provide an electronic device, the device including: a processor, a memory, and a system bus;

[0038] The processor and the memory are connected via the system bus;

[0039] The memory is used to store one or more programs, the one or more programs including instructions that, when executed by the processor, cause the processor to perform any possible feature extraction and matching method in the first aspect.

[0040] Fourthly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any possible feature extraction and matching method in the first aspect.

[0041] Compared with existing technologies, this application has the following beneficial effects: This application provides a feature extraction and matching method, system, electronic device, and medium. In this method, firstly, a point cloud set and the first semantic features of multiple frames of images are fused. Geometric errors are calculated using the projected pixel set generated by projecting the point cloud onto the pixel plane and the semantic boundary pixel set. The first semantic feature region is optimized by combining the observation error set with a preset semantic geometric prior, effectively correcting problems such as broken semantic boundaries and jagged noise, and supplementing boundary integrity. This ensures that the extracted semantic features both conform to geometric constraints and possess stable semantic attributes, solving the defect in traditional methods where similar geometric features are difficult to distinguish. Simultaneously, through cross-frame semantic feature optimization, tracking filtering, constraint correction, and temporal smoothing of continuous multi-frame features are used to reduce single-frame noise and offset, generating temporally continuous stable features. This further enhances the robustness of features under complex scenarios such as illumination changes and dynamic interference, improving the quality of feature extraction. In terms of feature matching efficiency, the descriptor that integrates semantic boundary attributes and environmental information is generated based on the optimized third semantic features, which greatly reduces the complexity of feature description, improves the discriminativeness and computational efficiency of the descriptor, and enables inter-frame matching to quickly and accurately establish correspondence, thereby improving the efficiency of feature matching. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 A flowchart illustrating a feature extraction and matching method provided in an embodiment of this application;

[0044] Figure 2 A flowchart illustrating a second semantic feature region generation method provided in an embodiment of this application;

[0045] Figure 3 A flowchart illustrating a third semantic feature generation method provided in an embodiment of this application;

[0046] Figure 4 This is a schematic diagram of the structure of a feature extraction and matching system provided in an embodiment of this application;

[0047] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with specific embodiments and accompanying drawings. It should be particularly noted that the embodiments described in this application are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0049] It should be noted that, unless otherwise defined, the technical or scientific terms used in the embodiments of this application should have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. The terms "first," "second," and similar terms used in the embodiments of this application do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed after the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are only used to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0050] As described above, current feature extraction and matching schemes typically rely on semantic segmentation technology to identify different feature regions in vehicle environment images and perform feature matching. However, feature recognition based on semantic segmentation technology cannot distinguish between geometrically similar structures and background interference, and directly using semantic information will result in large noise in the extracted features, leading to problems such as low feature extraction quality and low feature matching efficiency.

[0051] To address the aforementioned issues, this application provides a feature extraction and matching method, system, electronic device, and medium. In this method, a point cloud set and the first semantic features of multiple frames of images are first fused. Geometric errors are calculated using the projected pixel set generated by projecting the point cloud onto a pixel plane and the semantic boundary pixel set. The first semantic feature region is then optimized by combining the observation error set with a preset semantic geometric prior. This effectively corrects issues such as broken semantic boundaries and jagged noise, and improves boundary integrity, ensuring that the extracted semantic features both conform to geometric constraints and possess stable semantic attributes. This overcomes the difficulty in distinguishing similar geometric structures in traditional methods. Furthermore, through cross-frame semantic feature optimization, tracking filtering, constraint correction, and temporal smoothing of continuous multi-frame features are used to reduce single-frame noise and offset, generating temporally continuous and stable features. This further enhances the robustness of features under complex scenarios such as illumination changes and dynamic interference, improving the quality of feature extraction. In terms of feature matching efficiency, the descriptor that integrates semantic boundary attributes and environmental information is generated based on the optimized third semantic features, which greatly reduces the complexity of feature description, improves the discriminativeness and computational efficiency of the descriptor, and enables inter-frame matching to quickly and accurately establish correspondence, thereby improving the efficiency of feature matching.

[0052] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0053] See Figure 1 The figure is a flowchart illustrating a feature extraction and matching method provided in an embodiment of this application, specifically including the following steps:

[0054] S101: Based on the point cloud information of the vehicle within a preset body range and multiple frames of vehicle environment images continuously acquired within a preset time period, perform data preprocessing to determine the point cloud set for the vehicle environment and the first semantic feature for each frame of the vehicle environment image; the first semantic feature includes: a set of semantic boundary pixels and a first semantic feature region.

[0055] In the vehicle used in this embodiment, a LiDAR for acquiring point cloud information and an onboard camera for acquiring images of the vehicle's environment are installed inside the vehicle. At the point cloud information acquisition level, the vehicle relies on the onboard LiDAR to perform a three-dimensional scan of a preset area around the vehicle. The LiDAR is typically deployed on the roof or at the four corners of the vehicle to ensure full-angle or key area coverage around the vehicle, such as 150 meters in front, 80 meters behind, and 50 meters to the left and right. In one possible implementation, considering the high requirements of autonomous driving for environmental detail, the selection of LiDAR prioritizes high resolution (e.g., 128 lines, 64 lines) and high frame rate (e.g., scanning frequency above 10Hz) to capture the three-dimensional coordinates and reflection intensity information of dynamic traffic participants and static infrastructure (e.g., road markings, guardrails) in real time. This type of point cloud data acquisition possesses high density and high precision characteristics, accurately describing the geometric contours of the environment surrounding the vehicle, providing an original spatial coordinate basis for the subsequent construction of point cloud sets, and playing a crucial role in the detailed representation of scenarios such as complex intersections and curved road sections.

[0056] The acquisition of multi-frame vehicle environment images relies on an onboard camera array. In one possible implementation, multiple cameras such as front-view, rear-view, and surround-view cameras are usually configured on the vehicle to cover a 270-degree to 360-degree field of view around the vehicle. Each camera uses a different focal length according to the viewing angle requirements (such as a front-view telephoto lens for distant target detection and a side-view wide-angle lens for blind spot compensation). The continuous acquisition within a preset time (0.5 seconds) is achieved through a synchronous triggering mechanism. The frame rate is usually matched with the autonomous driving control cycle to ensure temporal continuity.

[0057] After acquiring point cloud information and multiple frames of vehicle environment images through vehicle-mounted LiDAR and vehicle-mounted cameras, the point cloud information within a preset vehicle range around the vehicle is first processed. The preset vehicle range ensures coverage of the core perception area for autonomous driving while avoiding redundant data interference. After preprocessing such as denoising filtering and spatial clustering, the point cloud information forms a point cloud set focused on the vehicle environment, fully preserving the three-dimensional coordinates and reflection intensity information of key environmental elements such as road surfaces, guardrails, and building facades, providing a precise spatial location benchmark for subsequent geometric constraint analysis. Simultaneously, multiple frames of vehicle environment images acquired within a preset time period (e.g., 20 consecutive frames within 0.5 seconds) are used to extract the first semantic features using semantic segmentation technology and a semantic segmentation network. The semantic boundary pixel set accurately locates the contour edges of each semantic target (such as lane lines, traffic signs, and obstacles), while the first semantic feature region delineates the pixel-level range of each semantic target. Together, they construct a semantic structure representation at the image level.

[0058] This step aims to overcome the limitations of single-image semantic segmentation by injecting geometric prior constraints into feature extraction through the fusion of point clouds and images. Traditional feature extraction schemes relying solely on image semantic segmentation are susceptible to interference from lighting variations and occlusion, leading to blurred boundaries or misclassification of regions. The introduction of point cloud sets provides a three-dimensional spatial location verification basis for semantic features. For example, point cloud projection can help verify whether the semantic region corresponding to the marked "lane lines" in the vehicle environment image matches the actual road surface point cloud distribution, thus filtering out obviously unreasonable semantic segmentation results during the preprocessing stage. Simultaneously, the processing strategy for consecutive multi-frame images constructs a temporal data foundation. In subsequent processing, it can further correct single-frame semantic segmentation noise through inter-frame comparison and provide an initial feature sequence for subsequent cross-frame feature optimization, ensuring the semantic accuracy and geometric consistency of feature extraction from the source.

[0059] S102: For each frame of the vehicle environment image, the point cloud set is transformed by coordinate projection based on the first semantic feature region to generate a first projection pixel set; the first projection pixel set is divided into regions based on the semantic type of each projection point.

[0060] The core of this step lies in accurately binding the spatial location information represented by the point cloud set to the semantic region of the image, providing fused data with both geometric accuracy and semantic attributes for subsequent error analysis and feature optimization. Specifically, the first projected pixel set generated in this step is a structured data set that fuses 3D point cloud geometric information and 2D image semantic features. Each projected point in the set contains both 3D spatial coordinates and a 2D semantic label. Through precise projection transformation of the intrinsic and extrinsic parameters of the vehicle-mounted camera, the 3D point cloud is mapped to the 2D pixel plane of the corresponding frame, ensuring that the spatial position of each projected point is strictly aligned with the semantic region of the image, forming a direct mapping relationship of "semantic label - spatial coordinates," providing a fundamental association for cross-modal data fusion.

[0061] Meanwhile, based on the preset first semantic feature region in the image, the projection points are divided into independent subsets according to semantic type, such as road surface, curb, dynamic vehicles, and pedestrians. Each subset contains only projection points of a specific semantic category, ensuring that subsequent processing can formulate differentiated strategies for different categories of semantic features, thereby improving the targeting and efficiency of feature processing.

[0062] Specifically, in one possible implementation, the semantic types of the projection points can be divided into three types: static targets, dynamic targets, and environmental background. Correspondingly, when dividing the region based on the semantic types of each projection point in the first projection pixel set, only the regions corresponding to projection points with the semantic type of static target or environmental background need to be divided; there is no need to divide the regions containing projection points with the semantic type of dynamic target. This setting aims to ensure the stability and reliability of subsequent feature extraction. This is because the non-fixed position of dynamic targets would introduce noise if they participated in region division, leading to false associations during feature matching. Static targets, on the other hand, have long-term spatial consistency with the environmental background, making them more suitable as stable localization and mapping features in structured scenes.

[0063] Specifically, the process of performing coordinate projection transformation on the point cloud set based on the first semantic feature region to generate the first projected pixel set is achieved through the following two steps:

[0064] Step 1: Based on the intrinsic and extrinsic parameters of the vehicle-mounted camera, project the point cloud set onto a two-dimensional pixel plane to generate a second projected pixel set.

[0065] First, a coordinate projection transformation is performed based on the camera's intrinsic and extrinsic parameters. The intrinsic parameters of the vehicle-mounted camera include inherent attributes such as focal length and principal point coordinates, used to correct lens distortion and transform the 3D point cloud from the camera coordinate system to the pixel plane. Extrinsic parameters describe the camera's position and orientation in the vehicle coordinate system. Through rotation and translation transformations, the point cloud set is precisely mapped from the global 3D space to the corresponding 2D imaging plane of each camera. This process uses perspective projection formulas to convert each 3D point (X, Y, Z) in the point cloud set into pixel coordinates (u, v), generating a second projected pixel set containing the positions of all projected points. The second projected pixel set only completes the cross-modal transformation of geometric positions and has not yet been associated with semantic information, but it provides a pixel-level spatial localization foundation for subsequent semantic segmentation. That is, each projected point clearly corresponds to a specific pixel position in the image, ensuring that the 3D information of the point cloud, such as distance and orientation, is strictly aligned with the pixel coordinates of the image, creating conditions for the spatial binding of semantic features.

[0066] Step 2: For each frame of the vehicle environment image, the second projection pixel set is divided into the semantic boundary pixel set and the semantic feature region to generate the first projection pixel set.

[0067] After obtaining the second projected pixel set, semantic attributes are assigned and classified to the projected points in the second projected pixel set based on the first semantic feature region. Specifically, if a projected point is located within the pixel range of a certain semantic feature region, it is assigned to the corresponding semantic feature region subset. If a projected point falls on the boundary edge of multiple semantic regions, or on the boundary line of the region defined by semantic segmentation, it is assigned to the semantic boundary pixel set. This division deconstructs the second projected pixel set into a structured subset with clear semantic attributes. That is, points within the semantic feature region carry labels of the corresponding category (such as "lane line" or "pedestrian"), while points in the boundary pixel set record the edge position information across regions, which is used for subsequent contour optimization or ambiguous region verification of the semantic segmentation results. Through this process, the second projected pixel set is transformed from a simple set of geometric coordinates into a first projected pixel set that integrates semantic information. Each projected point in the first projected pixel set not only retains its three-dimensional coordinates and pixel position, but is also assigned its own semantic category or boundary attribute, forming a multi-dimensional data structure of "geometric position - semantic label - boundary state". This method of dividing the first projection pixel set not only achieves semantic filtering of the point cloud, but also preserves the spatial correlation between semantic feature regions through the boundary point set, ensuring that the point cloud projection result can accurately correspond to the semantic content of the image and reflect the spatial structural features of the real environment.

[0068] S103: Based on the semantic type of each projection point in the first projection pixel set and the semantic boundary pixel set, calculate the geometric error between each projection point and the corresponding semantic boundary to generate an observation error set; the observation error set consists of a list of observation errors corresponding to multiple different semantic types.

[0069] After obtaining the first set of projected pixels that integrates semantic information and spatial coordinates, it is necessary to further analyze the matching degree between each projected point in the set and its corresponding semantic boundary. This process is achieved by calculating geometric errors. First, each projected point is classified into a specific semantic category based on its own semantic type (such as lane lines, road surface, obstacles, etc.), while the semantic boundary pixel set records the edge position information of each semantic region. For any projected point in the first set of projected pixels, the calculation of geometric errors needs to be combined with the boundary features of its semantic type. If the projected point belongs to a certain semantic feature region, the distance between the point and the semantic boundary of that category needs to be calculated. This may manifest as the vertical distance from the point to the boundary line segment, the closest distance between the point and the boundary contour, or the deviation between the corresponding position of the point in three-dimensional space and the geometry defined by the semantic boundary.

[0070] For projection points located within the set of semantic boundary pixels (i.e., points at the intersection of multiple semantic regions), error calculation focuses more on cross-regional positional consistency. For example, it determines whether the point simultaneously conforms to the spatial constraints of adjacent semantic categories, or detects whether its 3D coordinates conflict with the geometric characteristics of the semantic regions on both sides (e.g., whether the height difference between the boundary points of "road surface" and "curbstone" conforms to the real scene). This error calculation not only quantifies the positional deviation between the projection point and the semantic boundary, but also reflects the degree of fit between the semantic segmentation result and the point cloud geometric information. That is, if the geometric error of a large number of projection points in a certain semantic region exceeds the threshold, it may mean that there is a deviation in the semantic boundary division of the region, or that there is a calibration error in the point cloud projection process.

[0071] The final generated set of observation errors is not a single data list, but rather structured data organized according to semantic types. The system creates a separate observation error list for each semantic type (such as lane lines, obstacles, road surfaces, etc.), and each list contains the geometric error values ​​and corresponding spatial location information of all projection points under that category.

[0072] Taking the semantic type "sedan" as an example, the geometric error between each projection point and the outline boundary of the sedan in the image is calculated. A projection point corresponds to the position of the left front wheel of the sedan in 3D space. After being projected onto the image, it is located within the semantic region of the sedan, but there is a 3D distance deviation of 20 cm from the outline boundary of the semantic feature region (represented as a vertical distance of 5 pixels in the pixel plane). This error value is recorded in the observation error list for the semantic type "sedan". If multiple projection points located at the bottom of the sedan show a height error of more than 15 cm from the "road surface" boundary (i.e., the point cloud height is significantly higher than the normal height range of the road surface semantic region), it indicates that the semantic boundary at this location may have mistakenly classified the bottom of the sedan into the road surface region, and the segmentation outline needs to be corrected.

[0073] Taking the "lane line" semantic type as an example, the projection points of this type must conform to the geometric features of a straight line. If a point deviates from the fitted straight line of the lane line semantic boundary by more than 8 centimeters, it is marked as a significant error, indicating that there may be a break or offset in the lane line segmentation, and the error value is recorded in the observation error list of the semantic type "lane line".

[0074] Ultimately, the observation error set contains multiple independent error lists, such as "car" and "lane line," with each list recording the error value, spatial coordinates, and deviation direction of all projection points under the corresponding semantic type. By analyzing the concentrated error regions for the car category, the vehicle contour detection accuracy of the image segmentation model can be optimized; for significant errors in the lane line category, the existence of translational deviations in the camera calibration parameters can be verified. This error set categorized by semantic type enables the system to accurately locate inconsistencies in cross-modal data, providing specific quantitative basis for subsequent semantic segmentation optimization and sensor calibration.

[0075] S104: Using the observation error set and the preset semantic geometric prior, perform regional feature optimization on the first semantic feature region to generate a second semantic feature region with geometric constraints, and determine the second semantic feature region and the set of semantic boundary pixels as the second semantic feature; the preset semantic geometric prior is used to characterize the geometric rules set for different semantic features.

[0076] First, the observation error set records the geometric deviation between the projection points of each semantic type and the corresponding semantic boundaries. These error values ​​directly reflect the deficiencies in the spatial localization of the first semantic feature region. For example, the boundary of a certain semantic region may be offset due to the limitations of the image segmentation algorithm (such as insufficient resolution or illumination interference), or spatial misalignment may occur due to calibration errors during point cloud projection. The pre-defined semantic geometric priors are rules set for the inherent geometric properties of different semantic features. For example, "lane lines" should conform to the geometric shape of continuous straight lines or curves, "vehicles" should have the spatial characteristics of a rigid cuboid structure, and "road surface" should satisfy the high continuity of horizontal or gradually changing slopes. This prior knowledge provides a clear geometric constraint framework for region optimization, avoiding unreasonable partitioning that may result from relying solely on single-modal data (such as image semantic segmentation).

[0077] During the optimization of the first semantic feature region, it is necessary to identify regions with significant errors (such as where the geometric error of multiple consecutive projection points near a semantic boundary exceeds a preset threshold) and determine whether the semantic segmentation result of this region conflicts with the point cloud geometric information. For example, if the height of the projected point cloud at the bottom of the "vehicle" semantic region is generally higher than the normal height range of the "road surface" semantic region, it indicates that the bottom outline of the vehicle may be incorrectly segmented, and the boundary of this region needs to be adjusted downwards to match the actual height distribution of the point cloud.

[0078] By using observation errors as the quantitative basis for optimization and pre-setting semantic geometric priors as constraints, the contour, range, and shape of the first semantic feature region are corrected region by region to generate a second semantic feature region with geometric constraints. This process not only utilizes the spatial accuracy of point cloud data to compensate for positional deviations in image semantic segmentation, but also eliminates segmentation results that do not conform to physical reality through prior geometric rules, so that the second semantic feature region achieves consistency in two-dimensional image representation and three-dimensional spatial positioning.

[0079] To further understand the generation process of the second semantic feature region in step S104, the following will describe the generation process of the second semantic feature region with reference to the accompanying drawings of a specific implementation embodiment. See [link to documentation]. Figure 2 The figure is a flowchart illustrating a second semantic feature region generation method provided in an embodiment of this application, specifically including the following steps:

[0080] S1041: For all feature points within the first semantic feature region, noise points are removed according to the observation error list corresponding to each feature point, so as to remove feature points with errors greater than a preset first threshold, thereby obtaining a feature point set.

[0081] Each feature point within the first semantic feature region corresponds to an observation error list for its respective semantic type. These error values ​​quantify the degree of deviation between the feature point and the corresponding semantic boundary or geometric constraint. For example, the error of a feature point belonging to the "vehicle" semantic type may reflect the distance deviation between the point and the vehicle contour boundary in three-dimensional space, or the pixel distance deviating from the semantic segmentation contour on the two-dimensional pixel plane.

[0082] In practice, all feature points within the first semantic feature region are iterated one by one. For each feature point, its error value in the corresponding observation error list is extracted and compared with a preset first threshold. When the error value of a feature point exceeds the corresponding threshold, the point is determined to be a noise point and is removed. For example, if the error value of a feature point with the semantic type "lane line" shows that its deviation from the fitted straight line of the lane line exceeds 8 centimeters (exceeding the preset 5-centimeter threshold), then the point is considered to be a false detection point caused by road surface reflection or debris interference and is removed. In one possible implementation, the preset first threshold is set based on the characteristics of the semantic type and the actual scene requirements. For semantic categories with strong geometric regularity (such as "lane line", which needs to meet continuous linear features), the threshold is usually set more strictly to filter out noise points that significantly deviate from the straight line distribution; for semantic categories that allow a certain degree of flexibility (such as "pedestrian", due to morphological changes in body movements), the threshold is relatively loose to avoid mistakenly deleting feature points with reasonable fluctuations.

[0083] S1042: Based on the spatial clustering algorithm, feature points in the feature point set whose pixel spacing is not greater than a preset second threshold are merged to generate multiple semantic feature candidate regions.

[0084] After noise removal from the feature point set, the remaining feature points need to be further organized and aggregated spatially to form semantic regions with actual physical meaning. This process is achieved through spatial clustering algorithms, the core of which lies in merging neighboring points based on pixel spacing.

[0085] Specifically, each point in the feature point set carries two-dimensional pixel coordinates. These points, in terms of spatial distribution, should correspond to different positional features of the same semantic target in a real scene (such as multiple points on a vehicle outline or continuous sampling points on a lane line). However, due to factors such as sensor sampling density and the discreteness after noise removal, these points may exist in a scattered form. Therefore, clustering algorithms are needed to group spatially adjacent points into the same group. The choice of spatial clustering algorithm needs to be adapted to pixel-level distance metrics. Common methods include the density-based DBSCAN algorithm. The core processing logic of the algorithm is: if the Euclidean distance (i.e., pixel spacing) between two feature points on the pixel plane is not greater than a preset second threshold, then the two are determined to be potential members of the same semantic feature candidate region, and neighboring points are iteratively merged until all points that meet the distance condition are included in the corresponding group.

[0086] During clustering, the algorithm iterates through the feature point set, using each unlabeled point as a seed, searching for all neighboring points whose pixel distance is no greater than a preset second threshold to form initial clusters. Then, it recursively searches the neighborhoods of these neighboring points until no new neighboring points can be added, thus generating a complete semantic feature candidate region. This process automatically filters out discrete, isolated points and aggregates spatially continuous feature points into several compact regions. For example, multiple feature points belonging to the same car, although distributed at different locations on the vehicle body (front, rear, side) after point cloud projection, will be merged into a single candidate region containing the entire vehicle outline because their pixel distance is less than the threshold. Feature points belonging to different lane lines, however, will be divided into independent candidate regions because their distance exceeds the threshold. The generated semantic feature candidate region is essentially a spatial grouping of the feature point set. The points within each region form a tightly related set on the pixel plane, corresponding to potential carriers of the same semantic target or structure in the real scene. This transforms disordered feature points into candidate regions with clear spatial boundaries, laying the data foundation for subsequent accurate semantic region segmentation and ultimately improving the accuracy of feature extraction.

[0087] S1043: Optimize the regional features of each of the semantic feature candidate regions using the preset semantic geometry prior to generate the second semantic feature region.

[0088] Predefined semantic geometric priors are constraint rules predefined for typical geometric features of different semantic categories (such as vehicles, lane lines, pedestrians, road surfaces, etc.). For example, vehicles should have the spatial characteristics of a rigid cuboid structure, and lane lines should satisfy the geometric continuity of continuous curves or straight lines, etc. During the feature optimization of semantic feature candidate regions using predefined semantic geometric priors, it is necessary to call the semantic type of each semantic feature candidate region and call the predefined semantic geometric prior corresponding to that semantic type for constraint adjustments. Taking the semantic type of vehicle as an example of a semantic feature candidate region, if the region contour has obvious depressions or fragmentation, the missing contour edges will be filled in using a geometric fitting algorithm (such as least squares fitting of a 3D bounding box) based on the vehicle's cuboid prior, so that the region boundary conforms to the expected shape of a rigid structure. This optimizes the semantic feature candidate region, thus obtaining the final second semantic feature region.

[0089] Similarly, by using this method to traverse multiple frames of vehicle environment images acquired within the same image period, the interface determines the second semantic feature region corresponding to each vehicle environment image.

[0090] The above is a flowchart describing the second semantic feature region generation method in the embodiments of this application. Next, we will continue to combine... Figure 1 The following is a detailed description of steps S104-S105, which follow step S104:

[0091] S105: Based on the second semantic features of the same semantic type in multiple frames of the vehicle environment images, perform cross-frame semantic feature optimization to generate temporally continuous third semantic features.

[0092] After optimizing the geometric constraints of the semantic features in a single frame, it is necessary to further utilize the temporal series information of multi-frame data to integrate and optimize the second semantic features of the same semantic type across frames. This addresses issues that may exist in single-frame processing, such as residual noise, feature fragmentation caused by occlusion, and discontinuous dynamic target trajectories, ultimately generating a temporally continuous third semantic feature. The core of this process lies in transforming the independent semantic regions of a single frame into a dynamic expression with temporal consistency through the association of semantic features between multiple frames. This enables the autonomous driving system to perceive continuous changes in semantic targets in the environment, rather than isolated static fragments. Next, the generation method of the third semantic feature in step S105 will be described with reference to the accompanying drawings of a specific process embodiment.

[0093] See Figure 3 The figure is a flowchart illustrating a third semantic feature generation method provided in an embodiment of this application, specifically including the following steps:

[0094] S1051: Perform tracking filtering on the second semantic features of the same semantic type in multiple frames of the vehicle environment images to generate cross-frame feature trajectories and current frame features.

[0095] After completing the geometric optimization of the semantic features of a single frame and obtaining the second semantic features, it is necessary to perform tracking filtering on features of the same semantic type in multiple frames of images to establish the correlation of features in the time dimension and optimize the expression of the current frame, and finally generate cross-frame feature trajectories and reliable current frame features.

[0096] After completing the geometric optimization of the semantic features of a single frame to obtain the second semantic features, the system needs to perform tracking filtering on the features of the same semantic type in multiple frames of vehicle environment images in order to establish the temporal dimension association and generate cross-frame feature trajectories and reliable current frame features.

[0097] This process, through dynamic modeling and data fusion, solves the problems of multi-frame feature matching, trajectory continuity maintenance, and noise suppression, transforming isolated single-frame semantic features into a dynamic expression with temporal consistency. In specific implementation, a data association algorithm is first used to establish a correspondence between adjacent frames for the same semantic feature based on the spatial location, geometric shape, and motion priors of the second semantic feature. For association interruptions caused by occlusion or detection omissions, historical trajectory prediction maintains target continuity. Subsequently, a state-space model is used to model the motion state of the semantic features, and algorithms such as Kalman filtering are used to fuse the predicted values ​​with the current frame's detection values: the position and shape of the current feature are predicted based on historical states, while predictions are corrected using measured data. Single-frame detection noise is filtered out by dynamically adjusting weights; for example, abnormal detection values ​​exceeding a reasonable motion range are smoothed or discarded. After frame-by-frame processing, each semantic feature generates a cross-frame trajectory, recording its position, shape, and confidence changes in the time series, mitigating problems such as position jitter and size abrupt changes in single-frame detection.

[0098] For the features of the current frame, optimization is performed by combining historical trajectories and prediction models: when detecting missing information, geometric information is supplemented by the continuity of previous and subsequent frames; when detecting contradictions, stable historical states are given priority. The final generated cross-frame feature trajectory and current frame features retain the spatial geometric details of a single frame while ensuring physical rationality through dynamic constraints in the time dimension. The dynamic target trajectory is smooth and conforms to the laws of motion, the static target representation is stable and resistant to interference, and short-term occlusion or detection anomalies are effectively filtered out.

[0099] S1052: Based on the preset semantic geometric prior and the cross-frame feature trajectory, perform feature constraint processing on the current frame features to generate a set of current frame feature boundary points that conform to the preset cross-frame rules.

[0100] After generating cross-frame feature trajectories and current frame features through tracking filtering, the current frame features need to be subjected to dual constraint processing by combining the preset semantic geometric priors and cross-frame trajectory information to generate a feature boundary point set that conforms to physical laws and temporal logic, ensuring that the semantic feature geometric expression not only fits the inherent attributes but also matches the historical motion trajectory.

[0101] Among them, cross-frame feature trajectories provide dynamic constraints from a temporal dimension, and the recorded historical position, shape, and motion state of the target become the basis for predicting the features of the current frame. Dynamic targets calculate reasonable position and shape change ranges based on motion parameters of previous frames. If the detected boundary points exceed the range, an anomaly is judged and corrected. Static target trajectories should be stable or rigidly change with the vehicle body's motion. If the "static" feature boundary points have irregular displacement, noise is corrected by combining prior knowledge.

[0102] In practice, the current frame feature boundary points are compared with semantic and geometric priors. Deviation points are filtered or adjusted through size, shape, and structural constraints, while a predictive boundary model is constructed based on the trajectory. For example, missing detection segments are completed by predicting the extension direction based on the curvature of historical lane lines, and the displacement range of boundary points is limited by the vehicle trajectory speed, smoothing out abnormal detection points. For short-term occluded targets, boundary points are completed by combining geometric priors and trajectory predictions to avoid feature breaks. If the detected boundary points of static targets deviate from the positions derived from the vehicle's self-motion, historical boundary points with stable trajectories are used first in combination with priors for correction to ensure consistent spatial representation.

[0103] The final generated feature boundary point set for the current frame possesses both semantic and geometric rationality and cross-frame temporal consistency. Its dynamic target boundary point changes conform to kinematic laws, while its static target boundary points are stable and conform to spatial logic. Geometric distortions and temporal inconsistencies caused by detection noise are mitigated. This constraint, through prior knowledge and trajectory fusion, endows the features with stronger physical interpretability. Vehicle boundary point positions are mutually verified with historical motion and typical morphology, and lane line extensions conform to prior knowledge of road structure, avoiding misleading decisions due to local detection errors. It provides autonomous driving with environmental semantic boundaries that fit perception results and conform to physical laws, ensuring the reliability of inputs to modules such as collision detection and path planning, reducing decision-making risks caused by feature anomalies, and improving the accuracy and robustness of environmental understanding in complex scenarios.

[0104] S1053: Perform temporal smoothing on the current frame feature boundary point set that conforms to the preset cross-frame rules to generate the third semantic feature.

[0105] After cross-frame feature trajectory tracking and geometric constraint correction, the feature boundary point set of the current frame satisfies the preset cross-frame rules (such as the parallelism of lane lines and the perpendicularity of light poles). However, due to factors such as sensor noise, sudden changes in illumination, or dynamic occlusion, the boundary points may still exhibit local positional deviations or discontinuities. In this case, temporal smoothing processing associates the boundary point set of the current frame with the similar semantic feature boundaries of several previous and subsequent frames, and optimizes and adjusts the coordinates of each boundary point using filtering algorithms (such as moving average, Gaussian filtering, or median filtering). For example, for a lane line boundary point in a frame that is slightly offset due to a brief change in illumination, the algorithm will refer to the stable direction of the lane line at the same location in the preceding and following frames, and perform a weighted average of its coordinates to bring it back to a trajectory that conforms to temporal consistency. For local boundary breaks caused by dynamic object occlusion, interpolation will be used to complete the boundary information of adjacent frames to ensure the continuity of the boundary. This processing not only reduces the high-frequency noise that is unavoidable in single-frame segmentation, but also enhances the stability of feature boundaries over time, ensuring a smooth transition of boundary point sets for semantic features such as lane lines and traffic sign poles across consecutive frames and avoiding abrupt positional jumps. The resulting third semantic feature retains the precise semantic attributes (such as category and geometric shape) of single-frame semantic segmentation and possesses consistent spatiotemporal continuity across frames, providing a stable feature foundation for the generation of subsequent descriptors.

[0106] S106: Generate a descriptor for each vehicle environment image based on the third semantic feature in each frame of the vehicle environment image, and achieve inter-frame feature matching through the descriptor of each vehicle environment image.

[0107] Generating descriptors from the third semantic features in each frame of the vehicle's environmental image and achieving inter-frame feature matching is the core step in transforming stable semantic features, which have undergone multiple optimizations, into computable and comparable feature representations, thereby establishing cross-frame feature associations. The third semantic features, as input, have undergone point cloud geometric constraints and temporal smoothing, possessing clear semantic attributes, precise boundary contours, and temporally consistent spatiotemporal continuity, providing a high-quality feature base for descriptor generation. When generating descriptors, the algorithm focuses on extracting the core discriminative information from the third semantic features. On the one hand, it captures visual structural information within the feature region, such as characterizing texture details and edge directions by calculating the direction and magnitude of pixel gradients. On the other hand, it embeds semantic and geometric priors, such as feature category labels, boundary curvature, and relative positions in the vehicle's coordinate system, making the descriptors not only include visual appearance features but also semantic interpretability. This information is organically integrated into feature vectors, forming a compact description of the semantic features of each frame, preserving the uniqueness of the features while filtering out irrelevant noise.

[0108] Specifically, the descriptors for each vehicle body environment image are generated through the following two steps;

[0109] Step 1: Extract the pixel gradient information and associated environment information of the third semantic feature in each frame of the vehicle environment image, and divide the pixel gradient information into multiple gradient strips through the associated environment information;

[0110] Step 2: For each frame of the vehicle environment image, generate the descriptor based on the multiple gradient strips, the boundary attributes of the third semantic feature, and the vehicle environment information represented in the vehicle environment image.

[0111] When generating descriptors for vehicle environment images, the first step is to extract pixel gradient information from the third semantic feature of each frame and perform structured partitioning. As stable features optimized by geometric constraints and temporal processes, the third semantic feature covers semantic regions with clear visual boundaries and coherent structural features. Pixel gradient information can effectively capture texture changes and edge directions within these regions. Simultaneously, associated environmental information within a certain range around the feature is collected. Specifically, with the semantic feature boundary point as the center, a fixed pixel radius (e.g., 15 pixels up, down, left, and right) is extended outwards, and environmental pixel information within this range is collected, including contextual data such as texture, grayscale distribution, and neighborhood feature correlations. This information constitutes the spatial context of the feature, providing spatial constraints for the structured processing of gradient information.

[0112] Next, the pixel gradient information is striped based on the predefined associated environment intervals. According to the geometric orientation of the semantic features (such as the linear extension of lane lines and the vertical boundary of light poles) and the spatial range of the associated environment intervals, continuous feature contours and their corresponding environmental regions are divided into multiple local gradient strips along the main feature direction. For example, for a vertically extending lane line feature, based on the feature's starting point, the feature and its surrounding associated intervals are divided into continuous gradient strips of a fixed length (e.g., 32 pixels). Each strip contains both the gradient vector of the feature boundary (reflecting the edge direction) and the gradient distribution pattern within its associated environment interval (e.g., the gradient consistency of the shoulder texture). This partitioning method essentially decomposes long-distance features into spatially local feature units. Each strip focuses on the gradient cooperation relationship between the feature's local contour and its neighboring environment, avoiding computational redundancy in global processing while preserving the spatial dependency between the feature and its surrounding environment through the limitation of the associated intervals. Through this step, the gradient information of stable semantic features is organized into structured strip units. Each strip carries the gradient correlation features between the local contour of the feature and the environment within a specific range, providing a hierarchical data foundation for the subsequent generation of efficient descriptors that fuse boundary details and environmental context.

[0113] After completing the gradient strip segmentation, the algorithm further integrates gradient strip information, boundary attributes of the third semantic feature, and vehicle environment information to generate a descriptor that combines visual detail with semantic logic. Boundary attributes encompass the geometric parameters of feature edges (such as the curvature value of boundary points, the slope and intercept of straight line segments, and the perimeter-to-area ratio of closed regions). These parameters directly reflect the morphological characteristics of the features, for example, distinguishing between straight lane lines and curved road markings. Vehicle environment information includes the semantic category of the feature, its relative position in the vehicle coordinate system (e.g., 1.5 meters to the left of the vehicle, 30 meters ahead), and surrounding semantic relationships (e.g., whether an obstacle is adjacent to a lane line). When this information is combined with the gradient strip data, the algorithm forms a multi-dimensional descriptive vector through feature vector concatenation or weighted fusion. Gradient strips represent the visual texture structure, boundary attributes reinforce the uniqueness of the geometric shape, and vehicle environment information injects semantic contextual constraints.

[0114] For example, a descriptor might contain histogram data of the orientation of each gradient strip, statistical values ​​of boundary curvature, feature class encoding, and coordinate offset relative to the vehicle body. The organic integration of this information allows the descriptor not only to identify visually similar features but also to filter invalid matches through semantic and geometric consistency (e.g., avoiding mismatches between features of different categories). This generation method endows the descriptor with stronger discriminative power and environmental adaptability. Especially in complex traffic scenarios, it can effectively reduce the probability of mismatches between similar structures (such as parallel lane lines or similar traffic signs), providing a feature representation that combines accuracy and robustness for subsequent inter-frame feature matching.

[0115] In particular, the generated descriptors can be binary-coded. Matching binary codes only requires calculating the Hamming distance (counting the number of binary bit differences). Compared with the traditional Euclidean distance or cosine similarity calculation of floating-point vectors, it eliminates complex floating-point operations, increasing the speed by several to tens of times, thereby significantly improving computational efficiency.

[0116] This application provides a feature extraction and matching method. Firstly, it fuses a point cloud set with the first semantic features of multiple frames. Then, it calculates the geometric error using the projected pixel set generated by projecting the point cloud onto the pixel plane and the semantic boundary pixel set. Finally, it optimizes the first semantic feature region by combining the observation error set with a preset semantic geometric prior, effectively correcting issues such as broken semantic boundaries and jagged noise, and improving boundary integrity. This ensures that the extracted semantic features conform to geometric constraints and possess stable semantic attributes, overcoming the difficulty in distinguishing similar geometric structures in traditional methods. Simultaneously, through cross-frame semantic feature optimization, it utilizes tracking filtering, constraint correction, and temporal smoothing of continuous multi-frame features to reduce single-frame noise and offset, generating temporally continuous and stable features. This further enhances the robustness of features in complex scenarios such as illumination changes and dynamic interference, improving the quality of feature extraction. Regarding feature matching efficiency, it generates a descriptor that fuses semantic boundary attributes and environmental information based on the optimized third semantic features. This significantly reduces the complexity of feature description, improves the discriminative power and computational efficiency of the descriptor, and enables rapid and accurate inter-frame matching to establish correspondences, thereby improving feature matching efficiency.

[0117] The following describes a feature extraction and matching system provided by an embodiment of this application. The feature extraction and matching system described below can be referred to in correspondence with the feature extraction and matching method described above.

[0118] See Figure 4 The figure is a schematic diagram of the structure of a feature extraction and matching system provided in an embodiment of this application, which specifically includes the following modules:

[0119] The data preprocessing module 100 is used to perform data preprocessing based on the point cloud information of the vehicle within a preset body range and multiple frames of vehicle environment images continuously acquired within a preset time period, so as to determine the point cloud set for the vehicle environment and the first semantic feature for each frame of the vehicle environment image; the first semantic feature includes: a set of semantic boundary pixels and a first semantic feature region.

[0120] The projection module 200 is used to perform coordinate projection transformation on the point cloud set based on the first semantic feature region for each frame of the vehicle environment image to generate a first projection pixel set; the first projection pixel set is divided into regions based on the semantic type of each projection point.

[0121] The error observation module 300 is used to calculate the geometric error between each projection point and the corresponding semantic boundary based on the semantic type of each projection point in the first projection pixel set and the semantic boundary pixel set, so as to generate an observation error set; the observation error set is composed of observation error lists corresponding to multiple different semantic types.

[0122] The first feature optimization module 400 is used to optimize the region features of the first semantic feature region by using the observation error set and the preset semantic geometric prior, so as to generate a second semantic feature region with geometric constraints, and to determine the second semantic feature region and the set of semantic boundary pixels as the second semantic feature; the preset semantic geometric prior is used to characterize the geometric rules set for different semantic features.

[0123] The second feature optimization module 500 is used to perform cross-frame semantic feature optimization based on the second semantic features of the same semantic type in multiple frames of the vehicle environment image, so as to generate a temporally continuous third semantic feature.

[0124] The feature matching module 600 is used to generate descriptors for each vehicle environment image based on the third semantic features in each frame of the vehicle environment image, and to achieve inter-frame feature matching through the descriptors of each vehicle environment image.

[0125] See Figure 5 The figure is a schematic diagram of the structure of an electronic device provided in an embodiment of this application, including:

[0126] Memory 11 is used to store computer programs;

[0127] The processor 12 is configured to implement the steps of the feature extraction and matching method described in any of the above method embodiments when executing the computer program.

[0128] In this embodiment, the device can be an in-vehicle computer, a PC (Personal Computer), or a terminal device such as a smartphone, tablet computer, handheld computer, or portable computer.

[0129] The device may include a memory 11, a processor 12, and a bus 13.

[0130] The memory 11 includes at least one type of readable storage medium, such as flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 11 may be an internal storage unit of the device, such as the hard disk of the device. In other embodiments, the memory 11 may be an external storage device of the device, such as a plug-in hard disk, SmartMedia Card (SMC), Secure Digital (SD) card, Flash Card, etc. Furthermore, the memory 11 may include both internal and external storage units of the device. The memory 11 can be used not only to store application software and various types of data installed on the device, such as program code executing fault scenario screening methods, but also to temporarily store data that has been output or will be output. In some embodiments, the processor 12 may be a central processing unit (CPU).

[0131] In some embodiments, processor 12 may be a central processing unit (CPU), controller, microcontroller, microprocessor or other data processing chip, used to run program code stored in memory 11 or process data, such as executing program code for feature extraction and matching methods.

[0132] This bus 13 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 5 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0133] Furthermore, the device may also include a network interface 14, which may optionally include a wired interface and / or a wireless interface (such as a Wi-Fi interface, a Bluetooth interface, etc.), typically used to establish communication connections between the device and other electronic devices.

[0134] Optionally, the device may further include a user interface 15, which may include a display, an input unit such as a keyboard, and optionally, a standard wired interface or a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen, etc. The display may also be appropriately referred to as a screen or display unit, used to display information processed in the device and to display a visual user interface.

[0135] Figure 5 Only devices with components 11-15 are shown; those skilled in the art will understand that... Figure 5 The structure shown does not constitute a limitation on the device and may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0136] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, this application also provides a computer-readable storage medium storing computer instructions for causing the computer to execute the feature extraction and matching method as described in any of the above embodiments.

[0137] The computer-readable media in this application embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0138] The computer instructions stored in the storage medium of the above embodiments are used to cause the computer to execute the feature extraction and matching method as described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0139] It should be noted that the various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for the system, method, electronic device, and medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the description of the method embodiments. The system, method, electronic device, and medium embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components indicated as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the solution in this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0140] The above description is merely one specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A feature extraction and matching method, characterized in that, Applied to vehicles, the method includes: Based on the point cloud information of the vehicle within a preset body area and the multiple frames of vehicle environment images continuously acquired within a preset time period, data preprocessing is performed to determine the point cloud set for the vehicle environment and the first semantic feature for each frame of the vehicle environment image; the first semantic feature includes: a set of semantic boundary pixels and a first semantic feature region. For each frame of the vehicle environment image, the point cloud set is subjected to coordinate projection transformation based on the first semantic feature region to generate a first projection pixel set; the first projection pixel set is divided into regions based on the semantic type of each projection point. Based on the semantic type of each projection point in the first projection pixel set and the semantic boundary pixel set, the geometric error between each projection point and the corresponding semantic boundary is calculated to generate an observation error set; the observation error set consists of a list of observation errors corresponding to multiple different semantic types. The first semantic feature region is optimized using the observation error set and the preset semantic geometric prior to generate a second semantic feature region with geometric constraints. The second semantic feature region and the set of semantic boundary pixels are then defined as the second semantic feature. The preset semantic geometric prior is used to characterize the geometric rules set for different semantic features. Based on the second semantic features of the same semantic type in multiple frames of the vehicle environment image, cross-frame semantic feature optimization is performed to generate temporally continuous third semantic features; Based on the third semantic features in each frame of the vehicle environment image, a descriptor for each vehicle environment image is generated, and inter-frame feature matching is achieved through the descriptors of each vehicle environment image.

2. The method according to claim 1, characterized in that, The step of optimizing the first semantic feature region using the observation error set and a preset semantic geometric prior to generate a second semantic feature region with geometric constraints includes: For all feature points within the first semantic feature region, noise points are removed according to the observation error list corresponding to each feature point, so as to remove feature points with errors greater than a preset first threshold, thereby obtaining a feature point set. Based on the spatial clustering algorithm, feature points in the feature point set whose pixel spacing is not greater than a preset second threshold are merged to generate multiple semantic feature candidate regions. The second semantic feature region is generated by optimizing the regional features of each of the semantic feature candidate regions using the preset semantic geometry prior.

3. The method according to claim 1, characterized in that, The second semantic features of the same semantic type in multiple frames of the vehicle environment images are used to perform cross-frame semantic feature optimization to generate temporally continuous third semantic features, including: The second semantic features of the same semantic type in multiple frames of the vehicle environment image are subjected to tracking filtering to generate cross-frame feature trajectories and current frame features; Based on the preset semantic geometric prior and the cross-frame feature trajectory, feature constraint processing is performed on the current frame features to generate a set of current frame feature boundary points that conform to the preset cross-frame rules; The current frame feature boundary point set that conforms to the preset cross-frame rules is subjected to temporal smoothing processing to generate the third semantic feature.

4. The method according to claim 1, characterized in that, The step of generating a descriptor for each vehicle environment image based on the third semantic feature in each frame of the vehicle environment image includes: Extract the pixel gradient information and associated environment information of the third semantic feature in each frame of the vehicle environment image, and divide the pixel gradient information into multiple gradient strips through the associated environment information; For each frame of the vehicle environment image, the descriptor is generated based on the multiple gradient strips, the boundary attributes of the third semantic feature, and the vehicle environment information represented in the vehicle environment image.

5. The method according to claim 1, characterized in that, The vehicle includes multiple vehicle-mounted cameras; the coordinate projection transformation of the point cloud set based on the first semantic feature region to generate a first projected pixel set includes: Based on the intrinsic and extrinsic parameters of the vehicle-mounted camera, the point cloud set is projected onto a two-dimensional pixel plane to generate a second projected pixel set. For each frame of the vehicle environment image, the second projection pixel set is divided into the semantic boundary pixel set and the semantic feature region to generate the first projection pixel set.

6. The method according to claim 1, characterized in that, After generating descriptors for each of the vehicle body environment images, the method further includes: The descriptor is then binary-coded.

7. The method according to claim 1, characterized in that, The semantic types include: static targets, dynamic targets, and environmental background; the method of region division based on the semantic types of each projection point in the first projection pixel set includes: The region is divided into regions that contain only projection points whose semantic type is the static target or the environmental background.

8. A feature extraction and matching system, characterized in that, Applied to vehicles, the system includes: The data preprocessing module is used to perform data preprocessing based on the point cloud information of the vehicle within a preset body range and multiple frames of vehicle environment images continuously acquired within a preset time period, so as to determine the point cloud set for the vehicle environment and the first semantic feature for each frame of the vehicle environment image; the first semantic feature includes: a set of semantic boundary pixels and a first semantic feature region. The projection module is used to perform coordinate projection transformation on the point cloud set based on the first semantic feature region for each frame of the vehicle environment image to generate a first projection pixel set; the first projection pixel set is divided into regions based on the semantic type of each projection point. The error observation module is used to calculate the geometric error between each projection point and the corresponding semantic boundary based on the semantic type of each projection point in the first projection pixel set and the semantic boundary pixel set, so as to generate an observation error set; the observation error set is composed of observation error lists corresponding to multiple different semantic types. The first feature optimization module is used to optimize the region features of the first semantic feature region by using the observation error set and the preset semantic geometric prior, so as to generate a second semantic feature region with geometric constraints, and to determine the second semantic feature region and the set of semantic boundary pixels as the second semantic feature; the preset semantic geometric prior is used to characterize the geometric rules set for different semantic features. The second feature optimization module is used to perform cross-frame semantic feature optimization based on the second semantic features of the same semantic type in multiple frames of the vehicle environment image, so as to generate a temporally continuous third semantic feature. The feature matching module is used to generate descriptors for each vehicle environment image based on the third semantic features in each frame of the vehicle environment image, and to achieve inter-frame feature matching through the descriptors of each vehicle environment image.

9. An electronic device, characterized in that, The device includes: a processor, a memory, and a system bus; The processor and the memory are connected via the system bus; The memory is used to store one or more programs, the one or more programs including instructions that, when executed by the processor, cause the processor to perform the feature extraction and matching method according to any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the feature extraction and matching method as described in any one of claims 1-7.

Images