Dynamic scene-based automatic driving infrared semantic odometer implementation method and system

Abstract

The application discloses an automatic driving infrared semantic odometer implementation method and system based on a dynamic scene, and the method comprises the following steps: acquiring an environment infrared image and performing image enhancement; extracting ORB feature points and edge line features of a target in the infrared image, and performing feature fusion matching to obtain an infrared image point-line fusion feature matching result; performing semantic segmentation on a dynamic target in the infrared image to obtain a semantic segmentation result; screening static features based on the infrared image point-line fusion feature matching result and the semantic segmentation result; and obtaining an infrared semantic positioning result by adopting epipolar geometry motion estimation according to the static features. The semantic segmentation is used to realize segmentation and extraction of static targets and dynamic targets in the infrared image, the edge line feature and the feature point fusion matching mode are used to make the remaining infrared image feature information rich enough, the frame-to-frame matching can be stably performed, and the positioning precision and the robustness are enhanced.

Description

Technical Field

[0001] This invention belongs to the field of automotive intelligent driving positioning technology, and more specifically, relates to a method for implementing infrared semantic odometry for autonomous driving based on dynamic scenes. Background Technology

[0002] Visual odometry can assist in the localization and trajectory generation of unmanned vehicles when satellite signals are weak or lost, which is of great significance for improving the stability of unmanned vehicle positioning systems. Common visual odometry can be divided into two types according to the wavelength of light received by the sensor: visible light odometry and infrared odometry.

[0003] Currently, visible light odometry is widely used and performs well in general environments. However, its accuracy drops significantly in scenarios with strong light variations, smoke, or haze. In low-light or no-light conditions, or even when environmental image features are completely undetectable, it cannot complete the positioning task. In recent years, with the improvement in performance and reduction in cost of vehicle-mounted infrared thermal imaging products, research on infrared odometry has gradually unfolded. It utilizes infrared thermal imaging sensors to acquire environmental infrared thermal radiation image information, is insensitive to changes in light, and is unaffected by adverse environmental conditions such as rain, snow, haze, or smoke, enabling all-weather, all-time environmental perception and effectively solving the problem of poor robustness of visible light odometry under special conditions. At the same time, infrared odometry also suffers from problems such as limited infrared image texture, low signal-to-noise ratio, low contrast, and low resolution, resulting in fewer feature points available for inter-frame matching and poorer positioning accuracy.

[0004] In existing technologies, based on the assumption of scene staticity, feature points and edge points of environmental infrared images are fused, which can improve the positioning performance of infrared odometry to some extent. However, methods based on the assumption of scene staticity often cannot accurately distinguish between the movement of the vehicle and the movement of the target object when facing dynamic scenes of real unmanned vehicles. Although they can stably perform inter-frame feature matching, they introduce errors due to the movement of the target object, resulting in poor positioning accuracy in real-world scenarios. Summary of the Invention

[0005] To address the aforementioned deficiencies or improvement needs of existing technologies, this invention provides a method and system for implementing infrared semantic odometry for autonomous driving based on dynamic scenes. It utilizes a semantic segmentation network to segment and extract static targets such as lanes, signs, and buildings from infrared images, as well as dynamic targets such as vehicles and pedestrians. Dynamic targets are then removed using prior semantic information. After removing dynamic targets, an edge line feature and feature point fusion matching method is employed to ensure that the remaining infrared image features are sufficiently rich, enabling stable inter-frame matching and enhancing positioning accuracy and robustness.

[0006] To achieve the above objectives, according to one aspect of the present invention, a method for implementing infrared semantic odometry for autonomous driving based on dynamic scenes is provided, comprising:

[0007] Acquire environmental infrared images and perform image enhancement;

[0008] ORB feature points and edge line features of targets in infrared images are extracted, and feature fusion matching is performed to obtain the infrared image point-line fusion feature matching results.

[0009] Instance semantic segmentation is performed on dynamic targets in infrared images to obtain semantic segmentation results;

[0010] The first static feature is selected based on the infrared image point-line fusion feature matching result and semantic segmentation result. The first static feature includes absolute static features and potential static features.

[0011] Infrared semantic localization results are obtained by motion estimation based on the first static feature.

[0012] Further, the extraction of ORB feature points and edge line features of the target in the infrared image, and the performance of feature fusion matching to obtain the infrared image point-line fusion feature matching result, includes:

[0013] ORB feature points and edge line features of targets in infrared images are extracted and fused to obtain infrared image point-line fused features.

[0014] Based on the point-line fusion features of infrared images, feature matching is performed to obtain the infrared image point-line fusion feature matching results.

[0015] Further, the step of extracting ORB feature points and edge line features of the target in the infrared image and performing feature fusion to obtain infrared image point-line fusion features includes:

[0016] Construct a scale pyramid based on infrared images;

[0017] Extract multi-scale Oriented Fast keypoints from each layer of the scale pyramid and determine the BRIEF descriptors corresponding to the multi-scale Oriented Fast keypoints to obtain the first ORB feature points.

[0018] Based on the scale pyramid, the ED Lines algorithm is used to extract edge line features, and ORB feature transformation is performed on the edge line features to obtain the second ORB feature points.

[0019] The first ORB feature point is fused with the second ORB feature point to obtain the infrared image point-line fusion feature.

[0020] Furthermore, the edge line feature extraction based on the scale pyramid using the ED Lines algorithm includes:

[0021] Using images from each layer of the scale pyramid as input, Gaussian filtering is used for image smoothing and noise reduction. The Sobel gradient operator is used to calculate pixel gradients. The peak pixels of the gradient operator are selected as anchor points, and the gradient vectors are used to connect adjacent peak points to draw edges.

[0022] The least squares method is used to traverse the edge pixels to fit straight lines, segment and extract straight line segments in the edge, and generate edge straight line features.

[0023] Further, the ORB feature transformation of the edge line features includes:

[0024] The characteristics of a straight line are described by using its starting and ending points as key points.

[0025] Using the key points of the straight line as the center, 16 pixels on a circle with a radius of 3 are selected to generate feature corner points. The gray-scale centroid method is used to determine the direction information of the feature corner points and their corresponding BRIEF descriptors to complete the ORB feature transformation.

[0026] Based on the above method, the starting point and ending point are traversed to obtain the edge line ORB feature points, which are the second ORB feature points.

[0027] Further, the step of obtaining the infrared image point-line fusion feature matching result by performing feature matching based on the infrared image point-line fusion features includes:

[0028] For the point-line fusion features of the infrared image in frame t and frame t+1, the Hamming distance of the inter-frame feature BRIEF descriptor is used to measure feature similarity, and fast approximate nearest neighbor is used for feature matching to obtain the point-line fusion feature matching results of the infrared images in frames t and t+1.

[0029] Furthermore, the step of performing instance semantic segmentation on dynamic targets in the infrared image to obtain semantic segmentation results includes:

[0030] The Mask R-CNN network is used to perform instance semantic segmentation of dynamic targets in infrared images. For n dynamic targets in frame t, the set of mask results is as follows:

[0031] Furthermore, the first static feature is selected based on the infrared image point-line fusion feature matching result and semantic segmentation result. The first static feature includes absolute static features and potential static features, including:

[0032] Screening of absolutely static features based on infrared image point-line fusion feature matching results and semantic segmentation results

[0033]

[0034] Among them, M t For the dynamic target mask in frame t; The result of point-line fusion feature matching for the t-th frame of the infrared image; For the filtered features, These are the features to be selected.

[0035] Furthermore, the first static feature is selected based on the infrared image point-line fusion feature matching result and semantic segmentation result. The first static feature includes absolute static features and potential static features, and further includes:

[0036] Based on absolute static features, infrared image point-line fusion feature matching results, and semantic segmentation results, epipolar constraints are used to screen potential static features.

[0037] Furthermore, the step of using epipolar constraints to screen potential static features based on absolute static features, infrared image point-line fusion feature matching results, and semantic segmentation results includes:

[0038] Using the eight-point method, determine the absolute static features of frame t. and the absolute static features of frame t+1 The corresponding essential matrix E;

[0039] Based on the essential matrix E, latent static features are selected, and the latent static features satisfy:

[0040]

[0041] Where K is the intrinsic parameter matrix of the infrared thermal imager, and T is the static feature threshold. For the latent static features of frame t, This represents the potential static features of the (t+1)th frame.

[0042] Further, the step of obtaining the infrared semantic localization result by motion estimation based on the first static feature includes:

[0043] Infrared semantic localization results are obtained by using the epipolar geometry method to perform motion estimation based on the first static feature.

[0044] According to another aspect of the present invention, an infrared semantic odometry for autonomous driving based on dynamic scenes is provided, comprising:

[0045] The first main module is used to acquire environmental infrared images and perform image enhancement.

[0046] The second main module is used to extract ORB feature points and edge line features of targets in infrared images, and perform feature fusion matching to obtain infrared image point-line fusion feature matching results.

[0047] The third main module is used to perform instance semantic segmentation on dynamic targets in infrared images to obtain semantic segmentation results.

[0048] The fourth main module is used to filter the first static features based on the infrared image point-line fusion feature matching results and semantic segmentation results. The first static features include absolute static features and potential static features.

[0049] The fifth main module is used to obtain infrared semantic localization results by performing motion estimation using epipolar geometry based on the first static features.

[0050] According to another aspect of the invention, an electronic device is provided, including a processor and a memory, the processor and the memory being interconnected.

[0051] The memory is used to store computer programs;

[0052] The processor is configured to execute the dynamic scene-based autonomous driving infrared semantic odometry implementation method when the computer program is invoked.

[0053] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing a computer program, the computer program being executed by a processor to implement the aforementioned method for implementing infrared semantic odometry for autonomous driving based on dynamic scenes.

[0054] In summary, compared with the prior art, the above-described technical solutions conceived by this invention can achieve the following beneficial effects:

[0055] 1. The unmanned vehicle infrared semantic odometry implementation method of the present invention utilizes a semantic segmentation network to segment and extract static targets such as lanes, signs, and buildings from infrared images, and dynamic targets such as vehicles and pedestrians. Dynamic targets are then removed using prior semantic information. After removing dynamic targets, edge line features and feature point fusion matching are used to ensure that the remaining infrared image features are sufficiently rich, enabling stable inter-frame matching and enhancing positioning accuracy and robustness.

[0056] 2. The unmanned vehicle infrared semantic odometry implementation method of the present invention eliminates the positioning error caused by the movement of the target object in high dynamic scenes through semantic segmentation network, thereby improving the positioning accuracy of infrared semantic odometry;

[0057] 3. The unmanned vehicle infrared semantic odometry implementation method of the present invention proposes a point and line feature fusion method of infrared image to compensate for the reduction of features for inter-frame matching caused by the removal of dynamic targets, thereby enhancing the robustness of infrared semantic odometry. Attached Figure Description

[0058] Figure 1This is a flowchart of an infrared semantic odometry implementation method according to an embodiment of the present invention;

[0059] Figure 2 This is a flowchart illustrating the motion estimation process for an unmanned vehicle according to an embodiment of the present invention. Detailed Implementation

[0060] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0061] In the description of this invention, unless otherwise explicitly specified and limited, the terms "connected," "linked," and "fixed" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0062] Those skilled in the art will understand that, unless otherwise stated, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the word “comprising” as used in the specification of this application means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or combinations thereof.

[0063] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined as in the embodiments of this application.

[0064] In an embodiment of the present invention, an odometer is a method for estimating the change of an object's position over time using data obtained from a motion sensor.

[0065] An infrared thermal imager is a detection device that detects infrared heat non-contactly, converts it into a thermal image and temperature value, displays it on a monitor, and performs calculations on the temperature values. Infrared thermal imagers can accurately quantify the detected heat, enabling precise identification and rigorous analysis of overheated fault areas. They are very easy to use, with readily available thermal imaging. The operation and intuitive on-screen display guides allow for accurate measurements without professional training. Simply point at the target, focus the instrument, and it will automatically adjust the temperature range to display a clear and sharp image. Once the user presses the save button, the image and related measurement data are stored.

[0066] The present invention provides an infrared semantic odometry implementation method for autonomous driving based on dynamic scenes, which can be applied to fields such as autonomous driving, image recognition, positioning and motion tracking, and is particularly suitable for vehicle positioning and location tracking during autonomous driving.

[0067] like Figure 1 The diagram shows a flowchart of an infrared semantic odometry implementation method for autonomous driving based on dynamic scenes. Embodiments of this invention provide an infrared semantic odometry implementation method for autonomous driving based on dynamic scenes, specifically including steps S100 to S500. (See attached diagram). Figure 1 and Figure 2 .

[0068] Step S100: Acquire an ambient infrared image and perform image enhancement;

[0069] Specifically, in embodiments of the present invention, an infrared thermal imager is used to acquire environmental infrared images. In one embodiment of the present invention, at least one infrared thermal imager is installed on the vehicle to acquire environmental infrared images around the vehicle.

[0070] Infrared thermal imagers capture images by receiving ambient thermal radiation information. Their image quality depends on the thermal emissivity and temperature distribution of the target object, resulting in drawbacks such as high background, low contrast, and low signal-to-noise ratio in infrared images. This makes it difficult to directly extract edge features and perform semantic recognition on raw infrared images. Therefore, preprocessing such as contrast enhancement should be performed before feature extraction.

[0071] In one embodiment of the present invention, the most common histogram equalization algorithm in grayscale image processing is used for contrast enhancement, which takes into account both computational complexity and enhancement effect. If computational complexity is not considered, other grayscale image enhancement processing methods or image enhancement methods such as infrared image contrast enhancement can also be used.

[0072] In subsequent embodiments of the present invention, unless otherwise specified, the infrared images are all enhanced infrared images.

[0073] Step S200: Extract ORB feature points and edge line features of the target in the infrared image, and perform feature fusion matching to obtain the infrared image point-line fusion feature matching result;

[0074] The extraction of ORB feature points and edge line features of targets in infrared images, followed by feature fusion matching to obtain infrared image point-line fusion feature matching results, specifically includes:

[0075] ORB feature points and edge line features of targets in infrared images are extracted, and feature fusion is performed to obtain infrared image point-line fusion features; based on the infrared image point-line fusion features, feature matching is performed to obtain infrared image point-line fusion feature matching results.

[0076] Specifically, step S200 includes steps S201 to S205;

[0077] The process of extracting ORB feature points and edge line features of the target in the infrared image and performing feature fusion to obtain infrared image point-line fused features includes:

[0078] Step S201: Construct a scale pyramid based on the infrared image;

[0079] Gaussian low-pass smoothing filtering is applied to the enhanced infrared image, followed by alternating row and column downsampling. This process is repeated layer-by-layer to generate a high-scale pyramid image sequence (L0, L1, ..., L...). i In this context, L0 represents the original infrared image, and each subsequent upper layer is generated by downsampling the lower layer by a factor of 4.

[0080] Step S202: Extract multi-scale Oriented Fast keypoints from each layer of the scale pyramid and determine the BRIEF descriptors corresponding to the multi-scale Oriented Fast keypoints to obtain the first ORB feature points.

[0081] The method of this invention performs multi-scale Oriented Fast keypoint extraction on each layer of the image pyramid and calculates its corresponding BRIEF descriptor to obtain infrared image ORB features with good scale and rotation invariance.

[0082] Step S203: Based on the scale pyramid, the ED Lines algorithm is used to extract edge line features, and ORB feature transformation is performed on the edge line features to obtain the second ORB feature points;

[0083] In the embodiments of the present invention, the ED Lines algorithm is used, taking the images of each layer of the scale pyramid as input, to achieve high-precision real-time infrared image edge line feature extraction with relatively small computational resource consumption.

[0084] Step S203 specifically includes steps S2031 to S2035:

[0085] Step S2031: Using the images of each layer of the scale pyramid as input, Gaussian filtering is used to smooth and reduce noise in the image. The Sobel gradient operator is used to calculate the pixel gradient. The peak pixel of the gradient operator is selected as the anchor point. The gradient vector is used to connect adjacent peak points to draw the edge.

[0086] Step S2032: Use the least squares method to traverse the edge pixels to fit straight lines, segment and extract the straight line segments in the edge, and generate edge straight line features.

[0087] The edge straight line feature can be represented as (e1, e2, ..., e n );

[0088] The ORB feature transformation of the edge line features includes:

[0089] Step S2033: Describe the characteristics of the line using the starting point and ending point as key points;

[0090] Step S2034: Using the key points of the straight line as the center, select 16 pixels on a circle with a radius of 3 to generate feature corner points. Use the gray centroid method to determine the direction information of the feature corner points and their corresponding BRIEF descriptors to complete the ORB feature transformation.

[0091] It should be noted that when generating feature corner points, the selection of radius and number of pixels should be balanced according to the needs to balance the amount of computation. If the radius is too large, it will increase the amount of computation, and if the radius is too small, it will affect the feature matching effect. Those skilled in the art can adjust the selection of radius and number of pixels according to their needs. In a preferred embodiment of the present invention, 16 pixels on a circle with a radius of 3 are selected to generate feature corner points.

[0092] Step S2035: Based on the above method, traverse the starting point and ending point to obtain the edge line ORB feature points, which are the second ORB feature points.

[0093] Considering the limited texture in infrared images, which leads to poor stability in line feature extraction and difficulty in inter-frame line feature matching, embodiments of the present invention use a straight line starting point. end The key points are used to describe the features of the line. To further integrate the point feature extraction results, rotation invariance is introduced for the endpoints of the line. Embodiments of this invention use key points of the line... Using a circle as the center, select 16 pixels on a circle with a radius of 3 to generate feature corner points. Corner orientation information is calculated using the gray-scale centroid method, and its corresponding BRIEF descriptor is further calculated to complete ORB feature transformation. The starting and ending points are traversed to generate edge line ORB feature points.

[0094] Step S204: Fuse the first ORB feature point with the second ORB feature point to obtain the infrared image point-line fusion feature.

[0095] In an optional embodiment of the present invention, edge line features and feature point extraction tasks can be computed in parallel to ensure real-time feature extraction.

[0096] In an optional embodiment of the present invention, the step of obtaining the infrared image point-line fusion feature matching result by performing feature matching based on the infrared image point-line fusion features specifically includes:

[0097] For the dot-line fusion feature O of the t-th frame infrared image t and the dot-line fusion feature O of the infrared image in frame t+1. t+1 Hamming distance d from the BRIEF descriptor for inter-frame features is used. hm Feature similarity is measured, and fast approximate nearest neighbor is used for feature matching. Furthermore, to reduce the probability of false matches, matches with a Hamming distance less than twice the minimum distance (the minimum distance is the minimum Hamming distance between descriptors in the matching between two frames) are eliminated, resulting in coarse feature matching results for frame t and frame t+1. and This refers to the point-line fusion feature matching result of the infrared images in frame t and frame t+1.

[0098] The method of this invention addresses the limited texture information in infrared images, resulting in a small number of original feature points that can be extracted. Furthermore, removing feature points related to dynamic targets further reduces the number of feature points. To ensure stable and reliable operation of the infrared odometry within the required accuracy range, this invention directly extracts feature points and further integrates key points such as the start and end points of edge line features to enrich the feature point information required for inter-frame matching.

[0099] Step S300: Perform instance semantic segmentation on the dynamic target in the infrared image to obtain the semantic segmentation result;

[0100] The step of performing instance semantic segmentation on dynamic targets in infrared images to obtain semantic segmentation results includes:

[0101] The Mask R-CNN network is used to perform instance semantic segmentation of dynamic targets in infrared images. For n dynamic targets in frame t, the set of mask results is as follows:

[0102] The dynamic targets mentioned are common dynamic targets in actual road scenarios, such as pedestrians, cyclists, electric vehicle or motorcycle riders, cars, buses, trucks, trams, high-speed trains, and other vehicles.

[0103] The Mask R-CNN network is a mature network model that can be directly obtained from existing technologies. The specific process of instance semantic segmentation based on this network model will not be elaborated further. In another embodiment of this invention, other classification and recognition algorithms can also be used to implement the instance semantic segmentation process, as long as the algorithm can achieve the dynamic target.

[0104] Instance semantic segmentation: A concrete object of a class is called an instance. Instance segmentation not only requires pixel-level classification but also distinguishes different instances based on their specific categories. For example, if an image contains multiple people A, B, and C, their semantic segmentation result is all "people," but the instance segmentation result is different objects. Similarly, if an image contains multiple dynamic targets A, B, and C, the semantic segmentation result is all dynamic targets, but A, B, and C are different objects, such as people and cars.

[0105] In one embodiment of the present invention, to ensure the real-time performance of the algorithm, the instance semantic segmentation network is accelerated by GPU / NPU for inference and computed in parallel with step S200.

[0106] Step S400: Filter the first static features based on the infrared image point-line fusion feature matching results and semantic segmentation results, wherein the first static features include absolute static features and potential static features;

[0107] Specifically, step S400 includes steps S401 to S402.

[0108] Step S401: Infrared image point-line fusion feature matching result Dynamic target mask in frame t Filtering absolute static features on non-dynamic targets

[0109]

[0110] Among them, M t For the dynamic target mask in frame t; The result of point-line fusion feature matching in infrared images; For the filtered features, These are the features to be selected.

[0111] Step S402: Utilize epipolar constraints to screen potential static features:

[0112] Calculate the absolute static features of frame t using the eight-point method. and the absolute static features of frame t+1 The corresponding essential matrix E;

[0113] Based on the essential matrix E, potential static features that satisfy the following two conditions are selected.

[0114]

[0115] Where K is the intrinsic parameter matrix of the infrared thermal imager, and T is the static feature threshold. For the latent static features of frame t, This represents the potential static features of the (t+1)th frame.

[0116] Based on semantic information and after eliminating dynamic features using epipolar constraints, the first static feature of the t-th frame infrared image can be obtained.

[0117]

[0118] in, As a potential static feature, It is an absolutely static feature.

[0119] Step S500: Based on the first static feature, perform motion estimation using epipolar geometry to obtain the infrared semantic localization result.

[0120] Using the obtained static features Motion estimation is performed using the epipolar geometry method, enabling localization and motion tracking of unmanned vehicles without motion feature errors.

[0121] It should be noted that the use of epipolar geometry for motion estimation based on extracted features is an existing technique and will not be elaborated upon here.

[0122] The method of this invention utilizes a semantic segmentation network to segment and extract static targets such as lanes, signs, and buildings from infrared images, as well as dynamic targets such as vehicles and pedestrians. Dynamic targets are then removed using prior semantic information. After removing dynamic targets, an edge line feature and feature point fusion matching method is employed to ensure that the remaining infrared image features are sufficiently rich, enabling stable inter-frame matching and enhancing positioning accuracy and robustness.

[0123] The implementation of the various embodiments of this invention is based on programmed processing by a device with a central processing unit (CPU). Therefore, in practical engineering, the technical solutions and functions of the various embodiments of this invention can be encapsulated into various modules. Based on this reality, and building upon the above embodiments, the embodiments of this invention provide an autonomous driving infrared semantic odometry system based on dynamic scenes, used to execute the autonomous driving infrared semantic odometry implementation method based on dynamic scenes in the above method embodiments. This includes:

[0124] The first main module is used to acquire environmental infrared images and perform image enhancement.

[0125] The second main module is used to extract ORB feature points and edge line features of targets in infrared images, and perform feature fusion matching to obtain infrared image point-line fusion feature matching results.

[0126] The third main module is used to perform instance semantic segmentation on dynamic targets in infrared images to obtain semantic segmentation results.

[0127] The fourth main module is used to filter the first static features based on the infrared image point-line fusion feature matching results and semantic segmentation results. The first static features include absolute static features and potential static features.

[0128] The fifth main module is used to obtain infrared semantic localization results by performing motion estimation using epipolar geometry based on the first static features.

[0129] It should be noted that the system in the system embodiments provided by the present invention can be used not only to implement the methods in the above method embodiments, but also to implement the methods in other method embodiments provided by the present invention. The only difference is the setting of corresponding functional modules. Its principle is basically the same as that of the above system embodiments provided by the present invention. As long as those skilled in the art can improve the system in the above system embodiments by referring to the specific technical solutions in other method embodiments and combining technical features to obtain corresponding technical means and technical solutions composed of these technical means, on the basis of the above system embodiments, under the premise of ensuring the practicality of the technical solutions, so as to obtain corresponding system class embodiments for implementing the methods in other method class embodiments.

[0130] The methods in the embodiments of the present invention are implemented using electronic devices; therefore, it is necessary to describe the relevant electronic devices. For this purpose, embodiments of the present invention provide an electronic device comprising: at least one central processor, a communications interface, at least one memory, and a communication bus, wherein the at least one central processor, the communications interface, and the at least one memory communicate with each other via the communication bus. The at least one central processor can invoke logical instructions stored in the at least one memory to execute all or part of the steps of the methods provided in the foregoing method embodiments.

[0131] Furthermore, when the logical instructions in at least one of the aforementioned memories can be implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various method embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0132] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0133] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0134] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. Based on this understanding, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, or sometimes in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0135] In this application, the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes said element.

[0136] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for implementing infrared semantic odometry for autonomous driving based on dynamic scenes, characterized in that, include: Acquire environmental infrared images and perform image enhancement; ORB feature points and edge line features of targets in infrared images are extracted, and feature fusion matching is performed to obtain the infrared image point-line fusion feature matching results. Instance semantic segmentation is performed on dynamic targets in infrared images to obtain semantic segmentation results; The first static feature is selected based on the infrared image point-line fusion feature matching result and semantic segmentation result. The first static feature includes absolute static features and potential static features. The first static feature is selected based on the infrared image point-line fusion feature matching result and semantic segmentation result. The first static feature includes absolute static features and potential static features, including: Screening of absolutely static features based on infrared image point-line fusion feature matching results and semantic segmentation results : in, For the dynamic target mask in frame t; The result of point-line fusion feature matching for the t-th frame of the infrared image; For the filtered features, Features to be selected; The first static feature is selected based on the infrared image point-line fusion feature matching result and semantic segmentation result. The first static feature includes absolute static features and latent static features, and further includes: Based on absolute static features, infrared image point-line fusion feature matching results, and semantic segmentation results, antipolar constraints are used to screen potential static features. The step of selecting potential static features based on absolute static features, infrared image point-line fusion feature matching results, and semantic segmentation results, using epipolar constraints, includes: Using the eight-point method, determine the absolute static features of frame t. and the absolute static features of frame t+1 Corresponding essential matrix ; Based on the essential matrix Filter for latent static features that satisfy: Where K is the intrinsic parameter matrix of the infrared thermal imager, and T is the static feature threshold. For the latent static features of frame t, This represents the potential static features of the (t+1)th frame; Infrared semantic localization results are obtained by motion estimation based on the first static feature; The specific process of acquiring environmental infrared images and performing image enhancement is as follows: First, infrared images of the environment around the vehicle are acquired by setting up an infrared thermal imager; Secondly, infrared thermal imagers form images by receiving environmental thermal radiation information; Then, the most common histogram equalization algorithm in grayscale image processing is used for preprocessing, including infrared image contrast enhancement.

2. The method for implementing infrared semantic odometry for autonomous driving based on dynamic scenes according to claim 1, characterized in that, The step of extracting ORB feature points and edge line features of the target in the infrared image and performing feature fusion matching to obtain the infrared image point-line fusion feature matching result includes: ORB feature points and edge line features of targets in infrared images are extracted and fused to obtain infrared image point-line fused features. Based on the point-line fusion features of infrared images, feature matching is performed to obtain the infrared image point-line fusion feature matching results.

3. The method for implementing infrared semantic odometry for autonomous driving based on dynamic scenes according to claim 2, characterized in that, The process of extracting ORB feature points and edge line features of the target in the infrared image and performing feature fusion to obtain infrared image point-line fused features includes: Construct a scale pyramid based on infrared images; Extract multi-scale Oriented Fast keypoints from each layer of the scale pyramid and determine the BRIEF descriptors corresponding to the multi-scale Oriented Fast keypoints to obtain the first ORB feature points. Based on the scale pyramid, the ED Lines algorithm is used to extract edge line features, and ORB feature transformation is performed on the edge line features to obtain the second ORB feature points. The first ORB feature point is fused with the second ORB feature point to obtain the infrared image point-line fusion feature.

4. The method for implementing infrared semantic odometry for autonomous driving based on dynamic scenes according to claim 3, characterized in that, The edge line feature extraction based on the scale pyramid using the ED Lines algorithm includes: Using images from each layer of the scale pyramid as input, Gaussian filtering is used for image smoothing and noise reduction. The Sobel gradient operator is used to calculate pixel gradients. The peak pixels of the gradient operator are selected as anchor points, and the gradient vectors are used to connect adjacent peak points to draw edges. The least squares method is used to traverse the edge pixels to fit straight lines, segment and extract straight line segments in the edge, and generate edge straight line features.

5. The method for implementing infrared semantic odometry for autonomous driving based on dynamic scenes according to claim 3, characterized in that, The ORB feature transformation of the edge line features includes: The characteristics of a straight line are described by using its starting and ending points as key points. Using the key points of the straight line as the center, 16 pixels on a circle with a radius of 3 are selected to generate feature corner points. The gray-scale centroid method is used to determine the direction information of the feature corner points and their corresponding BRIEF descriptors to complete the ORB feature transformation. Based on the above method, the starting point and ending point are traversed to obtain the edge line ORB feature points, which are the second ORB feature points.

6. The method for implementing infrared semantic odometry for autonomous driving based on dynamic scenes according to claim 2, characterized in that, The step of obtaining the infrared image point-line fusion feature matching result by performing feature matching based on the point-line fusion features of the infrared image includes: For the point-line fusion features of the infrared image in frame t and frame t+1, the Hamming distance of the inter-frame feature BRIEF descriptor is used to measure feature similarity, and fast approximate nearest neighbor is used for feature matching to obtain the point-line fusion feature matching results of the infrared images in frames t and t+1.

7. The method for implementing infrared semantic odometry for autonomous driving based on dynamic scenes according to claim 1, characterized in that, The step of performing instance semantic segmentation on dynamic targets in infrared images to obtain semantic segmentation results includes: The Mask R-CNN network is used to perform instance semantic segmentation of dynamic targets in infrared images. For n dynamic targets in frame t, the set of mask results is as follows: .

8. The method for implementing infrared semantic odometry for autonomous driving based on dynamic scenes according to claim 1, characterized in that, The step of obtaining the infrared semantic localization result by motion estimation based on the first static feature includes: Infrared semantic localization results are obtained by using the epipolar geometry method to perform motion estimation based on the first static feature.

9. An autonomous driving infrared semantic odometry system based on dynamic scenes, used to implement the autonomous driving infrared semantic odometry implementation method based on dynamic scenes as described in claim 1, characterized in that, include: The first main module is used to acquire environmental infrared images and perform image enhancement. The second main module is used to extract ORB feature points and edge line features of targets in infrared images, and perform feature fusion matching to obtain infrared image point-line fusion feature matching results. The third main module is used to perform instance semantic segmentation on dynamic targets in infrared images to obtain semantic segmentation results. The fourth main module is used to filter the first static features based on the infrared image point-line fusion feature matching results and semantic segmentation results. The first static features include absolute static features and potential static features. The fifth main module is used to obtain infrared semantic localization results by performing motion estimation using epipolar geometry based on the first static features.

10. An electronic device, characterized in that, It includes a processor and a memory, which are interconnected; The memory is used to store computer programs; The processor is configured to execute, when the computer program is invoked, the method for implementing infrared semantic odometry for autonomous driving based on dynamic scenes as described in any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that is executed by a processor to implement the method for implementing infrared semantic odometry for autonomous driving based on dynamic scenes as described in any one of claims 1 to 8.

Images