Obstacle sensing method and device, vehicle, medium and program product
By using infrared image perception methods and obstacle recognition models, the problem of obstacle recognition accuracy under adverse weather conditions has been solved, enabling all-weather obstacle perception and classification, and meeting the real-time requirements of autonomous driving.
Patent Information
- Application Number
- CN202511578085.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-10-31
- Publication Date
- 2026-02-10
AI Technical Summary
In adverse weather conditions, visible light cameras are susceptible to light scattering and reflection, resulting in low image contrast and blurred target outlines; the effects of raindrops and fog particles on lidar increase the false detection rate of obstacles; millimeter-wave radar has insufficient resolution for non-metallic obstacles, making it difficult to achieve fine classification and accurate ranging.
An infrared image perception method is adopted to identify obstacle features through an obstacle recognition model and generate perception results based on motion features. An obstacle recognition model is constructed using the first and second recognition models to improve anti-interference ability and all-weather perception capability, and reduce false detection rate.
It achieves accurate identification and classification of obstacles under adverse weather conditions, improves the ability to resist environmental interference, meets the real-time requirements of autonomous driving, and achieves a balance between accuracy and efficiency.
Smart Images
Figure CN121505570A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of intelligent connected vehicle technology, and in particular to an obstacle sensing method, device, vehicle, medium, and program product. Background Technology
[0002] Among related technologies, visible light cameras can be used to capture environmental images, and convolutional neural networks can be used to extract features such as the contours and textures of obstacles. Then, deep learning algorithms can be combined to achieve target classification and localization. Alternatively, lidar can be used to emit laser pulses and measure the reflection time to generate high-precision three-dimensional point cloud data. Finally, cluster analysis can be used to identify the spatial position and shape of obstacles. Millimeter-wave radar can also be used to emit high-frequency electromagnetic waves, receive the echo signals reflected by obstacles, calculate the relative velocity of obstacles, and combine a constant false alarm rate algorithm to output the distance, velocity, and azimuth information of obstacles.
[0003] However, in related technologies, under adverse weather conditions such as rain, fog, and sandstorms, visible light cameras experience significantly enhanced light scattering and reflection effects, leading to compressed image dynamic range and blurred outlines of small targets (such as pedestrians and traffic cones). LiDAR is affected by raindrops / fog particles, and point cloud data is prone to dense outlier noise, resulting in an increased false detection rate. Millimeter-wave radar, due to its longer wavelength, has low sensitivity to the radar cross-section of non-metallic obstacles (such as pedestrians and animals), resulting in insufficient angular resolution and difficulty in achieving fine classification and accurate ranging, which urgently needs improvement. Summary of the Invention
[0004] This application provides an obstacle perception method, device, vehicle, medium, and program product to solve the technical problems in related technologies, such as: visible light cameras are easily affected by light scattering and reflection under adverse weather conditions, resulting in low image contrast and blurred target outlines; the influence of raindrops and fog particles on lidar leads to an increase in the false detection rate of obstacles; and millimeter-wave radar has insufficient resolution for non-metallic obstacles (such as pedestrians and animals), making it difficult to achieve fine classification and accurate ranging.
[0005] The first aspect of this application provides an obstacle perception method, comprising the following steps: acquiring an infrared image of a vehicle; inputting the infrared image into a pre-constructed obstacle recognition model to identify at least one obstacle feature in the infrared image using a first recognition model in the obstacle recognition model, and determining a final obstacle in the infrared image based on the at least one obstacle feature and a second recognition model in the obstacle recognition model; and generating a perception result of the final obstacle based on at least one motion feature of the final obstacle.
[0006] Optionally, in one embodiment of this application, before inputting the infrared image into a pre-built obstacle recognition model, the method further includes: acquiring a target infrared image of the target vehicle; extracting at least one image feature from the target infrared image to obtain multi-scale features of the target infrared image; determining at least one target obstacle feature of the target infrared image based on the multi-scale features; and constructing the first recognition model based on the at least one target obstacle feature to construct the obstacle recognition model according to the first recognition model.
[0007] Optionally, in one embodiment of this application, before inputting the infrared image into a pre-constructed obstacle recognition model, the method further includes: identifying at least one initial target obstacle in the target infrared image based on the features of the at least one target obstacle; determining whether the at least one initial target obstacle meets a preset obstacle condition; if the at least one initial target obstacle meets the preset obstacle condition, retaining the corresponding initial target obstacle to determine the final target obstacle based on the retained initial target obstacle; if the at least one initial target obstacle does not meet the preset obstacle condition, removing the corresponding initial target obstacle to determine the final target obstacle based on the removed initial target obstacle; and constructing a second recognition model based on the final target obstacle to construct the obstacle recognition model based on the second recognition model.
[0008] Optionally, in one embodiment of this application, acquiring the infrared image of the vehicle includes: detecting whether the infrared image meets preset visual conditions; if the infrared image does not meet the preset visual conditions, identifying the visual features to be processed in the infrared image, determining the processing instructions for the infrared image based on the visual features to be processed, and processing the infrared image according to the processing instructions until a processed infrared image that meets the preset visual conditions is obtained.
[0009] Optionally, in one embodiment of this application, generating a perception result of the final obstacle based on at least one motion feature of the final obstacle includes: determining a first weight of the corresponding final obstacle and a second weight of the corresponding motion feature based on the at least one motion feature; obtaining a danger level of the final obstacle based on the first weight, the second weight, the corresponding motion feature, and the corresponding final obstacle; and generating the perception result based on the danger level.
[0010] A second aspect of this application provides an obstacle sensing device, comprising: a first acquisition module for acquiring an infrared image of a vehicle; a first determination module for inputting the infrared image into a pre-constructed obstacle recognition model to identify at least one obstacle feature in the infrared image using a first recognition model in the obstacle recognition model, and determining a final obstacle in the infrared image based on the at least one obstacle feature and a second recognition model in the obstacle recognition model; and a sensing module for generating a sensing result of the final obstacle based on at least one motion feature of the final obstacle.
[0011] Optionally, in one embodiment of this application, the method further includes: a second acquisition module, configured to acquire a target infrared image of a target vehicle before inputting the infrared image into a pre-built obstacle recognition model; an extraction module, configured to extract at least one image feature from the target infrared image to obtain multi-scale features of the target infrared image; a second determination module, configured to determine at least one target obstacle feature of the target infrared image based on the multi-scale features; and a first construction module, configured to construct the first recognition model based on the at least one target obstacle feature, so as to construct the obstacle recognition model according to the first recognition model.
[0012] Optionally, in one embodiment of this application, it further includes: an identification module, configured to identify at least one initial target obstacle in the target infrared image based on the features of the at least one target obstacle before inputting the infrared image into a pre-built obstacle recognition model; a judgment module, configured to determine whether the at least one initial target obstacle meets a preset obstacle condition; a third determination module, configured to retain the corresponding initial target obstacle when the at least one initial target obstacle meets the preset obstacle condition, so as to determine the final target obstacle based on the retained initial target obstacle; a fourth determination module, configured to remove the corresponding initial target obstacle when the at least one initial target obstacle does not meet the preset obstacle condition, so as to determine the final target obstacle based on the removed initial target obstacle; and a second construction module, configured to construct the second identification model based on the final target obstacle, so as to construct the obstacle recognition model based on the second identification model.
[0013] Optionally, in one embodiment of this application, the first acquisition module includes: a detection unit, configured to detect whether the infrared image meets preset visual conditions; and a first generation unit, configured to, when the infrared image does not meet the preset visual conditions, identify visual features to be processed in the infrared image, determine processing instructions for the infrared image based on the visual features to be processed, and process the infrared image according to the processing instructions until a processed infrared image that meets the preset visual conditions is obtained.
[0014] Optionally, in one embodiment of this application, the perception module includes: a determining unit, configured to determine a first weight corresponding to the final obstacle and a second weight corresponding to the motion feature based on the at least one motion feature; a second generating unit, configured to obtain the danger level of the final obstacle based on the first weight, the second weight, the corresponding motion feature and the corresponding final obstacle; and a third generating unit, configured to generate the perception result based on the danger level.
[0015] A third aspect of this application provides a vehicle, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the obstacle perception method as described in the above embodiments.
[0016] A fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the obstacle perception method described above.
[0017] A fifth aspect of this application provides a computer program product, including a computer program that, when executed, implements the obstacle perception method described above.
[0018] This application embodiment can input the acquired infrared image of the vehicle into a pre-constructed obstacle recognition model. The first recognition model identifies obstacle features, and the second recognition model determines the final obstacle in the infrared image. Based on the motion characteristics of the final obstacle, a corresponding perception result is generated. This is unaffected by insufficient visible light or strong light interference, and it has strong penetration through rain and fog, improving environmental anti-interference capabilities and achieving all-weather perception. Hierarchical recognition reduces the false detection rate, solves the problem of reliance on single features, and achieves a balance between accuracy and efficiency, meeting the real-time requirements of autonomous driving. Therefore, it solves the technical problems in related technologies, such as the susceptibility of visible light cameras to light scattering and reflection under adverse weather conditions, resulting in low image contrast and blurred target outlines; the influence of raindrops and fog particles on lidar, leading to an increased false detection rate; and the insufficient resolution of millimeter-wave radar for non-metallic obstacles (such as pedestrians and animals), making it difficult to achieve fine classification and accurate ranging.
[0019] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description
[0020] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein: Figure 1This is a flowchart of an obstacle sensing method provided according to an embodiment of this application; Figure 2 This is a block diagram of an obstacle sensing device provided according to an embodiment of this application; Figure 3 This is a structural schematic diagram of a vehicle provided according to an embodiment of this application.
[0021] Figure label: Among them, 10-obstacle sensing device; 100-first acquisition module, 200-first determination module, 300-sensing module 300; 301-memory, 302-processor, 303-communication interface. Detailed Implementation
[0022] The embodiments of this application are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.
[0023] Before introducing the obstacle perception method proposed in the embodiments of this application, let's first introduce a vehicle equipped with an infrared thermal imager involved in the embodiments of this application.
[0024] The hardware of the vehicle equipped with the infrared thermal imager may include, but is not limited to, an infrared thermal imaging camera, a visible light camera, a lidar, an auxiliary computing platform, a main control system, and a CAN (Controller Area Network) bus interface. The specific configuration can be made by those skilled in the art according to the actual situation, and this application does not impose any specific restrictions.
[0025] The infrared thermal imaging camera is installed at the front of the vehicle and has the ability to image in night vision and low visibility environments to collect infrared images of the vehicle; the visible light camera and lidar are used to fuse with the infrared images; the auxiliary computing platform undertakes tasks such as image preprocessing and neural network inference; the main control system receives the processing results and controls the vehicle to execute decision commands (such as braking, steering, etc., which are not specifically limited in this application); the CAN bus interface is used to communicate with the vehicle control system.
[0026] The obstacle perception method, apparatus, vehicle, medium, and program product of this application are described below with reference to the accompanying drawings. Addressing the issues mentioned in the background art, such as the susceptibility of visible light cameras to light scattering and reflection under adverse weather conditions, resulting in low image contrast and blurred target outlines; the influence of raindrops and fog particles on lidar leading to increased false detection rates; and the insufficient resolution of millimeter-wave radar for non-metallic obstacles (such as pedestrians and animals), making it difficult to achieve fine classification and accurate ranging, this application provides an obstacle perception method. In this method, the acquired infrared image of the vehicle is input into a pre-constructed obstacle recognition model. A first recognition model is used to identify obstacle features, and a second recognition model is used to determine the final obstacle in the infrared image. The method then generates a corresponding perception result based on the motion characteristics of the final obstacle. This method is unaffected by insufficient visible light or strong light interference, and has strong penetration through rain and fog, improving environmental anti-interference capabilities and achieving all-weather perception. Hierarchical recognition reduces the false detection rate, solves the problem of reliance on single features, achieves a balance between accuracy and efficiency, and meets the real-time requirements of autonomous driving. This solves several technical problems in related technologies, such as: visible light cameras being susceptible to light scattering and reflection under adverse weather conditions, resulting in low image contrast and blurred target outlines; the impact of raindrops and fog particles on lidar leading to increased false detection rates of obstacles; and insufficient resolution of millimeter-wave radar for non-metallic obstacles (such as pedestrians and animals), making it difficult to achieve fine classification and accurate ranging.
[0027] Specifically, Figure 1 This is a flowchart of an obstacle sensing method provided according to an embodiment of this application.
[0028] like Figure 1 As shown, the obstacle perception method includes the following steps: In step S101, an infrared image of the vehicle is acquired.
[0029] In some embodiments, the present application can directly use an infrared thermal imaging camera to acquire infrared images of the vehicle; alternatively, it can simultaneously use an infrared thermal imaging camera, a visible light camera, and a lidar to fuse images from different sources to determine the infrared image. The visible light camera is used for image fusion, and the lidar provides distance information to assist in spatial positioning of the infrared image.
[0030] This application embodiment improves accuracy in complex scenarios and enhances system applicability by integrating visible light cameras and LiDAR.
[0031] Optionally, in one embodiment of this application, acquiring an infrared image of a vehicle includes: detecting whether the infrared image meets preset visual conditions; if the infrared image does not meet the preset visual conditions, identifying visual features to be processed in the infrared image, determining processing instructions for the infrared image based on the visual features to be processed, and processing the infrared image according to the processing instructions until a processed infrared image that meets the preset visual conditions is obtained.
[0032] It is understood that the visual features to be processed in the embodiments of this application may include, but are not limited to, sharpness features, contrast features, color features, edge features, etc., and this application does not impose specific limitations.
[0033] In some embodiments, this application can detect whether an infrared image meets certain visual conditions. If not, it identifies the visual features of the infrared image to be processed, and then determines the processing instructions for the infrared image. The infrared image is then processed according to the processing instructions until a processed infrared image that meets the certain visual conditions is obtained. These certain visual conditions can be set by those skilled in the art according to actual conditions, and this application does not impose specific limitations.
[0034] For example, in some embodiments, when the infrared image is detected to not meet certain visual conditions, the present application identifies the visual feature to be processed as a sharpness feature. In this case, the present application can determine the processing instructions for the infrared image through an improved dark channel prior algorithm to perform image dehazing, improve the sharpness feature, and thus obtain a processed infrared image that meets certain visual conditions.
[0035] Among them, the improved dark channel prior algorithm can estimate the fog concentration in the image and remove it, which can effectively restore the details of objects obscured by fog, make the outlines of objects in the image clearer, and improve the overall clarity of the image.
[0036] In some embodiments of this application, when an infrared image is detected to not meet certain visual conditions, the visual feature to be processed is identified as a contrast feature. In this case, the embodiment of this application can determine the processing instructions for the infrared image through an improved dark channel prior algorithm to eliminate the influence of fog on light propagation, restore the original brightness difference of the image, and thus obtain a processed infrared image that meets certain visual conditions.
[0037] Among them, the improved dark channel prior algorithm can eliminate the influence of fog on light propagation, restore the original brightness difference of the image, thereby enhancing the contrast of the image and making the image look more vivid.
[0038] In some embodiments, when the infrared image does not meet certain visual conditions, the present application identifies the visual feature to be processed as a color feature. In this case, the present application can determine the processing instructions for the infrared image through an improved dark channel prior algorithm to remove the influence of fog on color, restore the original color of the object in the image, make the color more realistic and natural, and thus obtain a processed infrared image that meets certain visual conditions.
[0039] Among them, the improved dark channel prior algorithm can remove the influence of fog on color, restore the original color of objects in the image, and make the color more realistic and natural.
[0040] In some embodiments, when the infrared image is detected to not meet certain visual conditions, the present application embodiments identify the visual features to be processed as edge features. In this case, the present application embodiments can determine the processing instructions for the infrared image through the CLAHE algorithm (Contrast Limited Adaptive Histogram Equalization), Gaussian filtering algorithm, and Sobel Edge Detection, thereby obtaining a processed infrared image that meets certain visual conditions.
[0041] Among them, the CLAHE algorithm can adaptively adjust the contrast of local areas of the image to highlight the edge information of hot targets, making the difference in pixel values more obvious at the edges of hot targets, thus making the edges look clearer and sharper and improving edge clarity.
[0042] The combination of Gaussian filtering and Sobel edge detection can extract significant thermal boundaries. Gaussian filtering smooths the image, removing noise and high-frequency interference, making the image smoother. Sobel edge detection, on the other hand, calculates the gradient of pixels in the image to detect edge information, thus more accurately extracting significant thermal boundaries of hot targets and reducing noise interference in edge detection.
[0043] Optionally, in one embodiment of this application, before inputting the infrared image into the pre-built obstacle recognition model, the method further includes: acquiring a target infrared image of the target vehicle; extracting at least one image feature from the target infrared image to obtain multi-scale features of the target infrared image; determining at least one target obstacle feature of the target infrared image based on the multi-scale features; and constructing a first recognition model based on the at least one target obstacle feature to construct an obstacle recognition model according to the first recognition model.
[0044] In some embodiments, the present application can first acquire the target infrared image of the target vehicle, and then use a convolutional neural network to extract image features from the target infrared image to obtain corresponding multi-scale features, thereby determining the target obstacle features of the target obstacle vehicle or the target obstacle pedestrian in the target infrared image, such as the size of the heat source, temperature gradient and heat diffusion features, etc. The present application does not impose specific limitations, and then constructs a first recognition model based on different target obstacle features to obtain an obstacle recognition model.
[0045] Among them, as the core architecture of deep learning in the field of image processing, the convolutional neural network can extend the image feature extraction capability of the convolutional neural network through techniques such as multi-scale convolutional kernels, feature pyramid networks and dilated convolution, thereby constructing the corresponding first recognition model.
[0046] For example, embodiments of this application can use a combination of feature pyramid networks and dilated convolutions to capture vehicle contours at a shallow level and identify engine heat sources at a deep level, thereby determining the target obstacle features of the vehicle and constructing a first recognition model; alternatively, a combination of multi-scale convolution kernels and skip connections can be used to fuse the overall thermal radiation and local limb features of the perceived target obstacle pedestrian, thereby determining the target obstacle features of the pedestrian and constructing a first recognition model.
[0047] Optionally, in one embodiment of this application, before inputting the infrared image into the pre-built obstacle recognition model, the method further includes: identifying at least one initial target obstacle in the target infrared image based on at least one target obstacle feature; determining whether the at least one initial target obstacle meets a preset obstacle condition; if the at least one initial target obstacle meets the preset obstacle condition, retaining the corresponding initial target obstacle to determine the final target obstacle based on the retained initial target obstacle; if the at least one initial target obstacle does not meet the preset obstacle condition, removing the corresponding initial target obstacle to determine the final target obstacle based on the removed initial target obstacle; and constructing a second recognition model based on the final target obstacle to construct an obstacle recognition model based on the second recognition model.
[0048] As one possible implementation, embodiments of this application can identify initial obstacles in a target infrared image based on the target obstacle features extracted by a first recognition model, and determine whether the initial obstacles meet certain obstacle conditions. If they do, the corresponding initial obstacles are retained to determine the final obstacle; otherwise, the corresponding initial obstacles are discarded to determine the final obstacle, thereby constructing a second recognition model, and an obstacle recognition model is constructed based on the second recognition model. The certain obstacle conditions can be set by those skilled in the art according to actual conditions, and this application does not impose specific limitations.
[0049] It should be noted that, in determining whether the initial obstacle of the target meets certain obstacle conditions, the embodiments of this application can collect thermal image samples under different climatic conditions, construct a rain, fog and noise thermal image sample template library, compare the feature deviation between the current target infrared image and the rain, fog and noise thermal image sample template library in real time, and dynamically adjust the filtering parameters through dynamic adaptive filtering to suppress false heat sources caused by raindrops and fog, thereby determining whether the initial obstacle of the target meets certain obstacle conditions.
[0050] For example, embodiments of this application can identify initial obstacles such as pedestrians, animals, and vehicles in target infrared images based on target obstacle features using a Transformer structure. Confidence fusion is performed using an infrared feature probability map, and Kalman filtering is applied to track the initial obstacles to determine their corresponding movement direction and speed. If the movement direction and speed meet certain obstacle conditions, such as the movement direction being towards a vehicle and the speed being relatively high, the corresponding initial obstacle is retained. Otherwise, if the movement direction is away from a vehicle and the speed is relatively high, the corresponding initial obstacle is identified, thus determining the final obstacle and constructing a second recognition model to obtain the corresponding obstacle recognition model.
[0051] In step S102, the infrared image is input into a pre-built obstacle recognition model to identify at least one obstacle feature in the infrared image using a first recognition model in the obstacle recognition model, and the final obstacle in the infrared image is determined based on at least one obstacle feature and a second recognition model in the obstacle recognition model.
[0052] In actual implementation, embodiments of this application can input infrared images into a pre-built obstacle recognition model, use a first recognition model to identify obstacle features in the infrared images, and use the obstacle features as input data to determine the final obstacles in the infrared images using a second recognition model.
[0053] In step S103, a perception result of the final obstacle is generated based on at least one motion feature of the final obstacle.
[0054] It is understood that, in the embodiments of this application, motion characteristics may include, but are not limited to, the type, location, speed, motion trend, relative distance, etc. of obstacles. This application does not impose specific limitations. Furthermore, in the embodiments of this application, the perception results may include, but are not limited to, the final obstacle's danger level, danger probability, driving suggestions, etc., and this application does not impose specific limitations.
[0055] For example, embodiments of this application can classify the danger level into three levels (e.g., low, medium, and high; this application does not impose specific limitations). For instance, embodiments of this application can calculate the final danger probability of an obstacle based on its type, location, speed, movement trend, relative distance, etc. If the danger probability is less than a first threshold (e.g., 0.3), the danger level is determined to be low. In this case, embodiments of this application can use sound or vibration to alert the user to "Caution: Obstacle Ahead." If the danger probability is greater than or equal to the first threshold (e.g., 0.3) but less than a second threshold (e.g., 0.65), the danger level is determined to be medium. In this case, embodiments of this application can issue an emergency voice alert to "Immediately Decelerate" or "Steering to Avoid," or automatically intervene in the control system to decelerate or increase vehicle speed, enhancing corresponding vehicle intervention. If the danger probability is greater than or equal to the second threshold (e.g., 0.65), the danger level is determined to be high. In this case, embodiments of this application can use high-frequency alarm sounds, seatbelt pretensioning, seat vibration, etc., to alert the user and automatically perform emergency braking or sharp steering. The first and second thresholds can be set by those skilled in the art according to actual conditions; this application does not impose specific limitations.
[0056] In some embodiments, the present application can generate corresponding perception results based on the motion characteristics of the final obstacle.
[0057] Optionally, in one embodiment of this application, generating a perception result of the final obstacle based on at least one motion feature of the final obstacle includes: determining a first weight of the corresponding final obstacle and a second weight of the corresponding motion feature based on at least one motion feature; obtaining a danger level of the final obstacle based on the first weight, the second weight, the corresponding motion feature, and the corresponding final obstacle; and generating a perception result based on the danger level.
[0058] It is understood that the embodiments of this application do not impose specific limitations on different obstacles (such as pedestrians, vehicles, animals, static objects, etc.) and their corresponding motion characteristics (such as type, position, speed, motion trend, relative distance, etc., which are not specifically limited). Therefore, the embodiments of this application can quantify the contribution of each factor to the hazard level by designing a dynamic weighting formula to obtain a hazard score. The calculation formula for the hazard score may be, but is not limited to, the following: , in, Indicate the type of obstacle (pedestrians, vehicles, animals, static objects, etc.); Relative distance, unit: meters; Relative velocity, unit: m / s; Relative acceleration, unit: ; This indicates the angle between the direction of motion and the body, where 0° represents approaching directly and 180° represents moving away. , , This represents the weighting coefficient, which is adaptively adjusted based on the scenario. Functions that represent obstacle types; Represents the distance function; This represents a function that indicates the trend of motion.
[0059] Furthermore, in the embodiments of this application, the expression of the obstacle type function may be, but is not limited to, as: , The expression for the distance function can be, but is not limited to, as follows: , in, This is expressed as a safe distance threshold (e.g., 2m, but this application does not specify a particular limit). This indicates the warning distance threshold (e.g., 5m; this application does not specify a specific limit).
[0060] The expression for the motion trend function can be, but is not limited to, as follows: , in, This indicates a speed threshold (e.g., 1 m / s; this application does not impose a specific limit).
[0061] Additionally, it should be noted that the weighting coefficients... , , The dynamic adjustment strategy can be: in urban roads, improve ,reduce To cope with frequent start-stop scenarios; on highways, improve ,reduce Prioritize response to high-speed approaching targets; enhance performance in nighttime / low-light conditions. To compensate for the error in distance estimation in infrared images, the specific settings can be made by those skilled in the art according to the actual situation, and this application does not impose specific limitations.
[0062] Furthermore, embodiments of this application can calculate the hazard score of different obstacles, determine the hazard probability, and then determine the corresponding hazard level, thereby generating a perception result.
[0063] For example, embodiments of this application can detect a pedestrian approaching from directly in front at a speed of 1.5 m / s. ), distance 3m ( ), calculate hazard score After normalization, the probability of danger is 0.7, which determines the danger level as high. The system will then attract the user's attention through high-frequency alarm sounds, seat belt pretensioning, seat vibration, etc., and automatically perform emergency braking.
[0064] The obstacle perception method proposed in this application can input the acquired infrared image of a vehicle into a pre-built obstacle recognition model. The first recognition model identifies obstacle features, and the second recognition model determines the final obstacle in the infrared image. Based on the motion characteristics of the final obstacle, a corresponding perception result is generated. This method is unaffected by insufficient visible light or strong light interference, and has strong penetration through rain and fog, improving environmental anti-interference capabilities and achieving all-weather perception. Hierarchical recognition reduces the false detection rate, solves the problem of reliance on single features, and achieves a balance between accuracy and efficiency, meeting the real-time requirements of autonomous driving. Therefore, it solves the technical problems in related technologies, such as the susceptibility of visible light cameras to light scattering and reflection under adverse weather conditions, resulting in low image contrast and blurred target outlines; the influence of raindrops and fog particles on lidar, leading to an increased false detection rate; and the insufficient resolution of millimeter-wave radar for non-metallic obstacles (such as pedestrians and animals), making it difficult to achieve fine classification and accurate ranging.
[0065] Next, the obstacle sensing device according to an embodiment of this application is described with reference to the accompanying drawings.
[0066] Figure 2 This is a block diagram of an obstacle sensing device provided according to an embodiment of this application.
[0067] like Figure 2 As shown, the obstacle sensing device 10 includes: a first acquisition module 100, a first determination module 200, and a sensing module 300.
[0068] The first acquisition module 100 is used to acquire infrared images of the vehicle.
[0069] The first determining module 200 is used to input an infrared image into a pre-built obstacle recognition model to identify at least one obstacle feature in the infrared image using a first recognition model in the obstacle recognition model, and to determine the final obstacle in the infrared image based on at least one obstacle feature and a second recognition model in the obstacle recognition model.
[0070] The perception module 300 is used to generate a perception result of the final obstacle based on at least one motion feature of the final obstacle.
[0071] Optionally, in one embodiment of this application, it further includes: a second acquisition module, an extraction module, a second determination module, and a first construction module.
[0072] The second acquisition module is used to acquire the target infrared image of the target vehicle before inputting the infrared image into the pre-built obstacle recognition model.
[0073] An extraction module is used to extract at least one image feature from the target infrared image to obtain multi-scale features of the target infrared image.
[0074] The second determining module is used to determine at least one target obstacle feature in the target infrared image based on multi-scale features.
[0075] A first construction module is used to construct a first recognition model based on at least one target obstacle feature, so as to construct an obstacle recognition model based on the first recognition model.
[0076] Optionally, in one embodiment of this application, it further includes: an identification module, a judgment module, a third determination module, a fourth determination module, and a second construction module.
[0077] The identification module is used to identify at least one initial target obstacle in the target infrared image based on at least one target obstacle feature before inputting the infrared image into the pre-built obstacle identification model.
[0078] The judgment module is used to determine whether at least one target initial obstacle meets the preset obstacle conditions.
[0079] The third determining module is used to retain the corresponding initial obstacle when at least one initial obstacle meets the preset obstacle conditions, so as to determine the final obstacle based on the retained initial obstacle.
[0080] The fourth determination module is used to remove the corresponding target initial obstacle when at least one target initial obstacle does not meet the preset obstacle conditions, so as to determine the target final obstacle based on the removed target initial obstacles.
[0081] The second building module is used to build a second recognition model based on the target final obstacle, so as to build an obstacle recognition model based on the second recognition model.
[0082] Optionally, in one embodiment of this application, the first acquisition module 100 includes a detection unit and a first generation unit.
[0083] The detection unit is used to detect whether the infrared image meets the preset visual conditions.
[0084] The first generation unit is used to identify the visual features to be processed in the infrared image when the infrared image does not meet the preset visual conditions, so as to determine the processing instructions of the infrared image according to the visual features to be processed, and to process the infrared image according to the processing instructions until a processed infrared image that meets the preset visual conditions is obtained.
[0085] Optionally, in one embodiment of this application, the sensing module 300 includes: a determining unit, a second generating unit, and a third generating unit.
[0086] The determining unit is used to determine a first weight corresponding to the final obstacle and a second weight corresponding to the motion feature based on at least one motion feature.
[0087] The second generation unit is used to obtain the danger level of the final obstacle based on the first weight, the second weight, the corresponding motion characteristics, and the corresponding final obstacle.
[0088] The third generation unit is used to generate perception results based on the hazard level.
[0089] It should be noted that the foregoing explanation of the obstacle sensing method embodiment also applies to the obstacle sensing device of this embodiment, and will not be repeated here.
[0090] The obstacle perception device proposed in this application can input the acquired infrared image of a vehicle into a pre-built obstacle recognition model. The first recognition model identifies obstacle features, and the second recognition model determines the final obstacle in the infrared image. Based on the motion characteristics of the final obstacle, a corresponding perception result is generated. This device is unaffected by insufficient visible light or strong light interference, and has strong penetration through rain and fog, improving environmental anti-interference capabilities and achieving all-weather perception. Hierarchical recognition reduces the false detection rate, solves the problem of reliance on single features, and achieves a balance between accuracy and efficiency, meeting the real-time requirements of autonomous driving. Therefore, it solves the technical problems in related technologies, such as the susceptibility of visible light cameras to light scattering and reflection under adverse weather conditions, resulting in low image contrast and blurred target outlines; the increased false detection rate of obstacles due to raindrops and fog particles affecting lidar; and the insufficient resolution of millimeter-wave radar for non-metallic obstacles (such as pedestrians and animals), making it difficult to achieve fine classification and accurate ranging.
[0091] Figure 3 This is a schematic diagram of the structure of a vehicle according to an embodiment of this application. The vehicle may include: The memory 301, the processor 302, and the computer program stored on the memory 301 and capable of running on the processor 302.
[0092] When the processor 302 executes the program, it implements the obstacle perception method provided in the above embodiments.
[0093] Furthermore, the vehicle also includes: Communication interface 303 is used for communication between memory 301 and processor 302.
[0094] The memory 301 is used to store computer programs that can run on the processor 302.
[0095] The memory 301 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.
[0096] If the memory 301, processor 302, and communication interface 303 are implemented independently, then the communication interface 303, memory 301, and processor 302 can be interconnected via a bus to complete communication between them. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 3 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.
[0097] Optionally, in a specific implementation, if the memory 301, processor 302, and communication interface 303 are integrated on a single chip, then the memory 301, processor 302, and communication interface 303 can communicate with each other through an internal interface.
[0098] Processor 302 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application.
[0099] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the obstacle perception method described above.
[0100] This application also provides a computer program product, including a computer program that, when executed, implements the obstacle perception method described above.
[0101] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.
[0102] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified.
[0103] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or N executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.
[0104] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). In addition, computer-readable media can even be paper or other suitable media on which programs can be printed, because programs can be obtained electronically by optically scanning paper or other media, then editing, interpreting or otherwise processing them as necessary, and then storing them in computer memory.
[0105] It should be understood that the various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. If implemented in hardware, as in another embodiment, it can be implemented using any one or more of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.
[0106] Those skilled in the art will understand that all or part of the steps of the methods described in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it includes one or a combination of the steps of the method embodiments.
[0107] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.
[0108] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.
Claims
1. An obstacle sensing method, characterized in that, Includes the following steps: Acquire infrared images of the vehicle; The infrared image is input into a pre-built obstacle recognition model to identify at least one obstacle feature in the infrared image using a first recognition model in the obstacle recognition model, and the final obstacle in the infrared image is determined based on the at least one obstacle feature and a second recognition model in the obstacle recognition model. Based on at least one motion feature of the final obstacle, a perception result of the final obstacle is generated.
2. The method according to claim 1, characterized in that, Before inputting the infrared image into the pre-built obstacle recognition model, the method further includes: Acquire the target infrared image of the target vehicle; At least one image feature is extracted from the infrared image of the target to obtain the multi-scale features of the infrared image of the target; Based on the multi-scale features, at least one target obstacle feature in the target infrared image is determined; Based on the features of the at least one target obstacle, the first recognition model is constructed to construct the obstacle recognition model according to the first recognition model.
3. The method according to claim 2, characterized in that, Before inputting the infrared image into the pre-built obstacle recognition model, the method further includes: Based on the features of the at least one target obstacle, identify at least one initial target obstacle in the target infrared image; Determine whether the at least one initial obstacle satisfies a preset obstacle condition; If the at least one initial target obstacle satisfies the preset obstacle condition, the corresponding initial target obstacle is retained, and the final target obstacle is determined based on the retained target obstacle. If the at least one target initial obstacle does not meet the preset obstacle condition, the corresponding target initial obstacle is removed, and the target final obstacle is determined based on the removed target initial obstacles; Based on the target final obstacle, a second recognition model is constructed to construct the obstacle recognition model according to the second recognition model.
4. The method according to claim 1, characterized in that, The acquisition of the vehicle's infrared image includes: Detect whether the infrared image meets preset visual conditions; If the infrared image does not meet the preset visual conditions, the visual features to be processed in the infrared image are identified, and the processing instructions for the infrared image are determined based on the visual features to be processed. The infrared image is then processed according to the processing instructions until a processed infrared image that meets the preset visual conditions is obtained.
5. The method according to claim 1, characterized in that, The generation of a perception result for the final obstacle based on at least one motion feature of the final obstacle includes: Based on the at least one motion feature, determine a first weight corresponding to the final obstacle and a second weight corresponding to the motion feature; Based on the first weight, the second weight, the corresponding motion characteristics, and the corresponding final obstacle, the danger level of the final obstacle is obtained; The perception result is generated based on the stated danger level.
6. An obstacle sensing device, characterized in that, include: The first acquisition module is used to acquire infrared images of the vehicle; The first determining module is used to input the infrared image into a pre-built obstacle recognition model, so as to use a first recognition model in the obstacle recognition model to identify at least one obstacle feature in the infrared image, and determine the final obstacle in the infrared image based on the at least one obstacle feature and a second recognition model in the obstacle recognition model; A perception module is used to generate a perception result of the final obstacle based on at least one motion feature of the final obstacle.
7. The apparatus according to claim 6, characterized in that, Also includes: The second acquisition module is used to acquire the target infrared image of the target vehicle before inputting the infrared image into the pre-built obstacle recognition model; An extraction module is used to extract at least one image feature from the target infrared image to obtain the multi-scale features of the target infrared image; The second determining module is used to determine at least one target obstacle feature of the target infrared image based on the multi-scale features; A first construction module is configured to construct the first recognition model based on the features of the at least one target obstacle, so as to construct the obstacle recognition model according to the first recognition model.
8. A vehicle, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, the processor executing the program to implement the obstacle sensing method as described in any one of claims 1-5.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by the processor to implement the obstacle perception method as described in any one of claims 1-5.
10. A computer program product, characterized in that, Includes a computer program, which, when executed, is used to implement the obstacle sensing method as described in any one of claims 1-5.