A road foreign object detection method, system, electronic equipment and product

Abstract

The present invention belongs to the field of road detection technology, and its purpose is to provide a method, system, electronic equipment and product for detecting foreign objects on a road. The method includes: obtaining a road video frame, point cloud data and environmental data; using environmental data to perform dynamic environmental compensation processing on the road video frame to obtain an environmentally compensated road video frame; performing road area segmentation processing on the environmentally compensated road video frame to obtain road area image data; performing foreign object area segmentation processing on the road area image data to obtain initial foreign object area image data; fusing the initial foreign object area image data with the point cloud data, and eliminating candidate foreign object areas without point cloud matching in the initial foreign object area image data to obtain final foreign object area image data; performing foreign object identification processing based on the final foreign object area image data to obtain foreign object information. The present invention has strong environmental adaptability and can improve detection accuracy and stability in complex environments.

Description

Technical Field

[0001] The present invention belongs to the technical field of road detection, and in particular relates to a method, system, electronic equipment and product for detecting foreign objects on a road. Background Art

[0002] With the rapid development of intelligent transportation systems and autonomous driving technologies, roadside foreign object detection has become a crucial technology for ensuring vehicle safety. Roadside foreign objects (such as rocks, lost tire parts, small animals, plastic bags, and other debris) pose a threat to driving safety, especially on highways and in complex traffic scenarios, where they can cause serious accidents. Therefore, efficient and accurate detection of roadside foreign objects is a pressing technical challenge.

[0003] Currently, the widely used road foreign object detection solutions mainly include image detection technology and millimeter-wave radar detection technology. Among them, image detection technology collects road image data through cameras and then uses image processing algorithms to detect and identify foreign objects. Its advantage is that it can identify the shape and texture information of foreign objects and is suitable for clear classification targets. However, this technology is very sensitive to ambient lighting conditions. In low light, rainy and foggy weather or strong light, the detection accuracy decreases significantly. Millimeter-wave radar detection technology uses point cloud data collected by radar to locate and estimate the size of foreign objects on the road. This technology has strong anti-interference capabilities, but the detection effect of small, low-reflective foreign objects such as plastic bags or paper scraps is poor, and it cannot directly provide the appearance and shape information of foreign objects.

[0004] To overcome the limitations of the above-mentioned single detection method, multi-sensor fusion detection technology using cameras and millimeter-wave radars has emerged in the existing technology. For example, Chinese patent publication number CN114419825A discloses a high-speed rail perimeter intrusion monitoring device and method based on millimeter-wave radar and cameras. By jointly analyzing the image data provided by the camera and the point cloud data provided by the millimeter-wave radar, it can improve the performance of foreign object detection by a single sensor to a certain extent.

[0005] However, in the process of using the existing technology, the inventors found that the existing technology has at least the following problems:

[0006] Existing technologies lack the ability to adapt to complex environments. Dynamic lighting, weather, and other environmental changes significantly affect detection performance, making it difficult to meet the needs of all-weather and dynamic traffic scenarios. Specifically, in the process of foreign object feature recognition based on point cloud data, the above-mentioned existing technologies use a constant false alarm detector for target detection, which relies on a set background noise threshold. In complex environments such as those with dynamic noise or interference signals around the road, this may lead to improper threshold adjustment. The detection accuracy of millimeter-wave radar may also be affected to a certain extent in bad weather, resulting in missed detections or false detections. In addition, in the existing technologies, foreign object feature recognition based on image data relies on a pre-established image database. When the ambient lighting changes drastically, such as at night, under strong direct light, or under shadows, the background difference method and the inter-frame difference method are easily ineffective, resulting in the inability to accurately extract foreign object targets. At the same time, rainy and foggy weather will significantly reduce the imaging quality of the camera, resulting in blurred image data, further affecting the foreign object detection effect. Summary of the Invention

[0007] The present invention aims to solve the above technical problems at least to a certain extent, and provides a method, system, electronic equipment and product for detecting foreign objects on the road.

[0008] In order to achieve the above object, the present invention adopts the following technical solutions:

[0009] In a first aspect, the present invention provides a method for detecting foreign objects on a road, comprising:

[0010] Obtain road video frames collected by cameras, point cloud data collected by millimeter-wave radars, and environmental data collected by environmental sensors;

[0011] Performing dynamic environmental compensation processing on the road video frame using the environmental data to obtain an environmentally compensated road video frame;

[0012] Performing road area segmentation processing on the road video frame after environmental compensation to obtain road area image data;

[0013] Performing foreign body region segmentation processing on the road region image data to obtain initial foreign body region image data;

[0014] Fusing the initial foreign body region image data with the point cloud data, and eliminating candidate foreign body regions without point cloud matching in the initial foreign body region image data to obtain final foreign body region image data;

[0015] Foreign body identification processing is performed based on the final foreign body area image data to obtain foreign body information.

[0016] In one possible design, the environmental data collected by the environmental sensor includes ambient light intensity and ambient transmittance; correspondingly, the road video frame is dynamically compensated for the environmental data to obtain an environmentally compensated road video frame, including:

[0017] The road video frame is subjected to illumination compensation processing using the ambient light intensity to obtain an illumination-compensated road video frame; wherein any pixel point ( x , y ) is:

[0018] ;

[0019] Where, , For any preset pixel point ( x , y ) reference brightness value, For any pixel ( x , y ) of the ambient light intensity; is any pixel point in the road video frame ( x , y )’s grayscale value; is the preset brightness compensation constant;

[0020] The environment transmittance and the environment light intensity are used to perform defogging on the illumination-compensated road video frame to obtain an environment-compensated road video frame; wherein any pixel point ( x , y ) is:

[0021] ;

[0022] Where, For any pixel ( x , y )’s ambient transmittance.

[0023] In one possible design, after obtaining the environment-compensated road video frame, the method further includes:

[0024] The environment-compensated road video frame is sequentially subjected to grayscale processing, denoising processing, histogram equalization processing, dynamic threshold segmentation processing, and edge detection processing to obtain a preprocessed road video frame, so as to perform road area segmentation processing on the preprocessed road video frame.

[0025] In one possible design, performing road region segmentation processing on the pre-processed road video frame to obtain road region image data includes:

[0026] Performing Hough transform processing on the pre-processed road video frame to detect and obtain a Hough transform straight line set in the environment-compensated road video frame;

[0027] performing vanishing point detection processing on the pre-processed road video frame according to the Hough transform line set to obtain vanishing points in the pre-processed road video frame;

[0028] extracting two road boundary lines from the preprocessed road video frame based on the vanishing point and the Hough transform line set, and treating an area between the two road boundary lines as a road area;

[0029] The area other than the road area in the preprocessed road video frame is segmented to obtain road area image data containing only the road area.

[0030] In one possible design, performing foreign body region segmentation processing on the road region image data to obtain initial foreign body region image data includes:

[0031] Performing an opening operation on the road area image data to obtain image data after the opening operation;

[0032] Performing a closing operation on the image data after the opening operation processing to obtain image data after the opening and closing operation processing;

[0033] Performing connected domain analysis on the image data processed by the opening and closing operations to extract candidate foreign body region image data from the image data processed by the opening and closing operations;

[0034] The candidate foreign body region image data is filtered to obtain initial foreign body region image data.

[0035] In one possible design, the initial foreign body region image data is fused with the point cloud data, and candidate foreign body regions with no point cloud matching in the initial foreign body region image data are eliminated to obtain final foreign body region image data, including:

[0036] Mapping the point cloud data to the image coordinate system where the initial foreign body area image data is located;

[0037] Matching the point cloud data with the initial foreign object area image data according to the projection position of the point cloud data on the initial foreign object area image, and eliminating point cloud data points in the point cloud data whose projection position is not within any candidate foreign object area in the initial foreign object area image data;

[0038] The number of point cloud data points located in each candidate foreign body area in the initial foreign body area image data is obtained respectively, and the candidate foreign body areas in which the number of point cloud data points in the initial foreign body area image data is less than a specified number are eliminated to obtain the final foreign body area image data.

[0039] In a second aspect, the present invention provides a road foreign object detection system, comprising:

[0040] The data acquisition module is used to obtain road video frames collected by cameras, point cloud data collected by millimeter-wave radars, and environmental data collected by environmental sensors;

[0041] a dynamic environment compensation module, communicatively connected to the data acquisition module, configured to perform dynamic environment compensation processing on the road video frame using the environmental data to obtain an environment-compensated road video frame;

[0042] A road area segmentation module is communicatively connected to the dynamic environment compensation module and is used to perform road area segmentation processing on the road video frame after environment compensation to obtain road area image data;

[0043] a foreign body region segmentation module, communicatively connected to the road region segmentation module, configured to perform foreign body region segmentation processing on the road region image data to obtain initial foreign body region image data;

[0044] a data fusion module, communicatively connected to the foreign body region segmentation module, configured to fuse the initial foreign body region image data with the point cloud data, and eliminate candidate foreign body regions without point cloud matching in the initial foreign body region image data to obtain final foreign body region image data;

[0045] The foreign body identification module is in communication with the data fusion module and is used to perform foreign body identification processing based on the final foreign body area image data to obtain foreign body information.

[0046] In a third aspect, the present invention provides an electronic device, comprising:

[0047] a memory for storing computer program instructions; and

[0048] A processor is configured to execute the computer program instructions to complete the operation of any one of the above-described methods for detecting foreign objects on a road.

[0049] In a fourth aspect, the present invention provides a computer program product, comprising a computer program or instructions, wherein the computer program or instructions, when executed by a computer, implements a road foreign object detection method as described in any one of the above.

[0050] The beneficial effects of the present invention are:

[0051] The present invention discloses a method, system, electronic device and product for detecting foreign objects on a road, which have strong environmental adaptability and can improve detection accuracy and stability in complex environments. Specifically, during the implementation process, the present invention obtains road video frames collected by a camera, point cloud data collected by a millimeter-wave radar and environmental data collected by an environmental sensor in real time, and uses the environmental data to perform dynamic environmental compensation processing on the road video frames to obtain environmentally compensated road video frames; then, the environmentally compensated road video frames are subjected to road region segmentation processing to obtain road region image data, and then the road region image data are subjected to foreign object region segmentation processing to obtain initial foreign object region image data; then, the initial foreign object region image data is fused with the point cloud data, and candidate foreign object regions without point cloud matching in the initial foreign object region image data are eliminated to obtain final foreign object region image data; finally, foreign object recognition processing is performed based on the final foreign object region image data to obtain foreign object information. During this process, dynamic environmental compensation can improve image quality, which not only improves detection accuracy, but also ensures the stability and robustness of the present invention in complex scenes, and is applicable to a variety of changing scenes; at the same time, the fusion processing of image data and point cloud data solves the limitations of a single sensor for road foreign object detection, and improves detection accuracy and environmental adaptability. In addition, the computational complexity of foreign object detection in the present invention is low and the computational efficiency is high, which is applicable to real-time detection scenarios of road foreign objects, and can better ensure the safety and intelligence level of vehicle driving.

[0052] Other beneficial effects of the present invention will be further described in the specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 is a flow chart of a method for detecting foreign objects on a road in an embodiment;

[0054] Figure 2 is a module block diagram of a road foreign object detection system in an embodiment;

[0055] Figure 3 It is a module block diagram of an electronic device in an embodiment. DETAILED DESCRIPTION

[0056] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the present invention will be briefly introduced below in conjunction with the drawings and the description of the embodiments or the prior art. Obviously, the following description of the structure of the drawings is only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work. It should be noted that the description of these embodiments is used to help understand the present invention, but does not constitute a limitation of the present invention.

[0057] Example 1:

[0058] This embodiment discloses a method for detecting foreign objects on a road surface, which can be executed by, but is not limited to, a computer device or a virtual machine with certain computing resources, such as a personal computer, a smart phone, a personal digital assistant, or a wearable device, or by a virtual machine.

[0059] like Figure 1 As shown, a method for detecting foreign objects on a road may include, but is not limited to, the following steps:

[0060] S1. Obtain road video frames captured by a camera, point cloud data collected by a millimeter-wave radar, and environmental data collected by environmental sensors. In this embodiment, the environmental sensors may include, for example, a light sensor and a rain and fog sensor, and the environmental data collected by the environmental sensors may include, for example, ambient light intensity and ambient transmittance, although this is not a limitation. Specifically, in this embodiment, the camera, millimeter-wave radar, and environmental sensors may be set to the same acquisition frequency to ensure temporal consistency of different data and reduce the difficulty of subsequent data processing. During implementation, various road detection data may be collected using devices such as drones.

[0061] It should be understood that in this embodiment, after obtaining the point cloud data collected by the millimeter wave radar, the point cloud data is also subjected to point cloud denoising and formatting processing to obtain pre-processed point cloud data for subsequent processing; wherein, the point cloud denoising processing is used to remove noise point clouds in the initial point cloud data whose reflection intensity is lower than a threshold, and the formatting processing is used to organize the denoised point cloud data into a data format including three-dimensional coordinates and reflection intensity.

[0062] S2. Dynamically compensating the road video frame using the environmental data to obtain an environmentally compensated road video frame. It should be noted that in this embodiment, dynamically compensating the road video frame using the environmental data, i.e., adjusting parameters in the road video frame under different weather and lighting conditions, can adapt the video frame to real-time scene changes.

[0063] Specifically, in this embodiment, the environmental data collected by the environmental sensor includes ambient light intensity and ambient transmittance, wherein the ambient transmittance is used to characterize environmental transparency. Correspondingly, in step S2, the road video frame is subjected to dynamic environmental compensation processing using the environmental data to obtain an environmentally compensated road video frame, including:

[0064] S201. Perform illumination compensation on the road video frame using the ambient light intensity to obtain an illumination-compensated road video frame; wherein any pixel point ( x , y) is:

[0065] ;

[0066] Where, , For any preset pixel point ( x , y ) reference brightness value, For any pixel ( x , y ) of the ambient light intensity; is any pixel point in the road video frame ( x , y )’s grayscale value; is the preset brightness compensation constant;

[0067] S202. Defogging the road video frame after illumination compensation is performed using the ambient transmittance and the ambient light intensity to obtain an environment-compensated road video frame; wherein any pixel point ( x , y ) is:

[0068] ;

[0069] Where, For any pixel ( x , y )’s ambient transmittance.

[0070] In this embodiment, after obtaining the environment-compensated road video frame, the method further includes:

[0071] The environmentally compensated road video frame is sequentially subjected to grayscale processing, denoising, histogram equalization, dynamic threshold segmentation, and edge detection to obtain a preprocessed road video frame, which is then used to perform road region segmentation on the preprocessed road video frame. Grayscale conversion converts an RGB format (a color space using the three primary colors red, green, and blue) image into a grayscale image to reduce computational complexity; denoising uses Gaussian filtering or median filtering to smooth the image while preserving edge information; histogram equalization enhances image contrast under varying lighting conditions; and dynamic threshold segmentation better preserves the edges and other important features of objects in the environmentally compensated road video frame, making it suitable for image segmentation under conditions of uneven lighting and complex backgrounds.

[0072] Specifically, in this embodiment, any pixel point ( x , y ) is:

[0073] ;

[0074] Where, is any pixel in the image after histogram equalization processing ( x , y )’s grayscale value; For any pixel point ( x , y ), , is any pixel point in the image after histogram equalization processing ( x , y )’s local average gray value, It is a preset sensitivity adjustment parameter used for dynamic threshold segmentation adjustment. is any pixel point in the image after histogram equalization processing ( x , y )’s local grayscale standard deviation.

[0075] In this embodiment, after the image is subjected to dynamic threshold segmentation, the image may be further subjected to edge detection, and the edge information of the image may be detected using the Canny operator to obtain the pre-processed road video frame. Specifically, in this embodiment, any pixel point ( x , y ) is:

[0076] ;

[0077] Where, Any pixel point in the image after the dynamic threshold segmentation process ( x , y ) in the horizontal direction, , Any pixel point in the image after the dynamic threshold segmentation process ( x , y ) in the vertical gradient, . Any pixel point in the pre-processed road video frame ( x , y ) is greater than a preset edge intensity threshold, the pixel is an edge point.

[0078] In this embodiment, non-maximum suppression and double threshold processing methods may be further used to extract effective edges, thereby obtaining the final pre-processed road video frame.

[0079] It should be noted that the pre-processed road video frame is an edge binary image. In this embodiment, during the dynamic threshold segmentation process of the image, by adaptively adjusting the threshold according to the local characteristics of the image, it can facilitate the subsequent effective segmentation and extraction of road areas and foreign object areas, further reduce the impact of lighting changes, and reduce false detections.

[0080] S3. Perform road region segmentation on the preprocessed road video frame to obtain road region image data. In this embodiment, the road region boundaries are extracted using a Hough transform and a vanishing point detection method. Non-road regions in the preprocessed road video frame are then excluded based on the road region boundaries (by modifying the pixel values of the non-road regions) to obtain the road region image data.

[0081] In step S3, the pre-processed road video frame is subjected to road region segmentation processing to obtain road region image data, including:

[0082] S301. Perform Hough transform on the pre-processed road video frame to detect and obtain a Hough transform line set in the environment-compensated road video frame. Specifically, in this embodiment, the Hough transform line set can be expressed as L ={ L 1, L 2,……, L n}, L 1, L 2,……, L n is the set of Hough transformed lines n Hough transform lines, n is a natural number greater than 1.

[0083] S302. Perform vanishing point detection on the preprocessed road video frame based on the Hough transform line set to obtain vanishing points in the preprocessed road video frame. Specifically, step S302 includes the following steps:

[0084] S3021. Extend all Hough transform lines in the Hough transform line set and calculate the intersection points between all Hough transform lines. In the implementation process, the linear equations of each Hough transform line can be obtained in advance, and then the intersection points between all Hough transform lines can be obtained by solving the linear equations. P ( x , y )express.

[0085] S3022. Count the position information of all intersections, and based on the position information of all intersections, use the intersections that are concentrated in the designated area of the pre-processed road video frame as vanishing points. It should be understood that the designated area of the pre-processed road video frame depends on the placement of the camera during road detection, and is not limited here. As an example, in this embodiment, if the designated area of the pre-processed road video frame is located in the upper area of the pre-processed road video frame, the vanishing point can be V ( x v , y v )express.

[0086] S303. Based on the vanishing point and the Hough transform line set, two road boundary lines are extracted from the pre-processed road video frame, and the area between the two road boundary lines is regarded as the road area; specifically, in step S303, the road boundary line between the vanishing point and the Hough transform line set is filtered. V ( x v , y v ) and extending from the bottom of the pre-processed road video frame to the vicinity of the vanishing point, as a candidate road boundary line. Assume that the selected candidate road boundary line is l left and l right , the road area can be confirmed based on the straight line equations of the two, and the polygon filling method can be used to generate a binary map of the road area.

[0087] S304. Segment the area excluding the road area in the preprocessed road video frame to obtain road area image data containing only the road area. Specifically, in step S304, the binary image of the road area can be used as a mask to set the grayscale of the non-road area in the preprocessed road video frame to 0, thereby obtaining road area image data containing only the road area.

[0088] In this embodiment, the Hough transform method and the vanishing point detection method are used to accurately extract the road area, adapt to road boundary detection in complex environments, eliminate interference from non-road areas, and provide accurate road area information for subsequent foreign object detection.

[0089] S4. Perform foreign object region segmentation on the road area image data to obtain initial foreign object region image data. It should be understood that during the foreign object segmentation process, if no foreign object region is found or the area of the foreign object region is smaller than a preset value, it is determined that no foreign object exists in the region of the current video frame, and no further processing is performed on the current video frame. In this embodiment, within the road area image data, discrete pixels are connected using morphological operations (opening and closing operations). Connected component analysis is then used to extract candidate foreign object regions. Non-foreign object regions can also be filtered based on criteria such as size and shape to obtain initial foreign object region image data.

[0090] In step S4, the road area image data is subjected to foreign body area segmentation processing to obtain initial foreign body area image data, including:

[0091] S401. Perform an opening operation on the road region image data to obtain opened image data. It should be noted that the opening operation is used to remove small area noise and disconnect fine connections in the road region image data. Specifically, in this embodiment, based on a preset pixel matrix element, such as a 3×3 or 5×5 matrix, an erosion operation and a dilation operation are sequentially performed on the road region image data to retain larger connected areas in the road region image data and remove isolated noise points therein.

[0092] S402. Perform closing operation on the image data after the opening operation processing to obtain the image data after the opening and closing operation processing; it should be noted that the closing operation processing adopts the opposite process to the opening operation processing, that is, the expansion operation and the corrosion operation are performed on the image data after the opening operation processing in sequence, which can further connect the scattered foreign object pixel areas in the image data after the opening operation processing and fill small holes to connect the close foreign object areas in the image data after the opening operation processing into a whole, and the obtained image data after the opening and closing operation processing contains the candidate foreign object image data.

[0093] S403. Perform connected domain analysis on the image data after the opening and closing operation processing, and extract the candidate foreign body area image data in the image data after the opening and closing operation processing; specifically, in this embodiment, a 4-connected (pixel connection in four directions of up, down, left, and right) or 8-connected (pixel connection in four diagonal directions of up, down, left, and right) method is used to scan the image data after the opening and closing operation processing, mark each connected domain, and assign a unique label to each connected area. Each connected area constitutes the candidate foreign body area image data in the image data after the opening and closing operation processing.

[0094] S404. Filter the image data of the candidate foreign object area to obtain the initial foreign object area image data. Specifically, in this embodiment, according to the characteristics of the area, shape, and bounding box of the connected domain, the candidate foreign object area image data in the image data after the opening and closing operations is extracted. For example, the candidate foreign object area image data with an area less than a preset area (such as 10 pixels) and an aspect ratio exceeding the preset range (such as 0.2 < r < 0.5) in the candidate foreign object area image data is screened out, and the remaining candidate foreign object area image data constitutes the initial foreign object area image data. The gray value of any pixel point ( x , y ) in the initial foreign object area image data is represented as .

[0095] In this embodiment, through the opening operation, closing operation, and connected domain analysis processing, the discrete foreign object areas in the road area image data can be connected and the noise can be removed, so as to obtain a more accurate foreign object target area image, effectively improving the accuracy of the foreign object target area extraction and providing clear image data for the subsequent identification of foreign object information such as foreign object types.

[0096] S5. Perform fusion processing on the initial foreign object area image data and the point cloud data, and剔除 the candidate foreign object areas without point cloud matching in the initial foreign object area image data to obtain the final foreign object area image data.

[0097] In step S5, performing fusion processing on the initial foreign object area image data and the point cloud data, and剔除 the candidate foreign object areas without point cloud matching in the initial foreign object area image data to obtain the final foreign object area image data includes:

[0098] S501. Map the point cloud data to the image coordinate system where the initial foreign object area image data is located; specifically, in this embodiment, the following perspective transformation formula is used to implement the mapping processing of the point cloud data:

[0099] ;

[0100] In the formula, ([[]] X , Y , Z ) is the initial coordinate of the point cloud data; ([[]] x , y ) is the image coordinate where the initial foreign object area image data is located, that is, the coordinate of the projection position of the point cloud data; H is the transformation matrix determined according to the internal and external parameter matrices of the camera, H =[[]] K · R 丨 t , Kis the internal parameter matrix of the camera, R | t is the external parameter matrix of the camera.

[0101] In this embodiment, after mapping the point cloud data to the image coordinate system where the initial foreign object area image data is located, points in the point cloud data that are far from the road area are removed according to a preset maximum effective distance threshold, so as to further process the point cloud data.

[0102] S502. According to the projection position of the point cloud data in the initial foreign object area image, match the point cloud data with the initial foreign object area image data, and剔除 the point cloud data points in the point cloud data whose projection positions are not in any candidate foreign object area in the initial foreign object area image data; specifically, in this embodiment, for each point cloud data point, according to its projection position in the initial foreign object area image data, check whether this position is located in any candidate foreign object area of the initial foreign object area image data, that is, check whether the pixel value at the position where this point cloud data point is located is 255. If so, it is determined that this point cloud data point is located within the candidate foreign object area, and this point cloud data point is保留, otherwise this point cloud data point is剔除 to实现 the screening process of the point cloud data.

[0103] S503. Respectively obtain the number of point cloud data points located in each candidate foreign object area in the initial foreign object area image data, and剔除 the candidate foreign object areas in the initial foreign object area image data where the number of point cloud data points is less than the specified number to obtain the final foreign object area image data. Specifically, in this embodiment, it is sequentially determined whether there are the specified number of point cloud data points in each candidate foreign object area. If not, it is determined that there is no point cloud matching in the current candidate foreign object area, and the current candidate foreign object area is剔除 from the initial foreign object area image data, that is, the pixel values of each pixel point in the current candidate foreign object area are modified to 0, otherwise no action is taken until the final foreign object area image data is obtained. The final foreign object area image data includes valid foreign object target areas.

[0104] In this embodiment, the effective fusion of point cloud data and image data is实现 through steps S501 to S503. Through the spatial information provided by millimeter wave radar or lidar, false detections can be effectively reduced and the accuracy of foreign object target detection can be提高.

[0105] S6. Perform foreign object recognition processing according to the final foreign object area image data to obtain foreign object information. Specifically, the foreign object information includes foreign object position information, foreign object size information, and foreign object type information. In this embodiment, the foreign object types include fixed obstacles and moving obstacles, etc.

[0106] It should be noted that there seem to be some inconsistent or unclear terms in the original Chinese text (such as "剔除" and "保留" which might need more context to accurately translate as the most appropriate English words). Here, I've translated them as "剔除" and "保留" for the time, but they could potentially be better translated as "exclude" and "retain" or other more context - appropriate terms. Also, "实现" is translated as "实现" which is actually the Chinese character for "achieve" in this context, and it should be "achieved" in English. Please review and correct as needed based on the actual meaning and context.In step S6, the foreign body type information is identified and processed using a rule-based classification method. Specifically, in this embodiment, foreign body classification rules are pre-established based on information such as size, shape, and texture characteristics, and the foreign body type classification of the final foreign body area image data is implemented based on these classification rules. This classification method is simple and fast.

[0107] In addition, in this embodiment, after obtaining foreign object information, a foreign object warning can be performed based on preset rules, which is not limited here.

[0108] This embodiment has strong environmental adaptability and can improve detection accuracy and stability in complex environments. Specifically, during the implementation process, this embodiment obtains road video frames collected by cameras, point cloud data collected by millimeter-wave radars, and environmental data collected by environmental sensors in real time, and uses the environmental data to perform dynamic environmental compensation processing on the road video frames to obtain environmentally compensated road video frames; then, the environmentally compensated road video frames are subjected to road area segmentation processing to obtain road area image data, and then the road area image data are subjected to foreign object area segmentation processing to obtain initial foreign object area image data; then, the initial foreign object area image data is fused with the point cloud data, and candidate foreign object areas without point cloud matching in the initial foreign object area image data are eliminated to obtain final foreign object area image data; finally, foreign object recognition processing is performed based on the final foreign object area image data to obtain foreign object information. During this process, dynamic environmental compensation can improve image quality, which not only improves detection accuracy, but also ensures the stability and robustness of this embodiment in complex scenarios, and is suitable for a variety of changing scenarios; at the same time, the fusion processing of image data and point cloud data solves the limitations of a single sensor for road foreign object detection, and improves detection accuracy and environmental adaptability. In addition, the computational complexity of foreign object detection in this embodiment is low and the computational efficiency is high, which is suitable for real-time road foreign object detection scenarios, and can better ensure the safety and intelligence level of vehicle driving.

[0109] Example 2:

[0110] This embodiment discloses a road foreign object detection system for implementing the road foreign object detection method in embodiment 1; Figure 2 As shown, the road foreign object detection system includes:

[0111] The data acquisition module is used to obtain road video frames collected by cameras, point cloud data collected by millimeter-wave radars, and environmental data collected by environmental sensors;

[0112] a dynamic environment compensation module, communicatively connected to the data acquisition module, configured to perform dynamic environment compensation processing on the road video frame using the environmental data to obtain an environment-compensated road video frame;

[0113] A road area segmentation module is communicatively connected to the dynamic environment compensation module and is used to perform road area segmentation processing on the road video frame after environment compensation to obtain road area image data;

[0114] a foreign body region segmentation module, communicatively connected to the road region segmentation module, configured to perform foreign body region segmentation processing on the road region image data to obtain initial foreign body region image data;

[0115] a data fusion module, communicatively connected to the foreign body region segmentation module, configured to fuse the initial foreign body region image data with the point cloud data, and eliminate candidate foreign body regions without point cloud matching in the initial foreign body region image data to obtain final foreign body region image data;

[0116] The foreign body identification module is in communication with the data fusion module and is used to perform foreign body identification processing based on the final foreign body area image data to obtain foreign body information.

[0117] It should be noted that the working process, working details and technical effects of the road foreign object detection system provided in this embodiment 2 can be found in embodiment 1 and will not be described in detail here.

[0118] Example 3:

[0119] Based on the embodiment 1 or 2, this embodiment discloses an electronic device, which may be a smart phone, a tablet computer, a laptop computer or a desktop computer. The electronic device may be called a user terminal, a portable terminal, a desktop terminal, etc. Figure 3 As shown, the electronic equipment includes:

[0120] a memory for storing computer program instructions; and

[0121] A processor is configured to execute the computer program instructions to thereby complete the operation of a road foreign object detection method as described in any one of the first embodiments.

[0122] Specifically, the processor 301 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 301 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 301 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 301 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen.

[0123] Memory 302 may include one or more computer-readable storage media, which may be non-transitory. Memory 302 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in memory 302 is used to store at least one instruction, which is executed by processor 301 to implement the road foreign object detection method provided in Example 1 of this application.

[0124] In some embodiments, the terminal may optionally include a communication interface 303 and at least one peripheral device. The processor 301, memory 302, and communication interface 303 may be connected via a bus or signal lines. Each peripheral device may be connected to the communication interface 303 via a bus, signal lines, or circuit boards. Specifically, the peripheral device may include at least one of a radio frequency circuit 304, a display screen 305, and a power supply 306.

[0125] The communication interface 303 can be used to connect at least one I / O (Input / Output)-related peripheral device to the processor 301 and the memory 302. In some embodiments, the processor 301, the memory 302, and the communication interface 303 are integrated on the same chip or circuit board. In other embodiments, any one or two of the processor 301, the memory 302, and the communication interface 303 can be implemented on separate chips or circuit boards, which is not limited in this embodiment.

[0126] The radio frequency circuit 304 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 304 communicates with a communication network and other communication devices via electromagnetic signals.

[0127] The display screen 305 is used to display a UI (User Interface). The UI may include any combination of graphics, text, icons, and videos.

[0128] The power supply 306 is used to supply power to various components in the electronic device.

[0129] Example 4:

[0130] Based on any one of Embodiments 1 to 3, this embodiment discloses a computer program product, including a computer program or instructions. When executed by a computer, the computer program or instructions implement the road foreign object detection method described in any one of Embodiment 1. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device.

[0131] Obviously, those skilled in the art will appreciate that the modules or steps of the present invention described above can be implemented using a general-purpose computing device. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Alternatively, they can be implemented using program code executable by a computing device, which can then be stored in a storage device and executed by the computing device. Alternatively, they can be fabricated into separate integrated circuit modules, or multiple modules or steps can be fabricated into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0132] Finally, it should be noted that the above embodiments are intended only to illustrate the technical solutions of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art will appreciate that modifications may be made to the technical solutions described in the above embodiments, or that some of the technical features may be replaced with equivalents. Such modifications or replacements do not deviate from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A method for detecting foreign objects on a road, characterized in that: include: Obtain road video frames collected by cameras, point cloud data collected by millimeter-wave radars, and environmental data collected by environmental sensors; Performing dynamic environmental compensation processing on the road video frame using the environmental data to obtain an environmentally compensated road video frame; Performing road area segmentation processing on the road video frame after environmental compensation to obtain road area image data; Performing foreign body region segmentation processing on the road region image data to obtain initial foreign body region image data; Fusing the initial foreign body region image data with the point cloud data, and eliminating candidate foreign body regions without point cloud matching in the initial foreign body region image data to obtain final foreign body region image data; Performing foreign body identification processing based on the final foreign body area image data to obtain foreign body information; The environmental data collected by the environmental sensor includes ambient light intensity and ambient transmittance; correspondingly, the road video frame is subjected to dynamic environmental compensation processing using the environmental data to obtain an environmentally compensated road video frame, including: The road video frame is subjected to illumination compensation processing using the ambient light intensity to obtain an illumination-compensated road video frame; wherein any pixel point ( x , y ) is: ； Where, , For any preset pixel point ( x , y ) reference brightness value, For any pixel ( x , y ) of the ambient light intensity; is any pixel point in the road video frame ( x , y )’s grayscale value; is the preset brightness compensation constant; The environment transmittance and the environment light intensity are used to perform defogging on the illumination-compensated road video frame to obtain an environment-compensated road video frame; wherein any pixel point ( x , y ) is: ； Where, For any pixel ( x , y )’s ambient transmittance.

2. A method for detecting foreign objects on a road according to claim 1, characterized in that: After obtaining the environment-compensated road video frame, the method further includes: The environment-compensated road video frame is sequentially subjected to grayscale processing, denoising processing, histogram equalization processing, dynamic threshold segmentation processing, and edge detection processing to obtain a preprocessed road video frame, so as to perform road area segmentation processing on the preprocessed road video frame.

3. A method for detecting foreign objects on a road according to claim 2, characterized in that: Performing road region segmentation processing on the pre-processed road video frame to obtain road region image data, including: Performing Hough transform processing on the pre-processed road video frame to detect and obtain a Hough transform straight line set in the environment-compensated road video frame; performing vanishing point detection processing on the pre-processed road video frame according to the Hough transform line set to obtain vanishing points in the pre-processed road video frame; extracting two road boundary lines from the preprocessed road video frame based on the vanishing point and the Hough transform line set, and treating an area between the two road boundary lines as a road area; The area other than the road area in the preprocessed road video frame is segmented to obtain road area image data containing only the road area.

4. The method for detecting foreign objects on a road according to claim 1, wherein: Performing foreign body region segmentation processing on the road region image data to obtain initial foreign body region image data includes: Performing an opening operation on the road area image data to obtain image data after the opening operation; Performing a closing operation on the image data after the opening operation processing to obtain image data after the opening and closing operation processing; Performing connected domain analysis on the image data processed by the opening and closing operations to extract candidate foreign body region image data from the image data processed by the opening and closing operations; The candidate foreign body region image data is filtered to obtain initial foreign body region image data.

5. The method for detecting foreign objects on a road according to claim 1, wherein: The initial foreign body region image data is fused with the point cloud data, and candidate foreign body regions with no point cloud matching in the initial foreign body region image data are eliminated to obtain final foreign body region image data, including: Mapping the point cloud data to the image coordinate system where the initial foreign body area image data is located; Matching the point cloud data with the initial foreign object area image data according to the projection position of the point cloud data on the initial foreign object area image, and eliminating point cloud data points in the point cloud data whose projection position is not within any candidate foreign object area in the initial foreign object area image data; The number of point cloud data points located in each candidate foreign body area in the initial foreign body area image data is obtained respectively, and the candidate foreign body areas in which the number of point cloud data points in the initial foreign body area image data is less than a specified number are eliminated to obtain the final foreign body area image data.

6. A road foreign object detection system, characterized in that: include: The data acquisition module is used to obtain road video frames collected by cameras, point cloud data collected by millimeter-wave radars, and environmental data collected by environmental sensors; a dynamic environment compensation module, communicatively connected to the data acquisition module, configured to perform dynamic environment compensation processing on the road video frame using the environmental data to obtain an environment-compensated road video frame; A road area segmentation module is communicatively connected to the dynamic environment compensation module and is used to perform road area segmentation processing on the road video frame after environment compensation to obtain road area image data; a foreign body region segmentation module, communicatively connected to the road region segmentation module, configured to perform foreign body region segmentation processing on the road region image data to obtain initial foreign body region image data; a data fusion module, communicatively connected to the foreign body region segmentation module, configured to fuse the initial foreign body region image data with the point cloud data, and eliminate candidate foreign body regions without point cloud matching in the initial foreign body region image data to obtain final foreign body region image data; A foreign body identification module is communicatively connected to the data fusion module and is used to perform foreign body identification processing based on the final foreign body area image data to obtain foreign body information; The environmental data collected by the environmental sensor includes ambient light intensity and ambient transmittance; correspondingly, the road video frame is subjected to dynamic environmental compensation processing using the environmental data to obtain an environmentally compensated road video frame, including: The road video frame is subjected to illumination compensation processing using the ambient light intensity to obtain an illumination-compensated road video frame; wherein any pixel point ( x , y ) is: ； Where, , For any preset pixel point ( x , y ) reference brightness value, For any pixel ( x , y ) of the ambient light intensity; is any pixel point in the road video frame ( x , y )’s grayscale value; is the preset brightness compensation constant; The environment transmittance and the environment light intensity are used to perform defogging on the illumination-compensated road video frame to obtain an environment-compensated road video frame; wherein any pixel point ( x , y ) is: ； Where, For any pixel ( x , y )’s ambient transmittance.

7. An electronic device, characterized in that: include: a memory for storing computer program instructions; as well as, A processor is configured to execute the computer program instructions to thereby complete the operation of the road foreign object detection method according to any one of claims 1 to 5.

8. A computer program product comprising a computer program or instructions, characterized in that When the computer program or the instruction is executed by a computer, the method for detecting foreign objects on a road surface according to any one of claims 1 to 5 is implemented.

Images