Road foreign matter detection method and system, electronic equipment and product

Abstract

The invention belongs to the technical field of road detection, and aims to provide a road foreign matter detection method and system, electronic equipment and a product. The method comprises the following steps: acquiring a road video frame, point cloud data and environment data; performing dynamic environment compensation processing on the road video frame by adopting the environment data to obtain an environment-compensated road video frame; performing road area segmentation processing on the road video frame after environment compensation to obtain road area image data; performing foreign matter region segmentation processing on the road region image data to obtain initial foreign matter region image data; performing fusion processing on the initial foreign matter area image data and the point cloud data, and removing candidate foreign matter areas without point cloud matching in the initial foreign matter area image data to obtain final foreign matter area image data; and performing foreign matter identification processing according to the final foreign matter area image data to obtain foreign matter information. The method is high in environmental adaptability, and can improve the detection precision and stability in a complex environment.

Description

Technical Field

[0001] The present invention belongs to the technical field of road detection, and particularly relates to a method, a system, an electronic device and a product for detecting road foreign objects. Background Art

[0002] With the rapid development of intelligent transportation systems and autonomous driving technologies, road foreign object detection has become one of the important technologies to ensure vehicle driving safety. Road foreign objects (such as stones, lost tire parts, small animals, plastic bags or other sundries) pose a threat to driving safety. Especially on highways or in complex traffic scenarios, foreign objects may cause serious traffic accidents. Therefore, how to efficiently and accurately detect road foreign objects is one of the technical problems that need to be solved urgently at present.

[0003] Currently, the widely used road foreign object detection solutions mainly include image detection technology and millimeter-wave radar detection technology. Among them, the image detection technology collects road image data through a camera and then uses image processing algorithms for foreign object detection and recognition. Its advantage is that it can identify the shape and texture information of foreign objects and is suitable for clear classification targets. However, this technology is very sensitive to environmental lighting conditions, and the detection accuracy drops significantly under low light, rainy or foggy weather or strong light illumination; while the millimeter-wave radar detection technology realizes the positioning and size estimation of foreign objects in the road through the point cloud data collected by the radar. This technology has strong anti-interference ability, but has poor detection effect on small and low-reflectivity foreign objects such as plastic bags or paper scraps, and cannot directly provide the appearance and shape information of foreign objects.

[0004] To overcome the limitations of the above single detection means, in the prior art, there has emerged a detection technology that uses multi-sensor fusion of a camera and a millimeter-wave radar. For example, Chinese Patent No. CN114419825A discloses a high-speed rail perimeter intrusion monitoring device and method based on a millimeter-wave radar and a camera, which can improve the performance of foreign object detection by a single sensor to a certain extent through joint analysis of the image data provided by the camera and the point cloud data provided by the millimeter-wave radar.

[0005] However, in the process of using the prior art, the inventors found that there are at least the following problems in the prior art: The prior art has insufficient adaptability to complex environments. Under dynamic environmental change conditions such as lighting and weather, it will significantly affect the detection effect, making it difficult to meet the requirements of all-weather and dynamic traffic scenarios. Specifically, in the process of foreign object feature recognition based on point cloud data, the method of using a constant false alarm detector for target detection depends on the set background noise threshold. In complex environments such as the appearance of dynamic noise or interference signals around the road, it may lead to improper threshold adjustment, and the detection accuracy of the millimeter-wave radar may also be affected to a certain extent in bad weather, resulting in missed detections or false detections. In addition, in the prior art, foreign object feature recognition based on image data relies on a pre-established image database, and when there are drastic changes in environmental lighting such as at night, direct strong light, or shadow coverage, the background difference method and the inter-frame difference method are prone to failure, resulting in the inability to accurately extract foreign object targets. At the same time, rainy and foggy weather will significantly reduce the imaging quality of the camera, resulting in blurred image data collected, further affecting the foreign object detection effect. Summary of the Invention

[0006] The present invention aims to solve at least to some extent the above technical problems, and provides a road foreign object detection method, system, electronic device and product.

[0007] To achieve the above object, the present invention adopts the following technical solutions: In a first aspect, the present invention provides a road foreign object detection method, including: Obtaining road video frames collected by a camera, point cloud data collected by a millimeter-wave radar, and environmental data collected by an environmental sensor; Performing dynamic environmental compensation processing on the road video frames by using the environmental data to obtain environmentally compensated road video frames; Performing road area segmentation processing on the environmentally compensated road video frames to obtain road area image data; Performing foreign object area segmentation processing on the road area image data to obtain initial foreign object area image data; Fusing the initial foreign object area image data with the point cloud data, and removing candidate foreign object areas in the initial foreign object area image data that have no point cloud matching to obtain final foreign object area image data; Performing foreign object recognition processing according to the final foreign object area image data to obtain foreign object information.

[0008] In a possible design, the environmental data collected by the environmental sensor includes environmental light intensity and environmental transmittance; correspondingly, performing dynamic environmental compensation processing on the road video frames by using the environmental data to obtain environmentally compensated road video frames includes: Perform illumination compensation processing on the road video frame using the ambient light intensity to obtain an illumination-compensated road video frame; wherein, the gray value of any pixel point ( x , y ) in the illumination-compensated road video frame is: ; In the formula, , is the reference brightness value of any preset pixel point ( x , y ), is the ambient light intensity of any pixel point ( x , y ); is the gray value of any pixel point ( x , y ) in the road video frame; is the preset brightness compensation constant; Perform defogging processing on the illumination-compensated road video frame using the ambient transmittance and the ambient light intensity to obtain an environment-compensated road video frame; wherein, the gray value of any pixel point ( x , y ) in the environment-compensated road video frame is: ; In the formula, is the ambient transmittance of any pixel point ( x , y ).

[0009] In a possible design, after obtaining the environment-compensated road video frame, the method further includes: Perform graying processing, denoising processing, histogram equalization processing, dynamic threshold segmentation processing, and edge detection processing on the environment-compensated road video frame in sequence to obtain a preprocessed road video frame for performing road area segmentation processing on the preprocessed road video frame.

[0010] In a possible design, performing road area segmentation processing on the preprocessed road video frame to obtain road area image data includes: Perform Hough transform processing on the preprocessed road video frame to detect the set of Hough transform lines in the environment-compensated road video frame; Perform vanishing point detection processing on the preprocessed road video frame according to the set of Hough transform lines to obtain the vanishing point in the preprocessed road video frame; Based on the vanishing point and the set of Hough transform lines, two road boundary lines in the preprocessed road video frame are extracted, and the area between the two road boundary lines is regarded as the road area; The area other than the road area in the preprocessed road video frame is segmented to obtain road area image data containing only the road area.

[0011] In a possible design, the road area image data is subjected to foreign object area segmentation processing to obtain initial foreign object area image data, including: The road area image data is subjected to opening operation processing to obtain image data after opening operation processing; The image data after opening operation processing is subjected to closing operation processing to obtain image data after opening and closing operation processing; The image data after opening and closing operation processing is subjected to connected component analysis processing to extract candidate foreign object area image data in the image data after opening and closing operation processing; The candidate foreign object area image data is filtered to obtain initial foreign object area image data.

[0012] In a possible design, the initial foreign object area image data is fused with the point cloud data, and candidate foreign object areas in the initial foreign object area image data without point cloud matching are removed to obtain final foreign object area image data, including: The point cloud data is mapped to the image coordinate system where the initial foreign object area image data is located; According to the projection position of the point cloud data in the initial foreign object area image, the point cloud data is matched with the initial foreign object area image data, and point cloud data points whose projection positions are not in any candidate foreign object area in the initial foreign object area image data are removed; The number of point cloud data points located in each candidate foreign object area in the initial foreign object area image data is obtained respectively, and candidate foreign object areas in the initial foreign object area image data with the number of point cloud data points less than the specified number are removed to obtain final foreign object area image data.

[0013] In a second aspect, the present invention provides a road foreign object detection system, including: A data acquisition module, configured to acquire road video frames collected by a camera, point cloud data collected by a millimeter wave radar, and environmental data collected by an environmental sensor; A dynamic environment compensation module, communicatively connected to the data acquisition module, configured to perform dynamic environment compensation processing on the road video frame by using the environmental data to obtain an environmentally compensated road video frame; A road area segmentation module, communicatively connected to the dynamic environment compensation module, is configured to perform road area segmentation processing on the road video frame after environment compensation to obtain road area image data; A foreign object area segmentation module, communicatively connected to the road area segmentation module, is configured to perform foreign object area segmentation processing on the road area image data to obtain initial foreign object area image data; A data fusion module, communicatively connected to the foreign object area segmentation module, is configured to perform fusion processing on the initial foreign object area image data and the point cloud data, and eliminate candidate foreign object areas in the initial foreign object area image data that have no point cloud matching, to obtain final foreign object area image data; A foreign object recognition module, communicatively connected to the data fusion module, is configured to perform foreign object recognition processing based on the final foreign object area image data to obtain foreign object information.

[0014] In a third aspect, the present invention provides an electronic device, including: A memory, configured to store computer program instructions; and, A processor, configured to execute the computer program instructions to complete the operations of a road foreign object detection method as described in any one of the above.

[0015] In a fourth aspect, the present invention provides a computer program product, including a computer program or instructions, where the computer program or the instructions, when executed by a computer, implement a road foreign object detection method as described in any one of the above.

[0016] The beneficial effects of the present invention are: The present invention discloses a method, system, electronic device and product for detecting road foreign objects, which has strong environmental adaptability and can improve the detection accuracy and stability in complex environments. Specifically, in the implementation process of the present invention, road video frames collected by a camera, point cloud data collected by a millimeter-wave radar, and environmental data collected by an environmental sensor are acquired in real time, and the road video frames are subjected to dynamic environmental compensation processing using the environmental data to obtain road video frames after environmental compensation; then, the road video frames after environmental compensation are subjected to road area segmentation processing to obtain road area image data, and the road area image data is further subjected to foreign object area segmentation processing to obtain initial foreign object area image data; then, the initial foreign object area image data is fused with the point cloud data, and candidate foreign object areas without point cloud matching in the initial foreign object area image data are removed to obtain final foreign object area image data; finally, foreign object recognition processing is performed based on the final foreign object area image data to obtain foreign object information. In this process, dynamic environmental compensation can improve the image quality, which can not only improve the detection accuracy, but also ensure the stability and robustness of the present invention in complex scenarios and is applicable to various changing scenarios; at the same time, the method of fusing image data and point cloud data solves the limitations of single-sensor road foreign object detection, improves the detection accuracy and environmental adaptability. In addition, the computational complexity of foreign object detection in the present invention is relatively low, the computational efficiency is high, it is applicable to the real-time detection scenario of road foreign objects, and can better ensure the safety and intelligent level of vehicle driving.

[0017] Other beneficial effects of the present invention will be further described in the specific implementation manner. Brief Description of the Drawings

[0018] Figure 1 is a flowchart of a method for detecting road foreign objects in an embodiment; Figure 2 is a block diagram of a system for detecting road foreign objects in an embodiment; Figure 3 is a block diagram of an electronic device in an embodiment. Specific Embodiment

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the present invention in combination with the drawings and the descriptions of the embodiments or the prior art. Obviously, the following descriptions of the structures of the drawings are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts. It should be noted here that the descriptions of these embodiments are used to help understand the present invention, but do not constitute a limitation to the present invention.

[0020] Embodiment 1: This embodiment discloses a method for detecting road foreign objects, which can be, but is not limited to, executed by a computer device or virtual machine with certain computing resources, such as an electronic device like a personal computer, smartphone, personal digital assistant, or wearable device, or by a virtual machine.

[0021] As Figure 1 shown, a method for detecting road foreign objects can, but is not limited to, include the following steps: S1. Obtain the road video frames collected by the camera, the point cloud data collected by the millimeter-wave radar, and the environmental data collected by the environmental sensor. In this embodiment, the environmental sensor includes, for example, a light sensor and a rain and fog sensor, and the environmental data collected by the environmental sensor includes, for example, environmental light intensity and environmental transmittance, which are not limited herein. Specifically, in this embodiment, the camera, millimeter-wave radar, and environmental sensor can be set to the same acquisition frequency to ensure the time consistency of different data and reduce the subsequent data processing difficulty. During implementation, the acquisition of various road detection data can be achieved through devices such as drones.

[0022] It should be understood that in this embodiment, after obtaining the point cloud data collected by the millimeter-wave radar, the point cloud data is also subjected to point cloud denoising processing and formatting processing to obtain preprocessed point cloud data for subsequent processing; among them, the point cloud denoising processing is used to remove the noise point clouds with a reflection intensity lower than the threshold in the initial point cloud data, and the formatting processing is used to organize the denoised point cloud data into a data format including three-dimensional coordinates and reflection intensity.

[0023] S2. Perform dynamic environmental compensation processing on the road video frames using the environmental data to obtain environmentally compensated road video frames. It should be noted that in this embodiment, performing dynamic environmental compensation processing on the road video frames using the environmental data, that is, adjusting the parameters in the road video frames under different weather and lighting conditions, can make the video frames adapt to real-time scene changes.

[0024] Specifically, in this embodiment, the environmental data collected by the environmental sensor includes environmental light intensity and environmental transmittance, where the environmental transmittance is used to characterize the environmental transparency; correspondingly, in step S2, performing dynamic environmental compensation processing on the road video frames using the environmental data to obtain environmentally compensated road video frames includes: S201. Perform light compensation processing on the road video frames using the environmental light intensity to obtain light-compensated road video frames; among them, the gray value of any pixel point ( x , y ) in the light-compensated road video frames is: ; In the formula, , is the reference luminance value of any preset pixel point ( x , y ); is the ambient light intensity of any pixel point ( x , y ); is the grayscale value of any pixel point ( x , y ) in the road video frame; is a preset luminance compensation constant; S202. Perform defogging processing on the light-compensated road video frame by using the ambient transmittance and the ambient light intensity to obtain an ambient-compensated road video frame; wherein, the grayscale value of any pixel point ( x , y ) in the ambient-compensated road video frame is: ; In the formula, is the ambient transmittance of any pixel point ( x , y ).

[0025] In this embodiment, after obtaining the ambient-compensated road video frame, the method further includes: Performing grayscale processing, denoising processing, histogram equalization processing, dynamic threshold segmentation processing, and edge detection processing on the ambient-compensated road video frame in sequence to obtain a preprocessed road video frame, so as to perform road region segmentation processing on the preprocessed road video frame. Among them, grayscale processing is used to convert an RGB format (a color space using a combination of three primary colors, red, green, and blue) image into a grayscale image to reduce computational complexity; denoising processing uses Gaussian filtering or median filtering to eliminate noise in the image to smooth the image and retain the edge information of the image; histogram equalization processing is used to enhance the contrast of the image under different lighting conditions; dynamic threshold segmentation processing can better retain the edges and other important features of the target object in the ambient-compensated road video frame, and is suitable for image segmentation in the case of uneven lighting and complex backgrounds.

[0026] Specifically, in this embodiment, the grayscale value of any pixel point ( x , y ) in the image after dynamic threshold segmentation processing is: ; In the formula, is the grayscale value of any pixel point ( x , y ) in the image after histogram equalization processing; is the said any pixel point ( x ,y The local threshold of , is the local average gray value of any pixel point ( x , y ) in the image after the histogram equalization process, is a preset sensitivity adjustment parameter for performing dynamic threshold segmentation adjustment, is the local gray standard deviation of any pixel point ( x , y ) in the image after the histogram equalization process.

[0027] In this embodiment, after performing the dynamic threshold segmentation process on the image, the image can be further subjected to edge detection processing, and the Canny operator is used to detect the edge information of the image, so as to obtain the preprocessed road video frame. Specifically, in this embodiment, the edge strength of any pixel point ( x , y ) in the preprocessed road video frame is: ; In the formula, is the gradient of any pixel point ( x , y ) in the horizontal direction in the image after the dynamic threshold segmentation process, , is the gradient of any pixel point ( x , y ) in the vertical direction in the image after the dynamic threshold segmentation process, . When the edge strength of any pixel point ( x , y ) in the preprocessed road video frame is greater than the preset edge strength threshold, this pixel point is an edge point.

[0028] In this embodiment, the non-maximum suppression and double-threshold processing methods can be further used to extract the effective edges, and then the final preprocessed road video frame can be obtained.

[0029] It should be noted that the preprocessed road video frame is an edge binary image. In this embodiment, during the process of performing the dynamic threshold segmentation process on the image, by adaptively adjusting the threshold according to the local characteristics of the image, it is beneficial to subsequently effectively perform the segmentation and extraction processing of the road area and the foreign object area, further reducing the influence brought by the illumination change and reducing the false detection.

[0030] S3. Perform road region segmentation processing on the preprocessed road video frames to obtain road region image data. In this embodiment, the Hough transform method and the vanishing point detection method are used to extract the road region boundaries, and then based on the road region boundaries, the non-road regions in the preprocessed road video frames are excluded (the pixel values of the non-road regions are modified), and thus the road region image data can be obtained.

[0031] In step S3, performing road region segmentation processing on the preprocessed road video frames to obtain road region image data includes: S301. Perform Hough transform processing on the preprocessed road video frames to detect the set of Hough transform lines in the environment-compensated road video frames; specifically, in this embodiment, the set of Hough transform lines can be expressed as L ={ L 1, L 2,……, L n}, L 1, L 2,……, L n is the n th Hough transform line in the set of Hough transform lines, n is a natural number greater than 1.

[0032] S302. According to the set of Hough transform lines, perform vanishing point detection processing on the preprocessed road video frames to obtain the vanishing point in the preprocessed road video frames; specifically, in step S302, it specifically includes the following steps: S3021. Extend all the Hough transform lines in the set of Hough transform lines and calculate the intersections between all the Hough transform lines; during implementation, the linear equations of each Hough transform line can be obtained in advance, and then according to the linear equations of each line, the intersections between all the Hough transform lines are solved, and the intersections can be represented by P ( x , y ).

[0033] S3022. Statistically analyze the position information of all the intersections, and according to the position information of all the intersections, take the intersections that are concentratedly distributed in the specified area of the preprocessed road video frames as the vanishing point. It should be understood that the specified area of the preprocessed road video frames is determined according to the placement position of the camera during road detection, which is not limited here. As an example, in this embodiment, the specified area of the preprocessed road video frames is, for example, the upper area of the preprocessed road video frames, and the vanishing point can be represented by V ( x v , y v) representation.

[0034] S303. Based on the vanishing point and the set of Hough transform lines, two road boundary lines in the preprocessed road video frame are extracted, and the area between the two road boundary lines is regarded as the road area; specifically, in step S303, by screening the set of Hough transform lines that are related to the vanishing point V ( x v , y v ) and extend from the bottom of the preprocessed road video frame to near the vanishing point, the Hough transform lines are used as candidate road boundary lines. Assume the selected candidate road boundary lines are l left and l right , then the road area can be confirmed according to their straight-line equations, and the binary image of the road area can be generated using the polygon filling method.

[0035] S304. The area other than the road area in the preprocessed road video frame is segmented to obtain road area image data containing only the road area. Specifically, in step S304, the binary image of the road area can be used as a mask to set the gray level of the non-road area in the preprocessed road video frame to 0, so as to obtain road area image data containing only the road area.

[0036] In this embodiment, by using the Hough transform method and the vanishing point detection method, the road area can be accurately extracted, adapting to road boundary detection in complex environments, excluding interference from non-road areas, and providing accurate road area information for subsequent foreign object detection.

[0037] S4. Perform foreign object area segmentation processing on the road area image data to obtain initial foreign object area image data. It should be understood that when performing foreign object segmentation processing, if no foreign object area is found or the area of the foreign object area is less than the preset value, it is determined that there is no foreign object in the area where the current video frame is located, and no subsequent processing is performed on the current video frame. In this embodiment, within the road area image data, discrete pixel points are connected through morphological operations (opening operation, closing operation), and then connected component analysis is used to extract candidate foreign object areas, and non-foreign object areas can be filtered according to rules such as size and shape, thereby obtaining initial foreign object area image data.

[0038] In step S4, performing foreign object area segmentation processing on the road area image data to obtain initial foreign object area image data includes: S401. Perform an opening operation on the road area image data to obtain the image data after the opening operation. It should be noted that the opening operation is used to remove small area noises in the road area image data and disconnect thin connections. Specifically, in this embodiment, based on a preset pixel matrix element such as 3×3 or 5×5, the erosion operation and the dilation operation are sequentially performed on the road area image data to retain the larger connected areas in the road area image data and remove the isolated noise points therein.

[0039] S402. Perform a closing operation on the image data after the opening operation to obtain the image data after the opening and closing operations. It should be noted that the closing operation adopts a process opposite to that of the opening operation, that is, the dilation operation and the erosion operation are sequentially performed on the image data after the opening operation, which can further connect the scattered foreign object pixel areas in the image data after the opening operation and fill small holes, so as to connect the adjacent foreign object areas in the image data after the opening operation into a whole, and the image data after the opening and closing operations contains candidate foreign object image data.

[0040] S403. Perform a connected component analysis on the image data after the opening and closing operations to extract the candidate foreign object area image data in the image data after the opening and closing operations. Specifically, in this embodiment, the 4-connected (pixel connection in the four directions of up, down, left, and right) or 8-connected (pixel connection in the four directions of up, down, left, and right and the four diagonal directions) method is used to scan the image data after the opening and closing operations, mark each connected component, and assign a unique label to each connected area. Each connected area constitutes the candidate foreign object area image data in the image data after the opening and closing operations.

[0041] S404. Filter the candidate foreign object area image data to obtain the initial foreign object area image data. Specifically, in this embodiment, according to the characteristics of the connected component area, shape, and bounding box, etc., the candidate foreign object area image data in the image data after the opening and closing operations is extracted. For example, the candidate foreign object area image data with an area less than a preset area (such as 10 pixels) and an aspect ratio exceeding a preset range (such as aspect ratio 0.2 < r < 0.5) in the candidate foreign object area image data is screened out, and the remaining candidate foreign object area image data constitutes the initial foreign object area image data. The gray value of any pixel point ( x , y ) in the initial foreign object area image data is expressed as .

[0042] In this embodiment, through opening operation processing, closing operation processing and connected domain analysis processing, the discrete foreign object areas in the road area image data can be connected and the noise can be removed, so as to obtain a more accurate foreign object target area image, effectively improving the accuracy of foreign object target area extraction, and providing clear image data for subsequent foreign object type and other foreign object information identification.

[0043] S5. The initial foreign body region image data is fused with the point cloud data, and candidate foreign body regions without point cloud matching in the initial foreign body region image data are eliminated to obtain final foreign body region image data.

[0044] In step S5, the initial foreign body region image data is fused with the point cloud data, and candidate foreign body regions without point cloud matching in the initial foreign body region image data are eliminated to obtain final foreign body region image data, including: S501. Mapping the point cloud data to the image coordinate system where the initial foreign body area image data is located; specifically, in this embodiment, the following perspective transformation formula is used to implement the mapping process of the point cloud data: ; In the formula, ( X , Y , Z ) is the initial coordinate of the point cloud data; ( x , y ) is the image coordinates of the initial foreign body area image data, that is, the coordinates of the projection position of the point cloud data; H is a transformation matrix determined according to the intrinsic and extrinsic parameter matrices of the camera, H = K ·[ R 丨 t ], K is the intrinsic parameter matrix of the camera, [ R 丨 t ] is the external parameter matrix of the camera.

[0045] In this embodiment, after mapping the point cloud data to the image coordinate system where the initial foreign object area image data is located, the points in the point cloud data that are far away from the road area are removed according to a preset maximum effective distance threshold, so as to achieve further processing of the point cloud data.

[0046] S502. According to the projection positions of the point cloud data on the initial foreign object area image, perform matching processing on the point cloud data and the initial foreign object area image data, and eliminate the point cloud data points in the point cloud data whose projection positions are not in any candidate foreign object area in the initial foreign object area image data; specifically, in this embodiment, for each point cloud data point, according to its projection position in the initial foreign object area image data, check whether this position is located in any candidate foreign object area of the initial foreign object area image data, that is, check whether the pixel value at the position where this point cloud data point is located is 255. If so, it is determined that this point cloud data point is located within the candidate foreign object area, and this point cloud data point is retained; otherwise, this point cloud data point is eliminated to implement the screening process of the point cloud data.

[0047] S503. Respectively obtain the number of point cloud data points located in each candidate foreign object area in the initial foreign object area image data, and eliminate the candidate foreign object areas in the initial foreign object area image data where the number of point cloud data points is less than the specified number to obtain the final foreign object area image data. Specifically, in this embodiment, sequentially determine whether there are the specified number of point cloud data points in each candidate foreign object area. If not, it is determined that there is no point cloud matching in the current candidate foreign object area, and the current candidate foreign object area is eliminated from the initial foreign object area image data, that is, the pixel values of each pixel point in the current candidate foreign object area are modified to 0; otherwise, no action is taken until the final foreign object area image data is obtained. The final foreign object area image data includes valid foreign object target areas.

[0048] In this embodiment, the effective fusion of the point cloud data and the image data is achieved through steps S501 to S503. Through the spatial information provided by the millimeter-wave radar or lidar, the false detection can be effectively reduced, and the accuracy of foreign object target detection can be improved.

[0049] S6. Perform foreign object recognition processing according to the final foreign object area image data to obtain foreign object information. Specifically, the foreign object information includes foreign object position information, foreign object size information, and foreign object type information. In this embodiment, the foreign object types include fixed obstacles and moving obstacles, etc.

[0050] In step S6, a rule-based classification method is used to perform the recognition processing of the foreign object type information. Specifically, in this embodiment, foreign object classification rules are formulated in advance according to information such as size, shape, and texture features, and the foreign object type classification of the final foreign object area image data is implemented based on this classification rule. This classification method is simple and fast.

[0051] In addition, in this embodiment, after obtaining the foreign object information, foreign object early warning can be performed based on preset rules, which is not limited here.

[0052] This embodiment has strong environmental adaptability and can improve the detection accuracy and stability in complex environments. Specifically, during the implementation of this embodiment, road video frames collected by a camera, point cloud data collected by a millimeter-wave radar, and environmental data collected by an environmental sensor are obtained in real time. The environmental data is used to perform dynamic environmental compensation processing on the road video frames to obtain road video frames after environmental compensation. Then, road region segmentation processing is performed on the road video frames after environmental compensation to obtain road region image data, and foreign object region segmentation processing is performed on the road region image data to obtain initial foreign object region image data. Next, the initial foreign object region image data is fused with the point cloud data, and candidate foreign object regions without point cloud matching in the initial foreign object region image data are removed to obtain final foreign object region image data. Finally, foreign object recognition processing is performed based on the final foreign object region image data to obtain foreign object information. During this process, dynamic environmental compensation can improve the image quality, not only improving the detection accuracy but also ensuring the stability and robustness of this embodiment in complex scenarios, and it is applicable to various changing scenarios. At the same time, the method of fusing image data and point cloud data solves the limitations of using a single sensor for road foreign object detection, improving the detection accuracy and environmental adaptability. In addition, the computational complexity of foreign object detection in this embodiment is relatively low, and the computational efficiency is high, making it applicable to real-time road foreign object detection scenarios and better ensuring the safety and intelligent level of vehicle driving.

[0053] Embodiment 2: This embodiment discloses a road foreign object detection system for implementing the road foreign object detection method in Embodiment 1; as Figure 2 shown, the road foreign object detection system includes: A data acquisition module, configured to acquire road video frames collected by a camera, point cloud data collected by a millimeter-wave radar, and environmental data collected by an environmental sensor; A dynamic environmental compensation module, communicatively connected to the data acquisition module, and configured to perform dynamic environmental compensation processing on the road video frames using the environmental data to obtain road video frames after environmental compensation; A road region segmentation module, communicatively connected to the dynamic environmental compensation module, and configured to perform road region segmentation processing on the road video frames after environmental compensation to obtain road region image data; A foreign object region segmentation module, communicatively connected to the road region segmentation module, and configured to perform foreign object region segmentation processing on the road region image data to obtain initial foreign object region image data; A data fusion module, communicatively connected to the foreign object region segmentation module, and configured to fuse the initial foreign object region image data with the point cloud data, and remove candidate foreign object regions without point cloud matching in the initial foreign object region image data to obtain final foreign object region image data; A foreign object recognition module, communicatively connected to the data fusion module, is configured to perform foreign object recognition processing based on the final foreign object area image data to obtain foreign object information.

[0054] It should be noted that for the working process, working details and technical effects of the road foreign object detection system provided in Embodiment 2, reference can be made to Embodiment 1, which will not be elaborated here.

[0055] Embodiment 3: Based on Embodiment 1 or 2, this embodiment discloses an electronic device, which may be a smart phone, a tablet computer, a notebook computer, a desktop computer, etc. The electronic device may be referred to as a user terminal, a portable terminal, a desktop terminal, etc. As Figure 3 shown, the electronic device includes: A memory, configured to store computer program instructions; and, A processor, configured to execute the computer program instructions to complete the operations of a road foreign object detection method as described in any one of Embodiment 1.

[0056] Specifically, the processor 301 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 301 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), PLA (Programmable Logic Array). The processor 301 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the wake state, also known as the CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 301 may be integrated with a GPU (Graphics Processing Unit), and the GPU is responsible for rendering and drawing the content to be displayed on the display screen.

[0057] The memory 302 may include one or more computer-readable storage media, and the computer-readable storage media may be non-transitory. The memory 302 may further include high-speed random access memory and non-volatile memory, such as one or more disk storage devices, flash storage devices. In some embodiments, the non-transitory computer-readable storage media in the memory 302 is used to store at least one instruction, and the at least one instruction is used to be executed by the processor 301 to implement the road foreign object detection method provided in Embodiment 1 of the present application.

[0058] In some embodiments, the terminal may further optionally include: a communication interface 303 and at least one peripheral device. The processor 301, the memory 302, and the communication interface 303 may be connected through a bus or signal lines. Each peripheral device may be connected to the communication interface 303 through a bus, signal lines, or a circuit board. Specifically, the peripheral device includes at least one of a radio frequency circuit 304, a display screen 305, and a power supply 306.

[0059] The communication interface 303 can be used to connect at least one peripheral device related to I / O (Input / Output) to the processor 301 and the memory 302. In some embodiments, the processor 301, the memory 302, and the communication interface 303 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 301, the memory 302, and the communication interface 303 can be implemented on a separate chip or circuit board, and this embodiment does not limit this.

[0060] The radio frequency circuit 304 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 304 communicates with a communication network and other communication devices through electromagnetic signals.

[0061] The display screen 305 is used to display a UI (User Interface). The UI may include any combination of graphics, text, icons, and videos.

[0062] The power supply 306 is used to supply power to each component in the electronic device.

[0063] Embodiment 4: Based on any one of Embodiments 1 to 3, this embodiment discloses a computer program product, including a computer program or instructions, and the computer program or the instructions, when executed by a computer, implement a road foreign object detection method as described in any one of Embodiments 1. Among them, the computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices.

[0064] Obviously, those skilled in the art should understand that the above-mentioned modules or steps of the present invention can be implemented by a general-purpose computing device. They can be concentrated on a single computing device or distributed on a network composed of multiple computing devices. Optionally, they can be implemented by program codes executable by the computing device, so that they can be stored in a storage device and executed by the computing device, or they can be separately fabricated into individual integrated circuit modules, or multiple modules or steps among them can be fabricated into a single integrated circuit module to implement. In this way, the present invention is not limited to any specific combination of hardware and software.

[0065] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements on some of the technical features. And these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for detecting road foreign objects, characterized in that, Including: Obtaining road video frames collected by a camera, point cloud data collected by a millimeter-wave radar, and environmental data collected by an environmental sensor; Performing dynamic environmental compensation processing on the road video frames using the environmental data to obtain environmentally compensated road video frames; Performing road region segmentation processing on the environmentally compensated road video frames to obtain road region image data; Performing foreign object region segmentation processing on the road region image data to obtain initial foreign object region image data; Performing fusion processing on the initial foreign object region image data and the point cloud data, and removing candidate foreign object regions in the initial foreign object region image data that have no point cloud matching to obtain final foreign object region image data; Performing foreign object recognition processing based on the final foreign object region image data to obtain foreign object information.

2. The road foreign object detection method according to claim 1, characterized in that, The environmental data collected by the environmental sensor includes environmental light intensity and environmental transmittance; correspondingly, performing dynamic environmental compensation processing on the road video frames using the environmental data to obtain environmentally compensated road video frames includes: Perform illumination compensation processing on the road video frame using the ambient light intensity to obtain an illumination-compensated road video frame; wherein, the gray value of any pixel point ( x , y ) in the illumination-compensated road video frame is: ； Wherein, , is the reference luminance value of any preset pixel point ( x , y ); is the ambient light intensity of any pixel point ( x , y ); is the gray value of any pixel point ( x , y ) in the road video frame; is the preset luminance compensation constant; Performing defogging processing on the road video frame after light compensation by using the environmental transmittance and the environmental light intensity to obtain a road video frame after environmental compensation; wherein, the gray value of any pixel point ( x , y ) in the road video frame after environmental compensation is: ； In the formula, is the environmental transmittance of any pixel point ( x , y ).

3. The method for detecting road foreign objects according to claim 1, wherein After obtaining the environmentally compensated road video frames, the method further includes: Performing grayscale processing, denoising processing, histogram equalization processing, dynamic threshold segmentation processing, and edge detection processing on the environmentally compensated road video frames in sequence to obtain preprocessed road video frames for performing road region segmentation processing on the preprocessed road video frames.

4. The method for detecting road foreign objects according to claim 3, wherein Performing road region segmentation processing on the preprocessed road video frames to obtain road region image data includes: Performing Hough transform processing on the preprocessed road video frames to detect a set of Hough transform lines in the environmentally compensated road video frames; Performing vanishing point detection processing on the preprocessed road video frames according to the set of Hough transform lines to obtain the vanishing point in the preprocessed road video frames; Based on the vanishing point and the set of Hough transform lines, extracting two road boundary lines in the preprocessed road video frames, and regarding the region between the two road boundary lines as the road region; Segmenting the region other than the road region in the preprocessed road video frames to obtain road region image data containing only the road region.

5. The method for detecting road foreign objects according to claim 1, wherein Performing foreign object region segmentation processing on the road region image data to obtain initial foreign object region image data includes: Performing opening operation processing on the road region image data to obtain image data after opening operation processing; Performing closing operation processing on the image data after opening operation processing to obtain image data after opening and closing operation processing; Performing connected component analysis processing on the image data after opening and closing operation processing to extract candidate foreign object region image data in the image data after opening and closing operation processing; Performing filtering processing on the candidate foreign object region image data to obtain initial foreign object region image data.

6. The method for detecting road foreign objects according to claim 1, wherein, Performing fusion processing on the initial foreign object region image data and the point cloud data, and removing candidate foreign object regions in the initial foreign object region image data that have no point cloud matching to obtain final foreign object region image data includes: Mapping the point cloud data to the image coordinate system where the initial foreign object region image data is located; Based on the projection positions of the point cloud data on the initial foreign object area image, match the point cloud data with the initial foreign object area image data, and remove the point cloud data points in the point cloud data whose projection positions are not within any candidate foreign object area in the initial foreign object area image data; Respectively obtain the number of point cloud data points located in each candidate foreign object area in the initial foreign object area image data, and remove the candidate foreign object areas in the initial foreign object area image data where the number of point cloud data points is less than the specified number to obtain the final foreign object area image data.

7. A road foreign object detection system, characterized in that, Comprising: A data acquisition module, configured to acquire road video frames collected by a camera, point cloud data collected by a millimeter-wave radar, and environmental data collected by an environmental sensor; A dynamic environment compensation module, communicatively connected to the data acquisition module, configured to perform dynamic environment compensation processing on the road video frames using the environmental data to obtain road video frames after environment compensation; A road area segmentation module, communicatively connected to the dynamic environment compensation module, configured to perform road area segmentation processing on the road video frames after environment compensation to obtain road area image data; A foreign object area segmentation module, communicatively connected to the road area segmentation module, configured to perform foreign object area segmentation processing on the road area image data to obtain initial foreign object area image data; A data fusion module, communicatively connected to the foreign object area segmentation module, configured to perform fusion processing on the initial foreign object area image data and the point cloud data, and remove the candidate foreign object areas in the initial foreign object area image data that have no point cloud matching to obtain the final foreign object area image data; A foreign object recognition module, communicatively connected to the data fusion module, configured to perform foreign object recognition processing based on the final foreign object area image data to obtain foreign object information.

8. An electronic device, characterized in that, Comprising: A memory, configured to store computer program instructions; And, A processor, configured to execute the computer program instructions to complete the operations of a road foreign object detection method as described in any one of claims 1 to 6.

9. A computer program product, comprising a computer program or instructions, characterized in that, The computer program or the instructions, when executed by a computer, implement a road foreign object detection method as described in any one of claims 1 to 6.

Images