Multi-sensor fusion detection method, device, equipment, medium and program product

Abstract

The present application relates to a multi-sensor fusion detection method, device, equipment, medium and program product. The method includes: obtaining laser radar data and binocular camera data, and determining the obstacle type and candidate contour information of the target obstacle according to the binocular camera data; marking the candidate contour information with confidence according to the laser radar data and binocular camera data to obtain the contour confidence of the target obstacle; when the contour confidence is lower than a preset threshold, obtaining line laser data; supplementing the candidate contour information with contour information according to the line laser data to obtain the target contour information; performing fusion detection according to the obstacle type, target contour information and laser radar data to obtain the fusion detection result. The use of this method can improve the accuracy of fusion detection and obtain more accurate fusion detection results.

Description

Technical Field

[0001] The present application relates to the technical field of sweeping robots, and in particular to a multi-sensor fusion detection method, device, equipment, medium and program product. Background Art

[0002] With the development of robot vacuum cleaner technology, people have a growing demand for the cleaning accuracy of robot vacuum cleaners, which is not limited to a single application scenario and a single obstacle recognition requirement. However, the sensors used in the obstacle detection and recognition process in the prior art are relatively simple, which results in poor obstacle avoidance of the robot vacuum cleaner in complex environments. For example, the robot vacuum cleaner is often trapped or collided. If the robot vacuum cleaner's ability to detect obstacles cannot be further improved, the intelligence of the robot vacuum cleaner cannot be further improved. Summary of the invention

[0003] Based on this, it is necessary to provide a multi-sensor fusion detection method, device, computer equipment, computer-readable storage medium and computer program product that can use multiple sensor data and effectively fuse the sensor data to achieve more accurate obstacle detection effect in response to the above technical problems.

[0004] In a first aspect, the present application provides a multi-sensor fusion detection method, comprising:

[0005] Acquire laser radar data and binocular camera data, and determine the obstacle type and candidate contour information of the target obstacle according to the binocular camera data;

[0006] Performing confidence marking on the candidate contour information according to the laser radar data and the binocular camera data to obtain the contour confidence of the target obstacle;

[0007] When the contour confidence is lower than a preset threshold, acquiring line laser data;

[0008] Supplementing the candidate contour information with contour information according to the line laser data to obtain target contour information;

[0009] Fusion detection is performed according to the obstacle type, the target contour information and the lidar data to obtain a fusion detection result.

[0010] In one embodiment, the step of labeling the candidate contour information with confidence according to the laser radar data and the binocular camera data to obtain the contour confidence of the target obstacle includes:

[0011] Performing confidence marking on the candidate contour information according to the laser radar data and a preset radar weight to obtain radar confidence;

[0012] Performing confidence marking on the candidate contour information according to the binocular camera data and the preset camera weight to obtain camera confidence;

[0013] The contour confidence is determined according to the radar confidence and the camera confidence.

[0014] In one embodiment, the method further comprises:

[0015] Acquire laser stability data of the laser radar data and camera stability data of the binocular camera data;

[0016] Performing environmental assessment according to the laser stability data and the camera stability data to obtain environmental status data;

[0017] Evaluate the preset radar weight and the preset camera weight according to the environmental status data to obtain an evaluation result;

[0018] The preset radar weight and the preset camera weight are updated according to the evaluation result.

[0019] In one embodiment, determining the obstacle type and candidate contour information of the target obstacle according to the binocular camera data includes:

[0020] Perform obstacle type identification on the binocular camera data according to a pre-built obstacle identification model to obtain the obstacle type;

[0021] Three-dimensional point cloud data is determined according to the binocular camera data, and the candidate contour information is determined according to the three-dimensional point cloud data.

[0022] In one embodiment, the step of supplementing the candidate contour information with contour information according to the line laser data to obtain target contour information includes:

[0023] Constructing a local map according to the line laser data and the first detection range of the line laser data to obtain a real-time local positioning map;

[0024] The candidate contour information is supplemented with contour information according to the real-time local positioning map to obtain the target contour information.

[0025] In one embodiment, the performing fusion detection according to the obstacle type, the target profile information and the lidar data to obtain a fusion detection result includes:

[0026] Constructing a real-time initial positioning map according to the radar data and a second detection range corresponding to the radar data;

[0027] The real-time initial positioning map is annotated by the obstacle type and the target contour information to obtain the fusion detection result.

[0028] In a second aspect, the present application also provides a multi-sensor fusion detection device, comprising:

[0029] A first acquisition module is used to acquire laser radar data and binocular camera data, and determine the obstacle type and candidate contour information of the target obstacle according to the binocular camera data;

[0030] A confidence labeling module, used to label the candidate contour information with confidence according to the laser radar data and the binocular camera data, so as to obtain the contour confidence of the target obstacle;

[0031] A second acquisition module, used for acquiring line laser data when the contour confidence is lower than a preset threshold;

[0032] A contour information supplementing module, used to supplement the candidate contour information according to the line laser data to obtain target contour information;

[0033] The fusion detection module is used to perform fusion detection according to the obstacle type, the target contour information and the lidar data to obtain a fusion detection result.

[0034] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0035] Acquire laser radar data and binocular camera data, and determine the obstacle type and candidate contour information of the target obstacle according to the binocular camera data;

[0036] Performing confidence marking on the candidate contour information according to the laser radar data and the binocular camera data to obtain the contour confidence of the target obstacle;

[0037] When the contour confidence is lower than a preset threshold, acquiring line laser data;

[0038] Supplementing the candidate contour information with contour information according to the line laser data to obtain target contour information;

[0039] Fusion detection is performed according to the obstacle type, the target contour information and the lidar data to obtain a fusion detection result.

[0040] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0041] Acquire laser radar data and binocular camera data, and determine the obstacle type and candidate contour information of the target obstacle according to the binocular camera data;

[0042] Performing confidence marking on the candidate contour information according to the laser radar data and the binocular camera data to obtain the contour confidence of the target obstacle;

[0043] When the contour confidence is lower than a preset threshold, acquiring line laser data;

[0044] Supplementing the candidate contour information with contour information according to the line laser data to obtain target contour information;

[0045] Fusion detection is performed according to the obstacle type, the target contour information and the lidar data to obtain a fusion detection result.

[0046] In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0047] Acquire laser radar data and binocular camera data, and determine the obstacle type and candidate contour information of the target obstacle according to the binocular camera data;

[0048] Performing confidence marking on the candidate contour information according to the laser radar data and the binocular camera data to obtain the contour confidence of the target obstacle;

[0049] When the contour confidence is lower than a preset threshold, acquiring line laser data;

[0050] Supplementing the candidate contour information with contour information according to the line laser data to obtain target contour information;

[0051] Fusion detection is performed according to the obstacle type, the target contour information and the lidar data to obtain a fusion detection result.

[0052] The above-mentioned multi-sensor fusion detection method, device, computer equipment, computer-readable storage medium and computer program product obtain laser radar data and binocular camera data, and determine the obstacle type and candidate contour information of the target obstacle according to the binocular camera data; mark the candidate contour information with confidence according to the laser radar data and binocular camera data to obtain the contour confidence of the target obstacle; when the contour confidence is lower than a preset threshold, obtain line laser data; supplement the candidate contour information with contour information according to the line laser data to obtain the target contour information; perform fusion detection according to the obstacle type, target contour information and laser radar data to obtain the fusion detection result. Therefore, by acquiring lidar data and binocular camera data, determining the obstacle type and candidate contour information of the target obstacle based on the binocular camera data, and then annotating the candidate contour information with confidence based on the lidar data and binocular camera data, the contour confidence of the target obstacle can be accurately confirmed, wherein the contour confidence can be used to assist subsequent fusion detection steps, thereby improving the accuracy of fusion detection. When the contour confidence is lower than a preset threshold, line laser data is acquired, and the contour information of the candidate contour is supplemented based on the line laser data to obtain the target contour information. Finally, fusion detection is performed based on the obstacle type, target contour information and lidar data, and a more accurate fusion detection result can be obtained. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0054] Figure 1 A diagram showing an application environment of a multi-sensor fusion detection method in one embodiment;

[0055] Figure 2 A schematic diagram of a multi-sensor fusion detection method according to an embodiment;

[0056] Figure 3 A schematic diagram of a multi-sensor fusion detection method according to an embodiment;

[0057] Figure 4 It is a structural block diagram of a multi-sensor fusion detection device in one embodiment;

[0058] Figure 5 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0060] The multi-sensor fusion detection method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the sweeper 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. The server 104 obtains the laser radar data and the binocular camera data, and determines the obstacle type and candidate contour information of the target obstacle according to the binocular camera data; the candidate contour information is marked with confidence according to the laser radar data and the binocular camera data to obtain the contour confidence of the target obstacle; when the contour confidence is lower than the preset threshold, the line laser data is obtained; the candidate contour information is supplemented with contour information according to the line laser data to obtain the target contour information; according to the obstacle type, the target contour information and the laser radar data, the fusion detection is performed to obtain the fusion detection result. Among them, the sweeper 102 can be, but is not limited to, various sweeper devices equipped with independent computers, tablets, Internet of Things devices and portable wearable devices. The server 104 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0061] In an exemplary embodiment, Figure 2 As shown, a multi-sensor fusion detection method is provided, which is applied to Figure 1 The server 104 in the example is used as an example to illustrate the method, which includes the following steps S202 to S210. Among them:

[0062] Step S202, acquiring laser radar data and binocular camera data, and determining the obstacle type and candidate contour information of the target obstacle according to the binocular camera data.

[0063] The laser radar data is obtained by a laser radar sensor on the sweeper, and the binocular camera data is obtained by a binocular camera on the sweeper. The laser radar sensor is located inside the sweeper, is hidden, and cannot be seen from the outside, and the binocular camera is located in front of the sweeper, but is not limited to this.

[0064] In some embodiments, the laser radar sensor can be used to collect data of 210 degrees in front of the sweeping machine in real time as laser radar data. The laser radar data of the rear part cannot be used due to obstruction by the sweeping machine. It should be noted that the front refers to the front in the field of view of the sweeping machine, and the field of view direction may be consistent with the movement direction of the sweeping machine or inconsistent with it, and is not limited to this.

[0065] In some embodiments, the obstacle type and candidate contour information of the target obstacle are determined based on the binocular camera data, including: identifying the obstacle type of the binocular camera data based on a pre-built obstacle recognition model to obtain the obstacle type; determining three-dimensional point cloud data based on the binocular camera data, and determining candidate contour information based on the three-dimensional point cloud data.

[0066] The pre-built obstacle recognition model refers to a YOLO model that is pre-trained based on different types of obstacle training data, and is used to implement a type detection function for obstacle types.

[0067] In some embodiments, the camera data acquired in real time by the left camera of the binocular camera is used through a pre-trained YOLO model to recognize the type of obstacle, thereby obtaining the obstacle type, thereby achieving an accurate and efficient obstacle type recognition process.

[0068] In some embodiments, three-dimensional point cloud data is output based on camera data acquired in real time by a binocular camera, and candidate contour information is determined based on the three-dimensional point cloud data. It should be noted that by using the output three-dimensional point cloud data, data at different heights of obstacles can be detected, thereby improving the accuracy of the collected candidate contour information.

[0069] In some embodiments, more detailed obstacle information can be annotated on the pre-constructed initial positioning map through obstacle type and candidate contour information, thereby obtaining a real-time global positioning map containing accurate obstacle information.

[0070] Step S204: Confidence annotation is performed on the candidate contour information according to the laser radar data and the binocular camera data to obtain the contour confidence of the target obstacle.

[0071] Among them, the confidence annotation is to mark the confidence of each contour part of the target obstacle in the candidate contour information.

[0072] In some embodiments, based on the laser radar data and the binocular camera data, each contour part in the candidate contour information of the target obstacle is labeled with a confidence level to obtain the contour confidence level of each contour part of the target obstacle.

[0073] In some embodiments, the candidate contour information is labeled with confidence based on the lidar data and the binocular camera data to obtain the contour confidence of the target obstacle, including: labeling the candidate contour information with confidence based on the lidar data and preset radar weights to obtain radar confidence; labeling the candidate contour information with confidence based on the binocular camera data and preset camera weights to obtain camera confidence; determining the contour confidence based on the radar confidence and the camera confidence.

[0074] Among them, the preset radar weight is the sensor weight corresponding to the lidar data, and the preset radar weight is the sensor weight dynamically adjusted according to the stability of the current sensor data. The preset camera weight is the sensor weight corresponding to the binocular camera data, and the preset camera weight is the sensor weight dynamically adjusted according to the stability of the current sensor data.

[0075] It should be noted that in some special cases, when the LiDAR illuminates an obstacle with a small reflective surface, the acquired LiDAR data is not very stable. At this time, the collected LiDAR data can only be used as a reference, so the preset radar weight can be set to 0.

[0076] In some embodiments, a confidence score is performed on the candidate contour information based on the lidar data to obtain a radar contour score, which is then annotated based on the radar contour score and a preset radar weight to obtain a radar confidence, a confidence score is performed on the candidate contour information based on the binocular camera data to obtain a camera contour score, which is then annotated based on the camera contour score and a preset camera weight to obtain a camera confidence.

[0077] In some embodiments, the radar confidence and the camera confidence may be calculated based on different data processing methods to determine the contour confidence. The different data processing methods may be mean, square difference, etc., but are not limited thereto.

[0078] Step S206: when the contour confidence is lower than a preset threshold, acquiring line laser data.

[0079] Among them, the preset threshold is pre-set through the performance of data collected by different sensors. The line laser data is obtained through three line laser sensors, two of which are located on the left and right sides of the front of the sweeper, which belong to the horizontal structure, and the other one is located on the right side of the sweeper, which belongs to the vertical structure.

[0080] In some embodiments, when the contour confidence is lower than a preset threshold, it indicates that the credibility of the candidate contour data currently acquired by the sensor is low, and line laser data needs to be acquired to supplement more detailed contour information of the target obstacle.

[0081] Among them, the situation where the credibility of the candidate contour data obtained by the current sensor is low may be when the lighting conditions are bad or when the laser radar data is irradiated to a small reflective surface.

[0082] In some embodiments, a line laser sensor can be used to obtain real-time three-dimensional point cloud data at a close distance in front of the sweeper, ensuring the detection accuracy of obstacles at close distances. Therefore, a high-precision local map can be provided for the sweeper at close range based on the line laser data obtained in real time at close range, so as to achieve more precise control of the sweeper based on the constructed high-precision local map. The line laser on the right side of the three line laser sensors is vertical, which can more conveniently detect the obstacle information on the right side quickly and accurately, so as to achieve close walking along obstacles.

[0083] Step S208, supplementing the candidate contour information according to the line laser data to obtain the target contour information.

[0084] Among them, contour information supplementation is the process of replacing or supplementing the contour information with lower credibility in the candidate contour information by high-precision line laser data.

[0085] In some embodiments, for parts with relatively low confidence in the candidate contour information of the target obstacle, the speed of the sweeper will be reduced when cleaning near the target obstacle, and the candidate contour information will be supplemented with contour information through line laser data to obtain target contour information, so as to construct high-precision target contour information of the target obstacle.

[0086] Step S210, performing fusion detection according to obstacle type, target contour information and lidar data to obtain a fusion detection result.

[0087] Among them, a specific map is required in the fusion detection process, and a specific fusion detection strategy is combined to realize the fusion detection process.

[0088] In some embodiments, contour information is supplemented for candidate contour information based on line laser data to obtain target contour information, including: constructing a local map based on the line laser data and a first detection range of the line laser data to obtain a real-time local positioning map; and supplementing contour information for candidate contour information based on the real-time local positioning map to obtain target contour information.

[0089] Among them, the local map construction process can adopt the SLAM (Simultaneous Localization And Mapping) map construction technology.

[0090] In some embodiments, a local map is constructed based on the line laser data and the second detection range corresponding to the line laser data, which can provide a high-precision real-time local positioning map for the sweeping machine at close range.

[0091] In some embodiments, the real-time local positioning map contains precise contour information of each obstacle within the second detection range. Therefore, by supplementing the candidate contour information with contour information according to the real-time local positioning map, accurate target contour information can be obtained, so as to achieve a more refined sweeper control effect. In addition, the line laser data obtained by the line laser sensor on the right side of the line laser data is obtained vertically, so that the obstacle information on the right side can be detected more conveniently, so as to achieve close walking along the obstacle.

[0092] The first detection range is the detection range of the line laser sensor, because the second detection range of the laser radar is usually larger than the first detection range of the line laser sensor.

[0093] In some embodiments, fusion detection is performed based on obstacle type, target contour information and lidar data to obtain a fusion detection result, including: building a real-time initial positioning map based on radar data and a second detection range corresponding to the radar data; annotating the real-time initial positioning map with obstacle type and target contour information to obtain a fusion detection result.

[0094] Among them, the second detection range is the detection distance range of the laser radar sensor, which is usually 0.1~10 meters, but not limited to this.

[0095] In some embodiments, by constructing a real-time initial positioning map based on radar data and a second detection range corresponding to the radar data, it is possible to establish a real-time initial positioning map capable of global real-time positioning, and annotate the real-time initial positioning map with obstacle type and target contour information to obtain a fused detection result of the real-time initial positioning map containing obstacle type and target contour information.

[0096] In the above-mentioned multi-sensor fusion detection method, by acquiring lidar data and binocular camera data, and determining the obstacle type and candidate contour information of the target obstacle based on the binocular camera data; annotating the candidate contour information with confidence based on the lidar data and binocular camera data to obtain the contour confidence of the target obstacle; when the contour confidence is lower than a preset threshold, acquiring line laser data; supplementing the candidate contour information with contour information based on the line laser data to obtain the target contour information; performing fusion detection based on the obstacle type, target contour information and lidar data to obtain a fusion detection result. Therefore, by acquiring lidar data and binocular camera data, determining the obstacle type and candidate contour information of the target obstacle based on the binocular camera data, and then annotating the candidate contour information with confidence based on the lidar data and binocular camera data, the contour confidence of the target obstacle can be accurately confirmed, wherein the contour confidence can be used to assist subsequent fusion detection steps, thereby improving the accuracy of fusion detection. When the contour confidence is lower than a preset threshold, line laser data is acquired, and the contour information of the candidate contour is supplemented based on the line laser data to obtain the target contour information. Finally, fusion detection is performed based on the obstacle type, target contour information and lidar data, and a more accurate fusion detection result can be obtained.

[0097] In an exemplary embodiment, Figure 3 As shown, the multi-sensor fusion detection method further includes a weight updating step, which includes steps S302 to S308. Among them:

[0098] Step S302, obtaining laser stability data of the laser radar data and camera stability data of the binocular camera data.

[0099] Among them, the laser stability data is the environmental data of the laser radar sensor and the camera stability data is the environmental data of the binocular camera sensor.

[0100] In some embodiments, laser stability data of lidar data and camera stability data of binocular camera data may be acquired through some other types of sensors to improve the accuracy of the acquired laser stability data and camera stability data.

[0101] Step S304, performing environmental assessment based on the laser stability data and the camera stability data to obtain environmental status data.

[0102] Among them, environmental assessment is a process of setting corresponding assessment data for different sensors and performing environmental assessment on stability data according to the assessment data corresponding to each sensor.

[0103] In some embodiments, an environmental assessment is performed on the laser stability data and the camera stability data according to preset environmental assessment data to obtain environmental status data corresponding to each sensor, so as to accurately determine the data performance of different sensors in the current environment.

[0104] An example should be given to illustrate that when the laser radar is irradiating a small reflective surface, the acquired laser radar data is not very stable. At this time, the laser radar data will only be used as a reference, and the current point will be marked as an unknown point to be confirmed. When the sweeper runs to this point, it will re-detect through a binocular camera or line laser. When the lighting conditions are poor, the credibility of the camera data will also be reduced, and sensor data with more stable data will be used instead.

[0105] Step S306, evaluating the preset radar weight and the preset camera weight according to the environmental status data to obtain an evaluation result.

[0106] Among them, the preset radar weight is the sensor weight corresponding to the lidar sensor, and the preset camera weight is the sensor weight corresponding to the binocular camera sensor.

[0107] In some embodiments, the preset radar weight and the preset camera weight are evaluated through the environmental status data, that is, it can be determined whether the current environmental status data matches the preset radar weight and the preset camera weight, thereby obtaining an evaluation result.

[0108] Step S308: updating the preset radar weight and the preset camera weight according to the evaluation result.

[0109] In some embodiments, if the evaluation result is unqualified, the preset radar weights and preset camera weights need to be updated according to the environmental status data recorded in the evaluation result, so as to re-evaluate the data confidence of the sensor to achieve optimal data collection.

[0110] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0111] Based on the same inventive concept, the embodiment of the present application also provides a multi-sensor fusion detection device for implementing the multi-sensor fusion detection method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more multi-sensor fusion detection device embodiments provided below can refer to the limitations of the multi-sensor fusion detection method above, and will not be repeated here.

[0112] In an exemplary embodiment, Figure 4 As shown, a multi-sensor fusion detection device 400 is provided, comprising: a first acquisition module 401, a confidence annotation module 402, a second acquisition module 403, a contour information supplement module 404 and a fusion detection module 405, wherein:

[0113] The first acquisition module 401 is used to acquire laser radar data and binocular camera data, and determine the obstacle type and candidate contour information of the target obstacle according to the binocular camera data;

[0114] The confidence marking module 402 is used to mark the candidate contour information with confidence according to the laser radar data and the binocular camera data to obtain the contour confidence of the target obstacle;

[0115] The second acquisition module 403 is used to acquire line laser data when the profile confidence is lower than a preset threshold;

[0116] The contour information supplementing module 404 is used to supplement the candidate contour information according to the line laser data to obtain the target contour information;

[0117] The fusion detection module 405 is used to perform fusion detection according to the obstacle type, target contour information and lidar data to obtain a fusion detection result.

[0118] In some embodiments, the confidence labeling module 402 is also used to label the candidate contour information with confidence based on the lidar data and the preset radar weight to obtain the radar confidence; to label the candidate contour information with confidence based on the binocular camera data and the preset camera weight to obtain the camera confidence; and to determine the contour confidence based on the radar confidence and the camera confidence.

[0119] In some embodiments, the device also includes: a weight update module, which is used to obtain laser stability data of lidar data and camera stability data of binocular camera data; perform environmental assessment based on the laser stability data and the camera stability data to obtain environmental status data; evaluate the preset radar weights and preset camera weights based on the environmental status data to obtain evaluation results; and update the preset radar weights and preset camera weights based on the evaluation results.

[0120] In some embodiments, the first acquisition module 401 is further used to identify the obstacle type of the binocular camera data according to a pre-built obstacle recognition model to obtain the obstacle type; determine the three-dimensional point cloud data according to the binocular camera data, and determine the candidate contour information according to the three-dimensional point cloud data.

[0121] In some embodiments, the contour information supplementation module 404 is also used to construct a local map based on the line laser data and the first detection range of the line laser data to obtain a real-time local positioning map; and to supplement the candidate contour information based on the real-time local positioning map to obtain target contour information.

[0122] In some embodiments, the fusion detection module 405 is also used to build a real-time initial positioning map according to the radar data and the second detection range corresponding to the radar data; the real-time initial positioning map is annotated by obstacle type and target contour information to obtain a fusion detection result.

[0123] In the above-mentioned multi-sensor fusion detection device, by acquiring lidar data and binocular camera data, and determining the obstacle type and candidate contour information of the target obstacle based on the binocular camera data; marking the candidate contour information with confidence based on the lidar data and binocular camera data to obtain the contour confidence of the target obstacle; when the contour confidence is lower than a preset threshold, acquiring line laser data; supplementing the candidate contour information with contour information based on the line laser data to obtain the target contour information; performing fusion detection based on the obstacle type, target contour information and lidar data to obtain a fusion detection result. Therefore, by acquiring lidar data and binocular camera data, determining the obstacle type and candidate contour information of the target obstacle based on the binocular camera data, and then annotating the candidate contour information with confidence based on the lidar data and binocular camera data, the contour confidence of the target obstacle can be accurately confirmed, wherein the contour confidence can be used to assist subsequent fusion detection steps, thereby improving the accuracy of fusion detection. When the contour confidence is lower than a preset threshold, line laser data is acquired, and the contour information of the candidate contour is supplemented based on the line laser data to obtain the target contour information. Finally, fusion detection is performed based on the obstacle type, target contour information and lidar data, and a more accurate fusion detection result can be obtained.

[0124] Each module in the above multi-sensor fusion detection device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module above.

[0125] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Figure 5 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store preset thresholds. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external sweeper through a network connection. When the computer program is executed by the processor, a multi-sensor fusion detection method is implemented.

[0126] Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0127] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps of the above-mentioned multi-sensor fusion detection method when executing the computer program.

[0128] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the multi-sensor fusion detection method are implemented.

[0129] In one embodiment, a computer program product is provided, including a computer program, which implements the steps of the above-mentioned multi-sensor fusion detection method when executed by a processor.

[0130] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0131] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0132] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0133] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A multi-sensor fusion detection method, characterized in that: The method comprises: Acquire laser radar data and binocular camera data, and determine the obstacle type and candidate contour information of the target obstacle according to the binocular camera data; The candidate contour information is labeled with confidence according to the laser radar data and the binocular camera data to obtain the contour confidence of the target obstacle; wherein the contour confidence is the labeling data of the confidence of each contour part of the target obstacle in the candidate contour information; When the contour confidence is lower than a preset threshold, acquiring line laser data; Supplementing the candidate contour information with contour information according to the line laser data to obtain target contour information; Perform fusion detection according to the obstacle type, the target contour information and the laser radar data to obtain a fusion detection result; The step of marking the candidate contour information with confidence according to the laser radar data and the binocular camera data to obtain the contour confidence of the target obstacle includes: Performing confidence marking on the candidate contour information according to the laser radar data and a preset radar weight to obtain radar confidence; Performing confidence marking on the candidate contour information according to the binocular camera data and the preset camera weight to obtain camera confidence; Determining the contour confidence according to the radar confidence and the camera confidence; Acquire laser stability data of the laser radar data and camera stability data of the binocular camera data; Performing environmental assessment according to the laser stability data and the camera stability data to obtain environmental status data; Evaluate the preset radar weight and the preset camera weight according to the environmental status data to obtain an evaluation result; The preset radar weight and the preset camera weight are updated according to the evaluation result.

2. The method according to claim 1, characterized in that The step of determining the obstacle type and candidate contour information of the target obstacle according to the binocular camera data includes: Perform obstacle type identification on the binocular camera data according to a pre-built obstacle identification model to obtain the obstacle type; Three-dimensional point cloud data is determined according to the binocular camera data, and the candidate contour information is determined according to the three-dimensional point cloud data.

3. The method according to claim 1, characterized in that The step of supplementing the candidate contour information according to the line laser data to obtain target contour information includes: Constructing a local map according to the line laser data and the first detection range of the line laser data to obtain a real-time local positioning map; The candidate contour information is supplemented with contour information according to the real-time local positioning map to obtain the target contour information.

4. The method according to claim 1, characterized in that The performing fusion detection according to the obstacle type, the target contour information and the laser radar data to obtain a fusion detection result includes: Constructing a real-time initial positioning map according to the radar data and a second detection range corresponding to the radar data; The real-time initial positioning map is annotated by the obstacle type and the target contour information to obtain the fusion detection result.

5. A multi-sensor fusion detection device, characterized in that: The device comprises: A first acquisition module is used to acquire laser radar data and binocular camera data, and determine the obstacle type and candidate contour information of the target obstacle according to the binocular camera data; A confidence labeling module, used to label the candidate contour information with confidence according to the laser radar data and the binocular camera data, so as to obtain the contour confidence of the target obstacle; wherein the contour confidence is labeling data of a confidence of each contour part of the target obstacle in the candidate contour information; A second acquisition module, used for acquiring line laser data when the contour confidence is lower than a preset threshold; A contour information supplementing module, used to supplement the candidate contour information according to the line laser data to obtain target contour information; A fusion detection module, used to perform fusion detection according to the obstacle type, the target contour information and the lidar data to obtain a fusion detection result; The confidence labeling module is further used to label the candidate contour information with confidence according to the laser radar data and the preset radar weight to obtain the radar confidence; label the candidate contour information with confidence according to the binocular camera data and the preset camera weight to obtain the camera confidence; determine the contour confidence according to the radar confidence and the camera confidence; A weight updating module is used to obtain the laser stability data of the laser radar data and the camera stability data of the binocular camera data; perform environmental assessment according to the laser stability data and the camera stability data to obtain environmental status data; evaluate the preset radar weight and the preset camera weight according to the environmental status data to obtain an assessment result; and update the preset radar weight and the preset camera weight according to the assessment result.

6. The device according to claim 5, characterized in that The first acquisition module is also used to identify the obstacle type of the binocular camera data according to a pre-built obstacle recognition model to obtain the obstacle type; determine the three-dimensional point cloud data according to the binocular camera data, and determine the candidate contour information according to the three-dimensional point cloud data.

7. The device according to claim 5, characterized in that The contour information supplementing module is further used to construct a local map based on the line laser data and the first detection range of the line laser data to obtain a real-time local positioning map; and to supplement the candidate contour information based on the real-time local positioning map to obtain the target contour information.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 4 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.

Images