Robot environment sensing method based on multi-sensor fusion

Abstract

The invention discloses a robot environment sensing method based on multi-sensor fusion, and relates to the technical field of environment sensing. Acquiring an image acquired by the camera and a laser point cloud acquired by the laser radar; performing time synchronization on the image and the laser point cloud, and determining a target image which is in time synchronization with the newly collected laser point cloud; performing obstacle detection on the target image; converting a three-dimensional point cloud coordinate of the newly collected laser point cloud into a pixel coordinate system coordinate; segmenting the converted laser points of which the corresponding pixel coordinate system coordinates are located in the pixel coordinate range of the obstacle target to obtain segmented laser point clouds; clustering the segmented laser point clouds to obtain at least one cluster; and taking an area enclosed by the point clouds corresponding to the clusters as an obstacle area in front of the robot. According to the invention, the calculation amount and the calculation complexity in the obstacle detection process can be reduced, the obstacle detection speed is improved, and the dynamic obstacle avoidance requirement of the robot moving at a high speed is met.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of environment perception, and particularly relates to a robot environment perception method based on multi-sensor fusion. BACKGROUND

[0002] In the field of robot autonomous navigation and intelligent interaction, environment perception is the core prerequisite for realizing safe operation and efficient decision-making. Traditional environment perception schemes mostly rely on a single sensor. Among them, laser radar has become the mainstream choice for obstacle detection due to its high-precision three-dimensional space modeling capability. This technology generates point cloud data by emitting laser beams and receiving reflected signals, and then identifies the position, size and motion state of obstacles in front of the robot through point cloud analysis and geometric modeling methods. However, as the application scenarios of robots develop towards high speed and complexity, the limitations of laser radar are increasingly highlighted.

[0003] Laser point cloud data has a natural high-dimensional characteristic, with a single frame data volume reaching tens of thousands to millions of point coordinates. It needs to be preprocessed through iterative closest point (ICP) and voxel grid filtering algorithms, and then combined with deep learning or traditional machine vision methods to realize obstacle classification. This process involves a large amount of matrix operations and feature matching, resulting in a real-time processing delay usually exceeding 100 ms, which is difficult to meet the dynamic obstacle avoidance needs of robots.

[0004] Therefore, how to provide an effective scheme to reduce the computational load of obstacle detection and realize dynamic obstacle avoidance of robots has become a difficult problem to be solved in the prior art. SUMMARY

[0005] The purpose of the present application is to provide a robot environment perception method based on multi-sensor fusion to solve the above problems existing in the prior art.

[0006] In order to achieve the above purpose, the present application adopts the following technical scheme: In a first aspect, the present application provides a robot environment perception method based on multi-sensor fusion, comprising: real-time acquisition of images of a region in front of the robot collected by a camera and laser point clouds of the region in front of the robot collected by a laser radar; time synchronization of the images collected by the camera and the laser point clouds collected by the laser radar to determine a target image time-synchronized with the latest collected laser point clouds; obstacle detection of the target image by a target detection algorithm to obtain an obstacle target in the target image; conversion of three-dimensional point cloud coordinates of the latest collected laser point clouds into pixel coordinate system coordinates; Segment the laser points after the conversion and corresponding to the pixel coordinate system coordinates located in the pixel coordinate range of the obstacle target to obtain segmented laser point cloud; Cluster the segmented laser point cloud by a clustering algorithm to obtain at least one cluster; The area surrounded by the point cloud corresponding to each cluster in the at least one cluster is taken as an obstacle area in front of the robot.

[0007] In one possible design, the image collected by the camera and the laser point cloud collected by the laser radar are time-synchronized to determine a target image time-synchronized with the latest collected laser point cloud, including: Calculate the difference between the time stamp corresponding to each image collected by the camera and the time stamp corresponding to the latest collected laser point cloud; Select the image with the smallest difference between the corresponding time stamp and the time stamp corresponding to the latest collected laser point cloud as the target image time-synchronized with the latest collected laser point cloud.

[0008] In one possible design, the three-dimensional point cloud coordinates of the latest collected laser point cloud are converted into pixel coordinate system coordinates, including: Convert the three-dimensional point cloud coordinates of the latest collected laser point cloud into space coordinates in the camera coordinate system; Convert the space coordinates converted into the camera coordinate system into pixel coordinate system coordinates.

[0009] In one possible design, the three-dimensional point cloud coordinates of the latest collected laser point cloud are converted into space coordinates in the camera coordinate system, including: Convert the three-dimensional point cloud coordinates of the latest collected laser point cloud into space coordinates in the camera coordinate system according to the following formula: Where x camera represents the coordinate of the converted point in the x-axis direction of the camera coordinate system, y camera represents the coordinate of the converted point in the y-axis direction of the camera coordinate system, and z camera represents the coordinate of the converted point in the z-axis direction of the camera coordinate system, R represents a rotation matrix, T represents a translation matrix, x ld represents the coordinate of the point in the x-axis direction of the laser point cloud coordinate system, y ld represents the coordinate of the point in the y-axis direction of the laser point cloud coordinate system, and z ld represents the coordinate of the point in the z-axis direction of the laser point cloud coordinate system; Convert the space coordinates converted into the camera coordinate system into pixel coordinate system coordinates, including: Convert the space coordinates converted into the camera coordinate system into pixel coordinate system coordinates according to the following formula: wherein u represents the coordinate of the converted point in the horizontal axis direction under pixel coordinates, v represents the coordinate of the converted point in the vertical axis direction under pixel coordinates, and K represents the camera intrinsic matrix.

[0010] In one possible design, the segmented laser point cloud is clustered by a clustering algorithm to obtain at least one cluster, including: The segmented laser point cloud is denoised to obtain a denoised laser point cloud. The reflection intensity of each point in the denoised laser point cloud is extracted. Based on the reflection intensity and the point cloud coordinates of each point in the denoised laser point cloud, a feature vector of each point in the denoised laser point cloud is constructed. Based on the feature vector of each point in the denoised laser point cloud, the denoised laser point cloud is clustered by a dynamic threshold adaptive clustering algorithm to obtain at least one cluster.

[0011] In one possible design, the target detection algorithm is a YOLOv5 algorithm.

[0012] In a second aspect, the present application provides a robot environment perception device based on multi-sensor fusion, including: The acquisition unit is configured to acquire images of a region in front of the robot collected by a camera and laser point clouds of the region in front of the robot collected by a laser radar in real time. The time synchronization unit is configured to time synchronize the images collected by the camera and the laser point clouds collected by the laser radar, and determine a target image time synchronized with the latest collected laser point cloud. The obstacle detection unit is configured to detect obstacles in the target image by a target detection algorithm to obtain obstacle targets in the target image. The conversion unit is configured to convert three-dimensional point cloud coordinates of the latest collected laser point cloud into pixel coordinate system coordinates. The segmentation unit is configured to segment laser points after conversion and corresponding pixel coordinate system coordinates located in a pixel coordinate range of the obstacle targets to obtain a segmented laser point cloud. The clustering unit is configured to cluster the segmented laser point cloud by a clustering algorithm to obtain at least one cluster. The determination unit is configured to determine a region surrounded by point clouds corresponding to each cluster in the at least one cluster as an obstacle region in front of the robot.

[0013] In a third aspect, the present application provides a robot comprising a memory, a processor and a transceiver connected in sequence and in communication, wherein the memory is configured to store a computer program, the transceiver is configured to transmit and receive messages, and the processor is configured to read the computer program and execute the robot environment perception method based on multi-sensor fusion according to the first aspect or any possible design of the first aspect.

[0014] In a fourth aspect, the present application provides a computer readable storage medium having instructions stored thereon, which, when executed on a computer, perform the robot environment perception method based on multi-sensor fusion according to the first aspect or any possible design of the first aspect.

[0015] In a fifth aspect, the present application provides a computer program product comprising instructions, which, when executed on a computer, cause the computer to perform the robot environment perception method based on multi-sensor fusion according to the first aspect or any possible design of the first aspect.

[0016] Advantages: The present application obtains the image of the area in front of the robot collected by the camera and the laser point cloud of the area in front of the robot collected by the laser radar in real time; synchronizes the image collected by the camera and the laser point cloud collected by the laser radar in time, and determines the target image synchronized in time with the latest collected laser point cloud; detects the obstacles in the target image through a target detection algorithm, and obtains the obstacle target in the target image; converts the three-dimensional point cloud coordinates of the latest collected laser point cloud into pixel coordinate system coordinates; segments the laser points corresponding to the pixel coordinate system coordinates in the range of the pixel coordinates of the obstacle target after conversion, and obtains the segmented laser point cloud; clusters the segmented laser point cloud through a clustering algorithm, and obtains at least one cluster; and the area surrounded by the point cloud corresponding to each cluster in the at least one cluster is taken as the obstacle area in front of the robot. In this way, the image is detected for obstacles, and the point cloud in the area of the obstacle is segmented from the three-dimensional point cloud according to the spatial relationship between the pixel coordinates and the three-dimensional point cloud coordinates, so as to eliminate a large number of point clouds irrelevant to the obstacle, and then the segmented point cloud is clustered through clustering, so as to identify the obstacle area according to the clustering result. Therefore, it is not necessary to analyze all the point cloud data, the operation amount and operation complexity in the obstacle detection process are greatly reduced, the detection speed of the obstacle is improved, the dynamic obstacle avoidance demand of the high-speed moving robot is met, the robot environment perception based on multi-sensor fusion can be widely used, and the practical application and popularization are facilitated. BRIEF DESCRIPTION OF DRAWINGS

[0017] Figure 1 The flowchart of the robot environment perception method based on multi-sensor fusion provided by the embodiments of the present application; Figure 2 A block diagram of a robot environment perception device based on multi-sensor fusion is provided for an embodiment of the present application. Figure 3 A block diagram of a robot is provided for an embodiment of the present application. DETAILED DESCRIPTION

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the present application will be briefly introduced below with reference to the drawings and the description of the embodiments or the prior art. Obviously, the following description of the drawings only constitutes some embodiments of the present application, and other drawings can be obtained by those skilled in the art without any creative effort. It should be noted that the description of these embodiments is used to help understand the present application, but does not constitute a limitation on the present application.

[0019] It should be understood that although the terms first, second, etc. may be used herein to describe various elements, these elements should not be limited by these terms. For example, a first element can be referred to as a second element, and similarly a second element can be referred to as a first element without departing from the scope of the example embodiments of the present application.

[0020] It should be understood that for the term "and / or" which may appear herein, it only describes the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B, which means that there are three cases of A alone, B alone, and A and B together; for the term " / and" which may appear herein, it describes another association object relationship, which means that there can be two relationships, for example, A / and B, which means that there are two cases of A alone and A and B together; in addition, for the character " / " which may appear herein, it generally means that the associated objects before and after are an "or" relationship.

[0021] EMBODIMENT In order to realize dynamic obstacle avoidance of the robot, an embodiment of the present application provides a robot environment perception method based on multi-sensor fusion, which can reduce the operation amount and operation complexity in the obstacle detection process, improve the detection speed of the obstacle, and meet the dynamic obstacle avoidance requirements of the high-speed moving robot.

[0022] The robot environment perception method based on multi-sensor fusion provided by the embodiment of the present application can be applied to a robot that can move autonomously. It can be understood that the execution subject does not constitute a limitation on the embodiment of the present application.

[0023] The robot environment perception method based on multi-sensor fusion provided by the embodiment of the present application will be described in detail below.

[0024] As Figure 1 shown in FIG. 1, which is a flowchart of a robot environment perception method based on multi-sensor fusion provided by a first aspect of the present application. The robot environment perception method based on multi-sensor fusion can include, but is not limited to, the following steps S101-S107.

[0025] Step S101. Real-time acquisition of images of the area in front of the robot collected by the camera and laser point clouds of the area in front of the robot collected by the laser radar.

[0026] In the present application, a camera and a laser radar can be provided on the robot, and the images of the area in front of the robot can be acquired in real time through the camera, and the laser point clouds of the area in front of the robot can be acquired in real time through the laser radar.

[0027] Step S102. Time synchronization of the images collected by the camera and the laser point clouds collected by the laser radar, and determination of the target image time-synchronized with the latest collected laser point cloud.

[0028] The sampling frequencies of the general camera and the laser radar are inconsistent, so when detecting obstacles in the area in front of the robot, the images collected by the camera and the laser point clouds collected by the laser radar need to be time-synchronized.

[0029] Specifically, the difference between the time stamp corresponding to each image collected by the camera and the time stamp corresponding to the latest collected laser point cloud can be calculated first. Then, the image with the smallest difference between the corresponding time stamp and the time stamp corresponding to the latest collected laser point cloud is selected as the target image time-synchronized with the latest collected laser point cloud.

[0030] Step S103. Obstacle detection on the target image through a target detection algorithm to obtain the obstacle target in the target image.

[0031] Among them, the obstacle detection on the target image can be achieved by existing conventional algorithms such as Fast R-CNN (Fast Region-based Convolutional Network), YOLOv5 (You Only Look Once version 5) algorithm, etc., but is not limited thereto, and will not be described in detail in the present application.

[0032] Step S104. Conversion of the three-dimensional point cloud coordinates of the latest collected laser point cloud into pixel coordinate system coordinates.

[0033] In the embodiments of the present application, the three-dimensional point cloud coordinates of the newly collected laser point cloud can be first converted into spatial coordinates in the camera coordinate system, and then the spatial coordinates converted into the camera coordinate system are converted into pixel coordinate system coordinates.

[0034] When converting the three-dimensional point cloud coordinates of the newly collected laser point cloud into spatial coordinates in the camera coordinate system, the three-dimensional point cloud coordinates of the newly collected laser point cloud can be converted into spatial coordinates in the camera coordinate system according to the following formula: where x represents the coordinate of the converted point in the x-axis direction of the camera coordinate system, y represents the coordinate of the converted point in the y-axis direction of the camera coordinate system, and z represents the coordinate of the converted point in the z-axis direction of the camera coordinate system, R represents a rotation matrix, T represents a translation matrix, x represents the coordinate of the point in the x-axis direction of the laser point cloud coordinate system, y represents the coordinate of the point in the y-axis direction of the laser point cloud coordinate system, and z represents the coordinate of the point in the z-axis direction of the laser point cloud coordinate system. camera camera camera ld ld ld The rotation matrix R and the translation matrix T can be obtained in advance through extrinsic calibration.

[0035] When converting the spatial coordinates converted into the camera coordinate system into pixel coordinate system coordinates, the spatial coordinates converted into the camera coordinate system can be converted into pixel coordinate system coordinates according to the following formula: where u represents the coordinate of the converted point in the horizontal axis direction of the pixel coordinate, v represents the coordinate of the converted point in the vertical axis direction of the pixel coordinate, and K represents a camera intrinsic matrix.

[0036] Step S105. The laser points converted and corresponding to pixel coordinate system coordinates located in the pixel coordinate range of the obstacle target are segmented to obtain segmented laser point clouds.

[0037] By segmenting the laser points converted and corresponding to pixel coordinate system coordinates located in the pixel coordinate range of the obstacle target, a large number of point clouds irrelevant to the obstacle can be removed, thereby greatly reducing the computational load of the subsequent operation and improving the detection speed of the obstacle.

[0038] Step S106. The segmented laser point clouds are clustered through a clustering algorithm to obtain at least one cluster.

[0039] ​​​​​The laser energy emitted by the laser radar irradiates the surface of the obstacle, and part of the energy is reflected. The reflection intensity is related to the reflectivity of the target surface. The same object usually has high consistency in material, surface roughness, color and the like, and thus the reflectivity remains consistent. Therefore, the embodiments of the present application improve the existing clustering algorithm by adding the reflection intensity of each point in the clustered point cloud feature to improve the accuracy of clustering.

[0040] Specifically, when clustering the segmented laser point cloud, the segmented laser point cloud can be first denoised to obtain a denoised laser point cloud, so as to remove some outlier noise points. Then the reflection intensity of each point in the denoised laser point cloud is extracted, which can be directly calculated and output by the laser radar hardware. Then, based on the reflection intensity and the point cloud coordinates of each point in the denoised laser point cloud, a feature vector of each point in the denoised laser point cloud can be constructed, and the feature vector can be normalized. Finally, based on the feature vector of each point in the denoised laser point cloud, the denoised laser point cloud can be clustered by dynamic threshold adaptive clustering (DTAC) to obtain at least one cluster.

[0041] It can be understood that in one or more embodiments, the denoised laser point cloud can also be clustered by a clustering algorithm such as K-means or DBSCAN (Density-Based Spatial Clustering of Applications with Noise).

[0042] Step S107. The region surrounded by the point cloud corresponding to each cluster in the at least one cluster is taken as an obstacle region in front of the robot.

[0043] The application provides a robot environment perception method based on multi-sensor fusion, which comprises the following steps: acquiring images of a region in front of a robot collected by a camera and laser point clouds of the region in front of the robot collected by a laser radar in real time; performing time synchronization on the images collected by the camera and the laser point clouds collected by the laser radar, and determining a target image which is time-synchronized with the latest collected laser point clouds; detecting obstacles in the target image by a target detection algorithm, and obtaining obstacle targets in the target image; converting three-dimensional point cloud coordinates of the latest collected laser point clouds into pixel coordinate system coordinates; segmenting laser points which are converted and correspond to pixel coordinate system coordinates located in a pixel coordinate range of the obstacle targets, and obtaining segmented laser point clouds; clustering the segmented laser point clouds by a clustering algorithm, and obtaining at least one cluster; and taking a region surrounded by point clouds corresponding to each cluster in the at least one cluster as an obstacle region in front of the robot. In this way, the obstacle targets are detected from the images, and the point clouds in the region where the obstacles are located are segmented from the three-dimensional point clouds according to the spatial relationship between the pixel coordinates and the three-dimensional point cloud coordinates, so that a large number of point clouds irrelevant to the obstacles are removed, then the segmented point clouds are clustered by the clustering, and thus the obstacle region is identified according to the clustering result, thereby without analyzing all the point cloud data, the operation amount and operation complexity in the obstacle detection process are greatly reduced, the detection speed of the obstacles is improved, the dynamic obstacle avoidance demand of the high-speed robot is met, the robot environment perception can be widely used, and the application and popularization are facilitated. Meanwhile, the reflection intensity of each point is added to the point cloud features in the clustering, so that the accuracy of the clustering is improved, and the detection accuracy of the obstacles is improved.

[0044] Please refer to Figure 2 The second aspect of the embodiments of the application provides a robot environment perception device based on multi-sensor fusion, which comprises: An acquisition unit is configured to acquire images of a region in front of a robot collected by a camera and laser point clouds of the region in front of the robot collected by a laser radar in real time; A time synchronization unit is configured to perform time synchronization on the images collected by the camera and the laser point clouds collected by the laser radar, and determine a target image which is time-synchronized with the latest collected laser point clouds; An obstacle detection unit is configured to detect obstacles in the target image by a target detection algorithm, and obtain obstacle targets in the target image; A conversion unit is configured to convert three-dimensional point cloud coordinates of the latest collected laser point clouds into pixel coordinate system coordinates; A segmentation unit is configured to segment laser points which are converted and correspond to pixel coordinate system coordinates located in a pixel coordinate range of the obstacle targets, and obtain segmented laser point clouds; a clustering unit, configured to cluster the segmented laser point cloud by a clustering algorithm to obtain at least one cluster; a determining unit, configured to determine a region surrounded by the point cloud corresponding to each cluster in the at least one cluster as an obstacle region in front of the robot.

[0045] The working process, working details and technical effects of the robot environment perception device based on multi-sensor fusion provided in the second aspect of the embodiment can be referred to the first aspect of the embodiment, and will not be described here.

[0046] As shown in Figure 3 The third aspect of the embodiment of the present application provides a robot, which comprises a memory, a processor and a transceiver connected in sequence and in communication, wherein the memory is configured to store a computer program, the transceiver is configured to transceive messages, and the processor is configured to read the computer program and execute the robot environment perception method based on multi-sensor fusion as described in the first aspect of the embodiment.

[0047] For example, the memory can include, but is not limited to, random access memory (RAM), read-only memory (ROM), flash memory, first-in-first-out memory (FIFO) and / or first-in-last-out memory (FILO) and the like; the processor can be, but is not limited to, a microprocessor of STM32F105 series, an ARM (Advanced RISC Machines) processor or an X86 architecture processor, or a processor integrated with NPU (neural-network processing units); and the transceiver can be, but is not limited to, a WiFi (Wireless Fidelity) wireless transceiver, a Bluetooth wireless transceiver, a General Packet Radio Service (GPRS) wireless transceiver, a ZigBee wireless transceiver, a 3G transceiver, a 4G transceiver and / or a 5G transceiver, etc.

[0048] The fourth aspect of the embodiment provides a computer-readable storage medium storing instructions of the robot environment perception method based on multi-sensor fusion as described in the first aspect of the embodiment, i.e., the computer-readable storage medium stores instructions, and when the instructions run on a computer, the robot environment perception method based on multi-sensor fusion as described in the first aspect is executed. The computer-readable storage medium refers to a carrier storing data, which can include, but is not limited to, floppy disks, optical disks, hard disks, flash memories, USB flash drives and / or Memory Sticks, etc., and the computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable devices.

[0049] The fifth aspect of the embodiment provides a computer program product containing instructions, which, when executed on a computer, cause the computer to perform the robot environment perception method based on multi-sensor fusion as described in the first aspect of the embodiment, wherein the computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices.

[0050] It should be appreciated that specific details are given in the following description in order to provide a thorough understanding. However, it should be appreciated that the example embodiments can be practiced in the absence of these specific details. For instance, systems can be illustrated in block diagram form in order not to obscure the examples in unnecessary detail.

[0051] Finally, it should be noted that the above-described embodiments are merely possible examples of implementing the present application, and thus do not limit the protective scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the protective scope of the present application.

Claims

1. A robot environment perception method based on multi-sensor fusion, characterized in that, The method comprises the following steps: real-time acquisition of images of a region in front of the robot collected by a camera and laser point clouds of the region in front of the robot collected by a laser radar; time synchronization of the images collected by the camera and the laser point clouds collected by the laser radar to determine a target image time-synchronized with the latest collected laser point cloud; obstacle detection on the target image by a target detection algorithm to obtain an obstacle target in the target image; conversion of three-dimensional point cloud coordinates of the latest collected laser point cloud into pixel coordinate system coordinates; segmentation of laser points corresponding to the converted pixel coordinate system coordinates located in a pixel coordinate range of the obstacle target to obtain segmented laser point clouds; clustering of the segmented laser point clouds by a clustering algorithm to obtain at least one cluster; regarding a region surrounded by point clouds corresponding to each cluster in the at least one cluster as an obstacle region in front of the robot. 2.The multi-sensor fusion based robot environment perception method according to claim 1, wherein, The time synchronization of the images collected by the camera and the laser point clouds collected by the laser radar to determine a target image time-synchronized with the latest collected laser point cloud comprises the following steps: calculating a difference value between a time stamp corresponding to each image in the images collected by the camera and a time stamp corresponding to the latest collected laser point cloud; selecting an image with the smallest difference value between the corresponding time stamp and the time stamp corresponding to the latest collected laser point cloud as a target image time-synchronized with the latest collected laser point cloud. 3.The multi-sensor fusion based robot environment perception method according to claim 1, wherein, The conversion of three-dimensional point cloud coordinates of the latest collected laser point cloud into pixel coordinate system coordinates comprises the following steps: conversion of three-dimensional point cloud coordinates of the latest collected laser point cloud into space coordinates in a camera coordinate system; conversion of the space coordinates converted into the camera coordinate system into pixel coordinate system coordinates.

4. The multi-sensor fusion based robot environment perception method according to claim 3, wherein, The conversion of three-dimensional point cloud coordinates of the latest collected laser point cloud into space coordinates in a camera coordinate system comprises the following steps: conversion of three-dimensional point cloud coordinates of the latest collected laser point cloud into space coordinates in a camera coordinate system according to the following formula: where x camera represents the coordinate of the converted point in the x-axis direction under the camera coordinate system, y camera represents the coordinate of the converted point in the y-axis direction under the camera coordinate system, z camera represents the coordinate of the converted point in the z-axis direction under the camera coordinate system, R represents a rotation matrix, T represents a translation matrix, x ld represents the coordinate of the point in the x-axis direction in the laser point cloud coordinate system, y ld represents the coordinate of the point in the y-axis direction in the laser point cloud coordinate system, z ld represents the coordinate of the point in the z-axis direction in the laser point cloud coordinate system; The conversion of the space coordinates converted into the camera coordinate system into pixel coordinate system coordinates comprises the following steps: conversion of the space coordinates converted into the camera coordinate system into pixel coordinate system coordinates according to the following formula: where u denotes the coordinate of the converted point in the horizontal axis direction under pixel coordinates, v denotes the coordinate of the converted point in the vertical axis direction under pixel coordinates, and K denotes a camera intrinsic matrix. 5.The multi-sensor fusion based robot environment perception method according to claim 1, wherein, The clustering of the segmented laser point clouds by a clustering algorithm to obtain at least one cluster comprises the following steps: de-noising processing of the segmented laser point clouds to obtain de-noised laser point clouds; extraction of reflection intensities of points in the de-noised laser point clouds; construction of feature vectors of points in the de-noised laser point clouds based on the reflection intensities and point cloud coordinates of the points in the de-noised laser point clouds; clustering of the de-noised laser point clouds based on the feature vectors of the points in the de-noised laser point clouds by a dynamic threshold adaptive clustering algorithm to obtain at least one cluster. 6.The multi-sensor fusion based robot environment perception method according to claim 1, wherein, The target detection algorithm is a YOLOv5 algorithm.

7. A multi-sensor fusion based robot environment perception device, characterized in that, The method comprises the following steps: an acquisition unit is configured to real-time acquisition of images of a region in front of the robot collected by a camera and laser point clouds of the region in front of the robot collected by a laser radar; a time synchronization unit is configured to time synchronization of the images collected by the camera and the laser point clouds collected by the laser radar to determine a target image time-synchronized with the latest collected laser point cloud; An obstacle detection unit is configured to perform obstacle detection on the target image by using a target detection algorithm to obtain an obstacle target in the target image. A conversion unit is configured to convert three-dimensional point cloud coordinates of the latest collected laser point cloud into pixel coordinate system coordinates. A segmentation unit is configured to segment the laser points after conversion and corresponding pixel coordinate system coordinates located in a pixel coordinate range of the obstacle target to obtain a segmented laser point cloud. A clustering unit is configured to perform clustering on the segmented laser point cloud by using a clustering algorithm to obtain at least one cluster. A determination unit is configured to take a region surrounded by point clouds corresponding to each cluster in the at least one cluster as an obstacle region in front of the robot.

8. A robot, characterized in that The computer readable storage medium stores instructions, when the instructions run on the computer, execute the robot environment perception method based on multi-sensor fusion as claimed in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores instructions, when the instructions run on the computer, execute the robot environment perception method based on multi-sensor fusion as claimed in any one of claims 1-6.

10. A computer program product comprising computer programs or instructions, characterized in that, The computer program or the instructions, when executed by a computer, implement the robot environment perception method based on multi-sensor fusion as claimed in any one of claims 1-6.

Images