An environmental monitoring system and method based on radar and vision fusion

Abstract

The present invention discloses an environment monitoring system and method based on radar and vision fusion. The environment monitoring system based on radar and vision fusion includes monitoring equipment, network equipment, and remote terminal. By aligning point cloud data and image data, each point in the point cloud is mapped to the pixel coordinate system of the visual module, and the coordinates of the abnormal point in the pixel coordinate system are obtained. Based on the coordinate data of the abnormal point in the pixel coordinate system of the visual module, the image data is fused and processed to generate a visual display image with abnormal point marks, thereby realizing the recognition of abnormal objects using radar detection results, and being able to intuitively display the abnormal objects in the image imaged by the camera, thereby improving the efficiency and accuracy of monitoring.

Description

Technical Field

[0001] The present invention belongs to the technical field of environmental monitoring, and in particular relates to an environmental monitoring system and method based on radar and vision fusion. Background Art

[0002] In the field of environmental monitoring technology, two types of sensors are commonly used: LiDAR and cameras. LiDAR is not affected by external conditions and can extract targets in the monitoring scene in bad weather or low visibility, but it cannot obtain optical information of the target and has a low resolution. The camera has a high resolution and is easy to perform target recognition based on neural networks, but it is seriously affected by external conditions.

[0003] In the prior art, the fusion of radar and camera mainly adopts the method of displaying the radar detection results in the imaging results of the camera. This method does not use the radar detection results to identify abnormal objects, nor can it intuitively display the abnormal objects in the image formed by the camera, which is easy to cause false alarms or missed reports.

[0004] Therefore, how to overcome the defects of the prior art has become a problem to be solved urgently in this technical field. Summary of the invention

[0005] The present invention provides an environment monitoring system and method based on radar and vision fusion, which specifically adopts the following technical solutions:

[0006] An environment monitoring system based on radar and vision fusion, the system comprising monitoring equipment, network equipment, and a remote terminal;

[0007] The network device is communicatively connected with the monitoring device and the remote terminal respectively;

[0008] The monitoring equipment includes a laser radar module, a vision module, a data processing module, a communication interface module, and a power module;

[0009] The laser radar module and the visual module are respectively connected to the data processing module, and are used to send the collected data to the data processing module;

[0010] The data processing module is connected to the communication interface module;

[0011] The communication interface module is used to connect to the external network device;

[0012] The data processing module of the monitoring device generates a radar point cloud display image, and based on the coordinate data of the abnormal point in the pixel coordinate system of the visual module, performs fusion processing on the image data to generate a visual display image with abnormal point marks, which is sent to the network device through the communication interface module.

[0013] The network device includes an interface server and a database;

[0014] The interface server is connected to the communication interface module, and is used to store the radar point cloud display image and the visual display image from the communication interface module into the database, and obtain the data in the database and send it to the mobile terminal according to the request of the mobile terminal;

[0015] The mobile terminal includes a processing module, a display module, an abnormality prompt module, and a user management module;

[0016] The processing module is used to send a request message to the interface server of the network device based on the user's instruction, and based on the interface server's response to the request message, receive the radar point cloud display image and the visual display image from the interface server, perform lightweight processing on them, send them to the display module for display, and send a prompt instruction to the abnormal prompt module when the visual display image includes an abnormal point mark;

[0017] The abnormal prompt module generates prompt information based on the prompt instruction and sends it to the display module for display;

[0018] The user management module is used to create, delete, modify, and set permissions for user information.

[0019] Further, the laser wavelength of the laser radar module is 905nm, the @100klx range is 190m@10% reflectivity, 260m@20% reflectivity, 320m@80% reflectivity, the FOV is 70 degrees, and the data rate is 240,000 points / second / first echo or strongest echo, 480,000 points / second / double echo, and 720,000 points / second / triple echo;

[0020] The vision module includes an RGB camera with 4K resolution.

[0021] Furthermore, the communication interface module includes a LAN interface, a fiber optic interface or a wireless communication interface.

[0022] Furthermore, the power supply module is connected to an external power supply and is connected to the laser radar module, vision module, data processing module, and communication interface module to provide power for the above modules.

[0023] Furthermore, the data processing module includes a comparison unit, a registration unit, and a fusion unit;

[0024] The comparison unit is used to receive and save the point cloud data collected by the laser radar module, wherein the point cloud data includes coordinate information and echo intensity information of all points in the point cloud in the laser radar module coordinate system, and compare the point cloud data at different sampling times to obtain the coordinates of the abnormal point in the laser radar module coordinate system;

[0025] The registration unit is used to receive and save the point cloud data collected by the laser radar module, the image data collected by the vision module, and the coordinate data of the abnormal point in the laser radar module coordinate system, register the above point cloud data and image data, map each point in the point cloud to the pixel coordinate system of the vision module, and obtain the coordinates of the abnormal point in the pixel coordinate system;

[0026] The fusion unit is used to receive and save the point cloud data collected by the laser radar module, the image data collected by the vision module, and the coordinate data of the abnormal point in the pixel coordinate system of the vision module, filter the point cloud data to generate a radar point cloud display image, and fuse the image data based on the coordinate data of the abnormal point in the pixel coordinate system of the vision module to generate a visual display image with abnormal point marks.

[0027] Furthermore, the comparison unit calculates the distance between any point in the point cloud at the current sampling moment and the corresponding point in the point cloud at the previous sampling moment, identifies the point whose distance exceeds a preset threshold as an abnormal point, and inputs the coordinates of the abnormal point in the laser radar module coordinate system into the alignment unit.

[0028] Further, the registration unit performs a random sampling consistent planarization process on the point cloud data to obtain point cloud plane data, performs random feature extraction on the point cloud plane data, and calculates the intersection coordinates of two intersecting edge approximate straight lines in the random features as random feature points;

[0029] The registration unit performs semantic analysis and recognition on the image data, segments the image feature points with the same names as the random feature points in the point cloud plane data, performs parameter calibration based on the random feature points and the image feature points, and obtains a parameter calibration matrix H:

[0030]

[0031] Wherein, R is the rotation matrix obtained according to parameter calibration, and T is the translation matrix obtained according to 3D-2D parameter calibration;

[0032] In the laser radar module coordinate system, the coordinates of the random feature points are (x r ,y r ,z r ), the coordinates of the image feature point with the same name as the random feature point in the image coordinate system of the visual module are (xc ,y c ,z c ), then the rotation matrix R and the translation matrix T satisfy the following relationship:

[0033]

[0034] The registration unit converts the coordinate information of the point cloud into the pixel coordinate system of the visual module:

[0035]

[0036] Among them, the coordinates (x p ,y p ) is the coordinate of a point in the point cloud (x r ,y r ,z r ) in the pixel coordinate system of the visual module, the coordinate (x p0 ,y p0 ) is the coordinate of the origin of the image coordinate system of the visual module in the pixel coordinate system of the visual module, f is the focal length of the visual module, p x is the x-dimension of the pixel, p y is the x-dimension of the pixel;

[0037] The registration unit converts and registers the coordinate information of the abnormal point into the pixel coordinate system of the visual module, generates the coordinates of the abnormal point in the pixel coordinate system of the visual module, and sends them to the fusion unit.

[0038] Furthermore, the abnormal point marking includes highlighting and / or changing the color and / or boxing the pixel area corresponding to the abnormal point in the visual display image;

[0039] The fusion unit sends the radar point cloud display image and the visual display image to the network device through the communication interface module.

[0040] The present invention also relates to an environment monitoring method based on radar and vision fusion, which is used in the environment monitoring system based on radar and vision fusion as described above, and specifically comprises the following steps:

[0041] S1. The laser radar module and the visual module of the monitoring device respectively collect point cloud data and image data of the environment, wherein the point cloud data includes coordinate information and echo intensity information of all points in the point cloud in the laser radar module coordinate system;

[0042] S2. The data processing module of the monitoring device receives and saves the point cloud data collected by the laser radar module, and compares the point cloud data at different sampling times to obtain the coordinates of the abnormal point in the laser radar module coordinate system;

[0043] S3. The data processing module of the monitoring equipment receives the image data collected by the visual module and the coordinate data of the abnormal point in the laser radar module coordinate system, aligns the above point cloud data and image data, maps each point in the point cloud to the pixel coordinate system of the visual module, and obtains the coordinates of the abnormal point in the pixel coordinate system.

[0044] S4. The data processing module of the monitoring device generates a radar point cloud display image, and based on the coordinate data of the abnormal point in the pixel coordinate system of the visual module, the image data is fused and processed to generate a visual display image with abnormal point marks, which is sent to the network device through the communication interface module;

[0045] S5. The network device sends data to the mobile terminal according to the request of the mobile terminal;

[0046] S6. Based on the user's instructions, the mobile terminal sends a request message to the interface server of the network device, and based on the interface server's response to the request message, receives the radar point cloud display image and the visual display image from the interface server, and performs lightweight processing on them for display.

[0047] Further, the step S2 comprises:

[0048] The comparison unit of the data processing module calculates the distance between any point in the point cloud at the current sampling moment and the corresponding point in the point cloud at the previous sampling moment, identifies the point whose distance exceeds a preset threshold as an abnormal point, and inputs the coordinates of the abnormal point in the laser radar module coordinate system into the alignment unit.

[0049] The technical solution of the present invention aligns point cloud data and image data, maps each point in the point cloud to the pixel coordinate system of the visual module, obtains the coordinates of the abnormal point in the pixel coordinate system, and fuses the image data based on the coordinate data of the abnormal point in the pixel coordinate system of the visual module to generate a visual display image marked with the abnormal point, thereby realizing the recognition of abnormal objects using radar detection results, and can intuitively display the abnormal objects in the image imaged by the camera, thereby improving the efficiency and accuracy of monitoring. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 The present invention is a schematic diagram of the structure of an environment monitoring system based on radar and vision fusion. DETAILED DESCRIPTION

[0051] The present invention will be further described below in conjunction with the accompanying drawings. The following examples are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the scope of protection of the present invention. It should be pointed out that the following detailed descriptions are all exemplary and are intended to provide further explanation of the present application.

[0052] Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those generally understood by those of ordinary skill in the art to which this application belongs. It should be noted that the terms used herein are only for the purpose of describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0053] Embodiment 1 of the present invention relates to an environment monitoring system based on radar and vision fusion, including a monitoring device, a network device, and a remote terminal, wherein the network device is communicatively connected to the monitoring device and the remote terminal respectively.

[0054] The monitoring device includes a laser radar module, a visual module, a data processing module, a communication interface module, and a power module. The laser radar module and the visual module are respectively connected to the data processing module to send the collected data to the data processing module. The data processing module is connected to the communication interface module, and the communication interface module is used to connect to the external network device. The data processing module sends the processed data to the network device through the communication interface.

[0055] The laser wavelength of the laser radar module is 905nm, the @100klx range is 190m@10% reflectivity, 260m@20% reflectivity, 320m@80% reflectivity, the FOV is 70 degrees, and the data rate is 240,000 points / second (first echo or strongest echo), 480,000 points / second (double echo), and 720,000 points / second (triple echo).

[0056] The vision module includes an RGB camera with 4K resolution.

[0057] The data processing module includes a comparison unit, a registration unit, and a fusion unit.

[0058] The comparison unit is used to receive and save the point cloud data collected by the laser radar module, the point cloud data includes the coordinate information and echo intensity information of all points in the point cloud in the laser radar module coordinate system, and compares the point cloud data at different sampling times to obtain the coordinates of the abnormal point in the laser radar module coordinate system. Specifically, the comparison unit calculates the distance between any point in the point cloud at the current sampling time and the corresponding point in the point cloud at the previous sampling time, identifies the point whose distance exceeds a preset threshold as an abnormal point, and inputs the coordinates of the abnormal point in the laser radar module coordinate system into the registration unit.

[0059] The registration unit is used to receive and save the point cloud data collected by the laser radar module, the image data collected by the vision module, and the coordinate data of the abnormal point in the laser radar module coordinate system. The registration unit registers the above point cloud data and image data so that each point in the point cloud is mapped to the pixel coordinate system of the vision module, and obtains the coordinates of the abnormal point in the pixel coordinate system.

[0060] The registration unit performs random sampling and consistent planarization processing on the point cloud data to obtain point cloud plane data, performs random feature extraction on the point cloud plane data, and calculates the intersection coordinates of two intersecting edge approximate straight lines in the random features as random feature points. The registration unit performs semantic analysis and recognition on the image data, segments the image feature points in the image with the same name as the random feature points in the point cloud plane data, performs parameter calibration based on the random feature points and the image feature points, and obtains the parameter calibration matrix H:

[0061]

[0062] Among them, R is the rotation matrix obtained according to parameter calibration, and T is the translation matrix obtained according to 3D-2D parameter calibration.

[0063] In the laser radar module coordinate system, the coordinates of the random feature points are (x r ,y r ,z r ), the coordinates of the image feature point with the same name as the random feature point in the image coordinate system of the visual module are (x c ,y c ,z c ), then the rotation matrix R and the translation matrix T satisfy the following relationship:

[0064]

[0065] The registration unit converts the coordinate information of the point cloud into the pixel coordinate system of the visual module:

[0066]

[0067] Among them, the coordinates (x p ,y p ) is the coordinate of a point in the point cloud (x r ,y r ,z r ) in the pixel coordinate system of the visual module, the coordinate (x p0 ,y p0 ) is the coordinate of the origin of the image coordinate system of the visual module in the pixel coordinate system of the visual module, f is the focal length of the visual module, p x is the x-dimension of the pixel, p y is the x-dimension of the pixel.

[0068] The registration unit converts and registers the coordinate information of the abnormal point into the pixel coordinate system of the visual module, generates the coordinates of the abnormal point in the pixel coordinate system of the visual module, and sends them to the fusion unit.

[0069] The fusion unit is used to receive and save the point cloud data collected by the laser radar module, the image data collected by the visual module, and the coordinate data of the abnormal point in the pixel coordinate system of the visual module. The fusion unit performs filtering processing on the point cloud data to generate a radar point cloud display image, and based on the coordinate data of the abnormal point in the pixel coordinate system of the visual module, performs fusion processing on the image data to generate a visual display image with abnormal point marks. The abnormal point mark is included in the visual display image, and the pixel area corresponding to the abnormal point is highlighted and / or changed in color and / or framed. The fusion unit sends the radar point cloud display image and the visual display image to the network device through the communication interface module. .

[0070] The communication interface module includes a LAN interface, an optical fiber interface or a wireless communication interface.

[0071] The power supply module is connected to an external power supply and is connected to the laser radar module, vision module, data processing module, and communication interface module to provide power for the above modules.

[0072] The network device includes an interface server and a database. The interface server is connected to the communication interface module and is used to store the radar point cloud display image and the visual display image from the communication interface module into the database, and obtain the data in the database and send it to the mobile terminal according to the request of the mobile terminal.

[0073] The mobile terminal includes a processing module, a display module, an abnormal prompt module, and a user management module. The processing module is used to send a request message to the interface server of the network device based on the user's instruction, and based on the response of the interface server to the request message, receive the radar point cloud display image and the visual display image from the interface server, perform lightweight processing on them, send them to the display module for display, and send a prompt instruction to the abnormal prompt module when the visual display image includes an abnormal point mark. The abnormal prompt module generates prompt information based on the prompt instruction and sends it to the display module for display. The user management module is used to create, delete, modify, and set permissions for user information.

[0074] Embodiment 2 of the present invention relates to a method for an environment monitoring system based on radar and vision fusion applied in Embodiment 1, specifically comprising the following steps:

[0075] S1. The laser radar module and the visual module of the monitoring equipment respectively collect point cloud data and image data of the environment, and the point cloud data includes the coordinate information and echo intensity information of all points in the point cloud in the laser radar module coordinate system.

[0076] S2. The data processing module of the monitoring device receives and saves the point cloud data collected by the laser radar module, and compares the point cloud data at different sampling times to obtain the coordinates of the abnormal point in the laser radar module coordinate system.

[0077] The comparison unit of the data processing module calculates the distance between any point in the point cloud at the current sampling moment and the corresponding point in the point cloud at the previous sampling moment, identifies the point whose distance exceeds a preset threshold as an abnormal point, and inputs the coordinates of the abnormal point in the laser radar module coordinate system into the alignment unit.

[0078] S3. The data processing module of the monitoring equipment receives the image data collected by the visual module and the coordinate data of the abnormal point in the laser radar module coordinate system, aligns the above point cloud data and image data, maps each point in the point cloud to the pixel coordinate system of the visual module, and obtains the coordinates of the abnormal point in the pixel coordinate system.

[0079] The registration unit of the data processing module performs a random sampling consistent planarization process on the point cloud data to obtain point cloud plane data, performs random feature extraction on the point cloud plane data, and calculates the intersection coordinates of two intersecting edge approximate straight lines in the random features as random feature points. The registration unit performs semantic analysis and recognition on the image data, segments the image feature points in the image with the same name as the random feature points in the point cloud plane data, performs parameter calibration based on the random feature points and the image feature points, and obtains the parameter calibration matrix H:

[0080]

[0081] Among them, R is the rotation matrix obtained according to parameter calibration, and T is the translation matrix obtained according to 3D-2D parameter calibration.

[0082] In the laser radar module coordinate system, the coordinates of the random feature points are (x r ,y r ,z r ), the coordinates of the image feature point with the same name as the random feature point in the image coordinate system of the visual module are (x c ,y c ,z c ), then the rotation matrix R and the translation matrix T satisfy the following relationship:

[0083]

[0084] The registration unit converts the coordinate information of the point cloud into the pixel coordinate system of the visual module:

[0085]

[0086] Among them, the coordinates (x p ,y p ) is the coordinate of a point in the point cloud (x r ,y r ,z r ) in the pixel coordinate system of the visual module, the coordinate (x p0 ,y p0 ) is the coordinate of the origin of the image coordinate system of the visual module in the pixel coordinate system of the visual module, f is the focal length of the visual module, p x is the x-dimension of the pixel, p y is the x-dimension of the pixel.

[0087] The registration unit converts and registers the coordinate information of the abnormal point into the pixel coordinate system of the visual module, generates the coordinates of the abnormal point in the pixel coordinate system of the visual module, and sends them to the fusion unit.

[0088] S4. The data processing module of the monitoring device generates a radar point cloud display image, and based on the coordinate data of the abnormal point in the pixel coordinate system of the visual module, performs fusion processing on the image data to generate a visual display image with abnormal point marks, and sends it to the network device through the communication interface module.

[0089] The fusion unit of the data processing module is used to receive and save the point cloud data collected by the laser radar module, the image data collected by the visual module, and the coordinate data of the abnormal point in the pixel coordinate system of the visual module. The fusion unit performs filtering processing on the point cloud data to generate a radar point cloud display image, and based on the coordinate data of the abnormal point in the pixel coordinate system of the visual module, performs fusion processing on the image data to generate a visual display image with abnormal point markings. The abnormal point marking includes highlighting and / or changing the color and / or boxing the pixel area corresponding to the abnormal point in the visual display image.

[0090] The fusion unit sends the radar point cloud display image and the visual display image to the network device through the communication interface module.

[0091] S5. The network device sends data to the mobile terminal according to the request of the mobile terminal.

[0092] The network device includes an interface server and a database. The interface server is connected to the communication interface module and is used to store the radar point cloud display image and the visual display image from the communication interface module into the database, and obtain the data in the database and send it to the mobile terminal according to the request of the mobile terminal.

[0093] S6. Based on the user's instructions, the mobile terminal sends a request message to the interface server of the network device, and based on the interface server's response to the request message, receives the radar point cloud display image and the visual display image from the interface server, and performs lightweight processing on them for display.

[0094] The mobile terminal includes a processing module, a display module, an abnormal prompt module, and a user management module. The processing module is used to send a request message to the interface server of the network device based on the user's instruction, and based on the response of the interface server to the request message, receive the radar point cloud display image and the visual display image from the interface server, perform lightweight processing on them, send them to the display module for display, and send a prompt instruction to the abnormal prompt module when the visual display image includes an abnormal point mark. The abnormal prompt module generates prompt information based on the prompt instruction and sends it to the display module for display. The user management module is used to create, delete, modify, and set permissions for user information.

[0095] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the protection scope of the present invention.

Claims

1. An environmental monitoring system based on radar and vision fusion, characterized in that: The system includes monitoring equipment, network equipment, and remote terminals; The network device is respectively connected to the monitoring device and the remote terminal in communication; The monitoring equipment includes a laser radar module, a vision module, a data processing module, a communication interface module, and a power module; The laser radar module and the vision module are respectively connected to the data processing module to send the collected data to the data processing module; The data processing module is connected to the communication interface module; The data processing module includes a comparison unit, a registration unit, and a fusion unit; The comparison unit is used to receive and save the point cloud data collected by the laser radar module, the point cloud data includes the coordinate information and echo intensity information of all points in the point cloud in the laser radar module coordinate system, and calculates the distance between any point in the point cloud at the current sampling time and the corresponding point in the point cloud at the previous sampling time, identifies the point whose distance exceeds a preset threshold as an abnormal point, and inputs the coordinates of the abnormal point in the laser radar module coordinate system into the registration unit; The registration unit converts the coordinate information of the abnormal point into the pixel coordinate system of the visual module, generates the coordinates of the abnormal point in the pixel coordinate system of the visual module, and sends them to the fusion unit, wherein the registration unit performs a random sampling consistent planarization process on the point cloud data to obtain the point cloud plane data, performs random feature extraction on the point cloud plane data, and calculates the intersection coordinates of two intersecting edge approximate straight lines in the random features as random feature points, and performs semantic analysis and recognition on the image data, segmenting the image feature points with the same name as the random feature points in the point cloud plane data, and performs parameter calibration according to the random feature points and the image feature points to obtain the parameter calibration matrix; The fusion unit generates a visual display image with abnormal point markings; The communication interface module is used to connect to external network equipment; The data processing module of the monitoring equipment generates a radar point cloud display image, and based on the coordinate data of the abnormal point in the pixel coordinate system of the visual module, fuses the image data to generate a visual display image with abnormal point marks, which is sent to the network device through the communication interface module. Network equipment includes interface servers and databases; The interface server is connected to the communication interface module, and is used to store the radar point cloud display image and the visual display image from the communication interface module into a database, and obtain the data in the database and send it to the mobile terminal according to the request of the mobile terminal; The mobile terminal includes a processing module, a display module, an abnormal prompt module, and a user management module; The processing module is used to send a request message to the interface server of the network device based on the user's instruction, and based on the response of the interface server to the request message, receive the radar point cloud display image and the visual display image from the interface server, perform lightweight processing on them, send them to the display module for display, and send a prompt instruction to the abnormal prompt module when the abnormal point mark is included in the visual display image; The abnormal prompt module generates prompt information based on the prompt instruction and sends it to the display module for display; The user management module is used to create, delete, modify, and set permissions for user information.

2. An environment monitoring system based on radar and vision fusion according to claim 1, characterized in that: The laser wavelength of the laser radar module is 905nm, the @100klx range is 190m@10% reflectivity, 260m@20% reflectivity, 320m@80% reflectivity, the FOV is 70 degrees, and the data rate is 240,000 points / second / first echo or strongest echo, 480,000 points / second / double echo, 720,000 points / second / triple echo; The vision module includes a 4K resolution RGB camera.

3. An environment monitoring system based on radar and vision fusion according to claim 1, characterized in that: The communication interface module includes a LAN interface, a fiber optic interface or a wireless communication interface.

4. An environment monitoring system based on radar and vision fusion according to claim 1, characterized in that: The power module is connected to an external power supply and is connected to the laser radar module, vision module, data processing module, and communication interface module to provide power for the above modules.

5. An environment monitoring system based on radar and vision fusion according to claim 1, characterized in that: The registration unit is used to receive and save the point cloud data collected by the laser radar module, the image data collected by the vision module, and the coordinate data of the abnormal point in the laser radar module coordinate system, register the above point cloud data and image data, map each point in the point cloud to the pixel coordinate system of the vision module, and obtain the coordinates of the abnormal point in the pixel coordinate system; The fusion unit is used to receive and save the point cloud data collected by the lidar module, the image data collected by the vision module, and the coordinate data of the abnormal point in the pixel coordinate system of the vision module, filter the point cloud data, generate a radar point cloud display image, and fuse the image data based on the coordinate data of the abnormal point in the pixel coordinate system of the vision module.

6. An environment monitoring system based on radar and vision fusion according to claim 1, characterized in that: The parameter calibration matrix is ​​H: ； Wherein, R is the rotation matrix obtained according to parameter calibration, and T is the translation matrix obtained according to 3D-2D parameter calibration; In the laser radar module coordinate system, the coordinates of the random feature points are , the coordinates of the image feature points with the same name as the random feature points in the image coordinate system of the visual module are , then the rotation matrix R and the translation matrix T satisfy the following relationship: ； The registration unit converts the coordinate information of the point cloud into the pixel coordinate system of the visual module: ； Among them, the coordinates is the coordinate of a point in the point cloud The corresponding coordinates in the pixel coordinate system of the visual module are is the coordinate of the origin of the image coordinate system of the visual module in the pixel coordinate system of the visual module, f is the focal length of the visual module, is the x-dimension of the pixel, is the x-dimension of the pixel.

7. An environment monitoring system based on radar and vision fusion according to claim 1, characterized in that: The abnormal point marking includes highlighting and / or changing the color and / or boxing the pixel area corresponding to the abnormal point in the visual display image; The fusion unit sends the radar point cloud display image and the visual display image to the network device through the communication interface module.

8. An environmental monitoring method based on radar and vision fusion, the method being used in an environmental monitoring system based on radar and vision fusion as claimed in any one of claims 1 to 7, characterized in that: The specific steps include: S1. The laser radar module and the visual module of the monitoring device respectively collect point cloud data and image data of the environment, and the point cloud data includes the coordinate information and echo intensity information of all points in the point cloud in the laser radar module coordinate system; S2. The data processing module of the monitoring device receives and saves the point cloud data collected by the laser radar module, and compares the point cloud data at different sampling times to obtain the coordinates of the abnormal point in the laser radar module coordinate system; the comparison unit of the data processing module calculates the distance between any point in the point cloud at the current sampling time and the corresponding point in the point cloud at the previous sampling time, identifies the point whose distance exceeds a preset threshold as an abnormal point, and inputs the coordinates of the abnormal point in the laser radar module coordinate system into the registration unit; S3. The data processing module of the monitoring device receives the image data collected by the visual module and the coordinate data of the abnormal point in the laser radar module coordinate system, aligns the above point cloud data and image data, maps each point in the point cloud to the pixel coordinate system of the visual module, and obtains the coordinates of the abnormal point in the pixel coordinate system; S4. The data processing module of the monitoring device generates a radar point cloud display image, and based on the coordinate data of the abnormal point in the pixel coordinate system of the visual module, the image data is fused to generate a visual display image with abnormal point marks, which is sent to the network device through the communication interface module; S5. The network device sends data to the mobile terminal according to the request of the mobile terminal; S6. The mobile terminal sends a request message to the interface server of the network device based on the user's instruction, and receives the radar point cloud display image and the visual display image from the interface server based on the interface server's response to the request message, and performs lightweight processing on them for display.

Images