Intelligent monitoring method and system of laser radar and vision fusion

Abstract

This application provides an intelligent monitoring method and system that fuses lidar and vision, specifically applied in the field of data fusion. The method includes: acquiring point cloud data, reflection intensity, and image data of a target scene; establishing two coordinate systems based on the point cloud data and reflection intensity; constructing a first mapping relationship and a second mapping relationship between the image data and the two coordinate systems; and fusing the point cloud data and image data of the target scene based on the first and second mapping relationships to achieve intelligent monitoring. This improved data fusion technology enhances the accuracy of target recognition, thereby optimizing the intelligent monitoring system's environmental perception capabilities.

Description

Technical Field

[0001] This application relates to the field of data fusion, and more specifically, to an intelligent monitoring method and system that integrates lidar and vision. Background Technology

[0002] The integrated LiDAR-based robotic automatic monitoring system is a measurement system centered around LiDAR and cameras. Utilizing data fusion technology, it intelligently monitors target scenes, assisting in the identification of defects, emergencies, and safety hazards, ensuring the operational safety of high-speed railway stations and improving the efficiency and quality of supervision.

[0003] Existing technologies typically use deep neural networks to extract point cloud data, project the point cloud data onto image data, and obtain a fusion result; or, they use deep neural networks to extract image data, project the image data onto point cloud data, and obtain a fusion result. However, the above methods only simply complete the matching between point cloud and image, without considering data errors caused by various environmental noise data.

[0004] Therefore, how to reduce errors and improve the accuracy of data fusion is a technical problem that urgently needs to be solved. Summary of the Invention

[0005] The purpose of this invention is to provide an intelligent monitoring method and system that integrates lidar and vision. The method divides data into background and foreground data, determines the mapping relationship between the data, thereby achieving the fusion of point cloud data and image data, improving the accuracy of target recognition, and ultimately optimizing the intelligent monitoring system's environmental perception capabilities. The specific technical solution is as follows:

[0006] In a first aspect of the present invention, a smart monitoring method integrating lidar and vision is provided, comprising:

[0007] Use lidar to collect point cloud data and reflection intensity of the target scene;

[0008] Use a monocular camera to acquire image data of the target scene;

[0009] A first coordinate system and a second coordinate system are established based on the point cloud data and the reflection intensity;

[0010] The image data is used to establish a first mapping relationship and a second mapping relationship with the first coordinate system and the second coordinate system, respectively;

[0011] The point cloud data and image data of the target scene are fused according to the first mapping relationship and the second mapping relationship to obtain the fusion result;

[0012] Intelligent monitoring is achieved based on the fusion results.

[0013] Optionally, establishing the first coordinate system and the second coordinate system based on the point cloud data and the reflection intensity includes:

[0014] Separate the background point cloud data and the foreground point cloud data from the point cloud data;

[0015] A first coordinate system is established based on the background point cloud data;

[0016] A second coordinate system is established based on the aforementioned foreground point cloud data;

[0017] The first coordinate system and the second coordinate system are adjusted based on the reflection intensity.

[0018] Optionally, adjusting the first coordinate system based on the reflection intensity includes:

[0019] The background point cloud data is divided into multiple first rectangular blocks of a preset size;

[0020] Determine the first sub-reflection intensity corresponding to each of the first rectangular blocks;

[0021] Construct a sub-first parameter matrix based on the first sub-reflection intensity;

[0022] Calculate the Hadamard product between the first sub-parameter matrix and the first rectangular block to obtain the first parameterized block;

[0023] Adjust the first coordinate system according to the first parameterized block.

[0024] Optionally, adjusting the second coordinate system based on the reflection intensity includes:

[0025] The foreground cloud data is divided into multiple second rectangular blocks of preset size;

[0026] Determine the second sub-reflection intensity corresponding to each of the second rectangular blocks;

[0027] Construct a second sub-parameter matrix based on the second sub-reflection intensity;

[0028] Calculate the Hadamard product between the second sub-parameter matrix and the second rectangular block to obtain the second parameterized block;

[0029] Adjust the second coordinate system according to the second parameterized block.

[0030] Optionally, constructing a first mapping relationship and a second mapping relationship between the image data and the first coordinate system and the second coordinate system respectively includes:

[0031] Separate the background image data and foreground image data from the image data;

[0032] Establish a first mapping relationship between the background image data and the adjusted first coordinate system;

[0033] A second mapping relationship is established between the foreground image data and the adjusted second coordinate system.

[0034] Optionally, establishing a first mapping relationship between the background image data and the adjusted first coordinate system includes:

[0035] Perform a local Fourier transform on the background image data to obtain the first encoding result;

[0036] Perform a Fourier transform on the first parameterized block to obtain the second encoding result;

[0037] The first encoding result and the second encoding result are input into a deep neural network model to obtain a first mapping relationship between the background image data and the adjusted first coordinate system.

[0038] Optionally, establishing a second mapping relationship between the foreground image data and the adjusted second coordinate system includes:

[0039] A local Fourier transform is performed on the foreground image data to obtain the third encoding result;

[0040] Perform a Fourier transform on the second parameterized block to obtain the fourth encoding result;

[0041] The third and fourth encoding results are input into a deep neural network model to obtain a second mapping relationship between the foreground image data and the adjusted second coordinate system.

[0042] Optionally, the step of fusing the point cloud data and image data of the target scene according to the first mapping relationship and the second mapping relationship to obtain the fusion result includes:

[0043] If the deviation between the first mapping relationship and the second mapping relationship is within a preset range, then a fusion rule is established based on the first mapping relationship and the second mapping relationship;

[0044] The point cloud data and image data of the target scene are fused based on the fusion rules to obtain the fusion result.

[0045] In another aspect of the present invention, an intelligent monitoring system integrating lidar and vision is provided, comprising: a data acquisition module for acquiring point cloud data and reflection intensity of a target scene using lidar; and acquiring image data of the target scene using a monocular camera;

[0046] A coordinate system establishment module is used to establish a first coordinate system and a second coordinate system based on the point cloud data and the reflection intensity.

[0047] The data mapping module is used to construct a first mapping relationship and a second mapping relationship between the image data and the first coordinate system and the second coordinate system, respectively;

[0048] The data fusion module is used to fuse the point cloud data and image data of the target scene according to the first mapping relationship and the second mapping relationship to obtain the fusion result;

[0049] The intelligent monitoring module is used to achieve intelligent monitoring based on the fusion results.

[0050] Optionally, the coordinate system establishment module is specifically used for:

[0051] Separate the background point cloud data and the foreground point cloud data from the point cloud data;

[0052] A first coordinate system is established based on the background point cloud data;

[0053] A second coordinate system is established based on the aforementioned foreground point cloud data;

[0054] The first coordinate system and the second coordinate system are adjusted based on the reflection intensity.

[0055] Optionally, adjusting the first coordinate system based on the reflection intensity includes:

[0056] The background point cloud data is divided into multiple first rectangular blocks of a preset size;

[0057] Determine the first sub-reflection intensity corresponding to each of the first rectangular blocks;

[0058] Construct a sub-first parameter matrix based on the first sub-reflection intensity;

[0059] Calculate the Hadamard product between the first sub-parameter matrix and the first rectangular block to obtain the first parameterized block;

[0060] Adjust the first coordinate system according to the first parameterized block.

[0061] Optionally, adjusting the second coordinate system based on the reflection intensity includes:

[0062] The foreground cloud data is divided into multiple second rectangular blocks of preset size;

[0063] Determine the second sub-reflection intensity corresponding to each of the second rectangular blocks;

[0064] Construct a second sub-parameter matrix based on the second sub-reflection intensity;

[0065] Calculate the Hadamard product between the second sub-parameter matrix and the second rectangular block to obtain the second parameterized block;

[0066] Adjust the second coordinate system according to the second parameterized block.

[0067] Optionally, the data mapping module is specifically used for:

[0068] Separate the background image data and foreground image data from the image data;

[0069] Establish a first mapping relationship between the background image data and the adjusted first coordinate system;

[0070] A second mapping relationship is established between the foreground image data and the adjusted second coordinate system.

[0071] Optionally, establishing a first mapping relationship between the background image data and the adjusted first coordinate system includes:

[0072] Perform a local Fourier transform on the background image data to obtain the first encoding result;

[0073] Perform a Fourier transform on the first parameterized block to obtain the second encoding result;

[0074] The first encoding result and the second encoding result are input into a deep neural network model to obtain a first mapping relationship between the background image data and the adjusted first coordinate system.

[0075] Optionally, establishing a second mapping relationship between the foreground image data and the adjusted second coordinate system includes:

[0076] A local Fourier transform is performed on the foreground image data to obtain the third encoding result;

[0077] Perform a Fourier transform on the second parameterized block to obtain the fourth encoding result;

[0078] The third and fourth encoding results are input into a deep neural network model to obtain a second mapping relationship between the foreground image data and the adjusted second coordinate system.

[0079] Optionally, the data fusion module is specifically used for:

[0080] If the deviation between the first mapping relationship and the second mapping relationship is within a preset range, then a fusion rule is established based on the first mapping relationship and the second mapping relationship;

[0081] The point cloud data and image data of the target scene are fused based on the fusion rules to obtain the fusion result.

[0082] Beneficial effects:

[0083] (1) This invention utilizes lidar and a monocular camera to collect point cloud data, reflection intensity, and image data of the target scene; establishes two coordinate systems based on the point cloud data and reflection intensity; constructs a first mapping relationship and a second mapping relationship between the image data and the two coordinate systems respectively; and fuses the point cloud data and image data of the target scene based on the first mapping relationship and the second mapping relationship to achieve intelligent monitoring. The improved data fusion technology enhances the accuracy of target recognition, thereby optimizing the intelligent monitoring system's ability to perceive the environment.

[0084] (2) This invention divides point cloud data into background point cloud data and foreground point cloud data, establishes a first coordinate system and a second coordinate system, and adjusts the first and second coordinate systems based on reflection intensity; divides the background / foreground point cloud data into multiple rectangular blocks of a preset size; determines the sub-reflection intensity corresponding to each rectangular block; constructs a sub-parameter matrix based on the sub-reflection intensity; calculates the Hadamard product of the sub-parameter matrix and the rectangular block to obtain the parameterized block; and adjusts the first / second coordinate system based on the parameterized block. The reference coordinate system established in this way can reduce data errors caused by noise and improve fusion accuracy.

[0085] (3) In order to accurately obtain the mapping relationship between data, this invention divides the image data into background image data and foreground image data, uses Fourier transform to encode the background / foreground image data and parameterized blocks respectively, introduces a deep neural network model, and establishes the mapping relationship between image data and point cloud data. This improves the accuracy and reliability of the fusion technology. Attached Figure Description

[0086] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0087] Figure 1 This is a flowchart of an intelligent monitoring method that fuses lidar and vision, as provided in an embodiment of this application.

[0088] Figure 2 This is a flowchart of the coordinate system adjustment method provided in the embodiments of this application;

[0089] Figure 3 This is a schematic diagram of the structure of an intelligent monitoring system that integrates lidar and vision, provided in an embodiment of this application. Detailed Implementation

[0090] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0091] Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0092] This application provides an intelligent monitoring method and system that fuses lidar and vision, including: acquiring point cloud data, reflection intensity, and image data of a target scene; establishing two coordinate systems based on the point cloud data and reflection intensity; constructing a first mapping relationship and a second mapping relationship between the image data and the two coordinate systems; and fusing the point cloud data and image data of the target scene based on the first mapping relationship and the second mapping relationship to achieve intelligent monitoring. The improved data fusion technology enhances the accuracy of target recognition, thereby optimizing the intelligent monitoring system's environmental perception capabilities.

[0093] This intelligent monitoring method and system, which integrates lidar and vision, can be integrated into electronic devices such as terminals and servers. The terminals can be mobile phones, tablets, smart Bluetooth devices, laptops, or personal computers (PCs); the servers can be a single server or a server cluster consisting of multiple servers.

[0094] It is understood that the intelligent monitoring method and system for fusion of lidar and vision described in this embodiment can be executed on a terminal, on a server, or jointly by a terminal and a server. The above examples should not be construed as limiting this application.

[0095] Artificial intelligence (AI) is the theory, methods, technology, and application devices that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. AI software technology mainly includes computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0096] In recent years, with the research and progress of artificial intelligence technology, it has been widely applied in many fields. The solutions provided in this disclosure involve computer vision technology, machine learning / deep learning technologies of artificial intelligence, etc., which are specifically illustrated through the following embodiments:

[0097] Example 1

[0098] Please see Figure 1 , Figure 1 The flowchart illustrates an intelligent monitoring method for fusion of lidar and vision according to an embodiment of the present disclosure. The method specifically includes the following steps:

[0099] Step S110: Use lidar to collect point cloud data and reflection intensity of the target scene.

[0100] Step S120: Use a monocular camera to acquire image data of the target scene.

[0101] Step S130: Establish a first coordinate system and a second coordinate system based on the point cloud data and the reflection intensity.

[0102] In one implementation, step S130 may specifically include the following steps:

[0103] Step S131: Separate the background point cloud data and the foreground point cloud data from the point cloud data.

[0104] Step S132: Establish a first coordinate system based on the background point cloud data.

[0105] Step S133: Establish a second coordinate system based on the foreground point cloud data.

[0106] Step S134: Adjust the first coordinate system and the second coordinate system based on the reflection intensity.

[0107] Specifically, the first coordinate system is adjusted based on the reflection intensity to obtain the adjusted first coordinate system; the second coordinate system is adjusted based on the reflection intensity to obtain the adjusted second coordinate system; step S140 is executed based on the adjusted first and second coordinate systems.

[0108] The coordinate system established in this way can reduce data errors caused by noise and improve fusion accuracy.

[0109] Step S140: Construct a first mapping relationship and a second mapping relationship between the image data and the first coordinate system and the second coordinate system, respectively.

[0110] In one implementation, step S140 may specifically include the following steps:

[0111] S141. Separate the background image data and foreground image data from the image data.

[0112] S142. Establish a first mapping relationship between the background image data and the adjusted first coordinate system.

[0113] Specifically, a local Fourier transform is performed on the background image data to obtain a first encoding result; a Fourier transform is performed on the first parameterized block to obtain a second encoding result; the first encoding result and the second encoding result are input into a deep neural network model to obtain a first mapping relationship between the background image data and the adjusted first coordinate system.

[0114] S143. Establish a second mapping relationship between the foreground image data and the adjusted second coordinate system.

[0115] Specifically, a local Fourier transform is performed on the foreground image data to obtain a third encoding result; a Fourier transform is performed on the second parameterized block to obtain a fourth encoding result; the third encoding result and the fourth encoding result are input into a deep neural network model to obtain a second mapping relationship between the foreground image data and the adjusted second coordinate system.

[0116] In this embodiment, to accurately obtain the mapping relationship between data, the image data is divided into background image data and foreground image data. Fourier transform is used to encode the background / foreground image data and parameterized blocks respectively. A deep neural network model is introduced to establish the mapping relationship between image data and point cloud data. This improves the accuracy and reliability of the data fusion technology.

[0117] Step S150: Fuse the point cloud data and image data of the target scene according to the first mapping relationship and the second mapping relationship to obtain the fusion result.

[0118] Specifically, if the deviation between the first mapping relationship and the second mapping relationship is within a preset range, a fusion rule is established based on the first mapping relationship and the second mapping relationship; the point cloud data and image data of the target scene are fused based on the fusion rule to obtain a fusion result.

[0119] The fusion rules can be as follows: fusion of background point cloud data and background image data according to the mapping relationship to obtain the background fusion result; fusion of foreground point cloud data and foreground image data according to the mapping relationship to obtain the foreground fusion result; and then splicing the two fusion results to obtain the final fusion result.

[0120] Step S160: Implement intelligent monitoring based on the fusion results.

[0121] Specifically, it can be integrated into a monitoring system to assist in identifying defects, emergencies, and safety hazards, ensuring the operational safety of high-speed railway stations and improving the efficiency and quality of supervision.

[0122] Therefore, by improving data fusion technology, the accuracy of target recognition can be enhanced, thereby optimizing the intelligent monitoring system's ability to perceive the environment.

[0123] Example 2

[0124] Figure 2 A flowchart of the coordinate system adjustment method provided in this application embodiment is shown. Please refer to it. Figure 2 Specifically, it includes the following steps:

[0125] Step S210: Establish the coordinate system to be adjusted based on the background / front view cloud data.

[0126] The method for establishing the coordinate system to be adjusted has been described in detail in Example 1, and will not be repeated here.

[0127] Step S220: Divide the background / foreground point cloud data into multiple rectangular blocks of a preset size.

[0128] It should be noted that the size of the rectangular blocks can be flexibly set according to the size of the target scene, and no specific limitation is made here.

[0129] Step S230: Determine the sub-reflection intensity corresponding to each rectangular block.

[0130] Step S240: Construct a sub-parameter matrix based on the sub-reflection intensity.

[0131] Step S250: Calculate the Hadamard product of the sub-parameter matrix and the rectangular block to obtain the parameterized block.

[0132] Specifically, the formula for calculating the Hadamard product is:

[0133]

[0134] Where H represents any parameterized block, Represents any rectangular block with m columns and n rows. Represents any subparameter matrix with m columns and n rows.

[0135] Repeat the above operation until all rectangular blocks are processed, resulting in multiple corresponding parameterized blocks.

[0136] Step S260: Adjust the coordinate system to be adjusted according to the parameterized block.

[0137] Specifically, a size compression operation is performed on each parameterized block. In particular, the maximum value in each parameterized block can be selected as the compression result. For example, if any parameterized block is... If the value is 10, then the compression result of this block is 10; alternatively, the average value of each parameterized block can be selected as the compression result, for example, if any parameterized block is 10, then the compression result is 10. The compression result of this block is (3+6+4+10+5+4+7+3+8) / 9=5.6; the compressed and recombined parts of all parameterized blocks are used as the adjustment result for the first / second coordinate system. This speeds up the calculation.

[0138] This embodiment establishes a coordinate system by introducing the Hadamard product, which can reduce data errors caused by noise and improve fusion accuracy.

[0139] Example 3

[0140] This embodiment also provides an intelligent monitoring system that integrates lidar and vision, such as... Figure 3 As shown, the intelligent monitoring system that integrates lidar and vision includes:

[0141] The data acquisition module 310 is used to acquire point cloud data and reflection intensity of the target scene using lidar; and to acquire image data of the target scene using a monocular camera.

[0142] The coordinate system establishment module 320 is used to establish a first coordinate system and a second coordinate system based on the point cloud data and the reflection intensity.

[0143] The data mapping module 330 is used to construct a first mapping relationship and a second mapping relationship between the image data and the first coordinate system and the second coordinate system, respectively.

[0144] The data fusion module 340 is used to fuse the point cloud data and image data of the target scene according to the first mapping relationship and the second mapping relationship to obtain the fusion result.

[0145] The intelligent monitoring module 350 is used to achieve intelligent monitoring based on the fusion results.

[0146] Optionally, the coordinate system establishment module 320 is specifically used for:

[0147] Separate the background point cloud data and the foreground point cloud data from the point cloud data; establish a first coordinate system based on the background point cloud data; establish a second coordinate system based on the foreground point cloud data; adjust the first coordinate system and the second coordinate system based on the reflection intensity.

[0148] Optionally, adjusting the first coordinate system based on the reflection intensity includes: dividing the background point cloud data into multiple first rectangular blocks of a preset size; determining the first sub-reflection intensity corresponding to each first rectangular block; constructing a sub-first parameter matrix based on the first sub-reflection intensity; calculating the Hadamard product of the first sub-parameter matrix and the first rectangular block to obtain a first parameterized block; and adjusting the first coordinate system based on the first parameterized block.

[0149] Optionally, adjusting the second coordinate system based on the reflection intensity includes: dividing the foreground point cloud data into multiple second rectangular blocks of a preset size; determining the second sub-reflection intensity corresponding to each second rectangular block; constructing a second sub-parameter matrix based on the second sub-reflection intensity; calculating the Hadamard product of the second sub-parameter matrix and the second rectangular block to obtain a second parameterized block; and adjusting the second coordinate system based on the second parameterized block.

[0150] Optionally, the data mapping module 330 is specifically used for:

[0151] Separate the background image data and foreground image data from the image data; establish a first mapping relationship between the background image data and the adjusted first coordinate system; establish a second mapping relationship between the foreground image data and the adjusted second coordinate system.

[0152] Optionally, establishing a first mapping relationship between the background image data and the adjusted first coordinate system includes: performing a local Fourier transform on the background image data to obtain a first encoding result; performing a Fourier transform on the first parameterized block to obtain a second encoding result; and inputting the first encoding result and the second encoding result into a deep neural network model to obtain a first mapping relationship between the background image data and the adjusted first coordinate system.

[0153] Optionally, establishing a second mapping relationship between the foreground image data and the adjusted second coordinate system includes: performing a local Fourier transform on the foreground image data to obtain a third encoding result; performing a Fourier transform on the second parameterized block to obtain a fourth encoding result; and inputting the third encoding result and the fourth encoding result into a deep neural network model to obtain a second mapping relationship between the foreground image data and the adjusted second coordinate system.

[0154] Optionally, the data fusion module 340 is specifically used to: if the deviation between the first mapping relationship and the second mapping relationship is within a preset range, then establish a fusion rule based on the first mapping relationship and the second mapping relationship; and fuse the point cloud data and image data of the target scene based on the fusion rule to obtain a fusion result.

[0155] By improving the accuracy of target recognition through enhanced data fusion technology, the system can be used to assist in the identification of defects, emergencies, and safety hazards, ensuring the operational safety of high-speed railway stations and improving the efficiency and quality of supervision.

[0156] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the modules / units / subunits / components in the above-described device can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0157] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Additionally, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0158] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0159] In addition, the functional units in the embodiments provided in this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0160] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0161] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In addition, the terms "first", "second", "third", etc. are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0162] Finally, it should be noted that the above-described embodiments are merely specific implementations of this application, used to illustrate the technical solutions of this application, and not to limit them. The protection scope of this application is not limited thereto. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this application; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application. All should be covered within the protection scope of this application. Therefore, the protection scope of this application should be determined by the protection scope of the claims.

Claims

1. An intelligent monitoring method integrating lidar and vision, characterized in that, include: Use lidar to collect point cloud data and reflection intensity of the target scene; Use a monocular camera to acquire image data of the target scene; Establish a first coordinate system and a second coordinate system based on the point cloud data and the reflection intensity; including: Separate the background point cloud data and the foreground point cloud data from the point cloud data; A first coordinate system is established based on the background point cloud data; A second coordinate system is established based on the aforementioned foreground point cloud data; The first coordinate system and the second coordinate system are adjusted based on the reflection intensity; The image data is used to establish a first mapping relationship and a second mapping relationship with the first coordinate system and the second coordinate system, respectively; The point cloud data and image data of the target scene are fused according to the first mapping relationship and the second mapping relationship to obtain the fusion result; Intelligent monitoring is achieved based on the fusion results.

2. The intelligent monitoring method for fusion of lidar and vision according to claim 1, characterized in that, Adjusting the first coordinate system based on the reflection intensity includes: The background point cloud data is divided into multiple first rectangular blocks of a preset size; Determine the first sub-reflection intensity corresponding to each of the first rectangular blocks; Construct a sub-first parameter matrix based on the first sub-reflection intensity; Calculate the Hadamard product between the first parameter matrix and the first rectangular block to obtain the first parameterized block; Adjust the first coordinate system according to the first parameterized block.

3. The intelligent monitoring method for fusion of lidar and vision according to claim 2, characterized in that, Adjusting the second coordinate system based on the reflection intensity includes: The foreground cloud data is divided into multiple second rectangular blocks of preset size; Determine the second sub-reflection intensity corresponding to each of the second rectangular blocks; Construct a second sub-parameter matrix based on the second sub-reflection intensity; Calculate the Hadamard product between the second sub-parameter matrix and the second rectangular block to obtain the second parameterized block; Adjust the second coordinate system according to the second parameterized block.

4. The intelligent monitoring method for fusion of lidar and vision according to claim 3, characterized in that, The step of constructing a first mapping relationship and a second mapping relationship between the image data and the first coordinate system and the second coordinate system respectively includes: Separate the background image data and foreground image data from the image data; Establish a first mapping relationship between the background image data and the adjusted first coordinate system; A second mapping relationship is established between the foreground image data and the adjusted second coordinate system.

5. The intelligent monitoring method for fusion of lidar and vision according to claim 4, characterized in that, The step of establishing a first mapping relationship between the background image data and the adjusted first coordinate system includes: Perform a local Fourier transform on the background image data to obtain the first encoding result; Perform a Fourier transform on the first parameterized block to obtain the second encoding result; The first encoding result and the second encoding result are input into a deep neural network model to obtain a first mapping relationship between the background image data and the adjusted first coordinate system.

6. The intelligent monitoring method for fusion of lidar and vision according to claim 5, characterized in that, The step of establishing a second mapping relationship between the foreground image data and the adjusted second coordinate system includes: A local Fourier transform is performed on the foreground image data to obtain the third encoding result; Perform a Fourier transform on the second parameterized block to obtain the fourth encoding result; The third and fourth encoding results are input into a deep neural network model to obtain a second mapping relationship between the foreground image data and the adjusted second coordinate system.

7. The intelligent monitoring method for fusion of lidar and vision according to claim 1, characterized in that, The step of fusing the point cloud data and image data of the target scene according to the first mapping relationship and the second mapping relationship to obtain the fusion result includes: If the deviation between the first mapping relationship and the second mapping relationship is within a preset range, then a fusion rule is established based on the first mapping relationship and the second mapping relationship; The point cloud data and image data of the target scene are fused based on the fusion rules to obtain the fusion result.

8. An intelligent monitoring system integrating lidar and vision, characterized in that, include: The data acquisition module is used to collect point cloud data and reflection intensity of the target scene using lidar; Use a monocular camera to acquire image data of the target scene; A coordinate system establishment module is used to establish a first coordinate system and a second coordinate system based on the point cloud data and the reflection intensity; and to separate the background point cloud data and the foreground point cloud data from the point cloud data. A first coordinate system is established based on the background point cloud data; A second coordinate system is established based on the aforementioned foreground point cloud data; The first coordinate system and the second coordinate system are adjusted based on the reflection intensity; The data mapping module is used to construct a first mapping relationship and a second mapping relationship between the image data and the first coordinate system and the second coordinate system, respectively; The data fusion module is used to fuse the point cloud data and image data of the target scene according to the first mapping relationship and the second mapping relationship to obtain the fusion result; The intelligent monitoring module is used to achieve intelligent monitoring based on the fusion results.

Images