Obstacle visualization method and device in working environment

Abstract

The invention discloses a method and a device for visualizing obstacles in a working environment. The method comprises the following steps: receiving sensing data of various sensors arranged on engineering machinery for the working environment; performing multi-source data fusion and obstacle recognition on the perception data through a pre-trained 2D image-3D point cloud fusion perception model to obtain obstacle information of a target obstacle existing in the working environment; and projecting the obstacle information to a pre-established 3D map to obtain a target 3D map containing the obstacle information of the target obstacle, and displaying the target 3D map. Therefore, by adopting the embodiment of the invention, the accuracy of obstacle recognition can be improved, meanwhile, real-time data processing can be carried out under high-load and high-speed conditions, and obstacles existing in the surrounding working environment can be visually visualized in a three-dimensional space, so that the accuracy of obstacle recognition is improved. Therefore, the probability that the engineering machinery collides with surrounding obstacles in the operation process is reduced.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method and device for visualizing obstacles in a working environment. Background Art

[0002] With the development of science and technology, construction machinery is increasingly used in modern industrial production, but its safety issues during operation are becoming increasingly prominent. Since the working environment of construction machinery is complex, the operating space is narrow, and it often needs to operate under high load and high speed conditions, how to effectively prevent construction machinery from colliding with surrounding obstacles during operation is an urgent problem to be solved in the current field of construction machinery safety.

[0003] In the related technology, a single sensor, such as a camera, is mainly installed on the construction machinery to obtain single environmental perception information around the construction machinery through these sensors, and obstacle judgment is performed based on the single environmental perception information. The data type of the single environmental perception information is single, resulting in poor obstacle recognition accuracy; at the same time, the processing process of the single environmental perception information cannot realize real-time data processing under high load and high speed conditions, and cannot realize obstacle visualization in the three-dimensional space of the working environment, thereby increasing the probability of the construction machinery colliding with surrounding obstacles during the operation process. Summary of the invention

[0004] The embodiments of the present application provide a method and apparatus for visualizing obstacles in a working environment. In order to have a basic understanding of some aspects of the disclosed embodiments, a simple summary is given below. This summary is not a general review, nor is it intended to identify key / important components or describe the scope of protection of these embodiments. Its only purpose is to present some concepts in a simple form as a preface to the detailed description that follows.

[0005] In a first aspect, an embodiment of the present application provides a method for visualizing obstacles in a working environment, the method comprising:

[0006] Receive data about the working environment from various sensors installed on the construction machinery;

[0007] Through the pre-trained 2D image-3D point cloud fusion perception model, multi-source data fusion and obstacle recognition are performed on the perception data to obtain obstacle information of the target obstacles in the working environment. The pre-trained 2D image-3D point cloud fusion perception model is a neural network based on deep learning for obstacle detection;

[0008] The obstacle information is projected onto a pre-established 3D map to obtain a target 3D map including the obstacle information of the target obstacle, and the target 3D map is displayed.

[0009] Optionally, the environmental perception data includes 2D image data and 3D point cloud data, and the pre-trained 2D image-3D point cloud fusion perception model includes a feature fusion module, an obstacle category determination module and an obstacle matching module, the feature fusion module is used to fuse feature vectors of the 2D image data and the 3D point cloud data, the obstacle category determination module is used to identify the obstacle category of the target obstacle in the working environment based on the fused features, and the obstacle matching module is used to associate the obstacle information of the target obstacle based on the feature vectors and obstacle categories of the 2D image data and the 3D point cloud data.

[0010] Optionally, through the pre-trained 2D image-3D point cloud fusion perception model, multi-source data fusion and obstacle recognition are performed on the perception data to obtain obstacle information of the target obstacles in the working environment, including:

[0011] Extract feature vectors from 2D image data and 3D point cloud data through a feature fusion module to obtain feature vectors of each pixel in the 2D image data and feature vectors of each point cloud position point in the 3D point cloud data;

[0012] The feature vector of each pixel point and the feature vector of each point cloud position point are fused through the feature fusion module to obtain the fused feature;

[0013] The fused features are input into the obstacle category determination module, and the obstacle category of the target obstacle in the working environment is output;

[0014] The feature vectors and obstacle categories of the 2D image data and 3D point cloud data are input into the obstacle matching module, and the obstacle information of the target obstacle is output.

[0015] Optionally, the feature vector of each pixel point is fused with the feature vector of each point cloud position point to obtain a fused feature, including:

[0016] According to the pre-established transformation relationship of the position coordinate system of each sensor, the feature vector of each pixel point is paired with the feature vector of each point cloud position point to obtain feature vectors of multiple groups of pixel points and feature vectors of point cloud position points under the same position coordinates;

[0017] The feature vector of each group of pixels is fused with the feature vector of the point cloud position point to obtain the fused feature.

[0018] Optionally, the various sensors provided on the engineering machinery include multiple cameras and millimeter wave radars, and the perception data also includes raw data of the millimeter wave radars and data of the laser radar;

[0019] Project obstacle information onto pre-built 3D maps, including:

[0020] Get the pixel coordinates of the target obstacle in the 2D image data taken by each camera;

[0021] Calculate the target distance between the target obstacle and the construction machinery based on the raw data of the millimeter wave radar and the data of the laser radar;

[0022] Determine the three-dimensional spatial coordinates of the target obstacle based on the pixel coordinates and the target distance between the construction machinery;

[0023] According to the three-dimensional spatial coordinates of the target obstacle, the obstacle information is projected into the pre-established 3D map.

[0024] Optionally, according to the three-dimensional spatial coordinates of the target obstacle, the obstacle information is projected into a pre-established 3D map, including:

[0025] From the pre-established 3D map, locate the target position point that is consistent with the three-dimensional spatial coordinates of the target obstacle;

[0026] Mark the obstacle information at the target location.

[0027] Optionally, generate a pre-built 3D map by following these steps, including:

[0028] Convert 3D point cloud data into a format that can be used for map construction to obtain basic map data of the working environment;

[0029] Divide the three-dimensional space of the working environment into discrete grids;

[0030] Fill the divided grids with basic map data to obtain a pre-built 3D map.

[0031] Optionally, a pre-trained 2D image-3D point cloud fusion perception model is generated by following the steps below, including:

[0032] Obtain a feature fusion module, an obstacle category determination module, and an obstacle matching module;

[0033] The feature fusion module, obstacle category determination module and obstacle matching module are integrated into the neural network to obtain a 2D image-3D point cloud fusion perception model;

[0034] Collect historical perception data of the working environment from various sensors on construction machinery;

[0035] Label the historical perception data with obstacle information to obtain model training samples;

[0036] The model training samples are used to perform deep learning on the 2D image-3D point cloud fusion perception model to obtain a pre-trained 2D image-3D point cloud fusion perception model.

[0037] Optionally, after displaying the target 3D map, the following is also included:

[0038] When the displayed obstacle information indicates that the distance between the target obstacle and the construction machinery is less than a preset threshold, a warning message is generated;

[0039] Display warning information.

[0040] In a second aspect, an embodiment of the present application provides a device for visualizing obstacles in a working environment, the device comprising:

[0041] A receiving module is used to receive the perception data of the working environment from various sensors installed on the engineering machinery;

[0042] The recognition module is used to perform multi-source data fusion and obstacle recognition on the perception data through a pre-trained 2D image-3D point cloud fusion perception model to obtain obstacle information of target obstacles in the working environment. The pre-trained 2D image-3D point cloud fusion perception model is a neural network for obstacle detection based on deep learning;

[0043] The display module is used to project the obstacle information into a pre-established 3D map, obtain a target 3D map containing the obstacle information of the target obstacle, and display the target 3D map.

[0044] The technical solution provided by the embodiments of the present application may have the following beneficial effects:

[0045] In the embodiment of the present application, on the one hand, through the pre-trained 2D image-3D point cloud fusion perception model, multi-source data fusion and obstacle identification are performed on the perception data. The perception data is fed back by multiple types of sensors. Through the fusion of multiple types of sensor data, all-round perception of the surrounding environment of the construction machinery is realized, and obstacles can be detected and identified more accurately, which greatly improves the accuracy of the system. At the same time, the pre-trained 2D image-3D point cloud fusion perception model is a neural network for obstacle detection based on deep learning, which can perform real-time data processing under high load and high speed conditions; on the other hand, by projecting the obstacle information into a pre-established 3D map, a target 3D map containing the obstacle information of the target obstacle is obtained. The 3D map can realize the visualization of obstacles in the three-dimensional space of the working environment, so that the operator can intuitively see the key information such as the location and distance of the obstacle, thereby improving the transparency of operation and reducing the probability of collision between the construction machinery and surrounding obstacles during operation.

[0046] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0048] Figure 1 It is a flowchart of a method for visualizing obstacles in a working environment provided by an embodiment of the present application;

[0049] Figure 2 It is a flowchart of a model training method provided by this application;

[0050] Figure 3 It is a structural schematic diagram of an obstacle visualization device in a working environment provided by the present application;

[0051] Figure 4 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0052] The following description and the accompanying drawings sufficiently illustrate specific embodiments of the present application to enable those skilled in the art to practice them.

[0053] It should be clear that the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0054] When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Instead, they are only examples of devices and methods consistent with some aspects of the present application as detailed in the attached claims.

[0055] In the description of the present application, it should be understood that the terms "first", "second", etc. are used for descriptive purposes only and should not be understood as indicating or implying relative importance. For those of ordinary skill in the art, the specific meanings of the above terms in the present application can be understood according to the specific circumstances. In addition, in the description of the present application, unless otherwise specified, "multiple" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects associated before and after are in an "or" relationship.

[0056] The present application provides a method and device for visualizing obstacles in an operating environment to solve the problems existing in the above-mentioned related technical problems. In the embodiment of the present application, on the one hand, through a pre-trained 2D image-3D point cloud fusion perception model, multi-source data fusion and obstacle identification are performed on the perception data. The perception data is fed back by multiple types of sensors. Through the fusion of multiple types of sensor data, all-round perception of the surrounding environment of the engineering machinery is realized, and obstacles can be detected and identified more accurately, greatly improving the accuracy of the system. At the same time, the pre-trained 2D image-3D point cloud fusion perception model is a neural network for obstacle detection based on deep learning, which can perform real-time data processing under high load and high speed conditions; on the other hand, by projecting obstacle information onto a pre-established 3D map, a target 3D map containing obstacle information of the target obstacle is obtained. The 3D map can realize the visualization of obstacles in the three-dimensional space of the operating environment, so that the operator can intuitively see the key information such as the location and distance of the obstacle, improve the transparency of the operation, and thus reduce the probability of collision between the engineering machinery and the surrounding obstacles during the operation. The following exemplary embodiments are used for detailed description.

[0057] The following will be combined with the attached Figure 1 -Attached Figure 2 , a method for visualizing obstacles in a working environment provided by an embodiment of the present application is introduced in detail. The method can be implemented by a computer program and can be run on an obstacle visualization device in a working environment based on a von Neumann system. The computer program can be integrated into an application or run as an independent tool application.

[0058] See also Figure 1 , which is a flow chart of a method for visualizing obstacles in a working environment provided by an embodiment of the present application. Figure 1 As shown, the method of the embodiment of the present application may include the following steps:

[0059] S101, receiving the perception data of the working environment from various sensors installed on the engineering machinery;

[0060] Among them, construction machinery is a mechanical vehicle that currently needs to be remotely controlled, such as an excavator. The various sensors installed on construction machinery are deployed on the target operating machinery by experts based on practical experience, and can be deployed in multiple different positions of the construction machinery. Various sensors include cameras, lidars, and millimeter-wave radars. Cameras provide high-resolution visual information (such as 2D image data), and lidars provide precise spatial information (such as 3D point cloud data). Lidars acquire 3D point cloud data of the environment by emitting laser pulses and measuring the reflected light, while millimeter-wave radars provide stable distance measurements.

[0061] In some embodiments, multiple sensors are installed at appropriate locations of the engineering machinery, including but not limited to cameras, LiDAR, millimeter-wave radars, etc. For example, the main LiDAR can use DJI livox_hap to sense the working area, and 5 Hesai FT120s are installed around it to sense the surroundings.

[0062] Then, the external parameters of the installed sensors can be calibrated to obtain the transformation relationship of each sensor coordinate system and the position transformation relationship with the construction machinery. The calibrateCamera function of openCV can be used to calculate the rotation and translation matrix between them, so as to unify the data of each sensor into one coordinate system. The internal parameter matrix K describes the internal geometric characteristics of the camera, including focal length, optical center, etc., in the following form:

[0063] K=[f x 0 c x 0f y c y 0 0 1];

[0064] Among them, fx, fy are the focal lengths of the camera in the x-axis and y-axis directions (in pixels), and cx, cy are the coordinates of the optical center (principal point) of the image.

[0065] The external parameters consist of the rotation matrix R and the translation vector T, which are used to describe the position change of the camera coordinate system relative to the world coordinate system:

[0066] R=[r 11 r 12 r 13 r 21 r 22 r 23 r 31 r 32 r 33 ];

[0067] T=[T x T y T z ];

[0068] For a 3D point Pw = [X, Y, Z]T in the world coordinate system, after the camera's projection model, it is finally mapped to a point p = [u, v]T in the image coordinate system. This process can be expressed by the following formula:

[0069] p = K[R|T]Pw;

[0070] Among them, [R|T] is an augmented matrix, that is,

[0071]

[0072] Finally, after the sensor calibration is completed, data collection begins. The camera continuously captures images around the construction machinery, and the laser radar scans the surrounding environment to generate 3D point cloud data. At the same time, the millimeter-wave radar measures the distance from the obstacle to the construction machinery, and transmits the above data to the computer in real time through the vehicle terminal. Among them, the raw data reported by the vehicle terminal has synchronized the data from different sensors in time and pre-processed the data to facilitate subsequent data fusion processing. Data preprocessing includes operations such as denoising and contrast enhancement of camera images to improve image quality, filtering and denoising of laser radar point cloud data, extracting effective point clouds, and performing signal processing on millimeter-wave radar data to extract distance information of obstacles.

[0073] Specifically, the image acquisition frequency formula is: n represents the number of frames acquired, and t represents the time length LiDAR point cloud data representation, where x, y, z coordinate values, mathcalC = {(x i ,y i ,z i ,I i )i=1,2,…,N}.

[0074] In some embodiments, the computer may receive perception data of the working environment from various sensors installed on the engineering machinery.

[0075] S102, performing multi-source data fusion and obstacle recognition on the perception data through a pre-trained 2D image-3D point cloud fusion perception model to obtain obstacle information of target obstacles in the working environment, wherein the pre-trained 2D image-3D point cloud fusion perception model is a neural network for obstacle detection based on deep learning;

[0076] Among them, the environmental perception data includes 2D image data and 3D point cloud data. The pre-trained 2D image-3D point cloud fusion perception model includes a feature fusion module, an obstacle category determination module and an obstacle matching module. The feature fusion module is used to fuse the feature vectors of the 2D image data and the 3D point cloud data. The obstacle category determination module is used to identify the obstacle category of the target obstacle in the working environment according to the fused features. The obstacle matching module is used to associate the obstacle information of the target obstacle according to the feature vectors and obstacle categories of the 2D image data and the 3D point cloud data.

[0077] In some embodiments, a specific process of performing multi-source data fusion and obstacle identification on the perception data through a pre-trained 2D image-3D point cloud fusion perception model to obtain obstacle information of a target obstacle in the working environment includes: extracting feature vectors from the 2D image data and the 3D point cloud data through a feature fusion module to obtain a feature vector of each pixel in the 2D image data and a feature vector of each point cloud position point in the 3D point cloud data; fusing the feature vector of each pixel with the feature vector of each point cloud position point through a feature fusion module to obtain a fused feature; inputting the fused feature into an obstacle category determination module to output the obstacle category of the target obstacle in the working environment; inputting the feature vectors and obstacle categories of the 2D image data and the 3D point cloud data into an obstacle matching module to output obstacle information of the target obstacle.

[0078] For example, I 2D is the feature vector of the 2D image, P 3D The feature representation of the 3D point cloud is obtained, fusion Represents a fusion function. At this time, the function expression of the feature fusion module is:

[0079] F combined =F fusion (I 2D ,P 3D );

[0080] At this time, the function expression of the obstacle category determination module is:

[0081] C obj =g(F combined );

[0082] At this time, the function expression of the obstacle matching module is:

[0083] S match =M(I 2D ,P 3D |C obj ).

[0084] Specifically, the feature vector of each pixel point and the feature vector of each point cloud position point are fused to obtain the fused feature. The specific process includes: pairing the feature vector of each pixel point with the feature vector of each point cloud position point according to the transformation relationship of the position coordinate system of each sensor established in advance to obtain the feature vectors of multiple groups of pixel points and the feature vectors of the point cloud position points under the same position coordinates; fusing the feature vector of each group of pixel points with the feature vector of the point cloud position point to obtain the fused feature.

[0085] In an embodiment of the present application, the specific process of generating a pre-trained 2D image-3D point cloud fusion perception model includes: obtaining a feature fusion module, an obstacle category determination module and an obstacle matching module; integrating the feature fusion module, the obstacle category determination module and the obstacle matching module into a neural network to obtain a 2D image-3D point cloud fusion perception model; collecting historical perception data of the working environment from various sensors on engineering machinery; annotating the historical perception data with obstacle information to obtain a model training sample; and using the model training sample to perform deep learning on the 2D image-3D point cloud fusion perception model to obtain a pre-trained 2D image-3D point cloud fusion perception model.

[0086] S103, projecting the obstacle information into a pre-established 3D map to obtain a target 3D map including the obstacle information of the target obstacle, and displaying the target 3D map.

[0087] Among them, various sensors installed on engineering machinery include multiple cameras and millimeter-wave radars, and the perception data also includes the raw data of the millimeter-wave radar and the data of the lidar.

[0088] In some embodiments, the specific process of projecting obstacle information into a pre-established 3D map includes: obtaining the pixel coordinates of the target obstacle in the 2D image data taken by each camera; calculating the target distance between the target obstacle and the construction machinery based on the original data of the millimeter wave radar and the data of the lidar; determining the three-dimensional spatial coordinates of the target obstacle based on the pixel coordinates and the target distance between the construction machinery; and projecting the obstacle information into the pre-established 3D map based on the three-dimensional spatial coordinates of the target obstacle.

[0089] Specifically, according to the three-dimensional spatial coordinates of the target obstacle, the specific process of projecting the obstacle information into the pre-established 3D map includes: locating a target position point consistent with the three-dimensional spatial coordinates of the target obstacle from the pre-established 3D map; and marking the obstacle information at the target position point.

[0090] Furthermore, the method further includes identifying the target obstacle in a pre-established 3D map according to the obstacle category of the target obstacle.

[0091] In an embodiment of the present application, the specific process of generating a pre-established 3D map includes: converting 3D point cloud data into a format that can be used for map construction to obtain basic map data of the working environment; dividing the three-dimensional space of the working environment into discrete grids; filling the basic map data into the divided grids to obtain a pre-established 3D map.

[0092] For example, given a point P(x,y,z) in a point cloud and a grid resolution of r, the coordinate system in the 3D world can be expressed as:

[0093]

[0094] Obstacle probability estimation, for a grid (i, j, k) if the number of its internal points is N ijk , obstacle occupancy probability P occ (i,j,k) can be expressed as:

[0095]

[0096] In an embodiment of the present application, after the target 3D map is displayed, a real-time collision warning can be performed. When the displayed obstacle information indicates that the distance between the target obstacle and the construction machinery is less than a preset threshold, a warning message is generated; and the warning message is displayed.

[0097] Among them, the present application displays the construction machinery and its surrounding environment in a 3D visualization in the control room or on the operation panel. Obstacles are distinguished by different colors and labels, and key information such as their category and distance are intuitively displayed to improve operational transparency.

[0098] In the embodiment of the present application, on the one hand, through the pre-trained 2D image-3D point cloud fusion perception model, multi-source data fusion and obstacle identification are performed on the perception data. The perception data is fed back by multiple types of sensors. Through the fusion of multiple types of sensor data, all-round perception of the surrounding environment of the construction machinery is realized, and obstacles can be detected and identified more accurately, which greatly improves the accuracy of the system. At the same time, the pre-trained 2D image-3D point cloud fusion perception model is a neural network for obstacle detection based on deep learning, which can perform real-time data processing under high load and high speed conditions; on the other hand, by projecting the obstacle information into a pre-established 3D map, a target 3D map containing the obstacle information of the target obstacle is obtained. The 3D map can realize the visualization of obstacles in the three-dimensional space of the working environment, so that the operator can intuitively see the key information such as the location and distance of the obstacle, thereby improving the transparency of operation and reducing the probability of collision between the construction machinery and surrounding obstacles during operation.

[0099] See also Figure 2 , which is a flow chart of a model training method provided in the embodiment of the present application. Figure 2 As shown, the method of the embodiment of the present application may include the following steps:

[0100] S201, acquiring a feature fusion module, an obstacle category determination module, and an obstacle matching module;

[0101] S202, integrating a feature fusion module, an obstacle category determination module, and an obstacle matching module into a neural network to obtain a 2D image-3D point cloud fusion perception model;

[0102] S203, collecting historical perception data of the working environment from various sensors on the construction machinery;

[0103] S204, annotating obstacle information on historical perception data to obtain model training samples;

[0104] S205, using the model training samples to perform deep learning on the 2D image-3D point cloud fusion perception model to obtain a pre-trained 2D image-3D point cloud fusion perception model.

[0105] In the embodiment of the present application, on the one hand, through the pre-trained 2D image-3D point cloud fusion perception model, multi-source data fusion and obstacle identification are performed on the perception data. The perception data is fed back by multiple types of sensors. Through the fusion of multiple types of sensor data, all-round perception of the surrounding environment of the construction machinery is realized, and obstacles can be detected and identified more accurately, which greatly improves the accuracy of the system. At the same time, the pre-trained 2D image-3D point cloud fusion perception model is a neural network for obstacle detection based on deep learning, which can perform real-time data processing under high load and high speed conditions; on the other hand, by projecting the obstacle information into a pre-established 3D map, a target 3D map containing the obstacle information of the target obstacle is obtained. The 3D map can realize the visualization of obstacles in the three-dimensional space of the working environment, so that the operator can intuitively see the key information such as the location and distance of the obstacle, thereby improving the transparency of operation and reducing the probability of collision between the construction machinery and surrounding obstacles during operation.

[0106] The following is an embodiment of the device of the present application, which can be used to execute the embodiment of the method of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the method of the present application.

[0107] See also Figure 3 , which shows a schematic diagram of the structure of an obstacle visualization device in a working environment provided by an exemplary embodiment of the present application. The obstacle visualization device in the working environment can be implemented as all or part of an electronic device through software, hardware or a combination of both. The device 1 includes a receiving module 10, an identification module 20, and a display module 30.

[0108] The receiving module 10 is used to receive the sensing data of the working environment from various sensors installed on the engineering machinery;

[0109] The recognition module 20 is used to perform multi-source data fusion and obstacle recognition on the perception data through a pre-trained 2D image-3D point cloud fusion perception model to obtain obstacle information of target obstacles in the working environment. The pre-trained 2D image-3D point cloud fusion perception model is a neural network for obstacle detection based on deep learning;

[0110] The display module 30 is used to project the obstacle information into a pre-established 3D map, obtain a target 3D map including the obstacle information of the target obstacle, and display the target 3D map.

[0111] It should be noted that the obstacle visualization device in the working environment provided by the above embodiment only uses the division of the above functional modules as an example when executing the obstacle visualization method in the working environment. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the obstacle visualization device in the working environment provided by the above embodiment and the obstacle visualization method embodiment in the working environment belong to the same concept, and the embodiment and implementation process thereof are detailed in the method embodiment, which will not be repeated here.

[0112] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0113] In the embodiment of the present application, on the one hand, through the pre-trained 2D image-3D point cloud fusion perception model, multi-source data fusion and obstacle identification are performed on the perception data. The perception data is fed back by multiple types of sensors. Through the fusion of multiple types of sensor data, all-round perception of the surrounding environment of the construction machinery is realized, and obstacles can be detected and identified more accurately, which greatly improves the accuracy of the system. At the same time, the pre-trained 2D image-3D point cloud fusion perception model is a neural network for obstacle detection based on deep learning, which can perform real-time data processing under high load and high speed conditions; on the other hand, by projecting the obstacle information into a pre-established 3D map, a target 3D map containing the obstacle information of the target obstacle is obtained. The 3D map can realize the visualization of obstacles in the three-dimensional space of the working environment, so that the operator can intuitively see the key information such as the location and distance of the obstacle, thereby improving the transparency of operation and reducing the probability of collision between the construction machinery and surrounding obstacles during operation.

[0114] The present application also provides a computer-readable medium having program instructions stored thereon, which, when executed by a processor, implement the obstacle visualization method in the working environment provided by the above-mentioned various method embodiments.

[0115] The present application also provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the obstacle visualization method in the working environment of each of the above method embodiments.

[0116] See also Figure 4 , is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 4 As shown, the electronic device 1000 may include: at least one processor 1001 , at least one network interface 1004 , a user interface 1003 , a memory 1005 , and at least one communication bus 1002 .

[0117] The communication bus 1002 is used to realize the connection and communication between these components.

[0118] The user interface 1003 may include a display screen (Display) and a camera (Camera), and the optional user interface 1003 may also include a standard wired interface and a wireless interface.

[0119] The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a WI-FI interface).

[0120] Among them, the processor 1001 may include one or more processing cores. The processor 1001 uses various interfaces and lines to connect various parts within the entire electronic device 1000, and executes various functions and processes data of the electronic device 1000 by running or executing instructions, programs, code sets or instruction sets stored in the memory 1005, and calling data stored in the memory 1005. Optionally, the processor 1001 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). The processor 1001 can integrate one or a combination of a central processing unit (Central Processing Unit, CPU), a graphics processing unit (Graphics Processing Unit, GPU) and a modem. Among them, the CPU mainly processes operating devices, user interfaces and applications, etc.; the GPU is responsible for rendering and drawing the content to be displayed on the display screen; the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 1001, and it can be implemented separately through a chip.

[0121] Among them, the memory 1005 may include a random access memory (Random Access Memory, RAM) and may also include a read-only memory (Read-Only Memory). Optionally, the memory 1005 includes a non-transitory computer-readable storage medium. The memory 1005 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 1005 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing the operating device, instructions for at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area may store data involved in the above-mentioned various method embodiments, etc. The memory 1005 may also be optionally at least one storage device located away from the aforementioned processor 1001. As Figure 4 As shown, the memory 1005 as a computer storage medium may include an operating device, a network communication module, a user interface module, and an obstacle visualization application in the working environment.

[0122] exist Figure 4 In the electronic device 1000 shown, the user interface 1003 is mainly used to provide an input interface for the user and obtain the data input by the user; and the processor 1001 can be used to call the obstacle visualization application in the working environment stored in the memory 1005, and specifically perform the following operations:

[0123] Receive data about the working environment from various sensors installed on the construction machinery;

[0124] Through the pre-trained 2D image-3D point cloud fusion perception model, multi-source data fusion and obstacle recognition are performed on the perception data to obtain obstacle information of the target obstacles in the working environment. The pre-trained 2D image-3D point cloud fusion perception model is a neural network based on deep learning for obstacle detection;

[0125] The obstacle information is projected onto a pre-established 3D map to obtain a target 3D map including the obstacle information of the target obstacle, and the target 3D map is displayed.

[0126] In one embodiment, when the processor 1001 executes the pre-trained 2D image-3D point cloud fusion perception model to perform multi-source data fusion and obstacle recognition on the perception data to obtain obstacle information of the target obstacle in the working environment, the processor 1001 specifically performs the following operations:

[0127] Extract feature vectors from 2D image data and 3D point cloud data through a feature fusion module to obtain feature vectors of each pixel in the 2D image data and feature vectors of each point cloud position point in the 3D point cloud data;

[0128] The feature vector of each pixel point and the feature vector of each point cloud position point are fused through the feature fusion module to obtain the fused feature;

[0129] The fused features are input into the obstacle category determination module, and the obstacle category of the target obstacle in the working environment is output;

[0130] The feature vectors and obstacle categories of the 2D image data and 3D point cloud data are input into the obstacle matching module, and the obstacle information of the target obstacle is output.

[0131] In one embodiment, when the processor 1001 fuses the feature vector of each pixel point with the feature vector of each point cloud position point to obtain the fused feature, the processor 1001 specifically performs the following operations:

[0132] According to the pre-established transformation relationship of the position coordinate system of each sensor, the feature vector of each pixel point is paired with the feature vector of each point cloud position point to obtain feature vectors of multiple groups of pixel points and feature vectors of point cloud position points under the same position coordinates;

[0133] The feature vector of each group of pixels is fused with the feature vector of the point cloud position point to obtain the fused feature.

[0134] In one embodiment, when the processor 1001 projects the obstacle information into the pre-established 3D map, the processor 1001 specifically performs the following operations:

[0135] Get the pixel coordinates of the target obstacle in the 2D image data taken by each camera;

[0136] Calculate the target distance between the target obstacle and the construction machinery based on the raw data of the millimeter wave radar and the data of the laser radar;

[0137] Determine the three-dimensional spatial coordinates of the target obstacle based on the pixel coordinates and the target distance between the construction machinery;

[0138] According to the three-dimensional spatial coordinates of the target obstacle, the obstacle information is projected into the pre-established 3D map.

[0139] In one embodiment, when the processor 1001 projects the obstacle information onto a pre-established 3D map according to the three-dimensional spatial coordinates of the target obstacle, the processor 1001 specifically performs the following operations:

[0140] From the pre-established 3D map, locate the target position point that is consistent with the three-dimensional spatial coordinates of the target obstacle;

[0141] Mark the obstacle information at the target location.

[0142] In one embodiment, after displaying the target 3D map, the processor 1001 further performs the following operations:

[0143] When the displayed obstacle information indicates that the distance between the target obstacle and the construction machinery is less than a preset threshold, a warning message is generated;

[0144] Display warning information.

[0145] In one embodiment, when generating a pre-trained 2D image-3D point cloud fusion perception model, the processor 1001 specifically performs the following operations:

[0146] Obtain a feature fusion module, an obstacle category determination module, and an obstacle matching module;

[0147] The feature fusion module, obstacle category determination module and obstacle matching module are integrated into the neural network to obtain a 2D image-3D point cloud fusion perception model;

[0148] Collect historical perception data of the working environment from various sensors on construction machinery;

[0149] Label the historical perception data with obstacle information to obtain model training samples;

[0150] The model training samples are used to perform deep learning on the 2D image-3D point cloud fusion perception model to obtain a pre-trained 2D image-3D point cloud fusion perception model.

[0151] In one embodiment, when the processor 1001 generates a pre-established 3D map, the processor 1001 specifically performs the following operations:

[0152] Convert 3D point cloud data into a format that can be used for map construction to obtain basic map data of the working environment;

[0153] Divide the three-dimensional space of the working environment into discrete grids;

[0154] Fill the divided grids with basic map data to obtain a pre-built 3D map.

[0155] In the embodiment of the present application, on the one hand, through the pre-trained 2D image-3D point cloud fusion perception model, multi-source data fusion and obstacle identification are performed on the perception data. The perception data is fed back by multiple types of sensors. Through the fusion of multiple types of sensor data, all-round perception of the surrounding environment of the construction machinery is realized, and obstacles can be detected and identified more accurately, which greatly improves the accuracy of the system. At the same time, the pre-trained 2D image-3D point cloud fusion perception model is a neural network for obstacle detection based on deep learning, which can perform real-time data processing under high load and high speed conditions; on the other hand, by projecting the obstacle information into a pre-established 3D map, a target 3D map containing the obstacle information of the target obstacle is obtained. The 3D map can realize the visualization of obstacles in the three-dimensional space of the working environment, so that the operator can intuitively see the key information such as the location and distance of the obstacle, thereby improving the transparency of operation and reducing the probability of collision between the construction machinery and surrounding obstacles during operation.

[0156] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware through a computer program, and the program for visualizing obstacles in the working environment can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium of the program for visualizing obstacles in the working environment can be a disk, an optical disk, a read-only storage memory, or a random access memory, etc.

[0157] The above disclosure is only the preferred embodiment of the present application, which certainly cannot be used to limit the scope of rights of the present application. Therefore, equivalent changes made according to the claims of the present application are still within the scope covered by the present application.

Claims

1. A method for visualizing obstacles in a working environment, characterized in that: The method comprises: Receive data about the working environment from various sensors installed on the construction machinery; Obstacle information of target obstacles in the working environment is obtained by performing multi-source data fusion and obstacle recognition on the perception data through a pre-trained 2D image-3D point cloud fusion perception model, wherein the pre-trained 2D image-3D point cloud fusion perception model is a neural network for obstacle detection based on deep learning; The obstacle information is projected onto a pre-established 3D map to obtain a target 3D map including the obstacle information of the target obstacle, and the target 3D map is displayed.

2. The method according to claim 1, characterized in that The environmental perception data includes 2D image data and 3D point cloud data. The pre-trained 2D image-3D point cloud fusion perception model includes a feature fusion module, an obstacle category determination module and an obstacle matching module. The feature fusion module is used to fuse the feature vectors of the 2D image data and the 3D point cloud data. The obstacle category determination module is used to identify the obstacle category of the target obstacle in the working environment according to the fused features. The obstacle matching module is used to associate the obstacle information of the target obstacle according to the feature vectors of the 2D image data and the 3D point cloud data and the obstacle category.

3. The method according to claim 2, characterized in that The pre-trained 2D image-3D point cloud fusion perception model is used to perform multi-source data fusion and obstacle recognition on the perception data to obtain obstacle information of target obstacles in the working environment, including: Extracting feature vectors from the 2D image data and the 3D point cloud data by the feature fusion module to obtain feature vectors of each pixel point in the 2D image data and feature vectors of each point cloud position point in the 3D point cloud data; The feature fusion module fuses the feature vector of each pixel point with the feature vector of each point cloud position point to obtain a fusion feature; Inputting the fusion feature into the obstacle category determination module, and outputting the obstacle category of the target obstacle existing in the working environment; The feature vectors of the 2D image data and the 3D point cloud data and the obstacle category are input into the obstacle matching module, and the obstacle information of the target obstacle is output.

4. The method according to claim 3, characterized in that The step of fusing the feature vector of each pixel point with the feature vector of each point cloud position point to obtain a fused feature includes: According to the pre-established transformation relationship of the position coordinate system of each sensor, the feature vector of each pixel point is paired with the feature vector of each point cloud position point to obtain feature vectors of multiple groups of pixel points and feature vectors of point cloud position points at the same position coordinates; The feature vector of each group of pixels is fused with the feature vector of the point cloud position point to obtain the fused feature.

5. The method according to claim 1, characterized in that The various sensors disposed on the engineering machinery include multiple cameras and millimeter wave radars, and the perception data also includes raw data of the millimeter wave radars and data of the laser radar; The step of projecting the obstacle information into a pre-established 3D map comprises: Obtain pixel coordinates of the target obstacle in the 2D image data taken by each camera; Calculating the target distance between the target obstacle and the engineering machinery according to the raw data of the millimeter wave radar and the data of the laser radar; Determining the three-dimensional spatial coordinates of the target obstacle according to the pixel coordinates and the target distance between the engineering machinery; According to the three-dimensional spatial coordinates of the target obstacle, the obstacle information is projected into a pre-established 3D map.

6. The method according to claim 5, characterized in that The step of projecting the obstacle information onto a pre-established 3D map according to the three-dimensional spatial coordinates of the target obstacle comprises: Locating a target position point that is consistent with the three-dimensional spatial coordinates of the target obstacle from a pre-established 3D map; The obstacle information is marked at the target location point.

7. The method according to any one of claims 2 to 6, characterized in that: Follow these steps to generate a pre-built 3D map, including: Converting the 3D point cloud data into a format that can be used for map construction to obtain basic map data of the working environment; Dividing the three-dimensional space of the working environment into discrete grids; The basic map data is filled into the divided grids to obtain a pre-established 3D map.

8. The method according to claim 1, characterized in that: Follow these steps to generate a pre-trained 2D image-3D point cloud fusion perception model, including: Obtain a feature fusion module, an obstacle category determination module, and an obstacle matching module; Integrating the feature fusion module, obstacle category determination module and obstacle matching module into a neural network to obtain a 2D image-3D point cloud fusion perception model; Collect historical perception data of the working environment from various sensors on construction machinery; Annotating the historical perception data with obstacle information to obtain a model training sample; The model training samples are used to perform deep learning on the 2D image-3D point cloud fusion perception model to obtain a pre-trained 2D image-3D point cloud fusion perception model.

9. The method according to claim 1, characterized in that: After displaying the target 3D map, the method further includes: When the displayed obstacle information indicates that the distance between the target obstacle and the engineering machinery is less than a preset threshold, generating a warning message; The warning information is displayed.

10. An obstacle visualization device in a working environment, characterized in that: The device comprises: A receiving module is used to receive the perception data of the working environment from various sensors installed on the engineering machinery; A recognition module, used to perform multi-source data fusion and obstacle recognition on the perception data through a pre-trained 2D image-3D point cloud fusion perception model, so as to obtain obstacle information of a target obstacle existing in the working environment, wherein the pre-trained 2D image-3D point cloud fusion perception model is a neural network for obstacle detection based on deep learning; The display module is used to project the obstacle information into a pre-established 3D map to obtain a target 3D map including the obstacle information of the target obstacle, and to display the target 3D map.

Images