Excavator environment virtual view angle display method and device

Abstract

The invention discloses an environment virtual view angle display method and device for an excavator, and the method comprises the steps: reading sensor data collected by various sensors disposed on the excavator in advance in an operation process of the excavator; calculating a target pose of the excavator and constructing a real-time rgb point cloud map of a surrounding scene of the excavator by combining a mapping positioning algorithm SLAM based on the sensor data and pre-calibrated internal and external parameters of the sensor, performing surface reconstruction on the real-time rgb point cloud map, and converting the real-time rgb point cloud map into a real-time three-dimensional scene of the surrounding scene of the excavator; projecting the original image to a corresponding position of a real-time three-dimensional scene to obtain a full three-dimensional scene of the excavator; and according to the full three-dimensional scene of the excavator, rendering an excavator operation scene under a preset visual angle in real time. By the adoption of the system, all-around display of the surrounding environment of the excavator can be achieved, meanwhile, visual angle and depth information which can be adjusted at will are provided, and therefore the efficiency and safety of remote operation of the excavator are improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method and device for displaying a virtual perspective of an excavator's environment. Background Art

[0002] In the field of construction machinery, excavators are a key operating equipment and are widely used in industries such as construction, mining and infrastructure construction. At present, the remote control system of excavators can display the surrounding environment of the excavator, allowing the operator to observe the surrounding environment of the excavator in real time.

[0003] In the related art, the remote control excavator system usually includes multiple cameras with fixed viewing angles to capture the environmental images around the excavator. However, the existing technology has the following technical defects: (1) The camera field of view and screen display are limited and cannot provide a sufficiently wide field of view; (2) The camera field of view of the existing system is fixed and cannot be dynamically adjusted according to the needs of the operator, making it difficult for the operator to obtain a full range of peripheral vision in a complex environment; (3) The main screen of the existing excavator remote control system lacks depth information. When displaying the environment, it cannot accurately reflect the distance relationship of objects, which affects the driver's judgment of the environment. Summary of the invention

[0004] The embodiments of the present application provide a method and device for displaying a virtual perspective of an excavator's environment. In order to have a basic understanding of some aspects of the disclosed embodiments, a simple summary is given below. This summary is not a general review, nor is it intended to identify key / important components or describe the scope of protection of these embodiments. Its only purpose is to present some concepts in a simple form as a preface to the detailed description that follows.

[0005] In a first aspect, an embodiment of the present application provides a method for displaying a virtual perspective of an excavator environment, the method comprising:

[0006] During the operation of the excavator, the sensor data collected by various sensors pre-installed on the excavator are read in real time. The sensor data includes original images, original point clouds, RTK data and IMU data;

[0007] Based on sensor data and pre-calibrated sensor internal and external parameters, combined with the mapping and positioning algorithm SLAM, the target posture of the excavator is calculated and a real-time RGB point cloud map of the scene around the excavator is constructed, and the real-time RGB point cloud map is reconstructed and converted into a real-time three-dimensional scene of the scene around the excavator;

[0008] According to the target posture and the pre-calibrated internal and external parameters of the sensor, the original image is projected to the corresponding position of the real-time 3D scene to obtain the full 3D scene of the excavator;

[0009] Based on the full 3D scene of the excavator, the excavator operation scene under the preset perspective is rendered in real time.

[0010] Optionally, before calculating the target pose of the excavator based on the sensor data and the pre-calibrated sensor internal and external parameters and combining the SLAM mapping and positioning algorithm, the following is also included:

[0011] According to the sensor data, various sensors are calibrated with internal and external parameters to obtain internal and external parameter data that unify the sensor data into the same coordinate system; the internal parameter calibration is used to determine the parameters of each sensor, and the external parameter calibration is used to determine the precise position and direction of the sensor relative to other sensors or the world coordinate system;

[0012] The internal and external parameter data are used as the pre-calibrated internal and external parameters of the sensor.

[0013] Optionally, various sensors include cameras and lidars. The intrinsic calibration of the camera can be described as:

[0014]

[0015] Where u and v are the image coordinates in the image captured by the camera, and f x and f y are the focal lengths in the horizontal and vertical directions, c x and c y is the optical axis coordinate, z is the distance from the three-dimensional point in the horizontal direction X and the vertical direction Y in the imaging plane;

[0016] The calibration process of the camera and lidar is as follows:

[0017]

[0018] Among them, R is a 3×3 rotation matrix, t is a 3×1 translation vector, z is the depth corresponding to the normalized plane of the camera, and [X, Y, Z, 1] is a point cloud point represented in a homogeneous manner. When there are multiple pairs of matching points between the point cloud and the image, the equations can be immediately solved for R and T.

[0019] Optionally, the sensor data includes RTK data and IMU data, RTK data is based on real-time motion measurement technology to achieve centimeter-level positioning data, and IMU data is collected by an inertial measurement unit;

[0020] Based on sensor data and pre-calibrated sensor internal and external parameters, the target position of the excavator is calculated in combination with the SLAM mapping and positioning algorithm, including:

[0021] According to the pre-calibrated sensor internal and external parameters, the RTK data is used as the initial value, and the LiDAR image imu odometer is constructed through the SLAM algorithm to obtain the target position and posture of the excavator;

[0022] Among them, the target posture calculation formula of the excavator is:

[0023] in,

[0024] min J(x) is the target position of the excavator, x is the sensor data, including position, attitude, speed, and bias information, N is the number of observations, and r i (x) is the residual vector of the ith observation, reflecting the difference between the observed value and the predicted value based on the current state estimate, W i is the weight matrix of the i-th observation, which is defined by the inverse covariance matrix of the observation noise in the SLAM algorithm, and T is the internal and external parameters of the sensor.

[0025] Optionally, based on the sensor data and pre-calibrated sensor internal and external parameters, and in combination with a mapping and positioning algorithm SLAM, the target posture of the excavator is calculated and a real-time RGB point cloud map of the scene around the excavator is constructed, including:

[0026] Extracting key frames according to the original image and the original point cloud read in real time;

[0027] Using the SLAM algorithm and the key frames to construct a local map;

[0028] Matching the local map with the current frame to determine the target position of the excavator;

[0029] Obtaining historical frames and observations collected by a camera installed on the excavator during historical operations;

[0030] Based on the historical frames and their observations, construct a factor graph, wherein the factor graph is used to represent the relationship between the sensor observations and the excavator posture, wherein the nodes in the factor graph represent the postures and the edges represent the observations;

[0031] Pre-integrate the original image, original point cloud, and IMU data in the sensor data to obtain an integration result;

[0032] Adding the integration result as a factor to the factor graph to obtain a target factor graph;

[0033] Globally optimizing a plurality of historical frames through the target factor graph to optimize the target pose to a global pose;

[0034] The local map is adjusted according to the global posture and the global map is constructed to obtain a real-time RGB point cloud map of the scene around the excavator.

[0035] Optionally, the surface reconstruction of the real-time RGB point cloud map is converted into a real-time three-dimensional scene of the scene around the excavator, including:

[0036] Meshing the global map by using the TSDF algorithm to generate a three-dimensional scene that can reflect the surrounding environment of the excavator in real time;

[0037] The three-dimensional scene that can reflect the surrounding environment of the excavator in real time is used as the real-time three-dimensional scene of the surrounding scene of the excavator.

[0038] Optionally, the method further includes:

[0039] The update equation of the TSDF algorithm is:

[0040] TSDF new (v) = w·(D(v)-D obs )+(1-w)·TSDF old (v);

[0041] Among them, TSDF new (v) is the meshing result, v is each voxel in the global map, w is the fusion weight, D(v) is the true distance from the voxel center to the nearest surface, Dobs is the observed distance, TSDFold(v) and TSDFnew(v) ​​are the old and new TSDF values ​​of the voxel, respectively. A voxel is an analogue of a pixel in three-dimensional space.

[0042] Optionally, according to the target posture and the pre-calibrated internal and external parameters of the sensor, the original image is projected to the corresponding position of the real-time three-dimensional scene to obtain the full three-dimensional scene of the excavator, including:

[0043] Through the pre-calibrated sensor internal and external parameters, the target posture is converted to the excavator's three-dimensional coordinate system to obtain the excavator's final posture information;

[0044] The original image is associated with the final pose information to determine the spatial position of each pixel in the original image in the three-dimensional coordinate system of the excavator;

[0045] According to the spatial position of each pixel point in the original image in the three-dimensional coordinate system of the excavator, each pixel point in the original image is projected to the same spatial position of the real-time three-dimensional scene to obtain the full three-dimensional scene of the excavator.

[0046] Optionally, the spatial position calculation formula of each pixel point in the original image in the three-dimensional coordinate system of the excavator is:

[0047] P world =R×P camera +t;

[0048] Among them, P world is the spatial position of each pixel in the original image in the three-dimensional coordinate system of the excavator, Pcamera is the coordinate of the pixel point in the original image in the camera coordinate system, R and t are the rotation matrix and translation vector of the image point cloud extrinsic parameters in the pre-calibrated sensor extrinsic parameters.

[0049] In a second aspect, an embodiment of the present application provides an environment virtual perspective display device for an excavator, the device comprising:

[0050] During the operation of the excavator, the sensor data collected by various sensors pre-installed on the excavator are read in real time. The sensor data includes original images, original point clouds, RTK data and IMU data;

[0051] Based on sensor data and pre-calibrated sensor internal and external parameters, the target position of the excavator is calculated in combination with the SLAM mapping and positioning algorithm;

[0052] According to the target posture and the pre-calibrated internal and external parameters of the sensor, the original image is projected to the corresponding position of the real-time three-dimensional scene to obtain the full three-dimensional scene of the excavator;

[0053] Based on the full 3D scene of the excavator, the excavator operation scene under the preset perspective is rendered in real time.

[0054] The technical solution provided by the embodiments of the present application may have the following beneficial effects:

[0055] In the embodiments of the present application, on the one hand, the present application projects the original image to the corresponding position of the real-time three-dimensional scene according to the target posture and the pre-calibrated internal and external parameters of the sensor, so as to obtain a full three-dimensional scene of the excavator. The full three-dimensional scene can simulate the actual surrounding environment of the excavator and realize a full display of the surrounding environment of the excavator. On the other hand, based on the full three-dimensional scene of the excavator, the excavator operation scene under a preset perspective can be rendered in real time. The preset perspective can be freely set, providing a freely adjustable perspective and depth information, thereby improving the efficiency and safety of remote operation of the excavator.

[0056] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0058] Figure 1 It is a flow chart of a method for displaying a virtual perspective of an excavator environment provided by an embodiment of the present application;

[0059] Figure 2 is a factor graph provided by this application;

[0060] Figure 3 It is a structural schematic diagram of an environmental virtual perspective display device for an excavator provided by the present application;

[0061] Figure 4 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0062] The following description and the accompanying drawings sufficiently illustrate specific embodiments of the present application to enable those skilled in the art to practice them.

[0063] It should be clear that the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0064] When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Instead, they are only examples of devices and methods consistent with some aspects of the present application as detailed in the attached claims.

[0065] In the description of the present application, it should be understood that the terms "first", "second", etc. are used for descriptive purposes only and should not be understood as indicating or implying relative importance. For those of ordinary skill in the art, the specific meanings of the above terms in the present application can be understood according to the specific circumstances. In addition, in the description of the present application, unless otherwise specified, "multiple" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects associated before and after are in an "or" relationship.

[0066] The present application provides a method and device for displaying a virtual perspective of an excavator's environment to solve the problems existing in the above-mentioned related technical issues. In an embodiment of the present application, on the one hand, the present application projects the original image to the corresponding position of the real-time three-dimensional scene according to the target posture and the pre-calibrated internal and external parameters of the sensor to obtain a full three-dimensional scene of the excavator. The full three-dimensional scene can simulate the actual surrounding environment of the excavator and realize a full display of the surrounding environment of the excavator; on the other hand, based on the full three-dimensional scene of the excavator, the excavator operation scene under a preset perspective can be rendered in real time. The preset perspective can be freely set, providing a freely adjustable perspective and depth information, thereby improving the efficiency and safety of the excavator's remote operation. The following is a detailed description using an exemplary embodiment.

[0067] The following will be combined with the attached Figure 1 -Attached Figure 2 , a method for displaying a virtual perspective of an excavator's environment provided by an embodiment of the present application is introduced in detail. The method can be implemented by a computer program and can be run on a device for displaying a virtual perspective of an excavator's environment based on a von Neumann system. The computer program can be integrated into an application or run as an independent tool application.

[0068] See also Figure 1 , which is a flow chart of a method for displaying a virtual perspective of an excavator environment according to an embodiment of the present application. Figure 1 As shown, the method of the embodiment of the present application may include the following steps:

[0069] S101, during the operation of the excavator, reading in real time sensor data collected by various sensors pre-installed on the excavator, the sensor data including original images, original point clouds, RTK data and IMU data;

[0070] Among them, an excavator is a heavy-duty construction machinery used to dig soil, rocks and other materials. Sensor data is the ability to detect and respond to physical quantities (such as temperature, pressure, position, speed, etc.) in the excavator and its surrounding environment. Various types of sensors refer to various types of sensors installed on excavators for different detection purposes, such as cameras, IMU (inertial measurement unit), GNSS-RTK (global navigation satellite system real-time motion measurement technology), and lidar. The raw image is the unprocessed image data captured by the camera sensor, which usually contains the scene information within the field of view of the excavator.

[0071] In an embodiment of the present application, for an excavator that needs to be remotely operated, the excavator is equipped with a variety of sensors, including high-definition cameras, laser radars, IMUs, and GNSS-RTK, for collecting environmental and mechanical status data to obtain sensor data. The goal of the present application is to provide an operator with a real-time, interactive three-dimensional view based on the sensor data so that the excavator can be accurately controlled.

[0072] In the embodiment of the present application, during the operation of the excavator, the system continuously reads data collected by various sensors from the excavator's intelligent control box in real time, including but not limited to raw images. These data are crucial for realizing remote control, environmental perception and operation decision-making of the excavator.

[0073] For example, the intelligent control box of the excavator uses high-definition cameras, lidar, IMU, GNSS-RTK and other sensors around the excavator to collect 360-degree images, point clouds and sensor data, ensure the time synchronization of each sensor during the data collection process, and use timestamps or synchronization signals to calibrate the data to ensure data consistency. The collected 360-degree images, point clouds and sensor data are stored in the intelligent control box.

[0074] S102, based on the sensor data and the pre-calibrated sensor internal and external parameters, and in combination with the SLAM mapping and positioning algorithm, the target posture of the excavator is calculated and a real-time RGB point cloud map of the scene around the excavator is constructed, and the real-time RGB point cloud map is surface reconstructed to convert it into a real-time three-dimensional scene of the scene around the excavator;

[0075] In some embodiments, the process of generating pre-calibrated sensor internal and external parameters includes: performing internal and external parameter calibration on various types of sensors according to sensor data, and obtaining internal and external parameter data that unifies the data of various types of sensors into the same coordinate system; wherein the internal parameter calibration is used to determine the inherent parameters of each type of sensor, and the external parameter calibration is used to determine the precise position and direction of the sensor relative to other sensors or the world coordinate system; and the internal and external parameter data are used as the pre-calibrated sensor internal and external parameters.

[0076] Specifically, various sensors include cameras and lidars. The intrinsic calibration of cameras can be described as:

[0077]

[0078] Where u and v are the image coordinates in the image captured by the camera, and f x and f y are the focal lengths in the horizontal and vertical directions, c x and c y is the optical axis coordinate, Z is the distance from the three-dimensional point in the horizontal direction X and the vertical direction Y in the imaging plane;

[0079] The calibration process of the camera and lidar is as follows:

[0080]

[0081] Among them, R is a 3×3 rotation matrix, t is a 3×1 translation vector, z is the depth corresponding to the normalized plane of the camera, and [X, Y, Z, 1] is a point cloud point represented in a homogeneous manner. When there are multiple pairs of matching points between the point cloud and the image, the equations can be immediately solved for R and T.

[0082] Among them, the sensor data includes RTK data and IMU data. RTK data is based on real-time motion measurement technology to achieve centimeter-level positioning data, and IMU data is collected by an inertial measurement unit.

[0083] In the embodiment of the present application, the process of calculating the target posture of the excavator based on the sensor data and the pre-calibrated sensor internal and external parameters, and combining the mapping and positioning algorithm SLAM includes: according to the pre-calibrated sensor internal and external parameters, using the RTK data as the initial value, through the SLAM algorithm, constructing the laser radar image imu odometer to obtain the target posture of the excavator; wherein the target posture calculation formula of the excavator is:

[0084] in,

[0085] min J(x) is the target position of the excavator, x is the sensor data, including position, attitude, speed, and bias information, N is the number of observations, and r i (x) is the residual vector of the ith observation, reflecting the difference between the observed value and the predicted value based on the current state estimate, W i is the weight matrix of the i-th observation, which is defined by the inverse covariance matrix of the observation noise in the SLAM algorithm, and T is the internal and external parameters of the sensor.

[0086] In an embodiment of the present application, the specific process of calculating the target pose of the excavator and constructing a real-time RGB point cloud map of the scene around the excavator based on the sensor data and the pre-calibrated sensor internal and external parameters, and combining the mapping and positioning algorithm SLAM includes: extracting key frames according to the original image and the original point cloud read in real time; using the SLAM algorithm and the key frames to construct a local map; matching the local map with the current frame to determine the target pose of the excavator; obtaining historical frames and observations collected by the camera set on the excavator during historical operations; based on the historical frames and their observations, constructing a factor graph, the factor graph is used to represent the relationship between sensor observations and excavator pose, the nodes in the factor graph represent poses, and the edges represent observations; pre-integrating the original image, original point cloud, and IMU data in the sensor data to obtain an integral result; adding the integral result as a factor to the factor graph to obtain a target factor graph; globally optimizing multiple historical frames through the target factor graph to optimize the target pose to a global pose; adjusting the local map according to the global pose and constructing a global map to obtain a real-time RGB point cloud map of the scene around the excavator.

[0087] In an embodiment of the present application, the specific process of surface reconstruction of the real-time RGB point cloud map and converting it into a real-time three-dimensional scene of the scene around the excavator includes: meshing the global map through the TSDF algorithm to generate a three-dimensional scene that can reflect the surrounding environment of the excavator in real time; using the three-dimensional scene that can reflect the surrounding environment of the excavator in real time as the real-time three-dimensional scene of the scene around the excavator.

[0088] Among them, factor graphs such as Figure 2 As shown in the figure, the hollow dots represent the pose, the solid dots represent the landmarks, and the lines between the pose and the landmarks represent the observations.

[0089] Specifically, according to the target factor graph, the specific process of generating a real-time three-dimensional scene includes: globally optimizing the target factor graph; using the optimized posture information to construct a global map of the environment around the excavator, and the global map is used to reflect the dynamic changes of the excavator in the environment; meshing the global map through the TSDF algorithm to generate a three-dimensional scene that can reflect the environment around the excavator in real time; and using the three-dimensional scene that can reflect the environment around the excavator in real time as the real-time three-dimensional scene.

[0090] Specifically, the update equation of the TSDF algorithm is:

[0091] TSDF new (v) = w·(D(v)-D obs )+(1-w)·TSDF old (v);

[0092] Among them, TSDF new (v) is the meshing result, v is each voxel in the global map, w is the fusion weight, D(v) is the true distance from the voxel center to the nearest surface, Dobs is the observed distance, TSDFold(v) and TSDFnew(v) ​​are the old and new TSDF values ​​of the voxel, respectively. A voxel is an analogue of a pixel in three-dimensional space.

[0093] S103, projecting the original image to a corresponding position of the real-time three-dimensional scene according to the target posture and the pre-calibrated internal and external parameters of the sensor to obtain a full three-dimensional scene of the excavator;

[0094] In some embodiments, the original image is projected to the corresponding position of the real-time three-dimensional scene according to the target posture and pre-calibrated internal and external parameters of the sensor to obtain the specific process of the full three-dimensional scene of the excavator, including: converting the target posture to the three-dimensional coordinate system of the excavator through the pre-calibrated internal and external parameters of the sensor to obtain the final posture information of the excavator; associating the original image with the final posture information to determine the spatial position of each pixel point in the original image in the three-dimensional coordinate system of the excavator; according to the spatial position of each pixel point in the original image in the three-dimensional coordinate system of the excavator, projecting each pixel point in the original image to the same spatial position of the real-time three-dimensional scene to obtain the full three-dimensional scene of the excavator.

[0095] The calculation formula for the spatial position of each pixel in the original image in the three-dimensional coordinate system of the excavator is:

[0096] Pworld =R×P camera +t

[0097] Among them, P world is the spatial position of each pixel in the original image in the three-dimensional coordinate system of the excavator, P camera is the coordinate of the pixel point in the camera coordinate system in the original image, R and t are the rotation matrix and translation vector of the image point cloud extrinsic parameters in the pre-calibrated sensor extrinsic parameters.

[0098] S104, based on the full three-dimensional scene of the excavator, rendering the excavator operation scene under a preset viewing angle in real time.

[0099] The preset perspective may be a conventional perspective of an excavator operator, such as a perspective in an excavator cab, a bird's-eye view, or a specific task perspective.

[0100] In some embodiments, the specific process of rendering the excavator operation scene under a preset perspective in real time based on the full 3D scene of the excavator includes: importing the full 3D scene and the excavator model into a real-time rendering engine, such as OpenGL or Unity3D, setting rendering parameters, including lighting, texture, shadow, etc., to ensure the authenticity and accuracy of the rendering effect, adjusting the position and direction of the virtual camera according to the preset perspective to align it with the current posture of the excavator and the operation scene, and the rendering engine renders the image of the excavator operation scene in real time according to the settings of the virtual camera. This includes the excavator model, the environment terrain, and possible dynamic elements, such as moving objects or changing lighting conditions.

[0101] Furthermore, the operator can interact with the rendered 3D scene through the user interface, such as changing the viewing angle, zooming or panning to obtain the best view required. The operator makes decisions based on the rendered view and sends control instructions to the excavator for precise control. The system continuously loops through the steps of data collection, fusion, rendering and control to maintain the real-time and accuracy of the view.

[0102] Through the embodiment of the present application, the operator can obtain a realistic, real-time updated view of the excavator operation scene, thereby achieving a full display and precise control of the excavator. This technology can not only improve the efficiency and safety of operation, but also be used for training and simulation operations to improve the operator's skills and ability to deal with complex scenarios.

[0103] It should be noted that the method of the present application can adapt to new remote operation modes such as VR, laying the foundation for subsequent VR operations.

[0104] In the embodiments of the present application, on the one hand, the present application projects the original image to the corresponding position of the real-time three-dimensional scene according to the target posture and the pre-calibrated internal and external parameters of the sensor, so as to obtain a full three-dimensional scene of the excavator. The full three-dimensional scene can simulate the actual surrounding environment of the excavator and realize a full display of the surrounding environment of the excavator. On the other hand, based on the full three-dimensional scene of the excavator, the excavator operation scene under a preset perspective can be rendered in real time. The preset perspective can be freely set, providing a freely adjustable perspective and depth information, thereby improving the efficiency and safety of remote operation of the excavator.

[0105] The following is an embodiment of the device of the present application, which can be used to execute the embodiment of the method of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the method of the present application.

[0106] See also Figure 3 , which shows a schematic diagram of the structure of an environment virtual perspective display device for an excavator provided by an exemplary embodiment of the present application. The environment virtual perspective display device for an excavator can be implemented as all or part of an electronic device through software, hardware or a combination of both. The device 1 includes a sensor data reading module 10, a posture construction module 20, an image projection module 30, and a work scene rendering module 40.

[0107] The sensor data reading module 10 is used to read the sensor data collected by various sensors pre-installed on the excavator in real time during the operation of the excavator, and the sensor data includes original images, original point clouds, RTK data and IMU data;

[0108] A posture construction module 20, used for a posture construction module, is used to calculate the target posture of the excavator and construct a real-time RGB point cloud map of the scene around the excavator based on the sensor data and the pre-calibrated sensor internal and external parameters, and in combination with a mapping and positioning algorithm SLAM;

[0109] The image projection module 30 is used to project the original image to the corresponding position of the real-time three-dimensional scene according to the target posture and the pre-calibrated internal and external parameters of the sensor to obtain the full three-dimensional scene of the excavator;

[0110] The operating scene rendering module 40 is used to render the operating scene of the excavator under a preset viewing angle in real time according to the full three-dimensional scene of the excavator.

[0111] It should be noted that the virtual perspective display device for the excavator environment provided by the above embodiment only uses the division of the above functional modules as an example when executing the virtual perspective display method for the excavator environment. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the virtual perspective display device for the excavator environment provided by the above embodiment and the virtual perspective display method embodiment for the excavator environment belong to the same concept. The implementation process is detailed in the method embodiment and will not be repeated here.

[0112] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0113] In the embodiments of the present application, on the one hand, the present application projects the original image to the corresponding position of the real-time three-dimensional scene according to the target posture and the pre-calibrated internal and external parameters of the sensor, so as to obtain a full three-dimensional scene of the excavator. The full three-dimensional scene can simulate the actual surrounding environment of the excavator and realize a full display of the surrounding environment of the excavator. On the other hand, based on the full three-dimensional scene of the excavator, the excavator operation scene under a preset perspective can be rendered in real time. The preset perspective can be freely set, providing a freely adjustable perspective and depth information, thereby improving the efficiency and safety of remote operation of the excavator.

[0114] The present application also provides a computer-readable medium having program instructions stored thereon, which, when executed by a processor, implement the method for displaying the virtual perspective of the environment of an excavator provided by the above-mentioned various method embodiments.

[0115] The present application also provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the method for displaying the virtual perspective of the environment of an excavator in each of the above-mentioned method embodiments.

[0116] See also Figure 4 , is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 4 As shown, the electronic device 1000 may include: at least one processor 1001 , at least one network interface 1004 , a user interface 1003 , a memory 1005 , and at least one communication bus 1002 .

[0117] The communication bus 1002 is used to realize the connection and communication between these components.

[0118] The user interface 1003 may include a display screen (Display) and a camera (Camera), and the optional user interface 1003 may also include a standard wired interface and a wireless interface.

[0119] The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a WI-FI interface).

[0120] Among them, the processor 1001 may include one or more processing cores. The processor 1001 uses various interfaces and lines to connect various parts within the entire electronic device 1000, and executes various functions and processes data of the electronic device 1000 by running or executing instructions, programs, code sets or instruction sets stored in the memory 1005, and calling data stored in the memory 1005. Optionally, the processor 1001 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). The processor 1001 can integrate one or a combination of a central processing unit (Central Processing Unit, CPU), a graphics processing unit (Graphics Processing Unit, GPU) and a modem. Among them, the CPU mainly processes the operating system, user interface and application programs; the GPU is responsible for rendering and drawing the content to be displayed on the display screen; the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 1001, and it can be implemented separately through a chip.

[0121] Among them, the memory 1005 may include a random access memory (Random Access Memory, RAM) and may also include a read-only memory (Read-Only Memory). Optionally, the memory 1005 includes a non-transitory computer-readable storage medium. The memory 1005 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 1005 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area may store data involved in the above-mentioned various method embodiments, etc. The memory 1005 may also be optionally at least one storage system located away from the aforementioned processor 1001. As Figure 4 As shown, the memory 1005 as a computer storage medium may include an operating system, a network communication module, a user interface module, and an application program for displaying the virtual perspective of the excavator's environment.

[0122] exist Figure 4 In the electronic device 1000 shown, the user interface 1003 is mainly used to provide an input interface for the user and obtain data input by the user; and the processor 1001 can be used to call the excavator environment virtual perspective display application stored in the memory 1005, and specifically perform the following operations:

[0123] During the operation of the excavator, the sensor data collected by various sensors pre-installed on the excavator are read in real time. The sensor data includes original images, original point clouds, RTK data and IMU data;

[0124] Based on sensor data and pre-calibrated sensor internal and external parameters, the target position of the excavator is calculated in combination with the SLAM mapping and positioning algorithm;

[0125] According to the target posture and the pre-calibrated internal and external parameters of the sensor, the original image is projected to the corresponding position of the real-time three-dimensional scene to obtain the full three-dimensional scene of the excavator;

[0126] Based on the full 3D scene of the excavator, the excavator operation scene under the preset perspective is rendered in real time.

[0127] In one embodiment, before the processor 1001 calculates the target posture of the excavator based on the sensor data and the pre-calibrated sensor internal and external parameters in combination with the SLAM mapping and positioning algorithm, it further performs the following operations:

[0128] According to the sensor data, various sensors are calibrated with internal and external parameters to obtain internal and external parameter data that unify the sensor data into the same coordinate system; the internal parameter calibration is used to determine the parameters of each sensor, and the external parameter calibration is used to determine the precise position and direction of the sensor relative to other sensors or the world coordinate system;

[0129] The internal and external parameter data are used as the pre-calibrated internal and external parameters of the sensor.

[0130] In one embodiment, the processor 1001 performs the following operations when calculating the target posture of the excavator based on the sensor data and the pre-calibrated sensor internal and external parameters in combination with the SLAM mapping and positioning algorithm:

[0131] According to the pre-calibrated sensor internal and external parameters, the RTK data is used as the initial value, and the LiDAR image imu odometer is constructed through the SLAM algorithm to obtain the target position and posture of the excavator;

[0132] Among them, the target posture calculation formula of the excavator is:

[0133] in,

[0134] min J(x) is the target position of the excavator, x is the sensor data, including position, attitude, speed, and bias information, N is the number of observations, and r i (x) is the residual vector of the ith observation, reflecting the difference between the observed value and the predicted value based on the current state estimate, W i is the weight matrix of the i-th observation, which is defined by the inverse covariance matrix of the observation noise in the SLAM algorithm, and T is the internal and external parameters of the sensor.

[0135] In one embodiment, the processor 1001 performs the following operations when calculating the target pose of the excavator and constructing a real-time RGB point cloud map of the scene around the excavator based on the sensor data and the pre-calibrated sensor internal and external parameters and combining the mapping and positioning algorithm SLAM:

[0136] Extracting key frames according to the original image and the original point cloud read in real time;

[0137] Using the SLAM algorithm and the key frames to construct a local map;

[0138] Matching the local map with the current frame to determine the target position of the excavator;

[0139] Obtaining historical frames and observations collected by a camera installed on the excavator during historical operations;

[0140] Based on the historical frames and their observations, construct a factor graph, wherein the factor graph is used to represent the relationship between the sensor observations and the excavator posture, wherein the nodes in the factor graph represent the postures and the edges represent the observations;

[0141] Pre-integrate the original image, original point cloud, and IMU data in the sensor data to obtain an integration result;

[0142] Adding the integration result as a factor to the factor graph to obtain a target factor graph;

[0143] Globally optimizing a plurality of historical frames through the target factor graph to optimize the target pose to a global pose;

[0144] The local map is adjusted according to the global posture and the global map is constructed to obtain a real-time RGB point cloud map of the scene around the excavator.

[0145] In one embodiment, when the processor 1001 performs surface reconstruction on the real-time RGB point cloud map and converts it into a real-time three-dimensional scene of the scene around the excavator, the processor 1001 specifically performs the following operations:

[0146] The global map is meshed by using a TSDF algorithm to generate a three-dimensional scene that can reflect the surrounding environment of the excavator in real time;

[0147] The three-dimensional scene that can reflect the surrounding environment of the excavator in real time is used as the real-time three-dimensional scene of the surrounding scene of the excavator.

[0148] In one embodiment, when the processor 1001 projects the original image to the corresponding position of the real-time three-dimensional scene according to the target posture and the pre-calibrated sensor internal and external parameters to obtain the full three-dimensional scene of the excavator, the processor 1001 specifically performs the following operations:

[0149] Through the pre-calibrated sensor internal and external parameters, the target posture is converted to the excavator's three-dimensional coordinate system to obtain the excavator's final posture information;

[0150] The original image is associated with the final pose information to determine the spatial position of each pixel in the original image in the three-dimensional coordinate system of the excavator;

[0151] According to the spatial position of each pixel point in the original image in the three-dimensional coordinate system of the excavator, each pixel point in the original image is projected to the same spatial position of the real-time three-dimensional scene to obtain the full three-dimensional scene of the excavator.

[0152] In the embodiments of the present application, on the one hand, the present application projects the original image to the corresponding position of the real-time three-dimensional scene according to the target posture and the pre-calibrated internal and external parameters of the sensor, so as to obtain a full three-dimensional scene of the excavator. The full three-dimensional scene can simulate the actual surrounding environment of the excavator and realize a full display of the surrounding environment of the excavator. On the other hand, based on the full three-dimensional scene of the excavator, the excavator operation scene under a preset perspective can be rendered in real time. The preset perspective can be freely set, providing a freely adjustable perspective and depth information, thereby improving the efficiency and safety of remote operation of the excavator.

[0153] A person skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program for displaying the virtual perspective of the excavator environment can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the storage medium of the program for displaying the virtual perspective of the excavator environment can be a disk, an optical disk, a read-only storage memory, or a random access memory, etc.

[0154] The above disclosure is only the preferred embodiment of the present application, which certainly cannot be used to limit the scope of rights of the present application. Therefore, equivalent changes made according to the claims of the present application are still within the scope covered by the present application.

Claims

1. A method for displaying an excavator's environment from a virtual perspective, characterized in that: The method comprises: During the operation of the excavator, the sensor data collected by various sensors pre-installed on the excavator are read in real time, and the sensor data includes original images, original point clouds, RTK data and IMU data; Based on the sensor data and the pre-calibrated sensor internal and external parameters, the target posture of the excavator is calculated and a real-time RGB point cloud map of the scene around the excavator is constructed in combination with the mapping and positioning algorithm SLAM, and the real-time RGB point cloud map is surface reconstructed and converted into a real-time three-dimensional scene of the scene around the excavator; According to the target posture and pre-calibrated internal and external parameters of the sensor, the original image is projected to a corresponding position of the real-time three-dimensional scene to obtain a full three-dimensional scene of the excavator; According to the full three-dimensional scene of the excavator, the excavator operation scene under a preset viewing angle is rendered in real time.

2. The method according to claim 1, characterized in that Before calculating the target posture of the excavator based on the sensor data and the pre-calibrated sensor internal and external parameters and in combination with the SLAM mapping and positioning algorithm, the method further includes: According to the sensor data, internal and external calibrations are performed on various sensors to obtain internal and external parameter data that unify the various sensor data into the same coordinate system; wherein the internal calibration is used to determine the own parameters of various sensors, and the external calibration is used to determine the precise position and direction of the sensor relative to other sensors or the world coordinate system; The internal and external parameter data are used as the pre-calibrated internal and external parameters of the sensor.

3. The method according to claim 2, characterized in that The various sensors include cameras and laser radars, and the intrinsic calibration of the camera can be described as: Where u and v are the image coordinates in the image captured by the camera, and f x and f y are the focal lengths in the horizontal and vertical directions, c x and c y is the optical axis coordinate, Z is the distance from the three-dimensional point in the horizontal direction X and the vertical direction Y in the imaging plane; The calibration process of the camera and lidar is as follows: Among them, R is a 3×3 rotation matrix, t is a 3×1 translation vector, z is the depth corresponding to the normalized plane of the camera, and [X,Y,Z,1] is a point cloud point represented in a homogeneous manner. When there are multiple pairs of matching points between the point cloud and the image, the equations can be immediately solved for R and T.

4. The method according to claim 1, characterized in that The RTK data is based on real-time motion measurement technology to achieve centimeter-level positioning data, and the IMU data is collected by an inertial measurement unit; The target posture of the excavator is calculated based on the sensor data and the pre-calibrated sensor internal and external parameters, and in combination with the SLAM mapping and positioning algorithm, including: According to the pre-calibrated sensor internal and external parameters, the RTK data is used as the initial value, and the laser radar image imu odometer is constructed through the SLAM algorithm to obtain the target posture of the excavator; Among them, the target posture calculation formula of the excavator is: in, min J(x) is the target position of the excavator, x is the sensor data, including position, attitude, speed, and bias information, N is the number of observations, and r i (x) is the residual vector of the ith observation, reflecting the difference between the observed value and the predicted value based on the current state estimate, W i is the weight matrix of the i-th observation, which is defined by the inverse covariance matrix of the observation noise in the SLAM algorithm, and T is the internal and external parameters of the sensor.

5. The method according to claim 1, characterized in that Based on the sensor data and the pre-calibrated sensor internal and external parameters, the target position of the excavator is calculated in combination with the SLAM mapping and positioning algorithm, and a real-time RGB point cloud map of the scene around the excavator is constructed, including: Extracting key frames according to the original image and the original point cloud read in real time; Using the SLAM algorithm and the key frames to construct a local map; Matching the local map with the current frame to determine the target position of the excavator; Obtaining historical frames and observations collected by a camera installed on the excavator during historical operations; Based on the historical frames and their observations, construct a factor graph, wherein the factor graph is used to represent the relationship between the sensor observations and the excavator posture, wherein the nodes in the factor graph represent the postures and the edges represent the observations; Pre-integrate the original image, original point cloud, and IMU data in the sensor data to obtain an integration result; Adding the integration result as a factor to the factor graph to obtain a target factor graph; Globally optimizing a plurality of historical frames through the target factor graph to optimize the target pose to a global pose; The local map is adjusted according to the global posture and the global map is constructed to obtain a real-time RGB point cloud map of the scene around the excavator.

6. The method according to claim 5, characterized in that The surface reconstruction of the real-time RGB point cloud map is converted into a real-time three-dimensional scene of the scene around the excavator, including: The global map is meshed by using a TSDF algorithm to generate a three-dimensional scene that can reflect the surrounding environment of the excavator in real time; The three-dimensional scene that can reflect the surrounding environment of the excavator in real time is used as the real-time three-dimensional scene of the surrounding scene of the excavator.

7. The method according to claim 6, characterized in that The update equation of the TSDF algorithm is: TSDF new (v)=w·(D(v)-D obs )+(1-w)·TSDF old (v); Among them, TSDF new (v) is the meshing result, v is each voxel in the global map, v is the fusion weight, D(v) is the true distance from the voxel center to the nearest surface, Dobs is the observed distance, TSDFold(v) and TSDFnew(v) ​​are the old and new TSDF values ​​of the voxel, respectively. A voxel is the analogue of a pixel in three-dimensional space.

8. The method according to claim 1, characterized in that According to the target posture and the pre-calibrated internal and external parameters of the sensor, the original image is projected to the corresponding position of the real-time three-dimensional scene to obtain the full three-dimensional scene of the excavator, including: The target posture is converted into the three-dimensional coordinate system of the excavator by using the pre-calibrated internal and external parameters of the sensor to obtain the final posture information of the excavator; Associating the original image with the final posture information to determine the spatial position of each pixel in the original image in the three-dimensional coordinate system of the excavator; According to the spatial position of each pixel point in the original image in the three-dimensional coordinate system of the excavator, each pixel point in the original image is projected to the same spatial position of the real-time three-dimensional scene to obtain the full three-dimensional scene of the excavator.

9. The method according to claim 8, characterized in that The calculation formula for the spatial position of each pixel point in the original image in the three-dimensional coordinate system of the excavator is: P world =R×P camera +t; Among them, P world is the spatial position of each pixel in the original image in the three-dimensional coordinate system of the excavator, P camera is the coordinate of the pixel point in the camera coordinate system in the original image, R and t are the rotation matrix and translation vector of the image point cloud extrinsic parameters in the pre-calibrated sensor extrinsic parameters.

10. An environmental virtual perspective display device for an excavator, characterized in that: The device comprises: The sensor data reading module is used to read the sensor data collected by various sensors pre-installed on the excavator in real time during the operation of the excavator, and the sensor data includes original images, original point clouds, RTK data and IMU data; A posture construction module is used to calculate the target posture of the excavator and construct a real-time RGB point cloud map of the scene around the excavator based on the sensor data and pre-calibrated sensor internal and external parameters and in combination with a mapping and positioning algorithm SLAM; An image projection module, used to project the original image to a corresponding position of a real-time three-dimensional scene according to the target posture and pre-calibrated sensor internal and external parameters, so as to obtain a full three-dimensional scene of the excavator; The operating scene rendering module is used to render the excavator operating scene under a preset viewing angle in real time according to the full three-dimensional scene of the excavator.

Images