Virtual reconstruction method for whole working space of working face based on laser slam

Abstract

The application discloses a working face whole working space virtual reconstruction method based on laser SLAM, comprising the steps of establishing a physical point cloud space, a virtual point cloud space and a virtual-real fusion channel. The physical point cloud space is used for collecting physical point cloud data through a laser radar installed on a coal mining machine and calculating pose information of the fully-mechanized mining equipment, so that the fully-mechanized mining equipment in the virtual point cloud module is reconstructed; the virtual point cloud space is used for establishing a digital twin of the fully-mechanized mining equipment and the laser radar and acquiring virtual point cloud of a hydraulic support; the virtual-real fusion channel is used for registering the physical point cloud and the virtual point cloud of the support, calculating a running track of the coal mining machine, constructing the running track into a space model, performing similarity analysis on the space model, planning the running track, and updating the running track by using real-time information.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of virtual reconstruction of fully mechanized mining equipment, in particular to a working face overall workspace virtual reconstruction method based on laser SLAM. BACKGROUND

[0002] With the development of automation and intelligentization of industrial production process, people have higher requirements for the production efficiency, personnel safety and information transparency of coal mining operation. Virtual reconstruction of working space is to reconstruct the pose information of shearer and hydraulic support in virtual space, and then accurately judge and evaluate the working condition of subsequent working face.

[0003] Coal mining is developing towards intelligentization and unmanned, and to achieve this goal, it is necessary to make the shearer, scraper conveyor and hydraulic support of the fully mechanized working face work together and improve the automation level of production. Among them, the real-time and accurate reconstruction of the running posture and position of fully mechanized equipment, the virtual fusion of virtual space and physical space, and the safe and reliable detection of the straightness of scraper conveyor, the running track of shearer and the pose information of hydraulic support are realized.

[0004] The "real-time three-dimensional imaging method and system for fully mechanized working face based on multiple laser radars" with publication number CN114859379A obtains the pose of any laser radar in the corresponding hydraulic support coordinate system, and the laser radar is preset on the corresponding hydraulic support. The original point data of the local fully mechanized working face in the field of view of any laser radar at the current time is obtained. The local real-time imaging of the local fully mechanized working face is carried out through the original point data. Taking a hydraulic support coordinate system as a reference, the result of any local real-time imaging is transformed to the hydraulic support coordinate system for global real-time imaging. The imaging system sets the laser radar on the hydraulic support, sets a sub-processor corresponding to each laser radar, connects each sub-processor with a total processor, and executes the software of the imaging method, realizes the local real-time imaging and global real-time imaging of the fully mechanized working face, and does not need to modify the original mine structure.

[0005] The patent with publication number CN114970073A "A coal winning machine virtual-real fusion positioning system based on laser radar" includes a bottom layer data processing system, a data correction system and a virtual reality system. The attitude data of the support and the coal winning machine are collected by two two-dimensional laser radars, an inclination sensor and an odometer installed on the coal winning machine, and the relative position information and the absolute position information of the coal winning machine in the working face are obtained through the solving module built in the fine positioning system and the coarse positioning system after correction by the data correction system. The simulation motion of the cutting and advancing process is realized in the virtual reality environment, and the position information of the coal winning machine is obtained through human-computer interaction while the positioning result confidence is determined. The present application provides a solution based on laser radar and virtual reality technology for the positioning of coal winning machine under complex floor conditions, solves the problem of accurate positioning of coal winning machine due to the inability of small and medium-sized coal mines to afford expensive inertial navigation instruments, and is conducive to reducing the investment cost of coal winning machine positioning. At the same time, it avoids the problem of long-term inaccurate positioning caused by the accumulation of inertial navigation positioning errors over time.

[0006] The patent with publication number CN110287974A "A method for quickly matching laser scanning three-dimensional model and GIS model of fully mechanized working face" installs a laser scanner on the coal winning machine to quickly obtain the laser scanning three-dimensional model of the fully mechanized working face; extracts the feature point set of the working face laser scanning three-dimensional model and the GIS model and calculates its surface normal vector; uses the RANSAC method to perform coarse matching on the feature point set; uses the improved ICP algorithm with normal vector constraint to register the feature points, uses the unit quaternion method to solve the coordinate transformation matrix, and checks whether the set threshold constraint condition is met; accurately aligns the coordinate systems of the working face laser scanning three-dimensional model and the GIS model, quickly matches the working face laser scanning three-dimensional model and the GIS model, and solves the problem of real-time synchronization of the actual space physical state and the GIS description state of the fully mechanized working face.

[0007] Research on rapid virtual reconstruction of mine production environment based on GeoSLAM: The overall process of underground production environment reconstruction based on GeoSLAM system is proposed, which mainly includes two key contents of data scanning and three-dimensional modeling, and is divided into two parts of field measurement and indoor modeling according to the operation process. A combined modeling method of underground production environment combining GeoSLAM system modeling and engineering survey data modeling is proposed.

[0008] However, the above method has the following defects:

[0009] (1) A large number of sensors need to be installed on the hydraulic support, and a large number of sensors are not easy to install on a large number of hydraulic supports and are expensive; the whole space imaging is completed by local imaging, which lacks real-time and accurate working space information.

[0010] (2) using the point cloud obtained by the laser radar to complete the modeling of the working space of the stack, unable to consider the coupling relationship with the existing fully mechanized mining equipment and the endogenous defects in the point cloud, respectively modeling each component to complete the reconstruction of the working space.

[0011] (3) using the feature point set of the roadway three-dimensional model and the feature point set of the working face coal seam three-dimensional model for rough matching, the geological information of the working face needs to be collected, the underground coal seam geological information is complex, and the sensor requirement is high;

[0012] (4) the point set of the feature is used for the improved ICP algorithm for normal vector constraint registration, which has a large amount of work to meet the distance threshold constraint condition and coordinate transformation, and the obtained coordinate information only has position information and lacks the pose information of the fully mechanized mining equipment.

[0013] In summary, in the prior art, only the reconstruction of the working space is completed, the virtual and real fusion of the virtual space and the physical space is lacking, the physical space cannot be stopped at any time and the environment is complex, and the collected point cloud information will have defects;The working plane of the virtual space is not used to deduce the running track and the cutting track. SUMMARY

[0014] The purpose of the present application is to provide a working face whole working space virtual reconstruction method based on laser SLAM, to establish a digital twin of the virtual space by combining physical point cloud information, and to adjust the fully mechanized mining equipment in the virtual space by using virtual and real fusion, and to analyze and predict the running track of the coal mining machine, so as to realize the virtual reconstruction of the whole working space.

[0015] To achieve the above purpose, the technical scheme adopted by the present application is:

[0016] A working face whole working space virtual reconstruction method based on laser SLAM, characterized in that:

[0017] Step one, establish a physical point cloud space, including:

[0018] - collection of physical point cloud data;Select the characteristic parts of the hydraulic support, and scan the whole hydraulic support group by the laser radar installed on the coal mining machine during the movement of the coal mining machine along the scraper conveyor to obtain the point cloud of the whole hydraulic support group, and then filter to extract the local feature point cloud of a single hydraulic support, complete the collection and processing of the physical point cloud data;

[0019] - solve the pose of the coal mining machine;Register the hydraulic support group point cloud scanned by the laser radar, use the quaternion and displacement vector obtained by registration to analyze the attitude angle of the laser radar during movement and fit the motion trajectory of the laser radar, since the laser radar is rigidly connected with the coal mining machine, the attitude and position information of the laser radar is regarded as the attitude and position information of the coal mining machine.

[0020] Step two, establish a virtual point cloud space, including:

[0021] Establish a digital twin; select Unity3d as the virtual reality environment platform, establish the digital twin model of the shearer, hydraulic support and scraper conveyor according to the three-machine assembly drawing of the working face and the sensor measurement results, and complete the initialization of the virtual reality scene;

[0022] Virtual point cloud acquisition; build a laser radar digital twin, add corresponding colliders to the target to be scanned, use the ray collision monitoring method to obtain the world coordinates of the feature component scanning points, and convert the world coordinates to the local coordinates with the laser radar as the coordinate system origin, to complete the simulation of the physical point cloud acquisition process;

[0023] Virtual-real coordinate system alignment; the quaternion and displacement vector of the rotation angle and position information of each frame of point cloud obtained by laser radar scanning are analyzed, so that it can be directly used to drive the motion of the digital twin of the shearer in the virtual reality environment;

[0024] Full-cycle point cloud acquisition and release; drive the digital twin of the shearer to move according to the method of reconstructing the pose of the shearer in the virtual space, and use the laser radar digital twin carried by it to scan the virtual support group one by one to obtain the feature component virtual point cloud data of all supports;

[0025] Step three, establish a virtual-real fusion channel, including:

[0026] Hydraulic support virtual-real point cloud registration; register the point clouds obtained in the physical and virtual spaces, transform the rotation and translation matrix obtained by registration, adjust the pose of the digital twin of the hydraulic support according to the transformation result to make it exactly the same as the pose of the hydraulic support in the physical space, and complete the reconstruction of the overall pose of a single hydraulic support;

[0027] Scraper conveyor pose solution; combine the position and attitude information in the shearer running track, and according to the coupling relationship of the shearer running on the scraper conveyor, solve the pitch angle of the middle trough of each section of the scraper conveyor to obtain the actual form of the scraper conveyor;

[0028] Trajectory prediction and correction iteration; analyze the obtained shearer running track, predict the pose trajectory in the virtual space; based on the support base pose information, obtain the roof point cloud information, and construct a space model with Unity3d, obtain the next cut theoretical trajectory through similarity analysis, and fuse the feedback real-time shearer trajectory information to correct the space model and the theoretical trajectory.

[0029] Further, in step one, ROS system is used to collect, record and analyze single-frame PCD point cloud files in bag files; PCL point cloud library is used to filter and segment local feature point clouds of single hydraulic support from overall scene point clouds, so as to complete collection and processing of physical point cloud data.

[0030] Further, in step one, A-Loam algorithm is used to correct the pose and running track of the coal mining machine calculated from the bag file in combination with equipment information, double vector pose determination and coordinate system transformation principles are used to determine parameters required for space pose calculation of the scraper, so as to complete relative pose calculation of the start and end joints.

[0031] Further, in step one, a space bounding box is drawn to extract target point clouds from overall point clouds by using straight-through filtering, parameters are determined according to the three-machine assembly diagram of the fully-mechanized coal mining face and the installation position and vertical scanning angle of the laser radar; appropriate sampling frequency is selected according to the moving speed of the coal mining machine and the point cloud publishing frequency, the point cloud PCD file published when the coal mining machine moves to the position where the laser radar is opposite to the electro-hydraulic controller of the hydraulic support is filtered, so as to obtain local feature point clouds of single hydraulic support in the physical space.

[0032] Further, in step one, motion distortion compensation is required before reconstructing the pose of the coal mining machine, the point cloud motion distortion compensation is to match the point cloud at the completion time of the current scanning with the complete point cloud of the last frame, to perform linear interpolation on the rotation and translation matrix, and to project all point clouds of the current frame to the completion time, so as to complete the point cloud motion distortion compensation, and after the motion distortion compensation, the point clouds are matched between frames.

[0033] Further, in step two, FileStream class is used to realize continuous reading, writing and storage of single-frame virtual PCD point cloud data; actual moving speed of the coal mining machine and point cloud publishing frequency are combined to select sampling frequency for successive scanning of the hydraulic support group in the virtual space, so as to obtain virtual point cloud data of feature components of all supports.

[0034] Further, in step three, NDT+PointtoPlane ICP algorithm is used for continuous registration to obtain action parameters for adjusting the overall pose of the support, the pose parameters of the support base are iteratively solved according to the coordinate axis direction in Unity3d; the virtual scanning is performed again according to the solving results of the virtual support group and is registered with the original physical point cloud, the virtual support is adjusted multiple times according to the action parameters obtained by registration until the error accuracy requirement is met, so as to obtain the pose of the support base; the virtual pose information of the middle trough is obtained based on the real-time pose information of the coal mining machine, so as to obtain the relative pose information of the start and end joints of the floating connection mechanism, so as to complete the reconstruction of the working space.

[0035] Further, in step three, based on the known cutting roof trajectory and floor trajectory data, the next three machine states and positions are predicted and judged, the top and bottom plate curves, hydraulic support support state and scraper conveyor arrangement state are extracted respectively, the three-dimensional form of the coal seam is depicted, the three-dimensional working face coal seam simulation model is constructed, and it is imported into Unity3d, the mesh division and construction of the coal seam floor are realized by connecting triangles, and the virtual reconstruction of the coal seam is realized by using MeshFilter and MeshRender components in Unity3d.

[0036] Compared with the prior art, the present application has the following beneficial effects:

[0037] (1) According to the working principle of three-dimensional laser radar, a method for constructing digital twin model in virtual space is proposed, which realizes the collection and storage of virtual point cloud; the point cloud motion distortion compensation is carried out on the physical space and matched with the high-precision virtual radar in the virtual space, so that it can be solved by simultaneous fusion, simultaneous iteration and simultaneous solution, so as to complete the virtual-real fusion collaborative work.

[0038] (2) The point cloud information is obtained by searching the physical radar and virtual radar, the virtual-real point cloud registration is carried out on the local feature point cloud of the hydraulic support group, so as to drive the pose reconstruction of the coal mining machine and the hydraulic support group; the position and pose information in the operation of the coal mining machine is analyzed, the coupling relationship of the coal mining machine in the operation of the scraper conveyor is established to obtain the actual form of the scraper machine, and the virtual reconstruction of the whole fully mechanized mining equipment is completed.

[0039] (3) The coordinate systems of physical space and virtual space are established and the motion direction is unified, the quaternion and displacement vector obtained by registration are analyzed, and the digital twin in the virtual space is adjusted according to the analysis result, so that it is the same as the pose of the physical space, and the virtual-real fusion is completed.

[0040] (4) Based on the contact between the front roller of the coal mining machine and the top beam of the support and the contact between the lower roller of the scraper machine and the base of the support and the floor, the curved surface of the top and bottom plate can be obtained by the fully mechanized mining equipment, and the coal seam simulation model is obtained, and the reconstruction of the overall environment of the working space and the coal seam is formed.

[0041] (5) The motion trajectory of the coal mining machine is analyzed and shown in the virtual space, the virtual path deduction is carried out by the calculation ability of Unity3d, the space model is constructed, the curved surface is analyzed by mesh division and similarity analysis, and the cutting trajectory is predicted, and the real-time coal mining machine pose information is used to update the path deduction and cutting trajectory. BRIEF DESCRIPTION OF DRAWINGS

[0042] Figure 1 is the technical roadmap of the real-time accurate virtual reconstruction method of the whole working space of the working face based on laser SLAM;

[0043] Figure 2 is a reconstruction schematic diagram of a hydraulic support of a shearer view angle;

[0044] Figure 3 is a physical three-level coordinate system schematic diagram of a shearer;

[0045] Figure 4 is a point cloud data writing logic of the application;

[0046] Figure 5 is a shearer pose reconstruction process schematic diagram of a shearer;

[0047] Figure 6 is a next cut theoretical trajectory illustration;

[0048] Figure 7 is a space model prediction next cut flow schematic diagram. DETAILED DESCRIPTION

[0049] A typical embodiment of the application provides a working face overall working space real-time accurate virtual reconstruction method based on laser SLAM, including a physical point cloud space, a virtual point cloud space and a virtual-real fusion channel.

[0050] As Figure 1 shown is a technical roadmap of a working face overall working space real-time accurate virtual reconstruction method based on laser SLAM, including a physical point cloud space, a virtual point cloud space and a virtual-real fusion channel. Among them, the physical point cloud space carries out physical point cloud data through the laser radar installed on the shearer, and calculates the pose information of the fully mechanized mining equipment, so as to facilitate the reconstruction of the fully mechanized mining equipment in the virtual point cloud module; the virtual point cloud space establishes the digital twin of the fully mechanized mining equipment and the laser radar, and obtains the virtual point cloud of the hydraulic support; the virtual-real fusion channel registers the support physical point cloud and the virtual point cloud, and calculates the shearer running track, which is constructed into a space model for similarity analysis to complete the planning of the running track, and is updated by using real-time information.

[0051] (1) Establishing a physical point cloud space

[0052] The physical point cloud space is to carry out physical point cloud data through the laser radar installed on the shearer, and calculate the pose information of the fully mechanized mining equipment; including hydraulic support feature component selection, laser radar calibration, point cloud data acquisition and processing and calculation of shearer pose, and the physical point cloud space obtains point cloud information, which is the premise of virtual space reconstruction.

[0053] The physical point cloud space selects the feature components to be scanned according to the support type and the parameters of each component, completes the selection of the laser radar, confirms the optimal installation position and the calibration work; uses the ROS system to collect, record and analyze the single-frame pcd point cloud file in the bag file; filters and segments the local feature point cloud of a single support from the overall scene point cloud by means of the PCL point cloud library, completes the collection and processing of the physical point cloud data; and uses the A-Loam algorithm to correct the pose and running track of the coal mining machine calculated from the bag file in combination with the inertial navigation and other equipment information of the coal mining machine, determines the parameters required for solving the spatial pose of the scraper by the double-vector pose determination and the principle of coordinate system transformation, and then completes the relative pose determination of the start and end joints.

[0054] The feature component selection and laser radar calibration are to select the feature components to be scanned according to the support type and the parameters of each component, complete the selection of the laser radar, confirm the optimal installation position and the calibration work. The three-dimensional laser radar has the characteristics of high resolution, high precision and strong anti-interference ability, and is suitable as a point cloud acquisition device for the fully-mechanized coal mining face.

[0055] As shown in Figure 2 , the laser radar is installed on the coal mining machine body and rigidly connected with the coal mining machine body through a magnetic base. In the process of moving along the scraper conveyor, the laser radar scans the whole hydraulic support group. According to the characteristics of the laser radar, the farther the scanning distance, the fewer the number of laser points on the same object surface, and the less obvious the collected object features. When the coal mining machine moves to the position where the laser radar is directly opposite to the hydraulic support, the point cloud information is the most abundant. Limited by the scanning angle of the laser radar in the vertical plane, the collected high-quality point cloud is mainly the local point cloud of the column part and the electro-hydraulic controller part of the hydraulic support, which is used as the source of the point cloud data in the physical space.

[0056] In order to facilitate the reconstruction of the equipment in the virtual space, a three-level coordinate system of the current cutting knife is established, as shown in Figure 3 , the midpoint of the upper edge line of the front baffle of the first hydraulic support base of the current cutting knife is selected as the marker point and a left-hand coordinate system is established, the vertical direction pointing to the roof beam is selected as the positive direction of the y axis, the horizontal direction pointing to the coal wall is selected as the positive direction of the x axis, and the direction of the tail of the scraper conveyor pointing to the head is selected as the positive direction of the z axis. The coordinate system is used as the world coordinate system (first-level coordinate system) of the current cutting knife, and the coordinates of the first hydraulic support in the world coordinate system are Since the coal mining machine and the laser radar are rigidly connected, the coordinate systems of the two are coincident, the coordinate system of the laser radar itself is selected as the secondary coordinate system, and the z-axis of the radar is vertically directed to the coal seam roof, the x-axis is horizontally directed to the hydraulic support group, and the y-axis is horizontally directed to the tail of the scraper conveyor during installation of the laser radar. Each hydraulic support establishes its own coordinate system according to the establishment method of the first hydraulic support coordinate system as the tertiary coordinate system. The position of a single hydraulic support is recorded as The angle of rotation around the z-axis of the hydraulic support itself coordinate system is defined as the pitch angle, the angle of rotation around the y-axis is defined as the yaw angle, and the angle of rotation around the z-axis is defined as the roll angle. The positive and negative values of the attitude angle are determined according to the right-hand rule. The angle of rotation around the x-axis of the coal mining machine is defined as the pitch angle, the angle of rotation around the y-axis is defined as the roll angle, and the angle of rotation around the z-axis is defined as the yaw angle. The positive and negative values of the attitude angle are determined according to the right-hand rule.

[0057] The point cloud data acquisition and processing uses the point cloud collected by the laser radar in the physical space to complete the pose reconstruction of the coal mining machine and the hydraulic support group. The pose reconstruction of the coal mining machine mainly registers the point cloud of the hydraulic support group scanned by the laser radar, uses the quaternion and displacement vector obtained by registration to analyze the attitude angle of the laser radar during motion and fit the motion trajectory of the laser radar. Since the laser radar and the coal mining machine are rigidly connected, the attitude and position information of the laser radar can be regarded as the attitude and position information of the coal mining machine. Using these information to drive the digital twin of the coal mining machine in the virtual reality environment can complete the reconstruction of the pose of the coal mining machine.

[0058] The coal mining machine carrying the laser radar scans the hydraulic support group and obtains the point cloud of the whole hydraulic support group. During one-way coal cutting operation, the hydraulic support only moves the support without pushing the machine during the process of cutting the coal wall, and the hydraulic support pushes the machine during the reverse floating coal. At this time, the overall pose of the hydraulic support will not change, so the scanning of the hydraulic support group is carried out in this process.

[0059] Since the VLP-16 laser radar is a mechanical rotating radar, it cannot scan a single hydraulic support alone. After obtaining the point cloud of the whole hydraulic support group, the point cloud needs to be filtered to extract the point cloud of a single hydraulic support. The straight-through filtering method is adopted, and a space bounding box is drawn to extract the target point cloud from the whole point cloud. The parameters are determined according to the three-machine assembly drawing of the fully mechanized coal face and the installation position and vertical scanning angle of the laser radar. According to the moving speed of the coal mining machine and the point cloud release frequency, a suitable sampling frequency is selected, and the point cloud PCD file released when the coal mining machine moves to the position where the laser radar is opposite to the hydraulic support electro-hydraulic controller is filtered to obtain the point cloud of a single hydraulic support in the physical space.

[0060] Wherein, the pose of the coal mining machine is solved, because the coal mining machine moves forward along the scraper conveyor during the scanning process of the laser radar, so it is necessary to compensate the motion distortion of the collected point cloud before reconstructing the pose of the coal mining machine. The reason for the motion distortion of the point cloud is that the coal mining machine carrying the laser radar moves along the scraper conveyor during the process of scanning and acquiring a complete frame of point cloud, which results in that the coordinate system origins of each point in the same frame of point cloud data do not coincide, so it is necessary to convert all points in a frame to the coordinate system at the same time.

[0061] The point cloud motion distortion compensation is to match the point cloud at the current scanning completion time with the last complete frame of point cloud, to linearly interpolate the rotation and translation matrix, to project all points in the current frame to the scanning completion time, so as to complete the point cloud motion distortion compensation. After the motion distortion compensation is completed, the point cloud can be matched between frames. First, the feature points of the point cloud need to be extracted, the curvature of each point is calculated, and the points are divided into corner feature points, surface feature points, secondary point surface feature points and non-feature points according to the curvature. After the extraction of the feature points is completed, the problem is converted into the inter-frame matching of the known feature points. The feature objects are selected as the corner feature points and the surface feature points, and the inter-frame matching of the point cloud can be completed by means of the ICP algorithm of point-to-surface and point-to-line, so as to obtain the pose increment of the last frame of point cloud to the current frame of point cloud. The pose of the point cloud in the world coordinate system is updated according to the pose increment. With the updated pose as the initial value, the points in the radar coordinate system are converted to the world coordinate system, and the delayed updated map is constructed in the world coordinate system. The feature objects are selected as the characteristic straight lines and characteristic planes fitted by the secondary point surface feature points, the converted current frame of point cloud is matched with the map, and the final quaternion and displacement vector are obtained, that is, the data source required for reconstructing the pose of the coal mining machine.

[0062] The above motion distortion compensation and the acquisition process of the quaternion and the displacement vector are realized by means of the A-LOAM algorithm. The laserOdometry part in the A-LOAM algorithm is responsible for the motion distortion compensation and the inter-frame matching, and the laserMapping part is responsible for matching the current frame of point cloud with the map and acquiring the final quaternion and displacement vector.

[0063] (2) Establish a virtual point cloud space

[0064] The virtual point cloud space is to establish a digital twin of the fully mechanized mining equipment and the laser radar in the virtual space, to acquire the virtual point cloud of the hydraulic support, to simulate the working process of the fully mechanized mining face, and to perform virtual-real fusion. It includes establishing a digital twin, acquiring a virtual point cloud, integrating virtual-real coordinate systems, and collecting and publishing all-time point clouds.

[0065] The virtual point cloud space is constructed according to the size parameters of the fully mechanized mining equipment and the laser radar, and the digital twin is constructed by using Unity3d software, and the fully mechanized working face is restored 1:1 in the virtual space; on this basis, the world coordinates of the scanning points of the column and other feature components are obtained by using the ray collision monitoring method; through coordinate conversion, the scanning points and the coordinate system of the laser radar are unified, and the mapping of the laser radar scanning function in the physical space in the virtual space is realized; through the FileStream class, the continuous reading and writing and storage of single-frame virtual pcd point cloud data are realized; combined with the actual moving speed of the coal mining machine and the point cloud publishing frequency, a suitable sampling frequency is selected to scan the hydraulic support group in the virtual space in sequence, and the virtual point cloud data of the feature components of all the supports are obtained.

[0066] In the formula, Unity3d is selected as the virtual reality environment platform for establishing the digital twin, and the digital twin models of the coal mining machine, the hydraulic support and the scraper conveyor are established according to the working face three-machine assembly drawing and the existing sensor measurement results, and the initialization of the virtual reality scene is completed.

[0067] The core of the three-dimensional laser radar digital twin is to simulate the emission of laser pulses by the laser, and the receiver receives the returned light beams, and the distance from the laser points on the surface of the object to the radar is calculated by the time difference between the emission and the reception. At the same time, the laser radar records the spatial coordinates, light intensity and scanning line bundle information of all the scanned points and outputs them by frame, and a frame of point cloud is generated by the laser radar rotating one round.

[0068] In order to achieve this purpose, the Physics.Raycast structure provided by Unity3d is needed. The Physics.Raycast structure can emit a ray from a specified position to a specified direction, and if the object contacted by the ray is added with a collision component within the detection distance, the coordinates of the ray collision point and other information are returned. Its logic is basically the same as the process of detecting the target by the laser radar. According to the scanning parameters of the VLP-16 laser radar, C# scripts are written to set the members in Raycast in turn. First, the ray, which contains two parameters, the origin Vector3 origin and the direction Vector3 direction of the ray, is set. The transform.position of the script-mounted object is set as the origin, and the direction is set as the direction of the laser radar. From formulas 1-3:

[0069]

[0070]

[0071] I={i n |i n= -(rayNum - 1) + 2*n, n≤rayNum, n∈N}, i∈I (3)

[0072] wherein represents a unit vector pointing to the positive direction of the z-axis, i represents the line number to which the current scanning point belongs, I represents the set of scanning line numbers, i n represents an element in the set, and rayNum represents the total number of lines of the laser radar.

[0073] Formulas (1)-(3) are implemented by the following statements:

[0074]

[0075] wherein Quaternion.AngleAxis() is an axis-angle rotation method provided by Unity3d, angle is the scanning angle range of the radar in the vertical plane, and angle = 30 in this case, transform.forward and transform.right correspond to the x-axis and z-axis of the object respectively. Then the declaration of the member variable hit is made and the member maxDistance is set to 100, so that the detection distance of the laser radar digital twin is the same as that of the actual radar, and the members layerMask and queryTriggerInteraction can be set according to the default settings. The visualization of the ray is realized by using the Debug.DrawLine(Vector3 start, Vector3 end, Color color) method, and start = transform.position, end = hit.point, and color = Color.blue, so that the blue ray from the laser to the ray collision point can be seen in the Sence interface. The transform.Rotate function is called frame by frame to control the rotation of the radar, and thus the construction of the laser radar digital twin is completed.

[0076] The acquisition of the above-mentioned virtual point cloud is obtained by using the collision detection function of Unity3d to detect the digital twin of the laser radar. Before using the digital twin, the target to be scanned needs to be added with a corresponding collision body. The point cloud data in the known physical space is mainly derived from the column part of the hydraulic support and the electro-hydraulic controller part, so the two parts of the hydraulic support digital twin are taken as the main scanning objects, and capsule collision bodies and Mesh collision bodies are added to them according to the component shapes. When the rays emitted from the radar contact the surface of the column and the electro-hydraulic controller model, the world coordinates of the collision points can be obtained through hit.position, and the world coordinates are converted to local coordinates with the radar as the coordinate system origin through the transform.InverseTransformPoint() method, thereby completing the simulation of the physical point cloud acquisition process.

[0077] After the acquisition of the virtual point cloud is completed, a common PCD file format is selected to store the point cloud data. A C# script is written to create a new PCD file at a specified file path, and the obtained point cloud is written into the file through the FileStream class. The PCD file output by the script is taken as the source of the point cloud data in the virtual space, and the data logic is as shown in Figure 4

[0078] Among them, the virtual-real coordinate system is to analyze the quaternion and displacement vector reflecting the rotation angle and position information of each frame of point cloud obtained by the laser radar scanning in the world coordinate system, so that it can be directly used to drive the motion of the digital twin of the coal mining machine in the virtual reality environment.

[0079] ​Firstly, the coordinate system used by the laser radar in the physical space is a right-handed coordinate system with the z-axis direction vertically upward, while the world coordinate system in Unity3d is a left-handed coordinate system with the y-axis direction vertically upward, and the x-axis directions of the two are opposite. Secondly, in the A-LOAM algorithm, the pose information of the point cloud is finally converted into a quaternion and a displacement vector in the world coordinate system, and its rotation order is Z-X-Y according to the Euler angle. However, in Unity3d, the rotation order of static Euler angle and dynamic Euler angle is different, the static Euler angle uses Z-X-Y order, and the dynamic Euler angle uses Y-X-Z order. Moreover, due to the different definitions of coordinate system and rotation direction, the rotation direction in the physical space and the virtual space is opposite when rotating around the same axis. In order to unify with the motion in the physical space, the quaternion is used to rotate in the world coordinate axis in Unity3d, and the displacement vector obtained in the previous text needs to exchange its y and z coordinates. The quaternion needs to be converted into a rotation matrix first, and the y and z axis related parameters in the rotation matrix need to be exchanged and the y axis rotation angle needs to be negated. The adjusted rotation matrix is converted into a quaternion to control the movement of the shearer. Align the scan start time with the reconstruction start time, adjust the frame rate of the Unity3d script to be the same as the frame rate of the laser radar, and draw the movement trajectory of the shearer through the Linerender component. The pose reconstruction of the shearer in the virtual reality environment can be completed, and the adjusted rotation matrix reflects the pitch angle, roll angle and yaw angle information of the shearer in the movement process, and the displacement vector reflects the position information of the shearer. The pose reconstruction process of the shearer is shown in Figure 5

[0080] Among them, the full-flight point cloud collection and release is to drive the digital twin of the shearer to move in the virtual space according to the method of pose reconstruction of the shearer, and to scan the virtual support group and obtain the point cloud of a single hydraulic support in the virtual space by means of the laser radar digital twin carried by it. Unlike in the physical space, the movement of the shearer in the virtual space can be stopped at any time, and because the scanning process is realized by using the collision body ray detection method, only the point cloud of a single support can be obtained by adding a collision body component to the target support and not adding a collision body to the non-target support during the scanning process. This approach can avoid filtering multiple PCD files of virtual point clouds and improve the reconstruction efficiency to some extent. Before scanning, the hydraulic support needs to be adjusted to the ideal position according to the three-machine assembly drawing of the fully mechanized working face, and its attitude angle needs to be set to zero to determine the initial state of the hydraulic support. According to the world coordinate relationship between the hydraulic support and the digital twin of the shearer, the laser radar starts to scan when it moves to the same position as the sampling position in the physical space. The scanning duration is determined according to the frame number of Unity3d and the rotation speed of the laser radar.

[0081] (3) Establishing a virtual-real fusion channel​

[0082] The virtual-real fusion channel registers the physical point cloud and the virtual point cloud of the support, and calculates the running track of the coal mining machine, constructs a space model, and completes the planning of the running track through similarity analysis, and updates it using real-time information; including coal mining machine track calculation, support virtual-real point cloud registration, and track prediction and correction iteration.

[0083] The virtual-real fusion channel adjusts the internal posture of each support in the virtual space based on the pose sensing information arranged on the support; the physical point cloud and the virtual point cloud are continuously registered using the NDT+PointtoPlane ICP algorithm to obtain the action parameters of the overall posture adjustment of the support, and the support base posture parameters are iteratively solved according to the coordinate axis direction in Unity3d; the virtual scanning is performed again according to the solving results of the virtual support group, and is registered with the original physical point cloud, and the virtual support is adjusted multiple times according to the action parameters obtained by registration until the error accuracy requirement is met, that is, the support base posture can be obtained; the virtual pose information of the middle trough is obtained based on the real-time pose information of the coal mining machine, and the relative pose information of the beginning and end joints of the floating connection mechanism is obtained, so as to complete the reconstruction of the working space; based on the obtained running track of the coal mining machine, the pose track is predicted in the virtual space; based on the support base pose information, the roof point cloud information is obtained, and a space model is constructed with the help of Unity3d; the next theoretical track is obtained through similarity analysis, and the space model and the theoretical track are corrected by fusing the feedback real-time coal mining machine track information.

[0084] The virtual-real point cloud registration of the hydraulic support is to register the point clouds obtained in the physical space and the virtual space, transform the rotation and translation matrix obtained by registration, adjust the pose of the digital twin of the hydraulic support according to the transformation result to make it completely the same as the pose of the hydraulic support in the physical space, thereby completing the reconstruction of the overall pose of a single hydraulic support. The NDT+PointtoPlane ICP algorithm is used for virtual-real point cloud registration. Due to the different directions of the coordinate axes of the laser radar in the virtual and real spaces and the different rotation directions of the point clouds, in order to facilitate the adjustment of the hydraulic support in the virtual space, the y and z coordinates of the point cloud in the physical space need to be exchanged before registration to convert the point cloud to the point cloud coordinate system in the virtual space. After registration, the y-axis rotation angle is negated and the x and z values in the displacement vector are negated. Finally, the hydraulic support digital twin is rotated and moved according to the world coordinate system provided by Unity3d, thereby iteratively solving the support base pose parameters.

[0085] The scraper conveyor pose calculation combines the position and attitude information in the running track of the coal mining machine, and calculates the pitch angle of the middle trough of each section of the scraper conveyor according to the coupling relationship of the coal mining machine running on the scraper conveyor, to obtain the actual shape of the scraper conveyor.

[0086] The scraper conveyor is connected by the dumbbell pin to form the middle trough of each section, so it can adapt to the bending within a certain range, thereby coupling with the bottom plate. The angle between the center line of the two support shoes of the shearer and the projection of the center line on the horizontal plane is the pitch angle of the shearer body. Since the two support shoes are in real-time contact with the middle trough, the spatial positional relationship between them will directly affect the pitch angle of the shearer body. During the operation of the shearer, the left and right support shoes are in contact with the coal shovel plate of the middle trough. In the process of solving the vertical plane shape of the scraper conveyor, in order to accurately identify the pitch angle of each section of the middle trough passed by the support shoes of the shearer, a coordinate system is established with the first section of the middle trough as the starting point. The pitch angles of multiple sections of the middle trough corresponding to the left support shoe and the two support shoes of the shearer at the initial position are measured and taken as known quantities. Combined with the solving model established based on the coupling relationship between the shearer and the scraper conveyor, the pitch angle of the middle trough contacted by the right support shoe of the shearer during the operation of the shearer can be calculated. According to the obtained pitch angles of the middle troughs, the actual shape of the scraper conveyor can be solved.

[0087] Among them, the trajectory prediction and correction iteration is based on the obtained running trajectory of the shearer to analyze and predict the pose trajectory in the virtual space; based on the pose information of the support base, the top plate point cloud information is obtained, and the space model is constructed by means of Unity3d; the next theoretical trajectory is obtained by similarity analysis, and the real-time shearer trajectory information is fused to correct the space model and the theoretical trajectory.

[0088] Due to the slow change of adjacent coal seams, the trend of adjacent coal seams changes slowly, and the shape of the top and bottom plates formed by the n-th cutting determines the laying shape of the scraper conveyor and the attitude of the n+2-th hydraulic support in the n+1-th coal mining cycle, which has a certain degree of approximation. Therefore, on the basis of the equipment positioning and orientation method, based on the known cutting top plate trajectory and bottom plate trajectory data, the next three-machine state and position are predicted and judged.

[0089] The top and bottom plate curves, hydraulic support support state and scraper conveyor arrangement state are extracted respectively, the three-dimensional shape of the coal seam is depicted, the three-dimensional working face coal seam simulation model is constructed, and it is imported into Unity3d. The mesh division and construction of the coal seam floor are realized by connecting triangles, and the virtual reconstruction of the coal seam is realized by using MeshFilter and MeshRender components in Unity3d. In order to facilitate the description of the coal seam shape along the advancing direction, the robot unit advancing plane as shown in Figure 6 is defined. The plane and the horizontal plane have an angle of the median value of the pitch angle of the support robot base. The tangent vector direction of the advancing plane perpendicular to the robot unit arrangement direction is the working face advancing direction. With the advancing plane as the reference, the straightness of the scraper robot is defined as the span of the projection of its laying trajectory on the advancing plane along the advancing direction. The projection center line is the ideal trajectory, and the difference is used as the compensation for the next advancing.

[0090] In the process of real-time propulsion of mining equipment, the method of real-time line drawing can be used to draw the path of the virtual scene as the equipment advances. This can be achieved by adding a Trail Renderer component to the model. After adding the Trail Renderer component, the path is generated accordingly as the object moves. Trail Renderer is a kind of trail renderer. Set the Time attribute to 10000, which is the duration of 10000s. This achieves the effect of long-time non-disappearance and continuous connection. The Width and Colors of the trail line can also be set to make the trail line effect better. In the case of known path, array points and line drawing method are used to predict the path before virtual propulsion. Define the variable line1 of type Draw, the variable a of type LineRenderer, and the array Vector3[]b for storing the points of line drawing. The specific line drawing programming is as follows:

[0091]

[0092]

[0093] Inputting Vector3[]b can achieve path drawing. The Color.blue method can edit the color of the line segment as desired, so that different paths can be distinguished. The predicted path is corrected through real-time pose information feedback of the coal mining machine. After predicting the running path and reconstructing the working space, relevant decisions can be made through the control program to construct a rough map of the coal seam and integrate the information of distinguishing coal and rock to make comprehensive decisions on the operation of the coal mining machine, so as to more accurately control the cutting height of the rocker arm and the straightness of the hydraulic support and the scraper conveyor.

[0094] As Figure 7As shown, by the virtual reconstruction of the coal seam and the predicted running path, the historical data of each cut of the shearer is fused to construct a spatial model with the mining data of each cut. The spatial model is grid-divided by the performance parameters in the cutting process. According to the data of each layer and the cutting depth, height and speed of the shearer, each divided grid has the same change trend, while the adjacent grids have certain different change trends, which is convenient for similarity analysis. According to the similarity principle between the change trends of adjacent grids, the grid of the model is analyzed from the aspects of pitch angle, roll angle and yaw angle, as well as cutting height and depth, such as the change range of pitch angle and the increasing and decreasing rate range of angle between adjacent grids. At the same time, the grid model will generate a curved surface containing change information in three dimensions. At a certain cutting moment, the shearer is set up a coordinate system with the shearer advancing direction as X, the working face advancing direction as Y and the cutting height direction as Z. If the shearer advancing direction is analyzed, the grid curved surface of X direction at the moment of the shearer is projected onto the XZ plane. Since the curved surface also has a change rate in the Z direction, it will cause errors in similarity analysis. Therefore, when grid-dividing, it is necessary to ensure that there is no mutation in each grid and the change trend is similar. Different parts of the curved surface are selected for projection to obtain curves with the same general trend. Through analysis, the change law of X and Z directions at this moment is obtained. Similarly, the remaining two plane directions are analyzed according to the same method to obtain the similarity law.

[0095] The current cutting data and the cutting data of the previous several cuts are analyzed according to the similarity to obtain the prediction of the cutting trajectory of the next cut and the adjacent several cuts. The real-time shearer pose information is fed back, and the spatial model of the coal seam and the running trajectory are updated in real time, so that the subsequent prediction can be more accurate and efficient.

Claims

1. A laser SLAM-based working face overall workspace virtual reconstruction method, characterized by: Step one, establishing a physical point cloud space, including: Physical point cloud data acquisition; select the characteristic components of the hydraulic support, and scan the entire hydraulic support group by the laser radar installed on the coal mining machine during the movement of the coal mining machine along the scraper conveyor to obtain the point cloud of the entire hydraulic support group. After obtaining the point cloud of the entire hydraulic support group, filtering is performed to extract the local characteristic point cloud of a single hydraulic support, thereby completing the acquisition and processing of physical point cloud data; Solving the pose of the coal mining machine; registering the hydraulic support group point cloud scanned by the laser radar, and analyzing the attitude angle of the laser radar during movement using the quaternion and displacement vector obtained by registration and fitting the motion trajectory of the laser radar. Since the laser radar is rigidly connected to the coal mining machine, the attitude and position information of the laser radar is regarded as the attitude and position information of the coal mining machine; Step two, establishing a virtual point cloud space, including: Establishing a digital twin; select Unity3d as the virtual reality environment platform, and establish the digital twin model of the coal mining machine, hydraulic support and scraper conveyor according to the working face three-machine assembly drawing and sensor measurement results, and complete the initialization of the virtual reality scene; Virtual point cloud acquisition; construct the laser radar digital twin, add the corresponding collision body to the target to be scanned, use the ray collision monitoring method to obtain the world coordinates of the feature component scanning points, and convert the world coordinates to the local coordinates with the laser radar as the coordinate system origin, thereby completing the simulation of the physical point cloud acquisition process; Virtual and real coordinate system integration; analyze the quaternion and displacement vector of the rotation angle and position information of each frame of point cloud in the world coordinate system, so that it can be directly used to drive the movement of the digital twin of the coal mining machine in the virtual reality environment; Full cycle point cloud acquisition and release; drive the digital twin of the coal mining machine to move in the virtual space according to the method of reconstructing the pose of the coal mining machine, and use the laser radar digital twin carried by the coal mining machine to scan the virtual support group one by one to obtain the virtual point cloud data of the feature components of all supports; Step three, establishing a virtual-real fusion channel, including: Hydraulic support virtual-real point cloud registration; register the point clouds obtained in the physical and virtual spaces, transform the rotation and translation matrix obtained by registration, adjust the pose of the hydraulic support digital twin according to the transformation result to make it identical to the pose of the hydraulic support in the physical space, and complete the reconstruction of the overall pose of a single hydraulic support; Scraper conveyor pose solving; combine the position and attitude information in the coal mining machine running track, and solve the pitch angle of the middle trough of each section of the scraper conveyor according to the coupling relationship of the coal mining machine running on the scraper conveyor, thereby obtaining the actual form of the scraper conveyor. Trajectory prediction and correction iteration; based on the obtained running trajectory of the coal mining machine, the pose trajectory is predicted in the virtual space; based on the pose information of the support base, the roof point cloud information is obtained, and the space model is constructed with the help of Unity3d, the next theoretical trajectory is obtained through similarity analysis, and the real-time coal mining machine trajectory information is fused to correct the space model and the theoretical trajectory.

2. The method of claim 1, wherein: In step one, ROS system is used to collect, record and analyze bag files of single frame PCD point cloud files; with the help of PCL point cloud library, local feature point cloud of single hydraulic support is obtained by filtering and segmenting from the whole scene point cloud, and the collection and processing of physical point cloud data are completed.

3. The method of claim 2, wherein: In step one, the A-Loam algorithm is used to correct the pose and running trajectory of the coal mining machine calculated from the bag file, and the space pose solution parameters of the scraper are determined through the principle of double vector pose determination and coordinate system transformation, and then the relative pose solution of the beginning and end joints is completed.

4. The method according to claim 2 or 3, characterized in that: In step one, the straight-through filtering method is adopted, and the target point cloud is extracted from the whole point cloud by setting a space bounding box, and the parameters are set according to the three-machine assembly drawing of the fully mechanized working face and the installation position and vertical scanning angle of the laser radar; according to the moving speed of the coal mining machine and the point cloud publishing frequency, the appropriate sampling frequency is selected, and the point cloud PCD file published when the coal mining machine moves to the position where the laser radar is opposite to the electro-hydraulic controller of the hydraulic support is filtered to obtain the local feature point cloud of the single hydraulic support in the physical space.

5. The method of claim 4, wherein: In step one, before reconstructing the pose of the coal mining machine, the collected point cloud needs to be compensated for motion distortion. The point cloud motion distortion compensation is to match the point cloud at the completion time of the current scanning with the complete point cloud of the last frame, and to project all the point clouds of the current frame to the completion time through linear interpolation of the rotation and translation matrix, so as to complete the point cloud motion distortion compensation. After the motion distortion compensation, the point cloud is matched between frames.

6. The method of claim 5, wherein: In step two, FileStream class is used to realize continuous reading, writing and storage of single frame virtual PCD point cloud data; combined with the actual moving speed of the coal mining machine and the point cloud publishing frequency, the sampling frequency is selected to scan the hydraulic support group in the virtual space one by one to obtain the virtual point cloud data of the feature components of all supports.

7. The method of claim 6, wherein: In step three, NDT+PointtoPlane ICP algorithm is used for continuous registration to obtain the action parameters of the overall attitude adjustment of the support; the support base pose parameters are iteratively solved according to the coordinate axis direction in Unity3d; the virtual support is adjusted according to the solving results of the virtual support group and the original physical point cloud registration for many times until the error accuracy requirement is met, that is, the support base pose can be obtained; based on the real-time pose information of the coal mining machine, the virtual pose information of the middle trough is obtained by inversion, that is, the relative pose information of the beginning and end joints of the floating connection mechanism is obtained, so as to complete the reconstruction of the working space.

8. The method of claim 7, wherein: In step three, based on the known cutting roof track and floor track data, the next three machine state and position are predicted and judged, the top and bottom plate curve, hydraulic support support state and apron conveyor arrangement state are extracted respectively, the three-dimensional form of coal seam is described, the three-dimensional working face coal seam simulation model is constructed, and it is imported into Unity3d, the mesh division and construction of coal seam floor are realized through connecting triangles, and the virtual reconstruction of coal seam is realized by using MeshFilter and MeshRender components in Unity3d.

Images