Mine mining straightness monitoring method and device and electronic equipment

Abstract

The invention provides a mine mining straightness monitoring method and device and electronic equipment, and relates to the technical field of intelligent coal mining. The monitoring equipment is provided with a calculation unit, a scanning unit, a vision unit and an inertial measurement unit, the method is applied to the calculation unit, and the method comprises the following steps: receiving point cloud data sent by the scanning unit, receiving image data sent by the vision unit, and receiving inertial measurement data sent by the inertial measurement unit; performing first data fusion processing on the point cloud data and the inertial measurement data to obtain geometric point cloud information, and performing second data fusion processing on the image data and the inertial measurement data to obtain appearance information; color point cloud information is obtained based on the geometric point cloud information and the appearance information, the straightness information of the working face is extracted from the color point cloud information, and the judgment precision of the straightness of the fully mechanized coal mining working face can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent coal mining, and in particular to a mine mining straightness monitoring method, device and electronic equipment. Background Art

[0002] In current coal production, fully mechanized coal mining technology is mainly used. Fully mechanized coal mining technology is a coal mining technology that integrates a variety of supporting equipment such as coal mining machines, support equipment, and transportation equipment. The main activity place of supporting equipment is the fully mechanized mining face. Under the current development trend of coal mining technology, the fully mechanized mining face is developing in the direction of intelligence and unmanned operation, and it has been able to realize automatic machine-following frame shifting and automated coal cutting on the fully mechanized mining face.

[0003] In the process of coal mining, the straightness of the working face is positively correlated with the quality of coal cutting by the coal mining machine. At present, laser radar scanning units are mostly used in actual projects to obtain the straightness data of the working face to determine the straightness of the fully-mechanized working face. However, the single laser radar scanning unit has a reduced working measurement accuracy and low reliability in environments such as coal dust, water mist, and smooth surfaces, which may cause large errors in the straightness monitoring of the working face trajectory, and then lead to large errors in the judgment of the straightness of the fully-mechanized working face. The fully-mechanized working face straightness detection and adaptive control system is an important issue that needs to be solved in the process of fully-mechanized working face intelligence. Summary of the invention

[0004] In view of the problems existing in the prior art, the present invention provides a mine mining straightness monitoring method, device and electronic equipment.

[0005] The present invention provides a method for monitoring the straightness of mining in a mine. A monitoring device is provided with a computing unit, a scanning unit, a visual unit and an inertial measurement unit. The method is applied to the computing unit and includes: Receiving point cloud data sent by the scanning unit, receiving image data sent by the visual unit, and receiving inertial measurement data sent by the inertial measurement unit; Performing a first data fusion process on the point cloud data and the inertial measurement data to obtain geometric point cloud information, and performing a second data fusion process on the image data and the inertial measurement data to obtain appearance information; Color point cloud information is obtained based on the geometric point cloud information and the appearance information, so as to extract straightness information of the working surface from the color point cloud information.

[0006] According to a mine mining straightness monitoring method provided by the present invention, before performing a first data fusion process on the point cloud data and the inertial measurement data to obtain geometric point cloud information and performing a second data fusion process on the image data and the inertial measurement data to obtain appearance information, the method further includes: Determining a timestamp of the point cloud data, determining a timestamp of the inertial measurement data, and determining a timestamp of the image data; Performing a first data fusion process on the point cloud data and the inertial measurement data to obtain geometric point cloud information, and performing a second data fusion process on the image data and the inertial measurement data to obtain appearance information, specifically includes: When it is determined that the difference between the timestamp of the point cloud data and the timestamp of the inertial measurement data is within a set range, performing a first data fusion process on the point cloud data and the inertial measurement data to obtain geometric point cloud information; When it is determined that the difference between the timestamp of the image data and the timestamp of the inertial measurement data is within a set range, a second data fusion process is performed on the image data and the inertial measurement data to obtain appearance information.

[0007] According to a mine mining straightness monitoring method provided by the present invention, after receiving the point cloud data sent by the scanning unit, receiving the image data sent by the visual unit, and receiving the inertial measurement data sent by the inertial measurement unit, the method further includes: performing a third data fusion process on the point cloud data and the inertial measurement data to obtain first mileage information, and determining the relative position of the monitoring device based on the first mileage information; or Performing a fourth data fusion on the image data and the inertial measurement data to obtain second mileage information, and determining the relative position of the monitoring device based on the second mileage information.

[0008] According to a mine mining straightness monitoring method provided by the present invention, after receiving the point cloud data sent by the scanning unit, receiving the image data sent by the visual unit, and receiving the inertial measurement data sent by the inertial measurement unit, the method further includes: Performing a third data fusion process on the point cloud data and the inertial measurement data to obtain first mileage information, and determining a first error factor of the first mileage information; Performing a fourth data fusion on the image data and the inertial measurement data to obtain second mileage information, and determining a second error factor of the second mileage information; The first error factor and the second error factor are fused through an error-based iterative Kalman filtering algorithm to obtain third mileage information, and the relative position of the monitoring device is determined based on the third mileage information.

[0009] According to a mine mining straightness monitoring method provided by the present invention, after performing a third data fusion process on the point cloud data and the inertial measurement data to obtain the first mileage information, the method further includes: determining a first error factor of the first mileage information, and sending a warning message when the first error factor exceeds a first preset threshold; the warning message is used to inform a user that the first mileage information is invalid; or After performing a fourth data fusion on the image data and the inertial measurement data to obtain second mileage information, the method further includes: Determine a second error factor of the second mileage information, and send an alarm message when the second error factor exceeds a first preset threshold; the alarm message is used to inform a user that the second mileage information is invalid.

[0010] According to a mine mining straightness monitoring method provided by the present invention, the monitoring device is further provided with a time synchronization unit, and the time synchronization unit is used to add a hardware timestamp to the point cloud data collected by the scanning unit, add a hardware timestamp to the image data sent by the visual unit, and add a hardware timestamp to the inertial measurement data sent by the inertial measurement unit; Receiving the point cloud data sent by the scanning unit, receiving the image data sent by the visual unit, and receiving the inertial measurement data sent by the inertial measurement unit, specifically includes: Receiving point cloud data with a timestamp sent by the scanning unit, receiving image data with a timestamp sent by the visual unit, and receiving inertial measurement data with a timestamp sent by the inertial measurement unit; or Before determining the timestamp of the point cloud data, determining the timestamp of the inertial measurement data, and determining the timestamp of the image data, the method further includes: The point cloud data is software-time stamped, the inertial measurement data is software-time stamped, and the image data is software-time stamped.

[0011] According to a mine mining straightness monitoring method provided by the present invention, the monitoring device has a communication unit, and the communication unit is connected to the calculation unit; After obtaining the color point cloud information based on the geometric point cloud information and the appearance information, the method further includes: The color point cloud information, the positioning mileage information and / or the warning information are sent to the communication unit to transmit the color point cloud information, the positioning mileage information and / or the warning information to a remote terminal.

[0012] The present invention also provides a mine mining straightness monitoring device, wherein the monitoring device is provided with a computing unit, a scanning unit, a visual unit and an inertial measurement unit, and the device is applied to the computing unit and comprises: A data receiving module, used for receiving the point cloud data sent by the scanning unit, receiving the image data sent by the visual unit, and receiving the inertial measurement data sent by the inertial measurement unit; A data processing module, configured to perform a first data fusion process on the point cloud data and the inertial measurement data to obtain geometric point cloud information, and perform a second data fusion process on the image data and the inertial measurement data to obtain appearance information; The point cloud generation module is used to obtain color point cloud information based on the geometric point cloud information and the appearance information, so as to extract the straightness information of the working surface from the color point cloud information.

[0013] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, any of the above-mentioned mine mining straightness monitoring methods is implemented.

[0014] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the mine mining straightness monitoring method as described in any one of the above is implemented.

[0015] The mine mining straightness monitoring method, device and electronic device provided in the embodiments of the present invention can receive point cloud data sent by a scanning unit, receive image data sent by a visual unit, and receive inertial measurement data sent by an inertial measurement unit; perform a first data fusion process on the point cloud data and the inertial measurement data to obtain geometric point cloud information, perform a second data fusion process on the image data and the inertial measurement data to obtain appearance information, and obtain color point cloud information based on the geometric point cloud information and the appearance information. This can reduce the risk of a single laser radar being unable to accurately obtain point cloud data in environments such as coal dust, water mist, and smooth surfaces, resulting in large errors in the straightness monitoring of the working face trajectory, and further resulting in the risk of large errors in the judgment of the straightness of the comprehensive mining working face. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 It is a flow chart of the mine mining straightness monitoring method provided by the present invention.

[0018] Figure 2 It is a structural schematic diagram of the mine mining straightness monitoring device provided by the present invention.

[0019] Figure 3 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0021] Combine the following Figure 1-Figure 3 The invention describes a mine mining straightness monitoring method, device and electronic equipment.

[0022] Figure 1 It is a flow chart of the mine mining straightness monitoring method provided by the present invention. The monitoring device is provided with a computing unit, a scanning unit, a visual unit and an inertial measurement unit. The method is applied to the computing unit, such as Figure 1 As shown, the method includes: Step 101: receiving point cloud data sent by the scanning unit, receiving image data sent by the vision unit, and receiving inertial measurement data sent by the inertial measurement unit.

[0023] The scanning unit refers to the device that obtains point cloud data, which can be a laser radar or a depth camera. Point cloud data refers to the point data of the specific structure of the lane where the monitoring device is located. Point cloud data can be the three-dimensional coordinate point information of the specific structure of the lane where the monitoring device is located.

[0024] The visual unit refers to a device for acquiring image data, which may be a panoramic camera or a binocular camera, etc. The image data refers to the data of each pixel in the pixel array, and the image data may be color RGB image data.

[0025] Inertial measurement data refers to data that describes the spatial posture of the monitoring device. The inertial measurement data can be the three-axis angular velocity data and three-axis acceleration data of the basic monitoring device in three-dimensional space, or it can be the three-axis angular velocity data, three-axis acceleration data and three-axis magnetic information of the monitoring device in three-dimensional space. The three-axis magnetic information can be the three axial components of the earth's magnetic field measured by three single-axis magnetometers.

[0026] For example, it can receive the three-dimensional coordinate point information of the specific structure of the lane where the monitoring equipment is located sent by the laser radar, can receive the color RGB image data sent by the panoramic camera, and can receive the three-axis angular velocity data and three-axis acceleration data sent by the inertial measurement unit (IMU).

[0027] Step 102: Perform a first data fusion process on the point cloud data and the inertial measurement data to obtain geometric point cloud information, and perform a second data fusion process on the image data and the inertial measurement data to obtain appearance information.

[0028] Point cloud information refers to a data structure composed of a large number of three-dimensional coordinate point information. Each point in the point cloud information represents a position in a physical space. Geometric point cloud information refers to point cloud information that can characterize the geometric structure in the tunnel environment. The geometric structure may include the tunnel roof, tunnel floor, and hydraulic support. Appearance information refers to the detailed features, material properties, and other information on the surface of objects in the environment.

[0029] For example, the point cloud data may be pre-processed by filtering, denoising, etc. to reduce the impact of invalid or interfering point data, and then the point cloud data and inertial measurement data may be subjected to a first data fusion process based on the state space filtering method to obtain geometric point cloud information. Similarly, the image data may be pre-processed by denoising, color correction, etc. to improve the image data quality, and then the image data and inertial measurement data may be subjected to a second data fusion process based on the graph optimization method to obtain appearance information.

[0030] Step 103 : obtaining color point cloud information based on the geometric point cloud information and the appearance information, so as to extract straightness information of the working surface from the color point cloud information.

[0031] Color point cloud information refers to point cloud information after attribute information is added to the three-dimensional coordinate point information. The attribute information may be a color attribute.

[0032] Exemplarily, the color attribute of the corresponding three-dimensional coordinate point can be extracted based on the appearance information, and the color attribute is added to the corresponding three-dimensional coordinate point to obtain the color point cloud information.

[0033] The straightness information of the working face includes the straightness information of the working face itself and the straightness information of the working face equipment equipped with the monitoring device. The working face equipment may include a coal mining machine, a scraper, a hydraulic support, and the like.

[0034] It can be understood that the color point cloud information obtained by data fusion processing based on the point cloud data, image data and inertial measurement data collected as the monitoring equipment moves can extract the straightness of the monitoring equipment in the mine working surface map environment, and output the corresponding position change curve, providing field data for intelligent planning and cutting. In addition, the flatness of the top and bottom plates can also be extracted, and the corresponding position change curve can be output.

[0035] In one embodiment, when the difference in three-dimensional coordinate point information of adjacent frames of geometric point cloud information is large, the three-dimensional coordinate point information can be corrected based on appearance information, previous frame point cloud information and inertial measurement data, and color point cloud information can be obtained based on the corrected geometric point cloud information and appearance information.

[0036] Exemplarily, texture features can be identified and extracted by comparing appearance information of adjacent frames, and the mode of change can be determined based on the texture features and inertial measurement data; point cloud features can be identified and extracted by comparing point cloud information of adjacent frames, and the point cloud information of the current frame can be corrected based on the point cloud features and the mode of change.

[0037] The mine mining straightness monitoring method provided by the embodiment of the present invention can receive point cloud data sent by a scanning unit, receive image data sent by a visual unit, and receive inertial measurement data sent by an inertial measurement unit after determining that the monitoring equipment is in a target state; perform a first data fusion process on the point cloud data and the inertial measurement data to obtain geometric point cloud information, perform a second data fusion process on the image data and the inertial measurement data to obtain appearance information, and obtain color point cloud information based on the geometric point cloud information and the appearance information. This can reduce the risk of a single laser radar being unable to accurately obtain point cloud data in environments such as coal dust, water mist, and smooth surfaces, resulting in large errors in the straightness monitoring of the working face trajectory, and further resulting in the risk of large errors in the judgment of the straightness of the comprehensive mining working face.

[0038] Based on the above embodiment, before performing a first data fusion process on the point cloud data and the inertial measurement data to obtain geometric point cloud information and performing a second data fusion process on the image data and the inertial measurement data to obtain appearance information, the method further includes: A timestamp of the point cloud data is determined, a timestamp of the inertial measurement data is determined, and a timestamp of the image data is determined.

[0039] Timestamp refers to a string of numbers that is generated based on a preset standard method to record and manage time information. Timestamp can uniquely identify a time point. For example, after obtaining point cloud data, the point cloud data can be analyzed to determine the timestamp of the point cloud data by keyword matching and other methods. The method for determining the timestamp of inertial measurement data and the method for determining the timestamp of image data are basically the same as the method for determining the timestamp of point cloud data, and will not be repeated here.

[0040] Among them, the keyword can be specifically determined according to a specific timestamp generation method.

[0041] Performing a first data fusion process on the point cloud data and the inertial measurement data to obtain geometric point cloud information, and performing a second data fusion process on the image data and the inertial measurement data to obtain appearance information, specifically includes: When it is determined that the difference between the timestamp of the point cloud data and the timestamp of the inertial measurement data is within a set range, performing a first data fusion process on the point cloud data and the inertial measurement data to obtain geometric point cloud information; When it is determined that the difference between the timestamp of the image data and the timestamp of the inertial measurement data is within a set range, a second data fusion process is performed on the image data and the inertial measurement data to obtain appearance information.

[0042] For example, the difference or ratio between the timestamp of the point cloud data and the timestamp of the inertial measurement data can be calculated. If the difference or ratio is less than a set threshold, it is determined that the difference is within the set range, and the point cloud data and the inertial measurement data are considered to be time synchronized, and data fusion is performed. Otherwise, the point cloud data or inertial measurement data is reacquired to ensure that the point cloud data for data fusion is time synchronized with the inertial measurement data. The process of performing the second data fusion processing on the image data and the inertial measurement data is basically the same as the process of performing the first data fusion processing on the point cloud data and the inertial measurement data, and will not be repeated here.

[0043] It is understandable that the setting range can be set according to the time synchronization accuracy requirements of the specific application. For example, the difference can be within the setting range, which can mean that the timestamp of the point cloud data and the timestamp of the inertial measurement data are the same.

[0044] For example, a data fusion algorithm can be used to fuse point cloud data and inertial measurement data, determine spatial coordinate information based on the three-dimensional coordinate point information of the point cloud data, and correct the motion error of the point cloud data based on the inertial measurement data to obtain more accurate geometric point cloud information, indicating the shape and position of objects in three-dimensional space, so as to facilitate subsequent modeling, measurement or path planning, etc. The data fusion algorithm can be a Kalman filter or a particle filter.

[0045] Image processing and visual processing algorithms can be used to fuse image data and inertial measurement data, correct image data or add depth information to image data, etc., to obtain more accurate appearance information for augmented reality applications, etc. Among them, the visual processing algorithm can be feature matching or deep learning.

[0046] The scanning unit, the vision unit and the inertial measurement unit can realize time synchronization through the hardware trigger of the time synchronization unit or through the software trigger of the computing unit for data acquisition.

[0047] Specifically, based on any of the above embodiments, the monitoring device is also provided with a time synchronization unit, and the time synchronization unit is used to add a hardware timestamp to the point cloud data collected by the scanning unit, add a hardware timestamp to the image data sent by the visual unit, and add a hardware timestamp to the inertial measurement data sent by the inertial measurement unit.

[0048] The time synchronization unit can provide a unified time reference through a high-precision clock source, which can be a GPS clock, an atomic clock, or a high-precision clock inside the system. The time synchronization unit can be electrically connected to the scanning unit, the vision unit, and the inertial measurement unit through a hardware interface to capture and synchronize the data collection moment of each unit. At the moment of each data point collection, the time synchronization unit automatically timestamps it to ensure the accurate time of data collection.

[0049] In addition, the time synchronization unit can synchronize the clock signals of the scanning unit, vision unit and inertial measurement unit, so that each unit uses the same clock source for data collection, thereby ensuring that the data collection time of each unit remains consistent, ensuring the consistency and accuracy of the timestamp, and realizing the synchronous collection of heterogeneous data.

[0050] Receiving the point cloud data sent by the scanning unit, receiving the image data sent by the visual unit, and receiving the inertial measurement data sent by the inertial measurement unit, specifically includes: The point cloud data with a timestamp sent by the scanning unit is received, the image data with a timestamp sent by the vision unit is received, and the inertial measurement data with a timestamp sent by the inertial measurement unit is received.

[0051] In another embodiment, before determining the timestamp of the point cloud data, determining the timestamp of the inertial measurement data, and determining the timestamp of the image data, the method further comprises: The point cloud data is software-time stamped, the inertial measurement data is software-time stamped, and the image data is software-time stamped.

[0052] Exemplarily, after collecting point cloud data, the scanning unit directly sends it to the computing unit, and the computing unit can add a timestamp to the point cloud data based on the system clock; after collecting inertial measurement data, the inertial measurement unit directly sends it to the computing unit, and the computing unit can add a timestamp to the inertial measurement data based on the system clock; after collecting image data, the visual unit directly sends it to the computing unit, and the computing unit can add a timestamp to the image data based on the system clock.

[0053] Point cloud data, inertial measurement data and image data are different types of data collected by different data acquisition units. In this embodiment, by marking timestamps for different types of data collected by different data acquisition units, the time synchronization between these data can be achieved, so as to facilitate data alignment during the data fusion process and increase the accuracy of data fusion processing.

[0054] Based on any of the above embodiments, after receiving the point cloud data sent by the scanning unit, receiving the image data sent by the vision unit, and receiving the inertial measurement data sent by the inertial measurement unit, the method further includes: performing a third data fusion process on the point cloud data and the inertial measurement data to obtain first mileage information, and determining the relative position of the monitoring device based on the first mileage information; or Performing a fourth data fusion on the image data and the inertial measurement data to obtain second mileage information, and determining the relative position of the monitoring device based on the second mileage information.

[0055] Among them, what is determined based on the first mileage information or the second mileage information may be the relative position of the monitoring device in the environment, such as the relative position relative to a specified reference point in the tunnel in a mine environment.

[0056] Exemplarily, the distance information of the monitoring device can be determined by point cloud data, and the motion state of the monitoring device can be determined by inertial measurement data. The point cloud data and inertial measurement data are fused to correct errors in the distance information of the monitoring device determined by the point cloud data, and the motion of the monitoring device can be compensated by inertial measurement data to obtain more accurate mileage information.

[0057] For example, the relative position information between the monitoring device and the surrounding environment can be obtained by processing the image data through the visual processing algorithm, the motion state of the monitoring device can be determined through the inertial measurement data, and the image data and the inertial measurement data can be fused to correct the error in the path of the monitoring device determined by the image data, and the motion of the monitoring device can be compensated through the inertial measurement data to obtain more accurate mileage information. Among them, the visual processing algorithm can be image recognition, feature point matching, etc.

[0058] Based on any of the above embodiments, after receiving the point cloud data sent by the scanning unit, receiving the image data sent by the vision unit, and receiving the inertial measurement data sent by the inertial measurement unit, the method further includes: Performing a third data fusion process on the point cloud data and the inertial measurement data to obtain first mileage information, and determining a first error factor of the first mileage information; Performing a fourth data fusion on the image data and the inertial measurement data to obtain second mileage information, and determining a second error factor of the second mileage information; The first error factor and the second error factor are fused through an error-based iterative Kalman filtering algorithm to obtain third mileage information, and the relative position of the monitoring device is determined based on the third mileage information.

[0059] The first error factor refers to a parameter for evaluating the accuracy of the first mileage information, and the second error factor refers to a parameter for evaluating the accuracy of the second mileage information. The specific method for obtaining the first error factor and the second error factor can be selected by those skilled in the art according to actual needs, and the embodiments of the present invention do not make further specific limitations on this.

[0060] It is understandable that in some environments, the point cloud data collected by the scanning unit or the image data collected by the vision unit may contain errors. In this embodiment, the third mileage information is obtained by fusing the first error factor of the first mileage information obtained based on the point cloud data and the second error factor of the second mileage information obtained based on the image data through an error-based iterative Kalman filtering algorithm. The relative position of the monitoring device is determined based on the third mileage information, which can achieve strong robust positioning and improve the reliability of positioning the monitoring device in a signal denial environment.

[0061] The signal denial environment refers to an environment where signals from other positioning systems cannot be received due to interference or shielding factors. The signal denial environment may refer to a GPS denial environment or an electromagnetic interference environment.

[0062] In another embodiment, after obtaining the first mileage information and the second mileage information, the first mileage information and the second mileage information may be backed up for each other and a closed-loop detection may be performed to timely determine whether the first mileage information and the second mileage information are invalid, so that in case the first mileage information or the second mileage information fails, an alarm message can be sent in time to facilitate maintenance and reduce the risk of production accidents caused by positioning failure.

[0063] Based on any of the above embodiments, after performing a third data fusion process on the point cloud data and the inertial measurement data to obtain the first mileage information, the method further includes: Determine a first error factor of the first mileage information, and send an alarm message when the first error factor exceeds a first preset threshold; the alarm message is used to inform a user that the first mileage information is invalid.

[0064] In this embodiment, whether the first mileage information is invalid can be determined by the first error factor of the first mileage information and the preset first preset threshold. When the first mileage information is invalid, an alarm message is sent to the user in time to facilitate maintenance and reduce the risk of production accidents caused by positioning failure.

[0065] In another embodiment, after performing a fourth data fusion on the image data and the inertial measurement data to obtain the second mileage information, the method further includes: Determine a second error factor of the second mileage information, and send an alarm message when the second error factor exceeds a first preset threshold; the alarm message is used to inform a user that the second mileage information is invalid.

[0066] The first preset threshold and the second preset threshold can be set according to the accuracy requirements under actual working conditions, and this embodiment does not make any further limitations on this.

[0067] In this embodiment, whether the second mileage information is invalid can be determined by the second error factor of the second mileage information and the preset second preset threshold. When the second mileage information is invalid, an alarm message is sent to the user in time to facilitate maintenance and reduce the risk of production accidents caused by positioning failure.

[0068] Based on any of the above embodiments, a communication unit is provided on the monitoring device, and the communication unit is connected to the computing unit; After obtaining the color point cloud information based on the geometric point cloud information and the appearance information, the method further includes: The color point cloud information, the positioning mileage information and / or the warning information are sent to the communication unit to transmit the color point cloud information, the positioning mileage information and / or the warning information to a remote terminal.

[0069] The remote terminal in this embodiment may refer to a remote terminal on the ground or a remote terminal underground.

[0070] In this embodiment, by setting up a communication unit, the data collected by real-time processing can be transmitted to the remote terminal to achieve data sharing, so as to facilitate monitoring of the effectiveness of real-time data collection during the operation process through the remote terminal.

[0071] In order to specifically illustrate the function of the mine mining straightness monitoring method provided by this embodiment, a specific example is provided below.

[0072] The monitoring device is equipped with a computing unit, a scanning unit, a vision unit, an inertial measurement unit, a time synchronization unit and a communication unit. The time synchronization unit is used to add a hardware timestamp to the point cloud data collected by the scanning unit, to add a hardware timestamp to the image data sent by the vision unit, and to add a hardware timestamp to the inertial measurement data sent by the inertial measurement unit.

[0073] A mine mining straightness monitoring method is applied to a computing unit of a monitoring device, the method comprising: determining that the monitoring device is in a target state; receiving point cloud data with a timestamp sent by the scanning unit, receiving image data with a timestamp sent by the visual unit, and receiving inertial measurement data with a timestamp sent by the inertial measurement unit; determining the timestamp of the point cloud data, determining the timestamp of the inertial measurement data, and determining the timestamp of the image data; when it is determined that the difference between the timestamp of the point cloud data and the timestamp of the inertial measurement data is within a set range, performing a first data fusion process on the point cloud data and the inertial measurement data to obtain geometric point cloud information; when it is determined that the difference between the timestamp of the image data and the timestamp of the inertial measurement data is within a set range, performing a second data fusion process on the image data and the inertial measurement data to obtain appearance information; obtaining color point cloud information based on the geometric point cloud information and the appearance information to extract the straightness information of the working surface from the color point cloud information; and sending the color point cloud information to the communication unit to transmit the color point cloud information to a remote terminal.

[0074] A mine mining straightness monitoring method provided by an embodiment of the present invention is based on a solution of multi-unit acquisition data fusion, has the advantage of multiple redundancy, and solves the problem of positioning failure of single-unit acquisition data in complex environments.

[0075] The mine mining straightness monitoring device provided by the present invention is described below. The mine mining straightness monitoring device described below and the mine mining straightness monitoring method described above can be referenced to each other.

[0076] Figure 2 Schematic diagram of the structure of the mine mining straightness monitoring device provided by the present invention. The monitoring device is provided with a computing unit, a scanning unit, a visual unit and an inertial measurement unit. The device is applied to the computing unit, such as Figure 2 As shown, the device comprises: A data receiving module 201 is used to receive the point cloud data sent by the scanning unit, receive the image data sent by the visual unit, and receive the inertial measurement data sent by the inertial measurement unit; A data processing module 202 is used to perform a first data fusion process on the point cloud data and the inertial measurement data to obtain geometric point cloud information, and to perform a second data fusion process on the image data and the inertial measurement data to obtain appearance information; The point cloud generation module 203 is used to obtain color point cloud information based on the geometric point cloud information and the appearance information, so as to extract the straightness information of the working surface from the color point cloud information.

[0077] Based on any of the above embodiments, the mine mining straightness monitoring device further includes a timestamp determination module, which is used to determine the timestamp of the point cloud data, determine the timestamp of the inertial measurement data, and determine the timestamp of the image data; The data processing module 202 is specifically used for: When it is determined that the difference between the timestamp of the point cloud data and the timestamp of the inertial measurement data is within a set range, performing a first data fusion process on the point cloud data and the inertial measurement data to obtain geometric point cloud information; When it is determined that the difference between the timestamp of the image data and the timestamp of the inertial measurement data is within a set range, a second data fusion process is performed on the image data and the inertial measurement data to obtain appearance information.

[0078] Based on any of the above embodiments, the mine mining straightness monitoring device further includes a mileage determination module, which is used to: Performing a third data fusion process on the point cloud data and the inertial measurement data to obtain first mileage information; performing a fourth data fusion process on the image data and the inertial measurement data to obtain second mileage information; When it is determined that the first mileage information and the second mileage information are the same, the first mileage information or the second mileage information is determined to be positioning mileage information, so as to locate the monitoring device in a signal denial environment.

[0079] Based on any of the above embodiments, the mileage determination module is specifically used to weight the first mileage information and the second mileage information based on a preset weight factor to obtain positioning mileage information when the first mileage information and the second mileage information are different and the number of points in the point cloud data is within a set number range, so as to locate the monitoring device in a signal denial environment.

[0080] Based on any of the above embodiments, the mine mining straightness monitoring device also includes an alarm module, which is used to send an alarm message when the first mileage information and the second mileage information are different and the number of points in the point cloud data is outside a set range; the alarm message is used to inform the user that the positioning mileage information is invalid.

[0081] Based on any of the above embodiments, the monitoring device is further provided with a time synchronization unit, and the time synchronization unit is used to add a hardware timestamp to the point cloud data collected by the scanning unit, add a hardware timestamp to the image data sent by the visual unit, and add a hardware timestamp to the inertial measurement data sent by the inertial measurement unit; The data receiving module 201 is specifically used to receive the point cloud data with a timestamp sent by the scanning unit, receive the image data with a timestamp sent by the visual unit, and receive the inertial measurement data with a timestamp sent by the inertial measurement unit.

[0082] In another embodiment, the mine mining straightness monitoring device further includes a timestamp marking module for software marking timestamps on the point cloud data, software marking timestamps on the inertial measurement data, and software marking timestamps on the image data.

[0083] Based on any of the above embodiments, the mine mining straightness monitoring device also includes a remote communication module, which is used to send the color point cloud information, the positioning mileage information and / or the alarm information to the communication unit, so as to transmit the color point cloud information, the positioning mileage information and / or the alarm information to a remote terminal.

[0084] Figure 3 An example of a structural diagram of an electronic device is shown in FIG. Figure 3 As shown, the electronic device may include: a processor 310, a communication interface 320, a memory 330 and a communication bus 340, wherein the processor 310, the communication interface 320 and the memory 330 communicate with each other through the communication bus 340. The processor 310 may call the logic instructions in the memory 330 to execute the mine mining straightness monitoring method, the method comprising: determining that the monitoring device is in a target state; receiving the point cloud data sent by the scanning unit, receiving the image data sent by the visual unit, and receiving the inertial measurement data sent by the inertial measurement unit; performing a first data fusion process on the point cloud data and the inertial measurement data to obtain geometric point cloud information, performing a second data fusion process on the image data and the inertial measurement data to obtain appearance information; obtaining color point cloud information based on the geometric point cloud information and the appearance information, so as to extract the straightness information of the working surface from the color point cloud information.

[0085] In addition, the logic instructions in the above-mentioned memory 330 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0086] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the mine mining straightness monitoring method provided by the above-mentioned methods, and the method includes: determining that the monitoring equipment is in a target state; receiving point cloud data sent by the scanning unit, receiving image data sent by the visual unit, and receiving inertial measurement data sent by the inertial measurement unit; performing a first data fusion process on the point cloud data and the inertial measurement data to obtain geometric point cloud information, and performing a second data fusion process on the image data and the inertial measurement data to obtain appearance information; obtaining color point cloud information based on the geometric point cloud information and the appearance information to extract the straightness information of the working surface from the color point cloud information.

[0087] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, it is implemented to execute the mine mining straightness monitoring method provided by the above-mentioned methods, and the method includes: determining that the monitoring device is in a target state; receiving the point cloud data sent by the scanning unit, receiving the image data sent by the visual unit, and receiving the inertial measurement data sent by the inertial measurement unit; performing a first data fusion processing on the point cloud data and the inertial measurement data to obtain geometric point cloud information, and performing a second data fusion processing on the image data and the inertial measurement data to obtain appearance information; obtaining color point cloud information based on the geometric point cloud information and the appearance information to extract the straightness information of the working surface from the color point cloud information.

[0088] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative work.

[0089] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0090] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A mine mining straightness monitoring method, characterized in that: The monitoring device is provided with a computing unit, a scanning unit, a visual unit and an inertial measurement unit. The method is applied to the computing unit and includes: Receiving point cloud data sent by the scanning unit, receiving image data sent by the visual unit, and receiving inertial measurement data sent by the inertial measurement unit; Performing a first data fusion process on the point cloud data and the inertial measurement data to obtain geometric point cloud information, and performing a second data fusion process on the image data and the inertial measurement data to obtain appearance information; Color point cloud information is obtained based on the geometric point cloud information and the appearance information, so as to extract straightness information of the working surface from the color point cloud information.

2. The mine mining straightness monitoring method according to claim 1 is characterized in that: Before performing a first data fusion process on the point cloud data and the inertial measurement data to obtain geometric point cloud information and performing a second data fusion process on the image data and the inertial measurement data to obtain appearance information, the method further includes: Determining a timestamp of the point cloud data, determining a timestamp of the inertial measurement data, and determining a timestamp of the image data; Performing a first data fusion process on the point cloud data and the inertial measurement data to obtain geometric point cloud information, and performing a second data fusion process on the image data and the inertial measurement data to obtain appearance information, specifically includes: When it is determined that the difference between the timestamp of the point cloud data and the timestamp of the inertial measurement data is within a set range, performing a first data fusion process on the point cloud data and the inertial measurement data to obtain geometric point cloud information; When it is determined that the difference between the timestamp of the image data and the timestamp of the inertial measurement data is within a set range, a second data fusion process is performed on the image data and the inertial measurement data to obtain appearance information.

3. The mine mining straightness monitoring method according to claim 1, characterized in that: After receiving the point cloud data sent by the scanning unit, receiving the image data sent by the vision unit, and receiving the inertial measurement data sent by the inertial measurement unit, the method further includes: performing a third data fusion process on the point cloud data and the inertial measurement data to obtain first mileage information, and determining the relative position of the monitoring device based on the first mileage information; or Performing a fourth data fusion on the image data and the inertial measurement data to obtain second mileage information, and determining the relative position of the monitoring device based on the second mileage information.

4. The mine mining straightness monitoring method according to claim 1, characterized in that: After receiving the point cloud data sent by the scanning unit, receiving the image data sent by the vision unit, and receiving the inertial measurement data sent by the inertial measurement unit, the method further includes: Performing a third data fusion process on the point cloud data and the inertial measurement data to obtain first mileage information, and determining a first error factor of the first mileage information; Performing a fourth data fusion on the image data and the inertial measurement data to obtain second mileage information, and determining a second error factor of the second mileage information; The first error factor and the second error factor are fused through an error-based iterative Kalman filtering algorithm to obtain third mileage information, and the relative position of the monitoring device is determined based on the third mileage information.

5. The mine mining straightness monitoring method according to claim 3 is characterized in that: After performing a third data fusion process on the point cloud data and the inertial measurement data to obtain the first mileage information, the method further includes: determining a first error factor of the first mileage information, and sending a warning message when the first error factor exceeds a first preset threshold; the warning message is used to inform a user that the first mileage information is invalid; or After performing a fourth data fusion on the image data and the inertial measurement data to obtain second mileage information, the method further includes: Determine a second error factor of the second mileage information, and send an alarm message when the second error factor exceeds a first preset threshold; the alarm message is used to inform a user that the second mileage information is invalid.

6. The mine mining straightness monitoring method according to claim 1, characterized in that: The monitoring device is also provided with a time synchronization unit, which is used to add a hardware timestamp to the point cloud data collected by the scanning unit, add a hardware timestamp to the image data sent by the visual unit, and add a hardware timestamp to the inertial measurement data sent by the inertial measurement unit; Receiving the point cloud data sent by the scanning unit, receiving the image data sent by the visual unit, and receiving the inertial measurement data sent by the inertial measurement unit, specifically includes: Receiving point cloud data with a timestamp sent by the scanning unit, receiving image data with a timestamp sent by the visual unit, and receiving inertial measurement data with a timestamp sent by the inertial measurement unit; or Before determining the timestamp of the point cloud data, determining the timestamp of the inertial measurement data, and determining the timestamp of the image data, the method further includes: The point cloud data is software-time stamped, the inertial measurement data is software-time stamped, and the image data is software-time stamped.

7. The mine mining straightness monitoring method according to claim 5, characterized in that: A communication unit on the monitoring device, wherein the communication unit is connected to the computing unit; After obtaining the color point cloud information based on the geometric point cloud information and the appearance information, the method further includes: The color point cloud information, the positioning mileage information and / or the warning information are sent to the communication unit to transmit the color point cloud information, the positioning mileage information and / or the warning information to a remote terminal.

8. A mine mining straightness monitoring device, characterized in that: The monitoring device is provided with a computing unit, a scanning unit, a visual unit and an inertial measurement unit, and the device is applied to the computing unit, including: A data receiving module, used for receiving the point cloud data sent by the scanning unit, receiving the image data sent by the visual unit, and receiving the inertial measurement data sent by the inertial measurement unit; A data processing module, configured to perform a first data fusion process on the point cloud data and the inertial measurement data to obtain geometric point cloud information, and perform a second data fusion process on the image data and the inertial measurement data to obtain appearance information; The point cloud generation module is used to obtain color point cloud information based on the geometric point cloud information and the appearance information, so as to extract the straightness information of the working surface from the color point cloud information.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the mine mining straightness monitoring method as described in any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the mine mining straightness monitoring method as claimed in any one of claims 1 to 7 is implemented.

Images