Well mining unmanned driving perception optimization method and device

Abstract

The invention provides an unmanned driving perception optimization method and device for a well mine. The method comprises the following steps: acquiring effective point cloud data, effective image data and effective ultrasonic data of an environment in the well mine in real time; performing fusion processing on the effective point cloud data, the effective image data and the effective ultrasonic data to obtain fusion data; inputting the fusion data into a perception model, and performing environment perception analysis on the fusion data by using the perception model to obtain an environment perception result; making a decision according to the environment sensing result and a preset decision rule, and generating a driving decision and a corresponding control signal, the driving decision being a decision scheme for driving the unmanned vehicle in the well mine; and controlling the unmanned vehicle to run according to the control signal and the driving decision. According to the method, the problem that the unmanned vehicle cannot accurately sense the surrounding environment in real time to make a correct driving decision due to the uncertainty and limitation of a single sensor in the prior art is solved.

Description

Technical Field

[0001] The present invention relates to the field of unmanned driving technology in underground mines, and in particular to an unmanned driving perception optimization method for underground mines, an unmanned driving perception optimization device for underground mines, a computer-readable storage medium, and an unmanned driving perception optimization system for underground mines. Background Art

[0002] Unmanned driving in underground mines is an important development direction in the mining industry, and its application can significantly improve production efficiency and safety. In traditional underground mining operations, a large number of workers are usually required to participate, and the application of unmanned driving technology can reduce manual operations, reduce accident risks, and improve production efficiency.

[0003] However, the mine environment is complex and changeable, including terrain, obstacles, lighting and other factors, which puts higher requirements on the perception performance of unmanned vehicles. In the mine environment, unmanned vehicles need to have high-precision and high-robust perception capabilities to accurately identify and avoid obstacles to ensure driving safety and stability.

[0004] At present, there are still some problems with the perception technology of unmanned vehicles in underground mines. First, the low perception accuracy is one of the main bottlenecks of the current perception technology of unmanned vehicles in underground mines. Due to the complexity and uncertainty of the underground mining environment, the perception system often has difficulty in accurately identifying and locating obstacles, resulting in increased risks of decision-making errors and safety accidents. Secondly, poor robustness is also a problem with the current perception technology of unmanned vehicles in underground mines. In the underground mining environment, factors such as changes in lighting and terrain may interfere with the perception system, resulting in unstable and inaccurate perception results. Summary of the invention

[0005] The main purpose of this application is to provide a method for optimizing perception of unmanned driving in underground mines, a device for optimizing perception of unmanned driving in underground mines, a computer-readable storage medium and a system for optimizing perception of unmanned driving in underground mines, so as to at least solve the problem in the prior art that the uncertainty and limitations of a single sensor cause the unmanned vehicle to be unable to accurately and in real time perceive the surrounding environment to make correct driving decisions.

[0006] In order to achieve the above-mentioned purpose, according to one aspect of the present application, a method for optimizing perception of unmanned driving in underground mines is provided, comprising: acquiring effective point cloud data, effective image data and effective ultrasonic data of the environment in the underground mine in real time; fusing the effective point cloud data, the effective image data and the effective ultrasonic data to obtain fused data; inputting the fused data into a perception model to perform environmental perception analysis on the fused data using the perception model to obtain environmental perception results, wherein the environmental perception results at least include obstacles, road signs and environmental features in the underground mine, the perception model is a convergence model obtained by iterative training by inputting multiple sets of training data into a predetermined model, and the training data includes sample fusion data and sample environmental perception results; making a decision according to the environmental perception result and a preset decision rule to generate a driving decision and a corresponding control signal, the driving decision being a decision plan for an unmanned vehicle to drive in the underground mine, and the control signal being used to control the unmanned vehicle to drive according to the driving decision; controlling the unmanned vehicle to drive according to the driving decision according to the control signal.

[0007] Optionally, obtaining valid point cloud data, valid image data and valid ultrasonic data of the environment in the underground mine includes: obtaining the distance between the laser radar and the object in the underground mine through a laser radar to obtain initial point cloud data, wherein the object includes at least a vehicle, an obstacle and a signboard, and the laser radar is located at the front and roof of the unmanned vehicle; capturing the image of the environment in the underground mine through a camera to obtain initial image data, and the camera is located at the front and side of the unmanned vehicle; obtaining the distance between the object in the underground mine and the ultrasonic sensor through an ultrasonic sensor to obtain initial ultrasonic data, and the ultrasonic sensor is located at the front, the roof and the side of the unmanned vehicle; formatting the initial point cloud data, the initial image data and the initial ultrasonic data to unify the data formats of the initial point cloud data, the initial image data and the initial ultrasonic data, and correspondingly obtaining first point cloud data, first image data and first ultrasonic data; performing preprocessing operations on the first point cloud data, the first image data and the first ultrasonic data to correspondingly obtain the valid point cloud data, the valid image data and the valid ultrasonic data.

[0008] Optionally, a preprocessing operation is performed on the first point cloud data, the first image data and the first ultrasonic data to obtain the valid point cloud data, the valid image data and the valid ultrasonic data, respectively, including: using a first filter to filter the first point cloud data and the first ultrasonic data to obtain second point cloud data and second ultrasonic data, respectively, and the first filter is a moving average filter or a Gaussian filter; using a second filter to filter the first image data to obtain second image data, and the second filter is a median filter or an edge-preserving filter; using a first denoising method to denoise the second point cloud data and the second ultrasonic data to obtain third point cloud data and third ultrasonic data, respectively, and the first denoising method at least includes: One of the following: including a density-based outlier detection method and a random sampling consensus algorithm; using a second denoising method to denoise the second image data to obtain third image data, and the second denoising method includes at least one of the following: morphological operations and threshold segmentation algorithms; performing data registration operations on the third point cloud data, the third ultrasonic data and the third image data, so that the third point cloud data, the third ultrasonic data and the third image data are represented in the same coordinate system, and fourth point cloud data, fourth ultrasonic data and fourth image data are obtained respectively; using a compression method to perform data compression processing on the fourth point cloud data, the fourth ultrasonic data and the fourth image data to obtain the corresponding valid point cloud data, the valid image data and the valid ultrasonic data.

[0009] Optionally, the valid point cloud data, the valid image data and the valid ultrasonic data are fused to obtain fused data, including: extracting features from the valid point cloud data, the valid image data and the valid ultrasonic data to obtain corresponding point cloud features, image features and ultrasonic features, respectively, wherein the point cloud features at least include the density, distribution and structure of the valid point cloud data, the image features at least include the image edge, image texture and image color of the valid image data, and the ultrasonic features at least include the distance to the obstacle and the shape of the obstacle; performing pattern recognition on the point cloud features, the image features and the ultrasonic features to associate the point cloud features, the image features and the ultrasonic features with specific targets in an observation environment to obtain description data, wherein the observation environment is the environment in the underground mine, and the specific target is at least one of the following: the obstacles, road signs and mine equipment in the observation environment; grouping all the description data according to the same specific target, and one specific target corresponds to a group of description data; fusing the description data in each specific target using a fusion algorithm to obtain the fused data.

[0010] Optionally, before inputting the fused data into the perception model so as to use the perception model to perform environmental perception analysis on the fused data to obtain the environmental perception result, the method further includes: using a historical data set to verify and evaluate the historical perception model to obtain an evaluation result, the historical perception model is a model that has been subjected to perception analysis after training the predetermined model before the current moment, and the evaluation result is one of the following: the historical perception model has poor performance, the historical perception model has medium performance, and the historical perception model has high performance; when the evaluation result is that the historical perception model has poor performance or the historical perception model has medium performance, the model parameters of the historical perception model are adjusted and the model structure of the historical perception model is optimized to obtain the perception model, the model parameters include at least weights, thresholds or learning rates, the model structure includes at least the number of layers of the historical perception model, activation functions and structures of each model layer in the historical perception model, and the model layers include at least input layers, output layers and pooling layers.

[0011] Optionally, making a decision based on the environmental perception result and a preset decision rule to generate a driving decision and a corresponding control signal includes: a decision-making step, making a decision based on the environmental perception result and a preset decision rule to generate a preliminary driving decision and a corresponding preliminary control signal; a matching step, matching the preliminary driving decision with the dynamic characteristics of the unmanned vehicle; in a case where the preliminary driving decision does not match the dynamic characteristics, repeating the decision-making step and the matching step in sequence at least once until the preliminary driving decision matches the dynamic characteristics; in a case where the preliminary driving decision matches the dynamic characteristics, determining the preliminary driving decision as the driving decision, and determining the corresponding preliminary control signal as the control signal.

[0012] Optionally, controlling the unmanned vehicle to drive according to the driving decision based on the control signal includes: obtaining a real-time driving result of the unmanned vehicle; comparing the real-time driving result with the driving decision to obtain a comparison result, wherein the comparison result is the similarity between the real-time driving result and the driving decision; when the similarity is greater than or equal to a similarity threshold, continuing to control the unmanned vehicle to drive according to the driving decision according to the control signal; when the similarity is less than the similarity threshold, re-adjusting the driving decision and the corresponding control signal, and controlling the unmanned vehicle to drive according to the adjusted driving decision according to the adjusted control signal.

[0013] According to another aspect of the present application, an unmanned driving perception optimization device for an underground mine is provided, the device comprising: an acquisition unit, for acquiring effective point cloud data, effective image data and effective ultrasonic data of an environment in the underground mine in real time; a fusion unit, for fusing the effective point cloud data, the effective image data and the effective ultrasonic data to obtain fused data; a perception unit, for inputting the fused data into a perception model, so as to perform environmental perception analysis on the fused data using the perception model to obtain environmental perception results, the environmental perception results at least including obstacles, road signs and environmental features in the underground mine, the perception model is a convergence model obtained by iterative training by inputting multiple sets of training data into a predetermined model, the training data comprising sample fusion data and sample environmental perception results; a decision unit, for making a decision according to the environmental perception result and a preset decision rule, generating a driving decision and a corresponding control signal, the driving decision being a decision plan for an unmanned vehicle to drive in the underground mine, the control signal being used to control the unmanned vehicle to drive according to the driving decision; a control unit, for controlling the unmanned vehicle to drive according to the driving decision according to the control signal.

[0014] According to another aspect of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute any one of the methods described.

[0015] According to another aspect of the present application, an unmanned driving perception optimization system for underground mines is provided, comprising: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include methods for executing any one of the methods described.

[0016] By applying the technical solution of the present application, in the unmanned driving perception optimization method for underground mines, first, effective point cloud data, effective image data and effective ultrasonic data of the environment inside the underground mine are obtained in real time; then, the effective point cloud data, the effective image data and the effective ultrasonic data are fused and processed to obtain fused data; then, the fused data is input into the perception model to use the perception model to perform environmental perception analysis on the fused data to obtain environmental perception results, which at least include obstacles, road signs and environmental features in the underground mine, and the perception model is a convergence model obtained by iterative training by inputting multiple sets of training data into a predetermined model, and the training data includes sample fusion data and sample environmental perception results; then, a decision is made according to the environmental perception results and preset decision rules to generate a driving decision and a corresponding control signal, the driving decision is a decision plan for the unmanned vehicle to drive in the underground mine, and the control signal is used to control the unmanned vehicle to drive according to the driving decision; finally, the unmanned vehicle is controlled to drive according to the driving decision according to the control signal. This application establishes multiple sensors for collection and preprocessing to obtain effective sensor data, namely effective point cloud data, effective image data and effective ultrasonic data, and fuses different sensor data. By fusing data from multiple sensors, more comprehensive and accurate information about the underground mining environment can be obtained, and the perception model can be optimized and updated according to actual production needs and historical data. Based on the optimized perception model and real-time perception data, the underground mining environment is perceived and decided in real time. Through real-time perception and decision-making, the driving and operation of unmanned vehicles can be guided to achieve safe and efficient unmanned driving in underground mines. This application solves the problem in the prior art that the uncertainty and limitations of a single sensor make it impossible for unmanned vehicles to perceive the surrounding environment in real time to make correct driving decisions. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A hardware structure block diagram of a mobile terminal for executing an unmanned driving perception optimization method for underground mines provided in an embodiment of the present application is shown;

[0018] Figure 2 A schematic diagram of a process flow of an unmanned driving perception optimization method for underground mines provided according to an embodiment of the present application is shown;

[0019] Figure 3 A schematic diagram of a flow chart of an optional underground mining unmanned driving perception optimization method provided according to an embodiment of the present application is shown;

[0020] Figure 4 A structural block diagram of an unmanned driving perception optimization device for underground mines provided according to an embodiment of the present application is shown.

[0021] The above drawings include the following reference numerals:

[0022] 102, processor; 104, memory; 106, transmission device; 108, input and output devices. DETAILED DESCRIPTION

[0023] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0024] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0025] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0026] As introduced in the background technology, due to the complexity and uncertainty of the underground mining environment in the prior art, the perception system often finds it difficult to accurately identify and locate obstacles, resulting in increased risks of decision-making errors and safety accidents. In order to solve the problem that the uncertainty and limitations of a single sensor in the prior art cause unmanned vehicles to be unable to accurately perceive the surrounding environment in real time to make correct driving decisions, the embodiments of the present application provide an underground mining unmanned driving perception optimization method, an underground mining unmanned driving perception optimization device, a computer-readable storage medium and an underground mining unmanned driving perception optimization system.

[0027] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0028] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on a mobile terminal as an example, Figure 1 1 is a hardware structure block diagram of a mobile terminal of an unmanned driving perception optimization method for underground mines according to an embodiment of the present invention. Figure 1 As shown, the mobile terminal may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data, wherein the mobile terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the mobile terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0029] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the unmanned driving perception optimization method of underground mining in the embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, the above method is implemented. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories can be connected to the mobile terminal via a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof. The transmission device 106 is used to receive or send data via a network. The above-mentioned specific examples of the network may include a wireless network provided by a communication provider of the mobile terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0030] In this embodiment, a method for optimizing perception of unmanned driving in underground mines that runs on a mobile terminal, a computer terminal or a similar computing device is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in an order different from that shown here.

[0031] Figure 2 1 is a flow chart of the unmanned driving perception optimization method for underground mining according to an embodiment of the present application. Figure 2 As shown, the method comprises the following steps:

[0032] Step S201, obtaining effective point cloud data, effective image data and effective ultrasonic data of the environment inside the underground mine in real time.

[0033] Specifically, Figure 3 As shown in the figure, by using a variety of sensors, including laser radar, camera, ultrasonic wave, etc., comprehensive collection of mine environment data is achieved. These sensors can adapt to the complex and changeable environment of mines, obtain key information such as terrain, obstacles, and lighting, and provide rich data for subsequent perception processing. The collected data is preprocessed to obtain the preprocessed data, namely the above-mentioned valid point cloud data, the above-mentioned valid image data, and the above-mentioned valid ultrasonic data.

[0034] Step S202: fusing the valid point cloud data, the valid image data and the valid ultrasonic data to obtain fused data.

[0035] Specifically, multi-sensor fusion technology is used to fuse the pre-processed data obtained by different sensors. By fusing the data of different sensors, more comprehensive and accurate information about the underground mining environment can be obtained. For example, the laser radar provides distance information of terrain and obstacles, the camera provides visual information of the terrain, and the ultrasound provides distance information of surrounding obstacles. This information is integrated through the fusion algorithm to generate more accurate perception data, that is, the above-mentioned fused data.

[0036] Step S203, input the above-mentioned fused data into the perception model, so as to use the above-mentioned perception model to perform environmental perception analysis on the above-mentioned fused data to obtain environmental perception results, and the above-mentioned environmental perception results at least include obstacles, road signs and environmental features in the above-mentioned underground mine. The above-mentioned perception model is a convergent model obtained by iterative training by inputting multiple groups of training data into a predetermined model, and the above-mentioned training data includes sample fusion data and sample environmental perception results.

[0037] Specifically, based on real-time data and historical data, the existing model is optimized and updated to obtain the above-mentioned perception model. This includes operations such as adjusting model parameters and updating model structure to improve the prediction accuracy and robustness of the perception model. At the same time, the model is updated online according to real-time perception data to adapt to changes in the underground mining environment. The above-mentioned fused data is input into the perception model to use the above-mentioned perception model to perform real-time perception analysis of the underground mining environment according to the above-mentioned fused data to obtain environmental perception results. This enables unmanned vehicles to perceive the surrounding environment in real time, make correct driving decisions, and ensure driving safety and stability. At the same time, real-time perception decision support can improve production efficiency and reduce manual intervention and accident risks.

[0038] Step S204, making a decision based on the above-mentioned environmental perception results and preset decision rules, generating a driving decision and a corresponding control signal, wherein the above-mentioned driving decision is a decision plan for the unmanned vehicle to drive in the above-mentioned underground mine, and the above-mentioned control signal is used to control the unmanned vehicle to drive according to the above-mentioned driving decision.

[0039] Specifically, after obtaining the environmental perception results, the decision-making phase begins. The goal of this phase is to make driving decisions based on the perception results and preset rules or strategies. Decision-making may involve multiple aspects, such as path planning, obstacle avoidance, speed control, etc. According to the complexity of the underground mining environment and specific application requirements, different decision rules and strategies can be formulated to generate driving decisions and control signals based on the results of the decision-making phase. These signals can include steering angles, acceleration / deceleration instructions, braking instructions, etc. to guide the driving of underground mining unmanned vehicles.

[0040] Step S205: Control the unmanned vehicle to travel according to the control signal and the driving decision.

[0041] Specifically, after the driving decision and the control signal are generated, the unmanned vehicle can be controlled to drive according to the driving decision and the control signal.

[0042] In this embodiment, first, effective point cloud data, effective image data and effective ultrasonic data of the environment in the underground mine are acquired in real time; then, the effective point cloud data, the effective image data and the effective ultrasonic data are fused to obtain fused data; then, the fused data is input into a perception model to use the perception model to perform environmental perception analysis on the fused data to obtain environmental perception results, wherein the environmental perception results at least include obstacles, road signs and environmental features in the underground mine, and the perception model is a convergence model obtained by iterative training by inputting multiple sets of training data into a predetermined model, and the training data includes sample fusion data and sample environmental perception results; then, a decision is made according to the environmental perception results and preset decision rules to generate a driving decision and a corresponding control signal, wherein the driving decision is a decision plan for the unmanned vehicle to drive in the underground mine, and the control signal is used to control the unmanned vehicle to drive according to the driving decision; finally, the unmanned vehicle is controlled to drive according to the driving decision according to the control signal. This application establishes multiple sensors for collection and preprocessing to obtain effective sensor data, namely effective point cloud data, effective image data and effective ultrasonic data, and fuses different sensor data. By fusing data from multiple sensors, more comprehensive and accurate information about the underground mining environment can be obtained, and the perception model can be optimized and updated according to actual production needs and historical data. Based on the optimized perception model and real-time perception data, the underground mining environment is perceived and decided in real time. Through real-time perception and decision-making, the driving and operation of unmanned vehicles can be guided to achieve safe and efficient unmanned driving in underground mines. This application solves the problem in the prior art that the uncertainty and limitations of a single sensor make it impossible for unmanned vehicles to perceive the surrounding environment in real time to make correct driving decisions.

[0043] In order to enable those skilled in the art to more clearly understand the technical solution of the present application, the implementation process of the underground mining unmanned driving perception optimization method of the present application will be described in detail below in combination with specific embodiments.

[0044] In order to improve the comprehensiveness of acquiring data in the underground mining environment, in an optional implementation manner, the above step S201 includes:

[0045] Step S2011, obtaining the distance between the laser radar and the object in the underground mine through the laser radar to obtain initial point cloud data, wherein the object includes at least a vehicle, an obstacle and a sign, and the laser radar is located at the front and roof of the unmanned vehicle;

[0046] Step S2012, capturing the above-mentioned internal environment image of the underground mine through a camera to obtain initial image data, wherein the above-mentioned camera is located at the above-mentioned front of the above-mentioned unmanned vehicle and the side of the above-mentioned unmanned vehicle;

[0047] Step S2013, obtaining the distance between the object in the underground mine and the ultrasonic sensor through an ultrasonic sensor to obtain initial ultrasonic data, wherein the ultrasonic sensor is located on the front, roof and side of the unmanned vehicle;

[0048] Step S2014, formatting the initial point cloud data, the initial image data and the initial ultrasonic data to unify the data formats of the initial point cloud data, the initial image data and the initial ultrasonic data, and obtaining first point cloud data, first image data and first ultrasonic data correspondingly;

[0049] Step S2015, performing a preprocessing operation on the first point cloud data, the first image data and the first ultrasonic data, and correspondingly obtaining the valid point cloud data, the valid image data and the valid ultrasonic data.

[0050] In the above embodiment, first, a perception data acquisition system is constructed through a variety of sensors (such as laser radar, camera, ultrasound, etc.). These sensors are distributed in different locations to fully cover the underground mining environment, and the underground mining environment data is obtained in real time through a variety of sensors, such as using laser radar to obtain initial point cloud data, cameras to obtain initial image data, and ultrasonic sensors to obtain initial ultrasonic data. Collect raw data from various sensors (laser radar, camera, ultrasound, etc.), which are usually stored in different formats and resolutions. Organize these data into a unified format and structure for subsequent processing. For example, laser radar data includes point cloud data, camera data may include image frames, and ultrasonic data may include distance readings. After formatting, preprocessing operations are performed to obtain the final valid point cloud data, the above-mentioned valid image data, and the above-mentioned valid ultrasonic data.

[0051] It should also be noted that the following types of sensors can be selected: LiDAR, LiDAR is a sensor used for distance measurement. It calculates the distance between the object and the sensor by emitting a laser beam and measuring the time it takes to reflect back. In underground mining environments, LiDAR can be used to obtain distance information of terrain and obstacles, providing accurate positioning and navigation support for unmanned vehicles. Camera, a camera is a sensor used for visual perception that can capture images of the surrounding environment. In underground mining environments, cameras can be used to obtain visual information of the terrain, such as terrain features, color, texture, etc. This visual information can help unmanned vehicles identify obstacles, road markings, and other key information. Ultrasonic sensor, an ultrasonic sensor is another type of sensor used for distance measurement. It is used to calculate the distance between an object and the sensor by sending ultrasonic waves and measuring the time it takes to reflect back. In underground mining environments, ultrasonic sensors can be used to detect the distance information of surrounding obstacles and provide obstacle perception capabilities for unmanned vehicles. Sensors are installed at different locations of underground mining unmanned vehicles to ensure comprehensive coverage of the surrounding environment. For example, LiDAR is installed at the front or top, cameras are installed at the front or side, and ultrasonic sensors can be installed at different locations of the vehicle as needed. The layout of sensors needs to consider factors such as their field of view, resolution, and accuracy to ensure that various information in the underground mining environment can be accurately perceived. Each sensor is equipped with a data transmission interface for transmitting the perception data to the data processing module. These interfaces can be wired or wireless, depending on the actual application scenario and requirements. Data transmission rate and stability are crucial for real-time perception, so it is necessary to select appropriate transmission methods and interface standards to ensure the real-time and accuracy of data.

[0052] In order to significantly improve the quality and consistency of sensor data, in an optional implementation manner, the above step S2015 includes:

[0053] Step S20151, using a first filter to filter the first point cloud data and the first ultrasonic data to obtain second point cloud data and second ultrasonic data respectively, wherein the first filter is a moving average filter or a Gaussian filter;

[0054] Step S20152, using a second filter to filter the first image data to obtain second image data, wherein the second filter is a median filter or an edge preserving filter;

[0055] Step S20153, using a first denoising method to denoise the second point cloud data and the second ultrasonic data to obtain third point cloud data and third ultrasonic data, respectively, wherein the first denoising method includes at least one of the following: a density-based outlier detection method and a random sampling consensus algorithm;

[0056] Step S20154, performing denoising on the second image data using a second denoising method to obtain third image data, wherein the second denoising method includes at least one of the following: morphological operation and threshold segmentation algorithm;

[0057] Step S20155, performing a data registration operation on the third point cloud data, the third ultrasonic data and the third image data, so that the third point cloud data, the third ultrasonic data and the third image data are represented in the same coordinate system, and obtaining fourth point cloud data, fourth ultrasonic data and fourth image data respectively;

[0058] Step S20156, using a compression method to perform data compression processing on the above-mentioned fourth point cloud data, the above-mentioned fourth ultrasonic data and the above-mentioned fourth image data, to obtain the corresponding above-mentioned valid point cloud data, the above-mentioned valid image data and the above-mentioned valid ultrasonic data.

[0059] In the above embodiment, the purpose of filtering is to eliminate random errors or noise in the sensor data. These noises may be caused by inherent errors of the sensor, environmental factors or interference during data transmission. For the point cloud data of the laser radar and the ultrasonic data of the ultrasonic sensor, a moving average filter or a Gaussian filter can be used to smooth the data and reduce noise points. For the image data of the camera, a median filter or an edge-preserving filter can be used to reduce noise and artifacts in the image while retaining important edge information. After filtering, the second point cloud data, the second ultrasonic data and the second image data are obtained. Denoising is a step to further clean the data, aiming to eliminate abnormal values ​​or outliers that are not related to the actual environment. These outliers may be caused by sensor failure, environmental factors or other interference. For point cloud data and ultrasonic data, statistical methods (such as random sampling consensus algorithms) or distance-based methods (such as density-based outlier detection) can be used to identify and remove outliers. For image data, morphological operations (such as open and closed operations) or pixel value-based methods (such as threshold segmentation) can be used to eliminate noise and artifacts. After denoising, the third point cloud data, the third ultrasonic data and the third image data are obtained. Registration is the process of aligning data from different sensors to ensure that they are represented in the same coordinate system. This is crucial for multi-sensor fusion and subsequent perception processing. For the data registration of lidar and camera, the method adopted is to estimate the spatial relationship (rotation and translation matrix) between them by finding common features (such as edges, corners, etc.) between the two sensors. This can be achieved by using ICP (iterative closest point) or feature-based methods (such as SIFT, SURF, etc.). For other types of sensors (such as ultrasonic), registration can be achieved by a timestamp-based method or by associating with data from other sensors. After the data registration operation, the fourth point cloud data, the fourth ultrasonic data and the fourth image data are obtained. After that, data enhancement processing is performed. Data enhancement is to increase the size and diversity of the data set by applying transformations (such as rotation, translation, scaling, etc.), thereby improving the generalization ability of the model. This is particularly useful when training perception models. Standardization is to adjust the data to a uniform scale or range to eliminate the influence of unit and range differences between different sensors. This helps to speed up the convergence of the algorithm and improve the performance of the perception model. In order to increase processing speed and reduce storage requirements, the data after data enhancement can be compressed. Common compression methods include principal component analysis or other dimensionality reduction techniques. The compressed data can be stored in an appropriate format (such as PCD files for point cloud data, JPEG or PNG for image data) for subsequent processing and analysis.Through the above detailed data preprocessing process, the quality and consistency of sensor data can be significantly improved, providing reliable basic data support for subsequent multi-sensor fusion, perception model optimization and real-time perception decision-making.

[0060] In order to improve the perception performance and decision-making accuracy of unmanned vehicles in underground mines, in an optional implementation, the above step S202 includes:

[0061] Step S2021, performing feature extraction on the valid point cloud data, the valid image data and the valid ultrasonic data to obtain corresponding point cloud features, image features and ultrasonic features, respectively, wherein the point cloud features at least include the density, distribution and structure of the valid point cloud data, the image features at least include the image edge, image texture and image color of the valid image data, and the ultrasonic features at least include the distance to the obstacle and the shape of the obstacle;

[0062] Step S2022, performing pattern recognition on the point cloud features, the image features and the ultrasonic features to associate the point cloud features, the image features and the ultrasonic features with specific targets in the observation environment to obtain description data, wherein the observation environment is the environment in the underground mine, and the specific target is at least one of the following: the obstacles, road signs and mine equipment in the observation environment;

[0063] Step S2023, grouping all the description data according to the same specific target, where one specific target corresponds to one group of description data;

[0064] Step S2024, fusing the description data in each of the specific targets using a fusion algorithm to obtain the fused data.

[0065] In the above embodiment, the preprocessed data is sent to the multi-sensor fusion module so that more comprehensive and accurate information about the underground mining environment can be obtained by fusing the data of different sensors. For example, the laser radar provides distance information of the terrain and obstacles, the camera provides visual information of the terrain, and the ultrasound provides distance information of the surrounding obstacles. This information is integrated through the fusion algorithm to generate more accurate perception data. The process of multi-sensor fusion is a process of comprehensively utilizing information from multiple sensors to optimize decision-making and estimation. The following is a detailed process of multi-sensor fusion: (1) Data collection: Collect various types of preprocessed sensor data (i.e., the above-mentioned point cloud data, the above-mentioned effective image data, and the above-mentioned effective ultrasonic data). These data may exist in the form of discrete or continuous time function data, output vectors, imaging data, or direct attribute descriptions. (2) Feature extraction: Extract features from the collected sensor data. This usually involves converting the raw data into feature vectors that represent the observed data that are easier to process and analyze. For example, for laser radar data, features may include the density, distribution, and structure of point cloud data; for camera data, features may include the edges, textures, and colors of the image; for ultrasonic data, features may include distance measurements and the shape of obstacles. (3) Pattern recognition: Perform pattern recognition processing on the extracted feature vectors. This can include clustering algorithms, adaptive neural networks, or other statistical pattern recognition methods that can convert feature vectors into target attribute judgments. Pattern recognition aims to associate sensor data with specific targets in the observed environment, thereby completing the description of the target by each sensor. (4) Association and association establishment: Group the description data of each sensor about a specific target according to the same specific target, that is, association. This involves determining which sensor data are related to the same target and integrating these data together. Association establishment is a key step in ensuring consistency and complementarity between multi-sensor data. (5) Application of fusion algorithm: Use fusion algorithm to synthesize the sensor data of the target to obtain a consistent interpretation and description of the target. Fusion algorithms can be selected according to different application scenarios and requirements, such as weighted average, Kalman filter, Bayesian network, etc. These algorithms can comprehensively utilize the information of multiple sensors to improve the accuracy and robustness of decision-making and estimation. (6) Output: After multi-sensor fusion processing, the consistent interpretation and description of the target obtained can be used for subsequent decision-making and control. For example, in unmanned vehicles in underground mining, the fused data can be used for tasks such as navigation, obstacle avoidance, and path planning. Through the above detailed multi-sensor fusion process, the complementary information of multiple sensors can be fully utilized to improve the perception performance and decision-making accuracy of unmanned vehicles in underground mining. This fusion method helps to reduce the limitations of a single sensor and improve the intelligence and robustness of the system.

[0066] In order to improve the representation and generalization capabilities of the perception model, in an optional implementation, before the above step S203, the method further includes:

[0067] Step S301, using the historical data set to verify and evaluate the historical perception model to obtain an evaluation result, the historical perception model is a model that has been subjected to perception analysis after the training of the predetermined model is completed before the current moment, and the evaluation result is one of the following: the historical perception model has low performance, the historical perception model has medium performance, and the historical perception model has high performance;

[0068] Step S302, when the evaluation result shows that the performance of the historical perception model is poor or the performance of the historical perception model is medium, the model parameters of the historical perception model are adjusted and the model structure of the historical perception model is optimized to obtain the perception model, wherein the model parameters at least include weights, thresholds or learning rates, and the model structure at least includes the number of layers of the historical perception model, the activation function and the structure of each model layer in the historical perception model, and the model layers at least include an input layer, an output layer and a pooling layer.

[0069] In the above embodiment, before optimizing and updating the perception model, it is first necessary to evaluate the existing model (i.e., the above-mentioned historical perception model). This can be done by re-verifying using historical data sets. Evaluation indicators may include accuracy, recall, F1 score, etc. to quantify the performance of the model. According to the evaluation results, the parameters of the perception model are adjusted. This may include adjusting the weights, thresholds, or other hyperparameters of the neural network. By adjusting the parameters, the predictive performance of the model can be optimized. If the performance of the existing model is poor, it may be necessary to optimize the architecture of the model. This may include increasing or decreasing the number of layers, changing the activation function, adjusting the pooling layer, etc. By optimizing the model structure, the representation and generalization capabilities of the perception model can be improved.

[0070] It should also be noted that in underground mining environments, environmental conditions and obstacles may change over time. Therefore, the perception model needs to have the ability to learn and update online. Through online learning, the model can update its parameters and structure in real time to adapt to changes in the environment. This can be achieved through methods such as incremental learning and online gradient descent. A feedback mechanism is established to evaluate and adjust the perception model based on real-time perception results. If there is a large difference between the actual underground mining environment and the perception results predicted by the perception model, the model can be improved by adjusting parameters, optimizing structure, or adding data. This feedback mechanism helps to continuously optimize the perception model and improve its adaptability and accuracy. After the perception model is optimized and updated, it needs to be re-verified and tested. This can be done by using new data sets or testing in actual underground mining environments. The purpose of verification and testing is to ensure that the optimized model can perform well in actual applications and meet the expected performance requirements. Through the above steps, the perception model can be continuously optimized and updated to improve its perception performance and robustness in underground mining environments. This optimization and update process can be flexibly adjusted according to actual needs and environmental changes to ensure that unmanned vehicles can operate safely and efficiently in underground mining environments.

[0071] In order to improve the perception accuracy of unmanned vehicles in underground mines, in an optional implementation, the above step S204 includes:

[0072] Step S2041, a decision-making step, making a decision based on the above-mentioned environmental perception result and a preset decision rule, generating a preliminary driving decision and a corresponding preliminary control signal;

[0073] Step S2042, a matching step, matching the preliminary driving decision with the dynamic characteristics of the unmanned vehicle;

[0074] Step S2043, when the preliminary driving decision does not match the dynamic characteristics, repeating the decision-making step and the matching step at least once in sequence until the preliminary driving decision matches the dynamic characteristics;

[0075] Step S2044, when the preliminary driving decision matches the dynamic characteristic, the preliminary driving decision is determined as the driving decision, and the corresponding preliminary control signal is determined as the control signal.

[0076] In the above embodiment, in the real-time perception stage, the optimized perception model is used to perceive the underground mining environment in real time. This includes obtaining real-time data from various sensors (such as lidar, camera, ultrasound, etc.) and using the perception model for prediction and interpretation. The goal of real-time perception is to obtain accurate perception results about the underground mining environment, obstacles, road markings and other key information. This includes characteristics such as position, size, shape, color, speed, etc. After obtaining the real-time perception results, the decision-making stage begins. The goal of this stage is to make driving decisions based on the perception results and preset rules or strategies. Decision-making may involve multiple aspects, such as path planning, obstacle avoidance, speed control, etc. Different decision rules and strategies can be formulated according to the complexity of the underground mining environment and specific application requirements. Driving decision output outputs driving decisions and control signals based on the results of the decision-making stage. These signals may include steering angles, acceleration / deceleration instructions, braking instructions, etc. to guide the driving of underground mining unmanned vehicles. Driving decisions need to match the vehicle's dynamic characteristics and control strategies to ensure that the vehicle can execute decisions safely and efficiently. Therefore, it is necessary to match and verify the above-mentioned preliminary driving decision with the dynamic characteristics of the above-mentioned unmanned vehicle; when the above-mentioned preliminary driving decision does not match the above-mentioned dynamic characteristics, the above-mentioned decision-making steps and the above-mentioned matching steps are repeated in sequence at least once until the above-mentioned preliminary driving decision matches the above-mentioned dynamic characteristics, so as to obtain the final driving decision and control signal.

[0077] In order to ensure the safety and stability during unmanned driving, in an optional implementation, the above step S205 includes:

[0078] Step S2051, obtaining the real-time driving result of the unmanned vehicle;

[0079] Step S2052, comparing the real-time driving result with the driving decision to obtain a comparison result, wherein the comparison result is a similarity between the real-time driving result and the driving decision;

[0080] Step S2053, when the similarity is greater than or equal to the similarity threshold, continue to control the unmanned vehicle to travel according to the driving decision based on the control signal;

[0081] Step S2054, when the similarity is less than the similarity threshold, readjust the driving decision and the corresponding control signal, and control the unmanned vehicle to drive according to the adjusted driving decision based on the adjusted control signal.

[0082] In the above embodiment, by comparing the actual driving result with the expected driving decision, the accuracy and effectiveness of the decision can be evaluated. If it is found that there is a large difference between the actual driving result and the expected one, that is, when the above similarity is greater than or equal to the similarity threshold, the decision rule or strategy can be adjusted in time, and the unmanned vehicle can adjust the parameters such as driving speed and direction in real time to ensure the safety and stability of driving.

[0083] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0084] The embodiment of the present application also provides an unmanned driving perception optimization device for underground mines. It should be noted that the unmanned driving perception optimization device for underground mines in the embodiment of the present application can be used to execute the unmanned driving perception optimization method for underground mines provided in the embodiment of the present application. The device is used to implement the above-mentioned embodiments and preferred implementation modes, and those that have been explained will not be repeated. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived.

[0085] The following is an introduction to the unmanned driving perception optimization device for underground mining provided in an embodiment of the present application.

[0086] Figure 4 1 is a structural block diagram of an unmanned driving perception optimization device for underground mining according to an embodiment of the present application. Figure 4 As shown, the device comprises:

[0087] The acquisition unit 10 is used to acquire effective point cloud data, effective image data and effective ultrasonic data of the environment inside the underground mine in real time.

[0088] Specifically, Figure 3 As shown in the figure, by using a variety of sensors, including laser radar, camera, ultrasonic wave, etc., comprehensive collection of mine environment data is achieved. These sensors can adapt to the complex and changeable environment of mines, obtain key information such as terrain, obstacles, and lighting, and provide rich data for subsequent perception processing. The collected data is preprocessed to obtain the preprocessed data, namely the above-mentioned valid point cloud data, the above-mentioned valid image data, and the above-mentioned valid ultrasonic data.

[0089] The fusion unit 20 is used to fuse the above-mentioned effective point cloud data, the above-mentioned effective image data and the above-mentioned effective ultrasonic data to obtain fused data.

[0090] Specifically, multi-sensor fusion technology is used to fuse the pre-processed data obtained by different sensors. By fusing the data of different sensors, more comprehensive and accurate information about the underground mining environment can be obtained. For example, the laser radar provides distance information of terrain and obstacles, the camera provides visual information of the terrain, and the ultrasound provides distance information of surrounding obstacles. This information is integrated through the fusion algorithm to generate more accurate perception data, that is, the above-mentioned fused data.

[0091] The perception unit 30 is used to input the above-mentioned fused data into the perception model, so as to use the above-mentioned perception model to perform environmental perception analysis on the above-mentioned fused data to obtain environmental perception results. The above-mentioned environmental perception results at least include obstacles, road signs and environmental features in the above-mentioned underground mine. The above-mentioned perception model is a convergent model obtained by iterative training by inputting multiple groups of training data into a predetermined model. The above-mentioned training data includes sample fusion data and sample environmental perception results.

[0092] Specifically, based on real-time data and historical data, the existing model is optimized and updated to obtain the above-mentioned perception model. This includes operations such as adjusting model parameters and updating model structure to improve the prediction accuracy and robustness of the perception model. At the same time, the model is updated online according to real-time perception data to adapt to changes in the underground mining environment. The above-mentioned fused data is input into the perception model to use the above-mentioned perception model to perform real-time perception analysis of the underground mining environment according to the above-mentioned fused data to obtain environmental perception results. This enables unmanned vehicles to perceive the surrounding environment in real time, make correct driving decisions, and ensure driving safety and stability. At the same time, real-time perception decision support can improve production efficiency and reduce manual intervention and accident risks.

[0093] The decision unit 40 is used to make decisions based on the above-mentioned environmental perception results and preset decision rules, and generate driving decisions and corresponding control signals. The above-mentioned driving decisions are decision plans for the unmanned vehicle to drive in the above-mentioned underground mine, and the above-mentioned control signals are used to control the unmanned vehicle to drive according to the above-mentioned driving decisions.

[0094] Specifically, after obtaining the environmental perception results, the decision-making phase begins. The goal of this phase is to make driving decisions based on the perception results and preset rules or strategies. Decision-making may involve multiple aspects, such as path planning, obstacle avoidance, speed control, etc. According to the complexity of the underground mining environment and specific application requirements, different decision rules and strategies can be formulated to generate driving decisions and control signals based on the results of the decision-making phase. These signals can include steering angles, acceleration / deceleration instructions, braking instructions, etc. to guide the driving of underground mining unmanned vehicles.

[0095] The control unit 50 is used to control the unmanned vehicle to travel according to the control signal and the driving decision.

[0096] Specifically, after the driving decision and the control signal are generated, the unmanned vehicle can be controlled to drive according to the driving decision and the control signal.

[0097] In this embodiment, the acquisition unit is used to acquire in real time the effective point cloud data, effective image data and effective ultrasonic data of the environment in the underground mine; the fusion unit is used to fuse the above-mentioned effective point cloud data, the above-mentioned effective image data and the above-mentioned effective ultrasonic data to obtain fused data; the perception unit is used to input the above-mentioned fused data into the perception model, so as to use the above-mentioned perception model to perform environmental perception analysis on the above-mentioned fused data to obtain environmental perception results, and the above-mentioned environmental perception results at least include obstacles, road signs and environmental features in the above-mentioned underground mine, and the above-mentioned perception model is a convergence model obtained by iterative training by inputting multiple groups of training data into a predetermined model, and the above-mentioned training data includes sample fusion data and sample environmental perception results; the decision unit is used to make a decision according to the above-mentioned environmental perception results and preset decision rules, and generate a driving decision and a corresponding control signal, the above-mentioned driving decision is a decision plan for the unmanned vehicle to drive in the above-mentioned underground mine, and the above-mentioned control signal is used to control the unmanned vehicle to drive according to the above-mentioned driving decision; the control unit is used to control the above-mentioned unmanned vehicle to drive according to the above-mentioned driving decision according to the above-mentioned control signal. This application establishes multiple sensors for collection and preprocessing to obtain effective sensor data, namely effective point cloud data, effective image data and effective ultrasonic data, and fuses different sensor data. By fusing data from multiple sensors, more comprehensive and accurate information about the underground mining environment can be obtained, and the perception model can be optimized and updated according to actual production needs and historical data. Based on the optimized perception model and real-time perception data, the underground mining environment is perceived and decided in real time. Through real-time perception and decision-making, the driving and operation of unmanned vehicles can be guided to achieve safe and efficient unmanned driving in underground mines. This application solves the problem in the prior art that the uncertainty and limitations of a single sensor make it impossible for unmanned vehicles to perceive the surrounding environment in real time to make correct driving decisions.

[0098] In order to improve the comprehensiveness of data acquired in the underground mining environment, in an optional implementation manner, the acquisition unit includes:

[0099] A first acquisition module acquires the distance between the laser radar and the object in the mine through the laser radar to obtain initial point cloud data, wherein the object includes at least a vehicle, an obstacle and a sign, and the laser radar is located at the front and roof of the unmanned vehicle;

[0100] A second acquisition module is configured to capture the image of the environment in the underground mine through a camera to obtain initial image data, wherein the camera is located at the front of the unmanned vehicle and the side of the unmanned vehicle;

[0101] A third acquisition module acquires the distance between the object in the mine and the ultrasonic sensor through an ultrasonic sensor to obtain initial ultrasonic data, wherein the ultrasonic sensor is located on the front, roof and side of the unmanned vehicle;

[0102] a formatting module, which formats the initial point cloud data, the initial image data and the initial ultrasonic data to unify the data formats of the initial point cloud data, the initial image data and the initial ultrasonic data, and obtains first point cloud data, first image data and first ultrasonic data correspondingly;

[0103] The preprocessing module performs preprocessing operations on the first point cloud data, the first image data and the first ultrasonic data to obtain the valid point cloud data, the valid image data and the valid ultrasonic data accordingly.

[0104] In the above embodiment, first, a perception data acquisition system is constructed through a variety of sensors (such as laser radar, camera, ultrasound, etc.). These sensors are distributed in different locations to fully cover the underground mining environment, and the underground mining environment data is obtained in real time through a variety of sensors, such as using laser radar to obtain initial point cloud data, cameras to obtain initial image data, and ultrasonic sensors to obtain initial ultrasonic data. Collect raw data from various sensors (laser radar, camera, ultrasound, etc.), which are usually stored in different formats and resolutions. Organize these data into a unified format and structure for subsequent processing. For example, laser radar data includes point cloud data, camera data may include image frames, and ultrasonic data may include distance readings. After formatting, preprocessing operations are performed to obtain the final valid point cloud data, the above-mentioned valid image data, and the above-mentioned valid ultrasonic data.

[0105] It should also be noted that the following types of sensors can be selected: LiDAR, LiDAR is a sensor used for distance measurement. It calculates the distance between the object and the sensor by emitting a laser beam and measuring the time it takes to reflect back. In underground mining environments, LiDAR can be used to obtain distance information of terrain and obstacles, providing accurate positioning and navigation support for unmanned vehicles. Camera, a camera is a sensor used for visual perception that can capture images of the surrounding environment. In underground mining environments, cameras can be used to obtain visual information of the terrain, such as terrain features, color, texture, etc. This visual information can help unmanned vehicles identify obstacles, road markings, and other key information. Ultrasonic sensor, an ultrasonic sensor is another type of sensor used for distance measurement. It is used to calculate the distance between an object and the sensor by sending ultrasonic waves and measuring the time it takes to reflect back. In underground mining environments, ultrasonic sensors can be used to detect the distance information of surrounding obstacles and provide obstacle perception capabilities for unmanned vehicles. Sensors are installed at different locations of underground mining unmanned vehicles to ensure comprehensive coverage of the surrounding environment. For example, LiDAR is installed at the front or top, cameras are installed at the front or side, and ultrasonic sensors can be installed at different locations of the vehicle as needed. The layout of sensors needs to consider factors such as their field of view, resolution, and accuracy to ensure that various information in the underground mining environment can be accurately perceived. Each sensor is equipped with a data transmission interface for transmitting the perception data to the data processing module. These interfaces can be wired or wireless, depending on the actual application scenario and requirements. Data transmission rate and stability are crucial for real-time perception, so it is necessary to select appropriate transmission methods and interface standards to ensure the real-time and accuracy of data.

[0106] In order to significantly improve the quality and consistency of sensor data, in an optional implementation, the preprocessing module includes:

[0107] A first filtering processing submodule, using a first filter to filter the first point cloud data and the first ultrasonic data to obtain second point cloud data and second ultrasonic data, respectively, wherein the first filter is a moving average filter or a Gaussian filter;

[0108] A second filtering processing submodule, using a second filter to filter the first image data to obtain second image data, wherein the second filter is a median filter or an edge preserving filter;

[0109] A first denoising submodule, which performs denoising on the second point cloud data and the second ultrasonic data using a first denoising method to obtain third point cloud data and third ultrasonic data, respectively, wherein the first denoising method includes at least one of the following: a density-based outlier detection method and a random sampling consensus algorithm;

[0110] A second denoising submodule, using a second denoising method to denoise the second image data to obtain third image data, wherein the second denoising method includes at least one of the following: morphological operation and threshold segmentation algorithm;

[0111] a data registration submodule, performing a data registration operation on the third point cloud data, the third ultrasonic data and the third image data, so that the third point cloud data, the third ultrasonic data and the third image data are represented in the same coordinate system, and obtaining fourth point cloud data, fourth ultrasonic data and fourth image data respectively;

[0112] The data compression submodule uses a compression method to perform data compression processing on the fourth point cloud data, the fourth ultrasonic data and the fourth image data to obtain the corresponding valid point cloud data, the valid image data and the valid ultrasonic data.

[0113] In the above embodiment, the purpose of filtering is to eliminate random errors or noise in the sensor data. These noises may be caused by inherent errors of the sensor, environmental factors or interference during data transmission. For the point cloud data of the laser radar and the ultrasonic data of the ultrasonic sensor, a moving average filter or a Gaussian filter can be used to smooth the data and reduce noise points. For the image data of the camera, a median filter or an edge-preserving filter can be used to reduce noise and artifacts in the image while retaining important edge information. After filtering, the second point cloud data, the second ultrasonic data and the second image data are obtained. Denoising is a step to further clean the data, aiming to eliminate abnormal values ​​or outliers that are not related to the actual environment. These outliers may be caused by sensor failure, environmental factors or other interference. For point cloud data and ultrasonic data, statistical methods (such as random sampling consensus algorithms) or distance-based methods (such as density-based outlier detection) can be used to identify and remove outliers. For image data, morphological operations (such as open and closed operations) or pixel value-based methods (such as threshold segmentation) can be used to eliminate noise and artifacts. After denoising, the third point cloud data, the third ultrasonic data and the third image data are obtained. Registration is the process of aligning data from different sensors to ensure that they are represented in the same coordinate system. This is crucial for multi-sensor fusion and subsequent perception processing. For the data registration of lidar and camera, the method adopted is to estimate the spatial relationship (rotation and translation matrix) between them by finding common features (such as edges, corners, etc.) between the two sensors. This can be achieved by using ICP (iterative closest point) or feature-based methods (such as SIFT, SURF, etc.). For other types of sensors (such as ultrasonic), registration can be achieved by a timestamp-based method or by associating with data from other sensors. After the data registration operation, the fourth point cloud data, the fourth ultrasonic data and the fourth image data are obtained. After that, data enhancement processing is performed. Data enhancement is to increase the size and diversity of the data set by applying transformations (such as rotation, translation, scaling, etc.), thereby improving the generalization ability of the model. This is particularly useful when training perception models. Standardization is to adjust the data to a uniform scale or range to eliminate the influence of unit and range differences between different sensors. This helps to speed up the convergence of the algorithm and improve the performance of the perception model. In order to increase processing speed and reduce storage requirements, the data after data enhancement can be compressed. Common compression methods include principal component analysis or other dimensionality reduction techniques. The compressed data can be stored in an appropriate format (such as PCD files for point cloud data, JPEG or PNG for image data) for subsequent processing and analysis.Through the above detailed data preprocessing process, the quality and consistency of sensor data can be significantly improved, providing reliable basic data support for subsequent multi-sensor fusion, perception model optimization and real-time perception decision-making.

[0114] In order to improve the perception performance and decision-making accuracy of unmanned vehicles in underground mines, in an optional implementation, the fusion unit includes:

[0115] a feature extraction module, performing feature extraction on the valid point cloud data, the valid image data and the valid ultrasonic data, and obtaining corresponding point cloud features, image features and ultrasonic features, respectively, wherein the point cloud features at least include the density, distribution and structure of the valid point cloud data, the image features at least include the image edge, image texture and image color of the valid image data, and the ultrasonic features at least include the distance to the obstacle and the shape of the obstacle;

[0116] A pattern recognition module performs pattern recognition on the point cloud features, the image features and the ultrasonic features to associate the point cloud features, the image features and the ultrasonic features with specific targets in an observation environment to obtain description data, wherein the observation environment is an environment in the underground mine, and the specific target is at least one of the following: the obstacles, road signs and mine equipment in the observation environment;

[0117] A grouping module groups all the description data according to the same specific target, where one specific target corresponds to one group of description data;

[0118] The fusion processing module fuses the above-mentioned description data in each of the above-mentioned specific targets using a fusion algorithm to obtain the above-mentioned fusion data.

[0119] In the above embodiment, the preprocessed data is sent to the multi-sensor fusion module so that more comprehensive and accurate information about the underground mining environment can be obtained by fusing the data of different sensors. For example, the laser radar provides distance information of the terrain and obstacles, the camera provides visual information of the terrain, and the ultrasound provides distance information of the surrounding obstacles. This information is integrated through the fusion algorithm to generate more accurate perception data. The process of multi-sensor fusion is a process of comprehensively utilizing information from multiple sensors to optimize decision-making and estimation. The following is a detailed process of multi-sensor fusion: (1) Data collection: Collect various types of preprocessed sensor data (i.e., the above-mentioned point cloud data, the above-mentioned effective image data, and the above-mentioned effective ultrasonic data). These data may exist in the form of discrete or continuous time function data, output vectors, imaging data, or direct attribute descriptions. (2) Feature extraction: Extract features from the collected sensor data. This usually involves converting the raw data into feature vectors that represent the observed data that are easier to process and analyze. For example, for laser radar data, features may include the density, distribution, and structure of point cloud data; for camera data, features may include the edges, textures, and colors of the image; for ultrasonic data, features may include distance measurements and the shape of obstacles. (3) Pattern recognition: Perform pattern recognition processing on the extracted feature vectors. This can include clustering algorithms, adaptive neural networks, or other statistical pattern recognition methods that can convert feature vectors into target attribute judgments. Pattern recognition aims to associate sensor data with specific targets in the observed environment, thereby completing the description of the target by each sensor. (4) Association and association establishment: Group the description data of each sensor about a specific target according to the same specific target, that is, association. This involves determining which sensor data are related to the same target and integrating these data together. Association establishment is a key step in ensuring consistency and complementarity between multi-sensor data. (5) Application of fusion algorithm: Use fusion algorithm to synthesize the sensor data of the target to obtain a consistent interpretation and description of the target. Fusion algorithms can be selected according to different application scenarios and requirements, such as weighted average, Kalman filter, Bayesian network, etc. These algorithms can comprehensively utilize the information of multiple sensors to improve the accuracy and robustness of decision-making and estimation. (6) Output: After multi-sensor fusion processing, the consistent interpretation and description of the target obtained can be used for subsequent decision-making and control. For example, in unmanned vehicles in underground mining, the fused data can be used for tasks such as navigation, obstacle avoidance, and path planning. Through the above detailed multi-sensor fusion process, the complementary information of multiple sensors can be fully utilized to improve the perception performance and decision-making accuracy of unmanned vehicles in underground mining. This fusion method helps to reduce the limitations of a single sensor and improve the intelligence and robustness of the system.

[0120] In order to improve the representation ability and generalization ability of the perception model, in an optional implementation, the device further includes:

[0121] an evaluation unit, for inputting the fused data into the perception model so as to use the perception model to perform environmental perception analysis on the fused data and obtaining an environmental perception result, and for verifying and evaluating the historical perception model using a historical data set to obtain an evaluation result, wherein the historical perception model is a model that has been subjected to perception analysis after training of the predetermined model is completed before the current moment, and the evaluation result is one of the following: the historical perception model has low performance, the historical perception model has medium performance, and the historical perception model has high performance;

[0122] An optimization unit is used to adjust the model parameters of the historical perception model and optimize the model structure of the historical perception model when the evaluation result shows that the performance of the historical perception model is poor or the performance of the historical perception model is medium, so as to obtain the above-mentioned perception model, wherein the model parameters at least include weights, thresholds or learning rates, and the model structure at least includes the number of layers of the historical perception model, activation functions and structures of each model layer in the historical perception model, and the model layers at least include input layer, output layer and pooling layer.

[0123] In the above embodiment, before optimizing and updating the perception model, it is first necessary to evaluate the existing model (i.e., the above-mentioned historical perception model). This can be done by re-verifying using historical data sets. Evaluation indicators may include accuracy, recall, F1 score, etc. to quantify the performance of the model. According to the evaluation results, the parameters of the perception model are adjusted. This may include adjusting the weights, thresholds, or other hyperparameters of the neural network. By adjusting the parameters, the predictive performance of the model can be optimized. If the performance of the existing model is poor, it may be necessary to optimize the architecture of the model. This may include increasing or decreasing the number of layers, changing the activation function, adjusting the pooling layer, etc. By optimizing the model structure, the representation and generalization capabilities of the perception model can be improved.

[0124] It should also be noted that in underground mining environments, environmental conditions and obstacles may change over time. Therefore, the perception model needs to have the ability to learn and update online. Through online learning, the model can update its parameters and structure in real time to adapt to changes in the environment. This can be achieved through methods such as incremental learning and online gradient descent. A feedback mechanism is established to evaluate and adjust the perception model based on real-time perception results. If there is a large difference between the actual underground mining environment and the perception results predicted by the perception model, the model can be improved by adjusting parameters, optimizing structure, or adding data. This feedback mechanism helps to continuously optimize the perception model and improve its adaptability and accuracy. After the perception model is optimized and updated, it needs to be re-verified and tested. This can be done by using new data sets or testing in actual underground mining environments. The purpose of verification and testing is to ensure that the optimized model can perform well in actual applications and meet the expected performance requirements. Through the above steps, the perception model can be continuously optimized and updated to improve its perception performance and robustness in underground mining environments. This optimization and update process can be flexibly adjusted according to actual needs and environmental changes to ensure that unmanned vehicles can operate safely and efficiently in underground mining environments.

[0125] In order to improve the perception accuracy of unmanned vehicles in underground mines, in an optional implementation manner, the decision unit includes:

[0126] A decision module is used to execute the decision steps, make decisions based on the above-mentioned environmental perception results and preset decision rules, and generate preliminary driving decisions and corresponding preliminary control signals;

[0127] A matching module, used to perform a matching step to match the preliminary driving decision with the dynamic characteristics of the unmanned vehicle;

[0128] a repeating module, for, when the preliminary driving decision does not match the dynamic characteristics, sequentially repeating the decision-making step and the matching step at least once, until the preliminary driving decision matches the dynamic characteristics;

[0129] A determination module determines the preliminary driving decision as the driving decision and determines the corresponding preliminary control signal as the control signal when the preliminary driving decision matches the dynamic characteristic.

[0130] In the above embodiment, in the real-time perception stage, the optimized perception model is used to perceive the underground mining environment in real time. This includes obtaining real-time data from various sensors (such as lidar, camera, ultrasound, etc.) and using the perception model for prediction and interpretation. The goal of real-time perception is to obtain accurate perception results about the underground mining environment, obstacles, road markings and other key information. This includes characteristics such as position, size, shape, color, speed, etc. After obtaining the real-time perception results, the decision-making stage begins. The goal of this stage is to make driving decisions based on the perception results and preset rules or strategies. Decision-making may involve multiple aspects, such as path planning, obstacle avoidance, speed control, etc. Different decision rules and strategies can be formulated according to the complexity of the underground mining environment and specific application requirements. Driving decision output outputs driving decisions and control signals based on the results of the decision-making stage. These signals may include steering angles, acceleration / deceleration instructions, braking instructions, etc. to guide the driving of underground mining unmanned vehicles. Driving decisions need to match the vehicle's dynamic characteristics and control strategies to ensure that the vehicle can execute decisions safely and efficiently. Therefore, it is necessary to match and verify the above-mentioned preliminary driving decision with the dynamic characteristics of the above-mentioned unmanned vehicle; when the above-mentioned preliminary driving decision does not match the above-mentioned dynamic characteristics, the above-mentioned decision-making steps and the above-mentioned matching steps are repeated in sequence at least once until the above-mentioned preliminary driving decision matches the above-mentioned dynamic characteristics, so as to obtain the final driving decision and control signal.

[0131] In order to ensure the safety and stability during unmanned driving, in an optional implementation, the control unit includes:

[0132] A fourth acquisition module is used to acquire the real-time driving result of the unmanned vehicle.

[0133] A comparison module compares the real-time driving result with the driving decision to obtain a comparison result, wherein the comparison result is a similarity between the real-time driving result and the driving decision;

[0134] A first control module, when the similarity is greater than or equal to a similarity threshold, continues to control the unmanned vehicle to travel according to the driving decision based on the control signal;

[0135] The second control module readjusts the driving decision and the corresponding control signal when the similarity is less than the similarity threshold, and controls the unmanned vehicle to drive according to the adjusted driving decision based on the adjusted control signal.

[0136] In the above embodiment, by comparing the actual driving result with the expected driving decision, the accuracy and effectiveness of the decision can be evaluated. If it is found that there is a large difference between the actual driving result and the expected one, that is, when the above similarity is greater than or equal to the similarity threshold, the decision rule or strategy can be adjusted in time, and the unmanned vehicle can adjust the parameters such as driving speed and direction in real time to ensure the safety and stability of driving.

[0137] The above-mentioned unmanned driving perception optimization device for underground mines includes a processor and a memory. The above-mentioned acquisition unit, fusion unit and perception unit are all stored in the memory as program units, and the processor executes the above-mentioned program units stored in the memory to realize the corresponding functions. The above-mentioned modules are all located in the same processor; or, the above-mentioned modules are located in different processors in any combination.

[0138] The processor contains a kernel, which retrieves the corresponding program unit from the memory. One or more kernels can be set, and the kernel parameters can be adjusted to solve the problem in the existing technology that the uncertainty and limitations of a single sensor make it impossible for the driverless vehicle to perceive the surrounding environment in real time and make correct driving decisions.

[0139] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0140] An embodiment of the present invention provides a computer-readable storage medium, which includes a stored program, wherein when the program is running, the device where the computer-readable storage medium is located is controlled to execute the unmanned driving perception optimization method for underground mining.

[0141] Specifically, the perception optimization method for unmanned driving in underground mines includes:

[0142] Step S201, real-time acquisition of valid point cloud data, valid image data and valid ultrasonic data of the environment inside the underground mine;

[0143] Step S202, fusing the valid point cloud data, the valid image data and the valid ultrasonic data to obtain fused data;

[0144] Step S203, inputting the fused data into a perception model, so as to use the perception model to perform environmental perception analysis on the fused data to obtain environmental perception results, wherein the environmental perception results at least include obstacles, road signs and environmental features in the underground mine, and the perception model is a convergent model obtained by iteratively training by inputting multiple sets of training data into a predetermined model, and the training data includes sample fusion data and sample environmental perception results;

[0145] Step S204, making a decision based on the above-mentioned environmental perception result and the preset decision rule, generating a driving decision and a corresponding control signal, wherein the above-mentioned driving decision is a decision plan for the unmanned vehicle to drive in the above-mentioned underground mine, and the above-mentioned control signal is used to control the unmanned vehicle to drive according to the above-mentioned driving decision;

[0146] Step S205: Control the unmanned vehicle to travel according to the control signal and the driving decision.

[0147] An embodiment of the present invention provides a processor, which is used to run a program, wherein the above-mentioned unmanned driving perception optimization method for underground mining is executed when the above-mentioned program is run.

[0148] An embodiment of the present invention provides an unmanned driving perception optimization system for underground mining. The unmanned driving perception optimization system for underground mining includes a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, at least the following steps are implemented:

[0149] Step S201, real-time acquisition of valid point cloud data, valid image data and valid ultrasonic data of the environment inside the underground mine;

[0150] Step S202, fusing the valid point cloud data, the valid image data and the valid ultrasonic data to obtain fused data;

[0151] Step S203, inputting the fused data into a perception model, so as to use the perception model to perform environmental perception analysis on the fused data to obtain environmental perception results, wherein the environmental perception results at least include obstacles, road signs and environmental features in the underground mine, and the perception model is a convergent model obtained by iteratively training by inputting multiple sets of training data into a predetermined model, and the training data includes sample fusion data and sample environmental perception results;

[0152] Step S204, making a decision based on the above-mentioned environmental perception result and the preset decision rule, generating a driving decision and a corresponding control signal, wherein the above-mentioned driving decision is a decision plan for the unmanned vehicle to drive in the above-mentioned underground mine, and the above-mentioned control signal is used to control the unmanned vehicle to drive according to the above-mentioned driving decision;

[0153] Step S205: Control the unmanned vehicle to travel according to the control signal and the driving decision.

[0154] The present application also provides a computer program product, which, when executed on a data processing device, is suitable for executing a program for initializing at least the following method steps:

[0155] Step S201, real-time acquisition of valid point cloud data, valid image data and valid ultrasonic data of the environment inside the underground mine;

[0156] Step S202, fusing the valid point cloud data, the valid image data and the valid ultrasonic data to obtain fused data;

[0157] Step S203, inputting the fused data into a perception model, so as to use the perception model to perform environmental perception analysis on the fused data to obtain environmental perception results, wherein the environmental perception results at least include obstacles, road signs and environmental features in the underground mine, and the perception model is a convergent model obtained by iteratively training by inputting multiple sets of training data into a predetermined model, and the training data includes sample fusion data and sample environmental perception results;

[0158] Step S204, making a decision based on the above-mentioned environmental perception result and the preset decision rule, generating a driving decision and a corresponding control signal, wherein the above-mentioned driving decision is a decision plan for the unmanned vehicle to drive in the above-mentioned underground mine, and the above-mentioned control signal is used to control the unmanned vehicle to drive according to the above-mentioned driving decision;

[0159] Step S205: Control the unmanned vehicle to travel according to the control signal and the driving decision.

[0160] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0161] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0162] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0163] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0164] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0165] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0166] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0167] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0168] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0169] From the above description, it can be seen that the above embodiments of the present application achieve the following technical effects:

[0170] 1) The unmanned driving perception optimization method of the underground mine of the present application first obtains effective point cloud data, effective image data and effective ultrasonic data of the environment in the underground mine in real time; then, the above-mentioned effective point cloud data, the above-mentioned effective image data and the above-mentioned effective ultrasonic data are fused and processed to obtain fused data; then, the above-mentioned fused data is input into the perception model, so as to use the above-mentioned perception model to perform environmental perception analysis on the above-mentioned fused data to obtain environmental perception results, and the above-mentioned environmental perception results at least include obstacles, road signs and environmental features in the above-mentioned underground mine. The above-mentioned perception model is a convergence model obtained by iterative training by inputting multiple groups of training data into a predetermined model, and the above-mentioned training data includes sample fusion data and sample environmental perception results; then, a decision is made according to the above-mentioned environmental perception results and preset decision rules to generate a driving decision and a corresponding control signal. The above-mentioned driving decision is a decision plan for the unmanned vehicle to drive in the above-mentioned underground mine, and the above-mentioned control signal is used to control the unmanned vehicle to drive according to the above-mentioned driving decision; finally, the above-mentioned unmanned vehicle is controlled to drive according to the above-mentioned driving decision according to the above-mentioned control signal. This application establishes multiple sensors for collection and preprocessing to obtain effective sensor data, namely effective point cloud data, effective image data and effective ultrasonic data, and fuses different sensor data. By fusing data from multiple sensors, more comprehensive and accurate information about the underground mining environment can be obtained, and the perception model can be optimized and updated according to actual production needs and historical data. Based on the optimized perception model and real-time perception data, the underground mining environment is perceived and decided in real time. Through real-time perception and decision-making, the driving and operation of unmanned vehicles can be guided to achieve safe and efficient unmanned driving in underground mines. This application solves the problem in the prior art that the uncertainty and limitations of a single sensor make it impossible for unmanned vehicles to perceive the surrounding environment in real time to make correct driving decisions.

[0171] 2) The unmanned driving perception optimization device for underground mines of the present application comprises an acquisition unit for acquiring effective point cloud data, effective image data and effective ultrasonic data of the environment inside the underground mine in real time; a fusion unit for fusing the effective point cloud data, the effective image data and the effective ultrasonic data to obtain fused data; a perception unit for inputting the fused data into a perception model to perform environmental perception analysis on the fused data using the perception model to obtain environmental perception results, wherein the environmental perception results at least include obstacles, road signs and environmental features inside the underground mine, the perception model is a convergence model obtained by iterative training by inputting multiple sets of training data into a predetermined model, and the training data includes sample fusion data and sample environmental perception results; a decision unit for making decisions based on the environmental perception results and preset decision rules to generate driving decisions and corresponding control signals, wherein the driving decision is a decision plan for the unmanned vehicle to drive in the underground mine, and the control signal is used to control the unmanned vehicle to drive according to the driving decision; and a control unit for controlling the unmanned vehicle to drive according to the driving decision according to the control signal. This application establishes multiple sensors for collection and preprocessing to obtain effective sensor data, namely effective point cloud data, effective image data and effective ultrasonic data, and fuses different sensor data. By fusing data from multiple sensors, more comprehensive and accurate information about the underground mining environment can be obtained, and the perception model can be optimized and updated according to actual production needs and historical data. Based on the optimized perception model and real-time perception data, the underground mining environment is perceived and decided in real time. Through real-time perception and decision-making, the driving and operation of unmanned vehicles can be guided to achieve safe and efficient unmanned driving in underground mines. This application solves the problem in the prior art that the uncertainty and limitations of a single sensor make it impossible for unmanned vehicles to perceive the surrounding environment in real time to make correct driving decisions.

[0172] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for optimizing perception of unmanned driving in underground mines, characterized in that: include: Real-time acquisition of effective point cloud data, effective image data and effective ultrasonic data of the underground mine environment; Fusing the valid point cloud data, the valid image data and the valid ultrasonic data to obtain fused data; The fused data is input into a perception model, so as to use the perception model to perform environmental perception analysis on the fused data to obtain environmental perception results, wherein the environmental perception results at least include obstacles, road signs and environmental features in the underground mine, and the perception model is a convergent model obtained by iterative training by inputting multiple sets of training data into a predetermined model, and the training data includes sample fused data and sample environmental perception results; Making a decision based on the environmental perception result and the preset decision rule, generating a driving decision and a corresponding control signal, wherein the driving decision is a decision plan for the unmanned vehicle to drive in the underground mine, and the control signal is used to control the unmanned vehicle to drive according to the driving decision; The unmanned vehicle is controlled to travel according to the control signal and the driving decision.

2. The method according to claim 1, characterized in that Acquire valid point cloud data, valid image data and valid ultrasonic data of the underground mine environment, including: Obtaining the distance between the object in the underground mine and the laser radar through the laser radar to obtain initial point cloud data, wherein the object includes at least a vehicle, an obstacle and a sign, and the laser radar is located at the front and roof of the unmanned vehicle; Capturing the environment image of the underground mine by a camera to obtain initial image data, wherein the camera is located at the front of the unmanned vehicle and the side of the unmanned vehicle; Acquiring the distance between the object in the underground mine and the ultrasonic sensor by means of an ultrasonic sensor to obtain initial ultrasonic data, wherein the ultrasonic sensor is located on the front, the roof and the side of the unmanned vehicle; Formatting the initial point cloud data, the initial image data and the initial ultrasonic data so that the data formats of the initial point cloud data, the initial image data and the initial ultrasonic data are unified, and correspondingly obtaining first point cloud data, first image data and first ultrasonic data; A preprocessing operation is performed on the first point cloud data, the first image data, and the first ultrasonic data to obtain the valid point cloud data, the valid image data, and the valid ultrasonic data accordingly.

3. The method according to claim 2, characterized in that Preprocessing the first point cloud data, the first image data, and the first ultrasonic data to obtain the valid point cloud data, the valid image data, and the valid ultrasonic data, respectively, includes: Using a first filter to filter the first point cloud data and the first ultrasonic data to obtain second point cloud data and second ultrasonic data, respectively, wherein the first filter is a moving average filter or a Gaussian filter; Using a second filter to filter the first image data to obtain second image data, where the second filter is a median filter or an edge-preserving filter; The second point cloud data and the second ultrasonic data are denoised by a first denoising method to obtain third point cloud data and third ultrasonic data, respectively, wherein the first denoising method comprises at least one of the following: a density-based outlier detection method and a random sampling consensus algorithm; Performing denoising on the second image data using a second denoising method to obtain third image data, wherein the second denoising method includes at least one of the following: morphological operation and threshold segmentation algorithm; Performing a data registration operation on the third point cloud data, the third ultrasonic data and the third image data so that the third point cloud data, the third ultrasonic data and the third image data are represented in the same coordinate system, and obtaining fourth point cloud data, fourth ultrasonic data and fourth image data respectively; The fourth point cloud data, the fourth ultrasonic data and the fourth image data are compressed by using a compression method to obtain the corresponding valid point cloud data, the valid image data and the valid ultrasonic data.

4. The method according to claim 1, characterized in that: The effective point cloud data, the effective image data and the effective ultrasonic data are fused to obtain fused data, including: Performing feature extraction on the valid point cloud data, the valid image data and the valid ultrasonic data to obtain corresponding point cloud features, image features and ultrasonic features respectively, wherein the point cloud features at least include the density, distribution and structure of the valid point cloud data, the image features at least include the image edge, image texture and image color of the valid image data, and the ultrasonic features at least include the distance to the obstacle and the shape of the obstacle; Performing pattern recognition on the point cloud features, the image features and the ultrasonic features to associate the point cloud features, the image features and the ultrasonic features with specific targets in an observation environment to obtain description data, wherein the observation environment is an environment in the underground mine, and the specific target is at least one of the following: the obstacles, road signs and mine equipment in the observation environment; Grouping all the description data according to the same specific target, where one specific target corresponds to one group of description data; The description data in each of the specific targets is fused using a fusion algorithm to obtain the fused data.

5. The method according to claim 1, characterized in that Before inputting the fused data into the perception model to perform environmental perception analysis on the fused data using the perception model to obtain an environmental perception result, the method further includes: Using the historical data set to verify and evaluate the historical perception model, obtaining an evaluation result, wherein the historical perception model is a model that has been subjected to perception analysis after the predetermined model has been trained before the current moment, and the evaluation result is one of the following: the historical perception model has low performance, the historical perception model has medium performance, and the historical perception model has high performance; When the evaluation result shows that the performance of the historical perception model is poor or the performance of the historical perception model is medium, the model parameters of the historical perception model are adjusted and the model structure of the historical perception model is optimized to obtain the perception model, the model parameters include at least weights, thresholds or learning rates, the model structure includes at least the number of layers of the historical perception model, the activation function and the structure of each model layer in the historical perception model, and the model layers include at least input layer, output layer and pooling layer.

6. The method according to claim 1, characterized in that Making a decision based on the environmental perception result and the preset decision rule to generate a driving decision and a corresponding control signal includes: A decision-making step, making a decision according to the environmental perception result and a preset decision rule, generating a preliminary driving decision and a corresponding preliminary control signal; a matching step of matching the preliminary driving decision with the dynamic characteristics of the unmanned vehicle; In the case where the preliminary driving decision does not match the dynamic characteristics, repeatedly executing the decision step and the matching step at least once in sequence until the preliminary driving decision matches the dynamic characteristics; In a case where the preliminary driving decision matches the dynamic characteristic, the preliminary driving decision is determined as the driving decision, and the corresponding preliminary control signal is determined as the control signal.

7. The method according to claim 1, characterized in that Controlling the unmanned vehicle to travel according to the control signal and the driving decision includes: Obtaining real-time driving results of the unmanned vehicle; Comparing the real-time driving result with the driving decision to obtain a comparison result, wherein the comparison result is a similarity between the real-time driving result and the driving decision; When the similarity is greater than or equal to the similarity threshold, continue to control the unmanned vehicle to travel according to the driving decision based on the control signal; When the similarity is less than the similarity threshold, the driving decision and the corresponding control signal are readjusted, and the unmanned vehicle is controlled to drive according to the adjusted driving decision based on the adjusted control signal.

8. An unmanned driving perception optimization device for underground mines, characterized in that: The device comprises: An acquisition unit is used to acquire effective point cloud data, effective image data and effective ultrasonic data of the environment inside the underground mine in real time; A fusion unit, used for fusing the valid point cloud data, the valid image data and the valid ultrasonic data to obtain fused data; A perception unit is used to input the fused data into a perception model, so as to use the perception model to perform environmental perception analysis on the fused data to obtain environmental perception results, wherein the environmental perception results at least include obstacles, road signs and environmental features in the underground mine, and the perception model is a convergent model obtained by iteratively training by inputting multiple sets of training data into a predetermined model, and the training data includes sample fusion data and sample environmental perception results; A decision-making unit, configured to make a decision based on the environmental perception result and a preset decision rule, and generate a driving decision and a corresponding control signal, wherein the driving decision is a decision-making scheme for the unmanned vehicle to drive in the underground mine, and the control signal is used to control the unmanned vehicle to drive according to the driving decision; A control unit is used to control the unmanned vehicle to travel according to the control signal and the driving decision.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the method according to any one of claims 1 to 7.

10. An unmanned driving perception optimization system for underground mines, characterized in that: include: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include methods for executing any one of claims 1 to 7.

Images