Trackless rubber-tyred vehicle multi-mode detection method and device

Abstract

The invention provides a trackless rubber-tyred vehicle multi-mode detection method and device. The method comprises the steps that a multi-mode detection model of a trackless rubber-tyred vehicle is acquired; the method comprises the following steps: performing panoramic detection on the real-time state of the trackless rubber-tyred vehicle to obtain a plurality of predictive factors related to the structure, performance and safety of the trackless rubber-tyred vehicle; and according to the evolution calculation of the multi-modal detection model on the plurality of prediction factors, obtaining a corresponding state evaluation value, and based on the state evaluation value, scheduling the trackless rubber-tyred vehicle. Carrying out evolution calculation on a plurality of prediction factors corresponding to the real-time state of the trackless rubber-tyred vehicle through a multi-modal detection model to obtain a state evaluation value; according to analysis of the state evaluation value, a prediction state related to the structure, the performance and the safety of the trackless rubber-tyred vehicle is obtained, and the trackless rubber-tyred vehicle is scheduled based on the prediction state, so that the capability of safe, efficient and orderly operation of the trackless rubber-tyred vehicle is greatly improved, and the method can better serve safe production of a mine field.

Description

Technical Field

[0001] The present application relates to the field of intelligent mining technology, and in particular to a multi-modal detection method and device for a trackless rubber-tyred vehicle. Background Art

[0002] With the continuous development of mining auxiliary transportation tools, trackless rubber-tyred vehicles play an important role in improving the safety and production efficiency of mines. Trackless rubber-tyred vehicles have the advantages of high transportation efficiency, strong climbing ability, low transportation cost, flexibility and efficiency. However, while improving the production efficiency of mines, the problems of trackless rubber-tyred vehicles in operation and transportation safety are becoming increasingly prominent. Therefore, how to achieve safe, efficient and orderly operation of trackless rubber-tyred vehicles and better serve the safety production of mines is an important issue that needs to be solved urgently.

[0003] It should be noted that the above introduction to the technical background is only for the convenience of providing a clear and complete description of the technical solutions of the present application and for the convenience of understanding by those skilled in the art. It cannot be considered that the above technical solutions are well known to those skilled in the art simply because they are described in the background technology section of the present application. Summary of the invention

[0004] The embodiments of the present application provide a multi-modal detection method and device for a rubber-tyred trackless vehicle.

[0005] The first embodiment of the present application proposes a multi-modal detection method for a trackless rubber-tyred vehicle, comprising:

[0006] Obtain a multimodal detection model for trackless rubber-tyred vehicles;

[0007] Based on the panoramic detection of the real-time status of the rubber-tyred trackless vehicle, multiple prediction factors related to the structure, performance and safety of the rubber-tyred trackless vehicle are obtained;

[0008] According to the evolution calculation of the plurality of prediction factors by the multimodal detection model, corresponding state evaluation values ​​are obtained, and the rubber-tyred trackless vehicles are dispatched based on the state evaluation values.

[0009] The second embodiment of the present application proposes a multi-modal detection device for a trackless rubber-tyred vehicle, comprising:

[0010] A first acquisition module, the first acquisition module is used to acquire a multimodal detection model of a trackless rubber-tyred vehicle;

[0011] A second acquisition module, the second acquisition module is used to acquire multiple prediction factors related to the structure, performance and safety of the trackless rubber-tyred vehicle based on a panoramic detection of the real-time state of the trackless rubber-tyred vehicle;

[0012] A scheduling module is used to calculate the evolution of multiple prediction factors according to the multimodal detection model, obtain corresponding state evaluation values, and schedule the rubber-tyred trackless vehicle based on the state evaluation values.

[0013] The third aspect embodiment of the present application proposes an electronic device, comprising: a processor; a memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions to implement the multimodal detection method for rubber-tyred trackless vehicles proposed in the first aspect embodiment of the present application.

[0014] The fourth aspect embodiment of the present application proposes a non-temporary computer-readable storage medium. When the instructions in the storage medium are executed by the processor of an electronic device, the electronic device can execute the method proposed in the first aspect embodiment of the present application.

[0015] The fifth aspect embodiment of the present application proposes a computer program product, including a computer program, which implements the method proposed in the first aspect embodiment of the present application when executed by a processor in a communication device.

[0016] The technical solution provided by the embodiments of the present application brings at least the following beneficial effects:

[0017] Through the multimodal detection model, multiple prediction factors corresponding to the real-time status of the rubber-tyred trackless vehicle are evolved and calculated to obtain the status evaluation value. According to the analysis of the status evaluation value, the predicted status related to the structure, performance and safety of the rubber-tyred trackless vehicle is obtained, and based on the predicted status, the rubber-tyred trackless vehicle is dispatched, which greatly improves the ability of the rubber-tyred trackless vehicle to operate safely, efficiently and orderly, and can better serve the safe production of the mine.

[0018] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0020] Figure 1 A schematic flow chart of a multi-modal detection method for a trackless rubber-tyred vehicle provided in an embodiment of the present application;

[0021] Figure 2 A schematic flow chart of another multi-modal detection method for a trackless rubber-tyred vehicle provided in an embodiment of the present application;

[0022] Figure 3 A schematic structural diagram of a multi-modal detection device for a trackless rubber-tyred vehicle provided in an embodiment of the present application;

[0023] Figure 4 A schematic diagram of the structure of an electronic device provided according to an embodiment of the present application;

[0024] Figure 5 It is a schematic diagram of the structure of another electronic device provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0025] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the embodiments of the present application as detailed in the appended claims.

[0026] The terms used in the embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to limit the embodiments of the present application. The singular forms of "a" and "the" used in the embodiments of the present application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0027] It should be understood that, although the terms first, second, third, etc. may be used to describe various information in the embodiments of the present application, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the embodiments of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein may be interpreted as "at..." or "when..." or "in response to determination".

[0028] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0029] It should be noted that the multimodal detection method for rubber-tyred trackless vehicles provided in any embodiment of the present application can be executed alone, or can be executed in combination with possible implementation methods in other embodiments, or can be executed in combination with any technical solution in the relevant technology.

[0030] The following describes a multi-modal detection method and device for a rubber-tyred trackless vehicle according to an embodiment of the present application with reference to the accompanying drawings.

[0031] Figure 1 A schematic flow chart of a multi-modal detection method for a trackless rubber-tyred vehicle provided in an embodiment of the present application. Figure 1 As shown, the method includes but is not limited to the following steps:

[0032] S101, obtaining a multimodal detection model of a rubber-tyred trackless vehicle.

[0033] In a feasible implementation, firstly, the specific objectives of the multimodal detection model of rubber-tyred trackless vehicles are clarified, such as the structure and appearance data of the rubber-tyred trackless vehicle (including the wear condition of the tires, whether the air pressure is normal, and whether there are safety hazards such as cracks and bulges; whether the integrity and functions of key parts such as the braking system, lighting system, and steering system are normal; check whether the joints of various components of the vehicle are loose, damaged, or leaking), vehicle performance data (including the exhaust emissions, braking distance, sideslip and wheel deviation of the rubber-tyred trackless vehicle, and obtaining the performance parameters of the rubber-tyred trackless vehicle such as speed and acceleration), safety warning data (including reversing warning data, collision warning data, etc., which can be obtained through a sensor network), image analysis data (including image information, motion trajectory, and potential collision objects during the operation of the rubber-tyred trackless vehicle), etc.

[0034] Next, select the data source. A 3D camera can be used as a visual acquisition tool to capture the image information of the rubber-tyred trackless vehicle and provide visual features such as the appearance, color, and texture of the rubber-tyred trackless vehicle. LiDAR can be used to provide high-precision, high-density, and high-resolution point cloud data to obtain the three-dimensional shape, size, and spatial distribution information of the surrounding environment of the rubber-tyred trackless vehicle. An inertial navigation system can be used to provide kinematic parameters such as acceleration, speed, and posture of the rubber-tyred trackless vehicle. Temperature sensors, pressure sensors, audio sensors, etc. can be used to detect specific parameters of the internal or external environment of the rubber-tyred trackless vehicle.

[0035] Next, the historical operation data of rubber-tyred trackless vehicles in various scenarios are preprocessed and fused to remove noise, outliers and duplicate data to ensure data accuracy and reliability. Data from different data sources are calibrated in time and space to ensure data consistency. Data from different modes (including vision, audio, pressure, temperature, kinematics, chemical composition, etc.) are fused to form a more comprehensive description of the vehicle status. Data fusion methods include Kalman filtering, particle filtering, etc.

[0036] Next, the fused data is used to extract key features that can reflect the status of the rubber-tyred trackless vehicle, such as appearance structure, internal performance, component status, travel status, etc. Based on machine learning or deep learning algorithms, such as support vector machines, random forests, neural networks, etc., the extracted key features are trained to establish a multimodal detection model. At the same time, the multimodal detection model is verified and optimized through actual data to improve the accuracy and robustness of the model.

[0037] S102, based on a panoramic detection of the real-time status of the rubber-tyred trackless vehicle, a plurality of prediction factors related to the structure, performance and safety of the rubber-tyred trackless vehicle are obtained.

[0038] In a feasible implementation, four wide-angle 3D cameras for front, rear, left and right views can be installed on the trackless rubber-wheeled vehicle to achieve 360-degree panoramic stitching, thereby obtaining visual information around the trackless rubber-wheeled vehicle. Then, edge detection is performed based on image processing technology to identify potential collision objects around the trackless rubber-wheeled vehicle, such as pedestrians, other trackless rubber-wheeled vehicles or obstacles. And road conditions such as curves, climbing, downhill, as well as the air door, road signs, lights, reflective strips, license plate numbers, chassis, etc. in the direction of travel are identified to provide accurate instructions and early warning information for the travel of the trackless rubber-wheeled vehicle. An ultrasonic radar can be set on the trackless rubber-wheeled vehicle to detect obstacles on the route to provide more accurate distance and position information, and to provide support for the automatic driving and avoidance of the trackless rubber-wheeled vehicle. A thermal imaging camera can also be set on the trackless rubber-wheeled vehicle to detect abnormal temperature fields on itself and the route, and to promptly discover potential fires or high-temperature areas to ensure the safe travel of the trackless rubber-wheeled vehicle. Tools such as portable brake performance testers can be used to monitor the braking force, braking distance, braking coordination time and other key parameters of rubber-tyred trackless vehicles in real time to evaluate the braking performance of rubber-tyred trackless vehicles. Sensors can be used to monitor the power system status of rubber-tyred trackless vehicles, including engine speed, power output, fuel consumption, etc., to ensure that the power performance of rubber-tyred trackless vehicles is stable and reliable. Strain gauges, acceleration sensors, etc. can be used to monitor the structural health of rubber-tyred trackless vehicles in real time to promptly detect potential safety hazards such as cracks and deformations.

[0039] Furthermore, based on the position data and motion information of the rubber-tyred trackless vehicle and the potential collision object, the collision risk is predicted using algorithms (such as basic safety information algorithms, algorithms based on trajectory prediction and circle detection, etc.), the collision risk level is determined, and warning information is sent to the vehicle terminal. Various data of the rubber-tyred trackless vehicle before starting and during driving are collected, including the appearance and structure data of the rubber-tyred trackless vehicle, acceleration, speed, turning angle, visual data, infrared thermal imaging data, braking force data, etc., and data analysis techniques (such as descriptive analysis, inferential analysis, exploratory analysis, machine learning algorithms, association rule mining, frequent item mining, etc.) are used for mining and analysis to extract multiple predictive factors related to the structure, performance and safety of the rubber-tyred trackless vehicle, and combined with the results of panoramic visual detection, radar and sensor detection, and vehicle status detection, the intelligent assisted driving functions of the rubber-tyred trackless vehicle, such as automatic avoidance and adaptive cruise, are realized.

[0040] S103, obtaining corresponding state evaluation values ​​according to the evolution calculation of multiple prediction factors by the multimodal detection model, and dispatching the rubber-tyred trackless vehicles based on the state evaluation values.

[0041] In a feasible implementation, the prediction factors include multimodal data related to the state of the rubber-tyred trackless vehicle, such as position, speed, acceleration, surrounding environment images, sound, etc. The multimodal detection model extracts features useful for the state assessment of the rubber-tyred trackless vehicle based on the characteristics of the collected data and machine learning or deep learning algorithms. Since the prediction factors include the state of the vehicle, changes in the surrounding environment, etc., the time series data of the prediction factors can be calculated to analyze their evolution trends and laws, providing a basis for state assessment.

[0042] Optionally, reasonable state evaluation indicators can be designed according to the needs of trackless rubber-tyred vehicle scheduling. These indicators may include vehicle operating efficiency, safety, energy consumption, etc. The multimodal detection model converts the data of the predictor into state evaluation values. According to the state evaluation values, reasonable scheduling strategies are designed, which may include path planning, speed control, priority allocation, etc. of the trackless rubber-tyred vehicle to ensure the optimization of the operating efficiency, safety and energy consumption of the trackless rubber-tyred vehicle. Then, the computer is used to receive the state evaluation values ​​in real time and make decisions according to the scheduling strategy; at the same time, the computer should have a good user interface and interactivity so that the operator can easily monitor and intervene in the scheduling process. On the other hand, the scheduling results can also be regularly evaluated, including the evaluation results of the operating efficiency, safety, energy consumption, etc. of the trackless rubber-tyred vehicle; according to the evaluation results, the scheduling strategy is continuously optimized and improved to improve the overall operation effect of the trackless rubber-tyred vehicle.

[0043] In summary, the multimodal detection model is used to perform evolutionary calculations on multiple prediction factors corresponding to the real-time state of the rubber-tyred trackless vehicle to obtain the state evaluation value; based on the analysis of the state evaluation value, the predicted state related to the structure, performance and safety of the rubber-tyred trackless vehicle is obtained, and based on the predicted state, the rubber-tyred trackless vehicle is dispatched, which greatly improves the ability of the rubber-tyred trackless vehicle to operate safely, efficiently and orderly, and can better serve the safe production of the mine.

[0044] Figure 2 A schematic flow chart of another multi-modal detection method for a trackless rubber-tyred vehicle provided in an embodiment of the present application. Figure 2 As shown, the method includes but is not limited to the following steps:

[0045] S201, obtaining historical operation data of the rubber-tyred trackless vehicle in various scenarios.

[0046] Optionally, as an example, obtain the operation log of the rubber-tyred trackless vehicle. Based on the data requirements, the operation logs are extracted and associated and fused to obtain the historical operation data of the rubber-tyred trackless vehicle in various scenarios. It should be noted that the data requirements, namely the types and scope of the required data, include but are not limited to the driving status, distance, load, energy consumption, environmental conditions (such as temperature, braking force, exhaust gas concentration, etc.), posture, location information of the rubber-tyred trackless vehicle, and any fault or alarm records related to the operation of the rubber-tyred trackless vehicle. The extraction of the operation log includes determining the source of the operation log, such as a local storage device, a remote server or cloud storage, and secondly, ensuring that you have the authority to access these operation logs and understand the format and source of the operation logs.

[0047] S202, performing feature extraction and modal fusion on historical operation data based on a deep learning framework to obtain a multimodal detection model for the rubber-tyred trackless vehicle.

[0048] Optionally, as an example, the first intermediate data is obtained based on the preprocessing of the historical operation data. The preprocessing of the extracted operation log includes: performing data cleaning to remove irrelevant error information, duplicate data, etc. The cleaned data (i.e., the first intermediate data) should retain the key operation information of the rubber-tyred trackless vehicle, such as time, speed, position, status, etc.

[0049] Next, feature extraction and modal fusion can be performed on the first intermediate data based on a deep learning framework to obtain a multimodal detection model for the rubber-tyred trackless vehicle.

[0050] It should be noted that the first intermediate data is associated with other relevant data of the rubber-tyred trackless vehicle (such as vehicle information, driver information, environmental information, etc.), and this process can be achieved through common fields in the data (such as timestamp, vehicle ID, etc.). Then the associated data is fused into a complete data set for subsequent analysis and mining. In the fusion process, attention should be paid to the consistency and integrity of the data to avoid data loss or duplication. The fusion process includes three stages: early fusion, deep fusion, and late fusion. In early fusion, data of different modalities are fused together before being input into the deep learning model. This is usually achieved by splicing, addition, or weighted summation. However, since data of different modalities may differ in representation, dimension, and distribution, early fusion may face some challenges. In deep fusion, data of different modalities are input into their respective sub-networks for feature extraction, and then fused in the middle layer or higher layer of the network. This method can maintain the independence of each modality data while utilizing the complementarity between different modalities. Deep fusion is usually achieved by feature splicing, attention mechanism, or gated unit. In late fusion, data from different modalities are input into their respective sub-networks for feature extraction and decision making, and then fused at the decision layer. This method is usually used for classification or regression tasks, where each modality contributes an independent prediction result, which is then fused by voting, averaging, or weighting.

[0051] As an example, the second intermediate data can be obtained based on image feature extraction, audio feature extraction and point cloud feature extraction of the first feature data. Then, according to the multiple modes related to the operation of the trackless rubber-tyred vehicle, the second intermediate data is aligned in time and space to obtain the third intermediate data. Next, based on the deep learning framework, the third intermediate data is trained to obtain the multimodal detection model of the trackless rubber-tyred vehicle. It should be noted that after obtaining the multimodal detection model, it also includes a training process, such as data preprocessing, model training, verification and testing steps. It should be added that in order to improve the generalization ability of the model, the training data can be enhanced, such as rotation, scaling, cropping, etc.; in order to prevent the model from overfitting, regularization techniques such as L1 regularization, L2 regularization, Dropout, etc. can be used; in addition, the learning rate is one of the important factors affecting the model training effect, and the learning rate can be adjusted according to the loss changes and learning curves during the training process.

[0052] After training is completed, the model needs to be evaluated and optimized. Evaluation indicators usually include accuracy, recall, F1 score, etc. If the model performance is poor, it can be optimized by adjusting the network structure, increasing the amount of data, using more advanced feature extraction and modality fusion methods, etc.

[0053] For further introduction to step S202, please refer to the relevant contents in the above embodiment, which will not be repeated here.

[0054] S203, based on a panoramic detection of the real-time status of the rubber-tyred trackless vehicle, a plurality of prediction factors related to the structure, performance and safety of the rubber-tyred trackless vehicle are obtained.

[0055] Optionally, as an example, environmental data related to the real-time status of the rubber-tyred trackless vehicle can be obtained first. Then, the appearance structure data, performance data and safety data of the rubber-tyred trackless vehicle are extracted from the environmental data. Then, according to the fault type of the rubber-tyred trackless vehicle, trend analysis is performed on the appearance structure data, performance data and safety data to obtain multiple prediction factors related to the structure, performance and safety of the rubber-tyred trackless vehicle.

[0056] For further introduction to step S203, please refer to the relevant contents in the above embodiment, which will not be repeated here.

[0057] S204, obtaining corresponding state evaluation values ​​according to the evolution calculation of multiple prediction factors by the multimodal detection model, and dispatching the rubber-tyred trackless vehicles based on the state evaluation values.

[0058] In a feasible implementation, the multimodal detection model is a detection method based on multimodal data. It is different from the traditional single-modal detection method and can make full use of the advantages of different modal data to improve the accuracy and comprehensiveness of detection. In the multimodal detection model, each modal data has its unique characteristics and advantages. By integrating the information of these different modal data, we can better understand the laws behind the data, thereby improving the detection effect.

[0059] Prediction factors may include temperature, vibration, noise, pressure, current and other aspects. Evolutionary computing is an optimization algorithm that simulates the operating state of rubber-tyred trackless vehicles. In the multimodal detection model, evolutionary computing can be used to iteratively optimize the prediction factors to improve the accuracy of state assessment. By setting the initial data related to the structure, performance and safety of the rubber-tyred trackless vehicle, time series functions, scheduling strategies, correlation operations between prediction factors, etc., the prediction factors are continuously iterated and optimized to make them more in line with the actual situation.

[0060] As an example, several main risk factors for potential failures of rubber-tyred trackless vehicles can be determined based on principal component analysis of multiple prediction factors. Further, fourth intermediate data can be obtained based on formatting of multiple prediction factors; the fourth intermediate data can be reconstructed based on the covariance matrix to obtain a first matrix; and several main risk factors for potential failures of rubber-tyred trackless vehicles can be determined based on principal component analysis of the first matrix.

[0061] After obtaining the main risk factors, regression analysis can be performed on multiple main risk factors based on the multimodal detection model to obtain the probability of occurrence and the degree of impact of the main risk factors; further, based on parameter combination and condition changes of multiple main risk factors, multiple input parameters can be obtained; regression analysis is performed on the input parameters according to the multimodal detection model to obtain the evolution curve of the input parameters; based on probability distribution fitting of the evolution curve, the probability of occurrence and the degree of impact of the main risk factors can be obtained.

[0062] After the probability and the degree of impact are obtained, the corresponding state evaluation value is obtained according to the multiplication calculation of the probability and the degree of impact.

[0063] For further introduction to step S204, please refer to the relevant contents in the above embodiment, which will not be repeated here.

[0064] In summary, the multimodal detection model is used to perform evolutionary calculations on multiple prediction factors corresponding to the real-time state of the rubber-tyred trackless vehicle to obtain the state evaluation value; based on the analysis of the state evaluation value, the predicted state related to the structure, performance and safety of the rubber-tyred trackless vehicle is obtained, and based on the predicted state, the rubber-tyred trackless vehicle is dispatched, which greatly improves the ability of the rubber-tyred trackless vehicle to operate safely, efficiently and orderly, and can better serve the safe production of the mine.

[0065] Figure 3 This is a schematic diagram of the structure of a multi-modal detection device for a trackless rubber-tyred vehicle provided in an embodiment of the present application. Figure 3 As shown, the multi-modal detection device 300 for rubber-tyred trackless vehicles comprises:

[0066] A first acquisition module 301, the first acquisition module 301 is used to acquire a multimodal detection model of a trackless rubber-tyred vehicle;

[0067] A second acquisition module 302, the second acquisition module 302 is used to acquire multiple prediction factors related to the structure, performance and safety of the rubber-tyred trackless vehicle based on a panoramic detection of the real-time state of the rubber-tyred trackless vehicle;

[0068] The scheduling module 303 is used to calculate the evolution of multiple prediction factors according to the multimodal detection model, obtain corresponding state evaluation values, and schedule the rubber-tyred trackless vehicle based on the state evaluation values.

[0069] Figure 4 The present invention is a schematic diagram of the structure of an electronic device provided according to an embodiment of the present application. Figure 4 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0070] like Figure 4As shown, the electronic device 400 includes a processor 401, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 402 or a program loaded from a memory 406 to a random access memory (RAM) 403. In the RAM 403, various programs and data required for the operation of the electronic device 400 are also stored. The processor 401, the ROM 402, and the RAM 403 are connected to each other via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0071] The following components are connected to the I / O interface 405: a memory 406 including a hard disk, etc.; and a communication part 407 including a network interface card such as a LAN (Local Area Network) card, a modem, etc., and the communication part 407 performs communication processing via a network such as the Internet; a drive 408 is also connected to the I / O interface 405 as needed.

[0072] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 407. When the computer program is executed by the processor 401, the above-mentioned functions defined in the method of the present application are executed.

[0073] In an exemplary embodiment, a storage medium including instructions is also provided, such as a memory including instructions, and the instructions can be executed by the processor 401 of the electronic device 400 to complete the above method. Optionally, the storage medium can be a non-transitory computer-readable storage medium, for example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0074] In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, apparatus, or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in combination with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0075] Figure 5 It is a schematic diagram of the structure of another electronic device provided according to an embodiment of the present application. Figure 5 The electronic device shown is only an example and should not limit the functions and scope of use of the embodiments of the present application. Figure 5 As shown, the electronic device 500 includes a processor 501 and a memory 502. The memory 502 is used to store program codes, and the processor 501 is connected to the memory 502 to read the program codes from the memory 502 to implement the multi-modal detection method of the trackless rubber-tyred vehicle in the above embodiment.

[0076] Optionally, the number of processors 501 may be one or more.

[0077] Optionally, the electronic device may further include an interface 503, and the number of the interface 503 may be multiple. The interface 503 may be connected to an application program and may receive data from an external device such as a sensor.

[0078] Those skilled in the art will readily appreciate other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses or adaptations of the present invention that follow the general principles of the present invention and include common knowledge or customary techniques in the art that are not disclosed in this application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0079] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A multi-modal detection method for a trackless rubber-tyred vehicle, characterized in that: include: Obtain a multimodal detection model for trackless rubber-tyred vehicles; Based on the panoramic detection of the real-time status of the rubber-tyred trackless vehicle, multiple prediction factors related to the structure, performance and safety of the rubber-tyred trackless vehicle are obtained; According to the evolution calculation of the plurality of prediction factors by the multimodal detection model, corresponding state evaluation values ​​are obtained, and the rubber-tyred trackless vehicles are dispatched based on the state evaluation values.

2. The method according to claim 1, characterized in that The method of obtaining a multimodal detection model of a trackless rubber-tyred vehicle includes: Obtain historical operation data of rubber-tyred trackless vehicles in various scenarios; Based on the preprocessing of the historical operation data, obtaining first intermediate data; Based on the deep learning framework, feature extraction and modal fusion are performed on the first intermediate data to obtain a multimodal detection model of the rubber-tyred trackless vehicle.

3. The method according to claim 2, characterized in that The acquisition of historical operation data of the rubber-tyred trackless vehicle in various scenarios includes: Get the operation log of the rubber-tyred trackless vehicle; Based on data requirements, the operation logs are extracted and correlated to obtain historical operation data of the rubber-tyred trackless vehicle in various scenarios.

4. The method according to claim 2, characterized in that: The method of performing feature extraction and modal fusion on the first intermediate data based on a deep learning framework to obtain a multimodal detection model of a trackless rubber-tyred vehicle includes: Acquire second intermediate data based on performing image feature extraction, audio feature extraction, and point cloud feature extraction on the first feature data; According to multiple modes related to the operation of the rubber-tyred trackless vehicle, the second intermediate data is time-space aligned to obtain third intermediate data; Based on a deep learning framework, the third intermediate data is trained to obtain a multimodal detection model of the rubber-tyred trackless vehicle.

5. The method according to claim 4, characterized in that The method of training the third intermediate data based on the deep learning framework to obtain a multimodal detection model of the trackless rubber-tyred vehicle includes: In response to the deep learning framework, a multimodal detection model of the rubber-tyred trackless vehicle is obtained based on performing data enhancement, regularization and learning rate adjustment operations on the third intermediate data.

6. The method according to any one of claims 1 to 5, characterized in that The method is based on performing a panoramic detection of the real-time status of the rubber-tyred trackless vehicle to obtain multiple prediction factors related to the structure, performance and safety of the rubber-tyred trackless vehicle, including: Obtain environmental data related to the real-time status of rubber-tyred trackless vehicles; Extracting appearance structure data, performance data and safety data of the rubber-tyred trackless vehicle from the environmental data; According to the fault type of the rubber-tyred trackless vehicle, trend analysis is performed on the appearance structure data, the performance data and the safety data to obtain multiple prediction factors related to the structure, performance and safety of the rubber-tyred trackless vehicle.

7. The method according to claim 1, characterized in that The step of calculating the evolution of the plurality of prediction factors according to the multimodal detection model to obtain corresponding state evaluation values ​​includes: Based on principal component analysis of the plurality of said prediction factors, several main risk factors for potential failure of the rubber-tyred trackless vehicle are determined; Performing regression analysis on the plurality of the main risk factors based on the multimodal detection model to obtain the probability of occurrence and the degree of impact of the main risk factors; According to the multiplication calculation of the probability and the influence degree, a corresponding state evaluation value is obtained.

8. The method according to claim 7, characterized in that Based on the principal component analysis of the plurality of prediction factors, several main risk factors for potential failure of the rubber-tyred trackless vehicle are determined, including: Based on formatting the plurality of prediction factors, obtaining fourth intermediate data; Based on the covariance matrix, reconstruct the fourth intermediate data to obtain a first matrix; Based on the principal component analysis of the first matrix, several main risk factors of potential failures of the rubber-tyred trackless vehicle are determined.

9. The method according to claim 7, characterized in that: The performing regression analysis on the plurality of the main risk factors based on the multimodal detection model to obtain the probability of occurrence and the degree of influence of the main risk factors includes: Based on parameter combination and condition change of a plurality of the main risk factors, a plurality of input parameters are obtained; Performing regression analysis on the input parameters according to the multimodal detection model to obtain an evolution curve of the input parameters; Based on the probability distribution fitting of the evolution curve, the probability of occurrence and the degree of impact of the main risk factors are obtained.

10. A multi-modal detection device for a trackless rubber-tyred vehicle, characterized in that: include: A first acquisition module, the first acquisition module is used to acquire a multimodal detection model of a trackless rubber-tyred vehicle; A second acquisition module, the second acquisition module is used to acquire multiple prediction factors related to the structure, performance and safety of the trackless rubber-tyred vehicle based on a panoramic detection of the real-time state of the trackless rubber-tyred vehicle; A scheduling module is used to calculate the evolution of multiple prediction factors according to the multimodal detection model, obtain corresponding state evaluation values, and schedule the rubber-tyred trackless vehicle based on the state evaluation values.

Images