Method, device, equipment and system for detecting the operating status of an unmanned vehicle

Abstract

The present invention relates to the field of abnormality detection and early warning technology, and provides a method, device, equipment and system for detecting the operating status of an unmanned vehicle, which is applied to a multi-dimensional data acquisition device deployed on a road section where the unmanned vehicle passes, and the method comprises: obtaining the status information of the unmanned vehicle on the road section; the status information comprises at least two of temperature data, audio data, image data and video data; pre-processing the status information and arranging it in chronological order to obtain multi-modal feature data; inputting the multi-modal feature data into a multi-dimensional algorithm model, and the multi-dimensional algorithm model obtains a target vector value according to the multi-modal feature data, and determines the status detection result of the unmanned vehicle according to the corresponding relationship between the vector value and the state label. The present invention adopts a multi-modal fusion method to utilize the richness and diversity of various information sources to effectively identify the real operating status of the unmanned vehicle.

Description

Technical Field

[0001] The present invention relates to the field of abnormality detection and early warning technology, and in particular to a method, device, equipment and system for detecting the operating status of an unmanned vehicle. Background Art

[0002] During the driving process, the system may have problems due to software failure, hardware damage or external interference. By monitoring the running status of the vehicle in real time, potential safety hazards can be discovered in time to avoid accidents.

[0003] For manned vehicles, it is the driver who is responsible for monitoring the vehicle's status. However, for unmanned vehicles, since the driver cannot directly understand the vehicle's fault conditions, when the vehicle is about to fail or has initial signs, if there is no corresponding means to detect and warn in advance, it will cause a greater impact. In the relevant technology, unmanned vehicles are generally maintained and inspected regularly, or sensors are set in unmanned vehicles to detect the operating status.

[0004] However, the regular inspection method cannot detect problems with driverless vehicles in a timely manner. The method of setting sensors in the vehicle requires deploying different types of sensors at multiple points. The cost of the entire machine is high, and the monitoring dimension of each sensor is single, which is easily affected by external interference and has low recognition accuracy. Summary of the invention

[0005] The present invention provides an unmanned vehicle operating status detection method, device, equipment and system to solve the technical problem in the prior art that the unmanned vehicle fault detection cannot effectively identify the actual operating status.

[0006] The present invention provides a method for detecting the running status of an unmanned vehicle, which is applied to a multi-dimensional data acquisition device, and the multi-dimensional data acquisition device is deployed on a road section where the unmanned vehicle passes. The method for detecting the running status of the unmanned vehicle comprises: obtaining status information of the unmanned vehicle on the road section where the unmanned vehicle passes; wherein the status information comprises at least two of temperature data, audio data, image data and video data; pre-processing the status information and arranging it in chronological order to obtain multi-modal feature data; inputting the multi-modal feature data into a multi-dimensional algorithm model to obtain a state detection result of the unmanned vehicle output by the multi-dimensional algorithm model; wherein the multi-dimensional algorithm model is used to obtain a target vector value according to the multi-modal feature data, and determine the state detection result of the unmanned vehicle according to the corresponding relationship between the vector value and the state label.

[0007] According to a method for detecting the operating status of an unmanned vehicle provided by the present invention, a multidimensional algorithm model is pre-set with a correspondence between a vector value and a state label, and the correspondence between the vector value and the state label is obtained by training and optimizing the multidimensional algorithm model based on the labeling results of an abnormal state training set; the state detection result of the unmanned vehicle output by the multidimensional algorithm model is obtained, including: the multidimensional algorithm model determines the state label corresponding to the target vector value based on the correspondence, and determines the state detection result of the unmanned vehicle based on the state label.

[0008] According to a method for detecting the operating status of an unmanned vehicle provided by the present invention, after inputting multimodal feature data into a multidimensional algorithm model to obtain the status detection result of the unmanned vehicle output by the multidimensional algorithm model, it also includes: when the status detection result of the unmanned vehicle is abnormal, reviewing and confirming the on-site situation of the unmanned vehicle; if the review confirms that the unmanned vehicle is operating abnormally, issuing an alarm message; if the review confirms that the unmanned vehicle is operating normally, re-marking the status detection result corresponding to the multimodal feature data.

[0009] According to a method for detecting the operating status of an unmanned vehicle provided by the present invention, status information is preprocessed and sorted in chronological order to obtain multimodal feature data, including: obtaining an audio sequence according to audio data, obtaining an image sequence according to image data, and obtaining a video sequence according to video data; position encoding the audio sequence, image sequence, and video sequence in chronological order to obtain sequence information; inputting the sequence information into a feature extraction model to obtain multimodal feature data output by the feature extraction model; wherein the feature extraction model includes an audio feature extraction model, an image feature extraction model, and a video feature extraction model; the audio feature extraction model is used to extract features from the audio sequence; the image feature extraction model is used to extract features from the image sequence; and the video feature extraction model is used to extract features from the video sequence.

[0010] According to a method for detecting the operating status of an unmanned vehicle provided by the present invention, multimodal feature data is input into a multidimensional algorithm model to obtain a state detection result of the unmanned vehicle output by the multidimensional algorithm model, including: the multidimensional algorithm model fuses various types of feature information in the multimodal feature data to obtain fused data; the multidimensional algorithm model processes the fused data using a multi-head attention mechanism to focus on different features in the fused data to obtain a first intermediate result; the multidimensional algorithm model integrates the first intermediate result by setting a context layer to consider the time dependency of the sequence information to obtain the state detection result of the unmanned vehicle.

[0011] According to a method for detecting the operating status of an unmanned vehicle provided by the present invention, status information of the unmanned vehicle on a passing road section is obtained, including: obtaining temperature information of the unmanned vehicle on the passing road section as temperature data through an infrared thermal imaging module, the temperature data including engine temperature and / or temperature distribution of the entire vehicle; obtaining sound information of the unmanned vehicle on the passing road section as sound data through a sound array sensor, the sound data including engine sound and / or sound distribution of the entire vehicle; and obtaining image information of the unmanned vehicle on the passing road section as image data through a camera.

[0012] The present invention also provides an unmanned vehicle operation status detection device, which is arranged on a multi-dimensional data acquisition device, and the multi-dimensional data acquisition device is deployed on a road section where the unmanned vehicle passes. The unmanned vehicle operation status detection device includes: a status information module, which is used to obtain the status information of the unmanned vehicle on the road section where the unmanned vehicle passes; wherein the status information includes at least two of temperature data, audio data, image data and video data; a multi-modal feature data module, which is used to pre-process the status information and sort it in chronological order to obtain multi-modal feature data; a multi-dimensional algorithm model module, which is used to input the multi-modal feature data into the multi-dimensional algorithm model to obtain the status detection result of the unmanned vehicle output by the multi-dimensional algorithm model; wherein the multi-dimensional algorithm model is used to obtain the target vector value according to the multi-modal feature data, and determine the status detection result of the unmanned vehicle according to the correspondence between the vector value and the status label.

[0013] The present invention also provides a multi-dimensional data acquisition device, including a sensor module, a memory, a processor, and a computer program stored in the memory and executable on the processor; the sensor module is used to collect status information of the unmanned vehicle on the road section it passes through; when the processor executes the program, it implements any of the above-mentioned unmanned vehicle operation status detection methods.

[0014] The present invention also provides an unmanned vehicle operation status detection system, comprising a plurality of the above-mentioned multi-dimensional data acquisition devices, each of which is deployed at a different position of a road section passed by the unmanned vehicle.

[0015] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the operating status detection method of any of the above-mentioned unmanned vehicles is implemented.

[0016] The present invention also provides a computer program product, including a computer program, which implements any of the above-mentioned methods for detecting the operating status of an unmanned vehicle when the computer program is executed by a processor.

[0017] The present invention provides a method, device, equipment and system for detecting the running state of an unmanned vehicle. The present invention uses a non-contact installation method, and a multi-dimensional data acquisition device is deployed on the road section where the unmanned vehicle passes. There is no need to modify the vehicle, and there is no need to deploy equipment on each vehicle. The method for detecting the running state of an unmanned vehicle includes: obtaining the state information of the unmanned vehicle on the road section; the state information includes at least two of temperature data, audio data, image data and video data; pre-processing the state information and arranging it in chronological order to obtain multi-modal feature data; inputting the multi-modal feature data into a multi-dimensional algorithm model, and the multi-dimensional algorithm model is used to obtain a target vector value according to the multi-modal feature data, and determine the state detection result of the unmanned vehicle according to the corresponding relationship between the vector value and the state label. The present invention adopts a multi-modal fusion method to make full use of the richness and diversity of various information sources, enhance the system's recognition ability and anti-interference stability for complex fault modes, and thus more effectively identify the true running state of the unmanned vehicle. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 This is one of the flow charts of the method for detecting the operating status of an unmanned vehicle provided in an embodiment of the present invention.

[0020] Figure 2 Schematic diagram of a system for detecting the operating status of an unmanned vehicle provided in an embodiment of the present invention.

[0021] Figure 3 This is the second flow chart of the method for detecting the operating status of an unmanned vehicle provided in an embodiment of the present invention.

[0022] Figure 4 It is a schematic diagram of the audio, video, image verification process and knowledge solidification process provided by an embodiment of the present invention.

[0023] Figure 5 It is a schematic diagram of a high-dimensional multimodal state recognition model provided by an embodiment of the present invention.

[0024] Figure 6 It is a structural schematic diagram of a running status detection device for an unmanned vehicle provided in an embodiment of the present invention.

[0025] Figure 7It is a schematic diagram of the physical structure of the multi-dimensional data acquisition device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0027] In the description of the embodiments of the present invention, it should be noted that the terms "center", "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the embodiments of the present invention and simplifying the description, and do not indicate or imply that the devices or elements referred to must have a specific orientation, be constructed and operate in a specific orientation, and therefore cannot be understood as limitations on the embodiments of the present invention. In addition, the terms "first", "second", and "third" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance.

[0028] In the description of the embodiments of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "connected" and "connection" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms in the embodiments of the present invention can be understood according to specific circumstances.

[0029] In the embodiments of the present invention, unless otherwise clearly specified and limited, the first feature being "above" or "below" the second feature may mean that the first and second features are in direct contact, or the first and second features are in indirect contact through an intermediate medium. Moreover, the first feature being "above", "above" and "above" the second feature may mean that the first feature is directly above or obliquely above the second feature, or simply means that the first feature is higher in level than the second feature. The first feature being "below", "below" and "below" the second feature may mean that the first feature is directly below or obliquely below the second feature, or simply means that the first feature is lower in level than the second feature.

[0030] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the embodiment of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0031] Open-pit coal mine unmanned mining trucks are special vehicles that mainly undertake transportation operations in open-pit coal mines. Open-pit coal mine unmanned mining trucks are a new generation of mining trucks that are continuously developing and maturing under the promotion of technology in the new era. They realize unmanned operation through the collaboration of artificial intelligence, visual computing, radar, and positioning systems.

[0032] Unmanned mining trucks in open-pit coal mines can effectively meet the safety and risk control needs of mining companies and greatly improve the transportation efficiency of mining areas. If an unmanned mining truck fails, it will have a significant impact on the production, safety and economic benefits of open-pit coal mines. The following is a detailed analysis of the impact caused by mining truck failures: 1) Unplanned production interruption: Unmanned mining trucks in open-pit coal mines are the main means of transporting coal. If a sudden failure occurs, it may cause the production line to stagnate, thereby affecting the production efficiency of the entire mining area.

[0033] 2) Equipment damage and increased maintenance costs: Failure of unmanned mining trucks in open-pit coal mines will not only increase maintenance costs, but may also lead to more serious mechanical damage if not resolved in a timely manner.

[0034] 3) Increased safety risks: Failure of unmanned mining trucks in open-pit coal mines may lead to other safety accidents, such as collision accidents causing casualties and damage to facilities.

[0035] 4) Decreased economic benefits. The failure of unmanned mining trucks in open-pit coal mines not only increases the operating costs of enterprises, but may also lead to a decline in corporate benefits. For example, production interruptions caused by truck failures will reduce coal production, which in turn affects the company's revenue.

[0036] When a mining truck is driven by someone, it is the driver who pays attention to the status of the vehicle. However, for unmanned mining trucks, since there is no driver in the cab to see, hear, or touch the vehicle, it is impossible to intuitively understand the vehicle's fault conditions. When the vehicle is about to fail or there are initial signs, if there is no corresponding means to detect and warn in advance, it will cause a greater impact. The following is a specific analysis of unmanned mining truck failures: 1) Abnormal noise from the engine of an unmanned mining truck: The sound of the engine when it is running is an important basis for judging its health. Abnormal noises, such as knocking, hissing, or roaring, often indicate mechanical problems. Knocking noises may come from pistons, connecting rods, or bearings. Hissing noises are usually related to air leaks, while roaring noises may be problems with the turbocharger or exhaust system.

[0037] 2) Abnormal exhaust of unmanned mining trucks: Abnormal exhaust is a common fault phenomenon. Abnormal exhaust is usually manifested as a change in the color of the smoke, such as black smoke, white smoke or blue smoke. Black smoke is usually caused by incomplete fuel combustion, which may be caused by air filter blockage or fuel injection system failure. White smoke may be a normal phenomenon during cold start, but continuous occurrence may be caused by coolant entering the combustion chamber or poor fuel atomization. Blue smoke usually indicates oil burning, which may be caused by engine wear or poor sealing.

[0038] 3) Engine heating of unmanned mining trucks: Due to long-term operation of mining trucks or failure of the cooling system, the engine will operate above the normal operating temperature, and a large amount of cooling water will be sprayed out from the radiator, causing the engine to malfunction. In mild cases, the engine will be underpowered and unable to perform transportation tasks. In severe cases, the engine will be damaged, further causing the vehicle to be scrapped.

[0039] 4) Mechanical abnormal noise of unmanned mining trucks: During the transportation process, due to the rough road surface or mechanical failure, long-term operation of the truck will cause excessive bearing clearance or severe wear, mainly due to the contact between the gear tooth top and the tooth root, the wrong pairing of the spiral bevel gear or the failure of the drive axle, which will cause greater vibration and noise.

[0040] In summary, the failure of unmanned mining trucks in open-pit coal mines has a profound impact on the production, safety and economic benefits of open-pit coal mines. Therefore, deepening the fault detection and predictive warning of unmanned mining trucks in open-pit coal mines is crucial to the production, safety and operation management of open-pit coal mines.

[0041] In this context, it is necessary to find a detection and early warning method and system that can detect and locate unmanned mining truck failures in open-pit coal mines, so as to promptly discover abnormal characteristics of the problem and reduce or avoid the risks and losses caused by sudden failures of unmanned mining trucks in open-pit coal mines.

[0042] Due to the complexity of the unmanned mining truck system, the fault characteristics of its engine, mechanical structure, and tires are also diverse. In the relevant technologies, there are mainly the following methods for detecting and warning the faults of unmanned mining trucks in open-pit coal mines: 1. Operation and maintenance personnel conduct visual, auditory and tactile inspections of unmanned mining trucks: Inspection personnel periodically (daily, weekly or monthly) conduct close inspections of unmanned mining trucks in open-pit coal mines. They walk around the vehicle to listen to the running sound of the unmanned mining truck, observe the smoke in the exhaust pipe, observe the pressure of the tires, and touch the protective cover near the engine. This method combines the advantages of visual, auditory and tactile inspections of unmanned mining trucks. The advantage is that maintenance can be carried out at the open-pit coal mining site, which is simple and economical. The disadvantage is that the inspection results are too dependent on the ability of the inspectors themselves and are subjective.

[0043] 2. Regular inspection of unmanned mining trucks by vehicle maintenance units: Unmanned mining trucks are taken over by drivers and regularly driven to vehicle maintenance units for inspection and maintenance. Vehicle maintenance units carry special equipment and instruments to inspect engines, transmission systems, etc. The advantage is that small problems can be corrected regularly to prevent them from evolving into big problems, and the overall situation of mining trucks can be fully tested; the disadvantage is that it cannot meet the needs of daily testing, affecting production and generating high testing costs.

[0044] 3. Pressure, position, speed, temperature and other sensors are deployed on unmanned mining trucks to collect the operating status parameters of the equipment under test. Through threshold judgment or mechanism judgment, remote monitoring, fault analysis and early warning can be displayed on the platform. The advantage is that the status of unmanned mining trucks can be monitored in real time. The disadvantage is that different types of sensors must be deployed at multiple points on each unmanned mining truck. The monitoring dimension of each sensor is single. Dozens of sensors need to be installed on an unmanned mining truck. The overall cost is high and it is difficult to use in batches.

[0045] In summary, the detection and early warning methods for unmanned mining truck failures in open-pit coal mines in related technologies have the following problems: 1. It is highly subjective and mainly depends on the personal qualities, work attitude and experience of the operation and maintenance personnel. Due to the harsh environment of open-pit coal mines, long hours of work consume a lot of physical energy and are prone to dangerous accidents such as falls and collisions. Due to the dangerous environment, operation and maintenance personnel often only want to complete the task quickly and find it difficult to concentrate on careful inspections, which makes it easy to miss inspections or make wrong inspections and ignore hidden dangers and faults.

[0046] 2. Vehicle maintenance units conduct inspections once a year, which is a low frequency. Especially for unmanned mine trucks working in the harsh environment of open-pit coal mines, safety hazards are prone to accidents. If the inspection is conducted once a month, it will consume a lot of financial and material resources, and the vehicle will be sent to the maintenance unit for a comprehensive inspection, which will delay a lot of production time.

[0047] 3. According to the monitoring area, contact sensors are deployed on the engine, exhaust pipe and other parts of the unmanned mining truck. On the one hand, the line of the unmanned mining truck needs to be modified, and on the other hand, data transmission, storage, network construction and operation need to be considered. Higher hardware costs and network usage fees need to be paid, which makes the cost of large-scale applications very high. Once the equipment needs to be upgraded or modified, the complexity is increased.

[0048] Based on this, the present invention provides a method for detecting the operating status of an unmanned vehicle, which can be applied to scenarios such as unmanned mining truck fault detection in open-pit coal mines. Multi-dimensional data acquisition equipment is deployed in a non-contact installation manner, and a multimodal fusion method is adopted to fully utilize the richness and diversity of various information sources, thereby more effectively identifying the actual operating status of the unmanned vehicle.

[0049] See also Figure 1 , Figure 1 This is one of the flow charts of the method for detecting the operating status of an unmanned vehicle provided by an embodiment of the present invention. The method for detecting the operating status of an unmanned vehicle may include steps S110 to S130, each of which is specifically as follows: S110: Obtaining status information of the unmanned vehicle on the road section it passes through; wherein the status information includes at least two of temperature data, audio data, image data and video data.

[0050] In this embodiment, the status information can be collected through a multi-dimensional data collection device. The multi-dimensional data collection device uses a non-contact installation method and is deployed on the road section where the unmanned vehicle passes. There is no need to modify the vehicle, and there is no need to deploy the device on each vehicle.

[0051] See also Figure 2 , Figure 2 Schematic diagram of a system for detecting the operating status of an unmanned vehicle provided in an embodiment of the present invention.

[0052] At least one multi-dimensional data acquisition device 220 is deployed on the road section where the unmanned vehicle 210 passes, forming a detection section 230. The multi-dimensional data acquisition device 220 may be provided with a sensor module 223 and an edge terminal 221. The sensor module 223 is provided with a variety of sensors for collecting state information of the unmanned vehicle 210 when passing through the detection section 230 from multiple dimensions; the edge terminal 221 is used to determine the operating state of the unmanned vehicle according to the state information.

[0053] Optionally, a solar panel 222 may be further provided in the multi-dimensional data acquisition device 220. The solar panel 222 enables the multi-dimensional data acquisition device 220 to reduce dependence on external power sources through self-sufficient solar power.

[0054] Optionally, a communication module may also be provided in the multi-dimensional data collection device 220, and the communication module enables the multi-dimensional data collection device 220 to receive administrator instructions or send alarm information by connecting to an upper-level management system.

[0055] The sensors installed in the multi-dimensional data acquisition equipment include but are not limited to infrared sensors, sound array sensors, cameras and other equipment; infrared radiation is detected by infrared sensors to detect heat sources and obtain temperature data; the sound array sensor measures the time difference (delay difference) or phase difference between the sound reaching different microphones, calculates the direction and position of the sound source, and obtains sound data; the camera can obtain image data and / or video data of the unmanned vehicle on the road section it passes through.

[0056] Optionally, the step of obtaining the status information of the driverless vehicle on the road section may specifically include: The temperature information of the unmanned vehicle on the road section it passes through is obtained as temperature data through the infrared thermal imaging module, and the temperature data includes the engine temperature and / or the temperature distribution of the entire vehicle; the sound information of the unmanned vehicle on the road section it passes through is obtained as sound data through the sound array sensor, and the sound data includes the engine sound and / or the sound distribution of the entire vehicle; the image information of the unmanned vehicle on the road section it passes through is obtained as image data through the camera.

[0057] S120: pre-processing the state information and arranging it in chronological order to obtain multimodal feature data.

[0058] Before inputting the state data into the multidimensional algorithm model, the state data needs to be preprocessed. The preprocessing may include data cleaning, data conversion and feature extraction.

[0059] Data cleaning refers to removing noise and outliers from state information; data transformation refers to standardizing or normalizing the cleaned data; feature extraction refers to extracting useful features from the transformed data to obtain various modal types of data.

[0060] After that, sensor data is usually collected in time series, that is, the data points are arranged in chronological order. The preprocessed data is arranged in chronological order to form a time series dataset. This ensures the continuity and consistency of the data in the time dimension. For example, the temperature data collected every second is sorted by timestamp.

[0061] Through the processing of this step, the features from different sensors or data sources can be integrated into one data set to form multimodal feature data. Multimodal feature data refers to multiple types of features extracted from different sensors or data sources. For example, temperature, image, speed, acceleration, sound, etc. can all be different modalities, and these data have been converted from their original state into features that can be used for analysis and modeling.

[0062] S130: Input the multimodal feature data into the multidimensional algorithm model to obtain the state detection result of the unmanned vehicle output by the multidimensional algorithm model; wherein the multidimensional algorithm model is used to obtain the target vector value according to the multimodal feature data, and determine the state detection result of the unmanned vehicle according to the correspondence between the vector value and the state label.

[0063] In this embodiment, a multi-dimensional algorithm model is provided for detecting the state of the unmanned vehicle. The multi-dimensional algorithm model is a machine learning or deep learning model that can process multi-dimensional and multi-type data. The multi-dimensional algorithm model can simultaneously consider multiple input features, conduct a comprehensive analysis of these features, and finally output a determination result on the current state of the unmanned vehicle.

[0064] The status detection results include, but are not limited to, normal vehicle operation and abnormal vehicle operation, where vehicle operation can be further subdivided into fault types, such as serious faults, minor faults, etc.

[0065] The correspondence between vector values ​​and state labels refers to the mapping relationship between vector values ​​and state labels. The correspondence can be obtained through model training, for example, through supervised learning during the training process. The multi-dimensional algorithm model finds the correspondence between vector values ​​and state labels by learning a large amount of labeled training data. In practical applications, the model obtains the target vector value based on the input multimodal feature data, and then determines the vehicle's state detection result through the established correspondence.

[0066] Specifically, the multi-dimensional algorithm model processes the multi-modal feature data to obtain a preliminary vector value, and based on the preliminary vector value, retrieves the closest vector value from the correspondence between the vector value and the state label as the target vector value, thereby determining the state detection result of the unmanned vehicle corresponding to the target vector value.

[0067] The multi-dimensional algorithm model can effectively utilize multi-modal feature data and provide accurate unmanned vehicle status detection results, thereby ensuring the safe operation and efficient maintenance of the vehicle.

[0068] Optionally, the multidimensional algorithm model is pre-set with a correspondence between the vector value and the state label, and the correspondence between the vector value and the state label is obtained by training and optimizing the multidimensional algorithm model based on the labeling results of the abnormal state training set; the step of obtaining the state detection result of the unmanned vehicle output by the multidimensional algorithm model may specifically include: The multi-dimensional algorithm model determines the state label corresponding to the target vector value based on the corresponding relationship, and determines the state detection result of the unmanned vehicle based on the state label.

[0069] The abnormal state training set is a training data set that contains a variety of abnormal state data, and each data is annotated with the corresponding state label. For example, one data may be labeled as "engine overheating", another may be labeled as "exhaust abnormality", and so on.

[0070] The multi-dimensional algorithm model learns how to convert the input multimodal feature data into vector values ​​through the supervised learning training process, and establishes the corresponding relationship between the target vector value and the state label.

[0071] During the training process, the model continuously optimizes its parameters to improve the accuracy and generalization of predictions. For example, you can use the validation set and test set to evaluate the performance of the model to ensure that the model performs well on unseen data.

[0072] As mentioned above, the embodiment of the present invention provides a method for detecting the running status of an unmanned vehicle. The embodiment of the present invention uses a non-contact installation method, and a multi-dimensional data acquisition device is deployed on the road section where the unmanned vehicle passes. There is no need to modify the vehicle, and there is no need to deploy the device on each vehicle. The method for detecting the running status of the unmanned vehicle includes: obtaining the state information of the unmanned vehicle on the road section; the state information includes at least two of temperature data, audio data, image data and video data; pre-processing the state information and arranging it in chronological order to obtain multi-modal feature data; inputting the multi-modal feature data into a multi-dimensional algorithm model, and the multi-dimensional algorithm model is used to obtain a target vector value according to the multi-modal feature data, and determine the state detection result of the unmanned vehicle according to the corresponding relationship between the vector value and the state label. The present invention adopts a multi-modal fusion method to make full use of the richness and diversity of various information sources, enhance the system's recognition ability and anti-interference stability for complex fault modes, and thus more effectively identify the true running status of the unmanned vehicle.

[0073] Based on any of the above embodiments, the steps after inputting the multimodal feature data into the multidimensional algorithm model to obtain the state detection result of the unmanned vehicle output by the multidimensional algorithm model may further include: When the status detection result of the unmanned vehicle is abnormal, the on-site situation of the unmanned vehicle is reviewed and confirmed; if the review confirms that the unmanned vehicle is operating abnormally, an alarm message is issued; if the review confirms that the unmanned vehicle is operating normally, the status detection result corresponding to the multimodal feature data is re-marked.

[0074] When the status detection result is "abnormal", it means that the model believes that the vehicle currently has some problems or potential risks. Therefore, in this embodiment, a review and confirmation step is also added. The review and confirmation can be manually reviewed by the operation and maintenance personnel, such as sending technicians to the site to check the status of the vehicle to confirm whether there is really an abnormality; the review and confirmation can also be automatically reviewed by collecting data through additional sensors or auxiliary systems.

[0075] When the unmanned vehicle is confirmed to be operating abnormally, it indicates that the unmanned vehicle is indeed abnormal and needs to issue an alarm. The alarm information may include information such as the type of abnormality, the location of the abnormality, and possible causes, so that relevant personnel can respond quickly.

[0076] When the review confirms that the unmanned vehicle is operating normally, it indicates that there is no abnormality in the unmanned vehicle, that is, the abnormal result detected by the model is a false alarm, so the state detection result corresponding to the multimodal feature data needs to be corrected.

[0077] Optionally, the corrected state labels are updated to the dataset for subsequent training and model optimization.

[0078] Through the review and confirmation steps, abnormal results in the status detection of unmanned vehicles can be effectively handled, the accuracy of the detection results can be ensured, and alarms can be issued in time when necessary to ensure the safe operation of the vehicle.

[0079] Based on any of the above embodiments, the step of preprocessing the state information and arranging it in chronological order to obtain multimodal feature data may specifically include: An audio sequence is obtained according to audio data, an image sequence is obtained according to image data, and a video sequence is obtained according to video data; the audio sequence, image sequence and video sequence are position-encoded in chronological order to obtain sequence information; the sequence information is input into a feature extraction model to obtain multimodal feature data output by the feature extraction model; wherein the feature extraction model includes an audio feature extraction model, an image feature extraction model and a video feature extraction model; the audio feature extraction model is used to extract features from the audio sequence; the image feature extraction model is used to extract features from the image sequence; and the video feature extraction model is used to extract features from the video sequence.

[0080] In this embodiment, the steps of feature processing and time splicing of various modal data are introduced respectively. In order to ensure the efficiency and accuracy of feature extraction, different feature extraction models can be used to extract feature data of different modalities, and then the feature data of different modalities are spliced ​​through time relationship.

[0081] Specifically, an audio sequence is to convert audio data into a series of time series data. For example, an audio signal is divided into multiple time windows, and the audio data in each window forms an audio sequence. An image sequence is to convert image data into a series of time series data. For example, multiple images taken continuously are arranged in time order to form an image sequence. A video sequence is to convert video data into a series of time series data. For example, a video is decomposed into frames of images, and each frame forms a video sequence.

[0082] Encode the audio, image, and video sequences in chronological order. This step is to preserve the temporal information of each sequence and ensure that the model can understand the temporal relationship of the data. For example, positional embedding or other temporal encoding techniques can be used to add temporal information to each data point.

[0083] Through these steps, multimodal feature data can be effectively extracted from different types of data, providing rich information support for subsequent multimodal data processing tasks.

[0084] Based on any of the above embodiments, the step of inputting the multimodal feature data into the multidimensional algorithm model to obtain the state detection result of the unmanned vehicle output by the multidimensional algorithm model may specifically include: The multidimensional algorithm model fuses various feature information in the multimodal feature data to obtain fused data; the multidimensional algorithm model uses a multi-head attention mechanism to process the fused data to focus on different features in the fused data to obtain a first intermediate result; the multidimensional algorithm model integrates the first intermediate result by setting a context layer to consider the time dependency of the sequence information to obtain the state detection result of the unmanned vehicle.

[0085] In this embodiment, a multi-dimensional algorithm model can be used to process multi-modal feature data, and features can be fused, different features can be focused on, and time dependencies can be considered through a multi-head attention mechanism and context layer, so as to finally obtain the state detection result of the unmanned vehicle. The fusion method can be simple concatenation, weighted average, or more complex deep learning technology (such as feature layer fusion) to ensure that features of different modalities can work together.

[0086] The Multi-Head Attention Mechanism is a mechanism used to capture the relationship between different features in the data, and is often used in the Transformer model. Its core idea is to weight the different parts of the input data (i.e., different features) to focus on the important parts of the data. The Multi-Head Attention Mechanism performs multiple independent attention calculations on the fused data, each time focusing on different features in the fused data. For example, in one attention calculation, the model may focus mainly on audio features; in another calculation, the model may focus mainly on image features or video features.

[0087] The context layer is designed to take into account the temporal dependency of sequence information. It can capture the changes in features in the temporal dimension, thereby better understanding the overall structure of sequence data. The context layer considers the temporal order in the sequence information, ensuring that the model can understand the association between features at different time points. For example, a feature at a certain time point in an audio sequence may be associated with a certain frame in a video sequence.

[0088] Exemplarily, the context layer captures the temporal dependencies of features in sequence data through a recurrent neural network (RNN), a long short-term memory network (LSTM), or a self-attention mechanism in a Transformer.

[0089] In some embodiments, multi-dimensional data collection can be performed through voiceprint recognition, sound source positioning, visible light AI visual recognition, infrared temperature measurement, etc., to achieve multi-dimensional detection of abnormal conditions of unmanned mining trucks using voiceprints, sound arrays, infrared thermal imaging, and cameras.

[0090] Using a contactless installation method, a detection device for abnormal conditions of unmanned mining trucks using voiceprints, sound arrays, infrared thermal imaging, and cameras is set up on the road sections where unmanned mining trucks pass. A model algorithm including voiceprints, sound arrays, infrared thermal imaging, and cameras is deployed in the device on the road sections where unmanned mining trucks pass to identify abnormal conditions of unmanned mining trucks.

[0091] See also Figure 3 , Figure 3 This is the second flow chart of the method for detecting the operating status of an unmanned vehicle provided in an embodiment of the present invention.

[0092] The process starts with the collection of field data, which is then enhanced to extract global features. Next, samples are selected and classified to prepare training datasets. These datasets are used for supervised pre-training networks, while unlabeled training samples are also used for self-supervised pre-training networks after global feature extraction.

[0093] In the model training phase, the results of supervised and self-supervised pre-training are combined to build a multi-dimensional algorithm model. The model is then validated on the test set, and global feature extraction is used to retrieve the vector closest to the test data and output the closest state label. The data is classified using an activation function to determine the type of anomaly. Once the system detects an anomaly, it issues an alarm.

[0094] After the alarm, manual intervention is required to confirm whether there is really an anomaly. If the anomaly is manually confirmed to exist, relevant personnel will be assigned to handle it. If the anomaly is manually confirmed to not exist, the data will be re-labeled, and the entire process ends after processing, ensuring the accuracy and timeliness of anomaly detection. This system improves the efficiency and accuracy of anomaly detection by combining supervised learning and self-supervised learning methods.

[0095] The following is a detailed description of the core steps of this solution: Step 1: Collect the engine temperature data, transmitter sound data, image data, and overall mechanical sound distribution data of the unmanned mining truck on the inspection section, and perform feature enhancement processing on them. Then use the spectrum recognition algorithm of multiple modes to propose multi-dimensional data features, select some stable and continuous data, and have experts with unmanned mining truck operation and maintenance experience and professionals who understand the vehicle working conditions label the selected data based on usage.

[0096] See also Figure 4 , Figure 4 It is a schematic diagram of the audio, video, image verification process and knowledge solidification process provided by an embodiment of the present invention.

[0097] During the data collection phase, the system collects on-site data through audio, video and image acquisition. The data is then verified by audio and video and image verification to ensure its accuracy and usability.

[0098] In the expert knowledge integration phase, experts and professionals use their expertise and equipment management experience to analyze the collected data. This phase includes knowledge consolidation, which converts the expert's knowledge and experience into actionable information, and knowledge application, which uses this information in the actual fault identification process.

[0099] In the model training phase, the model is optimized by identifying sample types and fine-tuning sample calibration. Model parameter training is based on the data processed and solidified knowledge in the previous steps to improve the accuracy of fault identification. Ultimately, this process aims to improve the system's fault identification capabilities, enabling it to more effectively detect and respond to potential problems.

[0100] Step 2: Label the data according to normal, serious fault, and minor fault conditions. After labeling, divide the data set into a training set and a test set. The training set and the test set serve as the training set data and the test set data of the supervised pre-trained network.

[0101] Step 3: Select massive open source data from open source databases, industrial equipment image libraries, industrial equipment voiceprint libraries, and industrial equipment temperature libraries, and train the initial pre-trained model through a self-supervised pre-trained network. Then use the training set data in step 2 to optimize and fine-tune the model and output a multi-dimensional algorithm model.

[0102] Step 4: Extract the multi-dimensional data features of the test set, input them into the multi-dimensional algorithm model in step 3, obtain the vector value, retrieve the closest vector, output the closest state label that has been marked in the training set, classify according to the set activation function, determine the abnormal type, and the system generates abnormal alarm information.

[0103] Step 5: The operation and maintenance personnel receive the abnormal alarm information generated by the system in step 4 in the system, and review and confirm the situation on site. If there is indeed an abnormal situation, the relevant operation and maintenance personnel will be arranged to repair it. If there is no abnormal situation, the data will be re-labeled in the system. In the next training task, the data label classification information in the label will be used to perform the iterative task from step 4.

[0104] This embodiment uses multimodal fusion to synchronously analyze voiceprints, image data and other sensor data, and integrates complementary information to improve diagnostic accuracy. Voiceprint data reflects the acoustic characteristics of the internal mechanical condition of the unmanned mining truck, and other sensor units such as infrared thermal imaging and video record state parameters at multiple levels such as thermal and electrical. The use of multimodal fusion methods can make full use of the richness and diversity of various information sources, enhance the system's ability to identify complex fault modes and anti-interference stability, and thus more effectively identify the true operating status of unmanned mining trucks. Therefore, a high-dimensional multimodal state recognition model algorithm that integrates voiceprint, vision and sensor data is designed, and the extraction and fusion of multimodal data features are fully considered. Based on this, a high-dimensional multimodal state recognition model structure is designed to perform the preprocessing of the above-mentioned state data and the recognition of the operating status.

[0105] See also Figure 5 , Figure 5 It is a schematic diagram of a high-dimensional multimodal state recognition model provided by an embodiment of the present invention.

[0106] Input layer: The input layer of the model needs to be able to receive and process multi-dimensional sensor data simultaneously. Design a specific preprocessing process to ensure that all data is in the same format and scale for subsequent analysis and presentation.

[0107] In the input layer, the original data set can be used to obtain a new data set after time alignment, and the original data sequence encoding such as audio sequence, image sequence and video sequence can be combined with position encoding to obtain a new data set.

[0108] Feature extraction layer: The next feature extraction layer should use deep neural networks such as convolutional neural networks (CNN) or recurrent neural networks (RNN) to extract useful high-level features from various data sources. These networks should be trained independently to capture the specific patterns of each data type.

[0109] For example, the encoded audio sequence can use the first Transformer feature model to obtain voiceprint features, the encoded image sequence can use the second Transformer feature model to obtain image features, and the encoded video sequence can use the third Transformer feature model to obtain video features.

[0110] Fusion strategy: After extracting high-level features, a strategy is needed to fuse these features together. Early fusion or late fusion strategies can be adopted here, depending on the characteristics of the data and application requirements.

[0111] In the process of feature fusion, features can be extracted and fused from multimodal data by means of dimension splicing, attention mechanism, cross-sequence dependency and global feature extraction.

[0112] Dimension concatenation refers to directly connecting different types of features in a certain dimension to form a larger feature vector or feature matrix. For example, concatenating audio features, image features, and video features in the time dimension.

[0113] The attention mechanism is a mechanism for capturing the important parts of the input data. It focuses on the key information in the data by assigning different weights to different parts of the input. For example, the multi-head attention mechanism.

[0114] When using a multi-head attention mechanism to process multimodal data, it allows the model to learn multiple representations of the data in different heads, with each head focusing on different features. For example, the model can simultaneously take into account instantaneous changes in voiceprint data and long-term trends in other sensor data.

[0115] Cross-sequence dependency refers to the existence of certain correlations between features of different sequences, and the model needs to capture these correlations to better understand the data. In multimodal tasks, there are temporal dependencies between audio sequences, image sequences, and video sequences. For example, a context layer can be used to process cross-sequence dependencies.

[0116] Specifically, after the attention layer, a context layer, such as a fully connected layer or an LSTM network, can be set to further integrate the information from the multi-head attention mechanism and consider the temporal dependency of sequence data.

[0117] Output layer: Finally, the output layer should be designed according to the specific classification or regression task. For unmanned mining truck abnormal state recognition, there may be multiple output classes (such as normal, different types of faults, etc.), so the softmax activation function can be used for multi-class classification.

[0118] Training strategy: When training such a large model, appropriate regularization measures such as weight decay, dropout, etc. need to be taken to prevent overfitting. In addition, transfer learning and fine-tuning strategies can be used to pre-train the model on a general dataset and then fine-tune it on a specific dataset.

[0119] Application layer: After the model library is built, the state change of the unmanned mining truck can be identified. The sample to be tested is inferred by the model to extract the feature vectors of sound, video, and image, and this feature is searched in the library to match the feature with the closest distance. According to the state label of the matching sample, the state of the sample to be tested is predicted, so as to judge the abnormal state of the unmanned mining truck.

[0120] The embodiments of the present invention provide a method, device, equipment and system for detecting the operating status of an unmanned vehicle. The collected data is processed, trained, and judged and warned based on a multimodal fusion method. The device corresponding to the method includes a multi-dimensional data acquisition device, an edge algorithm service, an antenna, a power supply unit, and a monitoring system. The operation and maintenance personnel who try this device can log in to the system remotely and view the vehicle's operating data and abnormal conditions in the system, which helps the management unit to reduce the operation and maintenance burden of the unmanned vehicle and increase efficiency by reducing manpower. In addition, the feature abstraction achieved by the high model capacity and deep structure of the large model can improve the generalization ability of the real operating status recognition model of the unmanned vehicle.

[0121] Taking unmanned mining trucks as an example, the embodiment of the present invention monitors unmanned mining trucks in open-pit coal mines through voiceprint recognition, sound source positioning, visible light AI visual recognition, and infrared temperature measurement multi-dimensional data collection. When engine heating, abnormal transmission noise, abnormal exhaust, and mechanical noise occur, the system judges based on the model deployed on the edge, comprehensively analyzes the abnormal state of the mining truck, and issues an alarm to the customer side. Specifically, it includes the following contents: 1. Use sound array sensors, infrared thermal imaging technology, and camera equipment to collect the voiceprints, temperature, sound intensity, and on-site video images of vehicles on site. Use vision, voiceprint models, and temperature thresholds to determine the abnormal status of unmanned mining trucks and display them on the system screen. After a period of model marking, training, and iteration, the accuracy of abnormal alarms is finally achieved to exceed 95%.

[0122] 2. Use a non-contact sound array sensor and an internally integrated infrared thermal imaging module to detect whether the unmanned mining truck has engine or mechanical abnormal noises through voiceprint recognition and acoustic positioning technology, detect whether the engine is hot through infrared thermal imaging technology, and detect whether the exhaust pipe smoke is abnormal through visual AI.

[0123] 3. Use the contactless installation method to select a 20-30 meter rectangular area on the road section where unmanned mining trucks pass every day, and deploy 4 sets of devices. When the mining truck passes through this detection section once, it is detected once. There is no need to modify the vehicle, and there is no need to deploy equipment on each vehicle.

[0124] The present invention also provides an operating status detection device for an unmanned vehicle. The operating status detection device for the unmanned vehicle provided by the present invention is described below. The operating status detection device for the unmanned vehicle described below and the operating status detection method for the unmanned vehicle described above can be referenced to each other.

[0125] See also Figure 6 , Figure 6 1 is a schematic diagram of the structure of the unmanned vehicle running state detection device provided by an embodiment of the present invention. In this embodiment, the unmanned vehicle running state detection device can be set in a multi-dimensional data acquisition device, and the multi-dimensional data acquisition device is deployed on the road section where the unmanned vehicle passes.

[0126] Specifically, the operating status detection device of the unmanned vehicle includes a status information module 610 , a multimodal feature data module 620 and a multidimensional algorithm model module 630 .

[0127] The state information module 610 is used to obtain the state information of the unmanned vehicle on the road section; wherein the state information includes at least two of temperature data, audio data, image data and video data; The multimodal feature data module 620 is used to pre-process the state information and sort it in chronological order to obtain multimodal feature data; The multidimensional algorithm model module 630 is used to input the multimodal feature data into the multidimensional algorithm model to obtain the state detection result of the unmanned vehicle output by the multidimensional algorithm model; wherein the multidimensional algorithm model is used to obtain the target vector value based on the multimodal feature data, and determine the state detection result of the unmanned vehicle based on the correspondence between the vector value and the state label.

[0128] Based on any of the above embodiments, the multidimensional algorithm model is pre-set with a correspondence between vector values ​​and state labels, and the correspondence between vector values ​​and state labels is obtained by training and optimizing the multidimensional algorithm model based on the labeling results of the abnormal state training set; the multidimensional algorithm model determines the state label corresponding to the target vector value based on the correspondence, and determines the state detection result of the unmanned vehicle based on the state label.

[0129] Based on any of the above embodiments, the multi-dimensional algorithm model module 630 can also be specifically used for: when the state detection result of the unmanned vehicle is abnormal, reviewing and confirming the on-site situation of the unmanned vehicle; if the review confirms that the unmanned vehicle is operating abnormally, issuing an alarm message; if the review confirms that the unmanned vehicle is operating normally, re-marking the state detection result corresponding to the multimodal feature data.

[0130] Based on any of the above embodiments, the multimodal feature data module 620 can be specifically used to: obtain an audio sequence based on audio data, obtain an image sequence based on image data, and obtain a video sequence based on video data; position-encode the audio sequence, image sequence, and video sequence in chronological order to obtain sequence information; input the sequence information into a feature extraction model to obtain multimodal feature data output by the feature extraction model; wherein the feature extraction model includes an audio feature extraction model, an image feature extraction model, and a video feature extraction model; the audio feature extraction model is used to extract features from the audio sequence; the image feature extraction model is used to extract features from the image sequence; and the video feature extraction model is used to extract features from the video sequence.

[0131] Based on any of the above embodiments, the multidimensional algorithm model module 630 can be specifically used for: the multidimensional algorithm model fuses various types of feature information in the multimodal feature data to obtain fused data; the multidimensional algorithm model uses a multi-head attention mechanism to process the fused data to focus on different features in the fused data to obtain a first intermediate result; the multidimensional algorithm model integrates the first intermediate result by setting a context layer to consider the time dependency of the sequence information to obtain the state detection result of the unmanned vehicle.

[0132] Based on any of the above embodiments, the status information module 610 can be specifically used to: obtain the temperature information of the unmanned vehicle on the road section it passes through as temperature data through an infrared thermal imaging module, the temperature data including the engine temperature and / or the temperature distribution of the entire vehicle; obtain the sound information of the unmanned vehicle on the road section it passes through as sound data through a sound array sensor, the sound data including the engine sound and / or the sound distribution of the entire vehicle; obtain the image information of the unmanned vehicle on the road section it passes through as image data through a camera.

[0133] On the other hand, the embodiment of the present invention also provides a multi-dimensional data acquisition device, see Figure 7 , Figure 7 is a schematic diagram of the physical structure of a multi-dimensional data acquisition device provided by an embodiment of the present invention, such as Figure 7 As shown, the multi-dimensional data acquisition device may include a sensor module (not shown), a memory 720 , a processor 710 , and a computer program stored in the memory 720 and executable on the processor 710 .

[0134] The sensor module is used to collect the status information of the unmanned vehicle on the road section it passes through; when the processor 710 executes the program, the operating status detection method of the unmanned vehicle provided by the above methods is implemented.

[0135] Optionally, the multi-dimensional data acquisition device may further include a communication bus 730 and a communication interface (Communications Interface) 740, wherein the processor 710, the communication interface 740, and the memory 720 communicate with each other through the communication bus 730. The processor 710 may call a computer program in the memory 720 to execute the operating state detection method of the unmanned vehicle, and the method may include: Obtaining status information of the unmanned vehicle on the road section it passes; wherein the status information includes at least two of temperature data, audio data, image data and video data; preprocessing the status information and arranging it in chronological order to obtain multimodal feature data; inputting the multimodal feature data into a multidimensional algorithm model to obtain a status detection result of the unmanned vehicle output by the multidimensional algorithm model; wherein the multidimensional algorithm model is used to obtain a target vector value based on the multimodal feature data, and determine the status detection result of the unmanned vehicle based on the correspondence between the vector value and the status label.

[0136] In addition, the logic instructions in the above-mentioned memory 720 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0137] On the other hand, the present invention also provides an unmanned vehicle operation status detection system, comprising a plurality of the above-mentioned multi-dimensional data acquisition devices, each of which is deployed at a different position of the road section where the unmanned vehicle passes.

[0138] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the unmanned vehicle operation status detection method provided by the above methods. Its steps and principles have been introduced in detail in the above methods and will not be repeated here.

[0139] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, it is implemented to execute the unmanned vehicle operation status detection method provided by the above-mentioned methods. Its steps and principles have been introduced in detail in the above-mentioned methods and will not be repeated here.

[0140] Non-transitory computer-readable storage media can be any available media or data storage devices that can be accessed by the processor, including but not limited to magnetic storage (such as floppy disks, hard disks, magnetic tapes, magneto-optical disks (MO)), optical storage (such as CDs, DVDs, BDs, HVDs, etc.), and semiconductor storage (such as ROM, EPROM, EEPROM, non-volatile memory (NANDFLASH), solid-state drives (SSDs)), etc.

[0141] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0142] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0143] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for detecting the operating status of an unmanned vehicle, characterized in that: Applied to a multi-dimensional data acquisition device, the multi-dimensional data acquisition device is deployed on a road section where an unmanned vehicle passes, and the operating status detection method of the unmanned vehicle includes: Acquire status information of the unmanned vehicle on the road section it passes through; wherein the status information includes at least two of temperature data, audio data, image data and video data; Preprocessing the state information and arranging it in chronological order to obtain multimodal feature data; Inputting the multimodal feature data into a multidimensional algorithm model to obtain a state detection result of the unmanned vehicle output by the multidimensional algorithm model; Among them, the multi-dimensional algorithm model is used to obtain the target vector value according to the multi-modal feature data, and determine the state detection result of the unmanned vehicle according to the corresponding relationship between the vector value and the state label.

2. The method for detecting the operating status of an unmanned vehicle according to claim 1, characterized in that: The multidimensional algorithm model is pre-set with the correspondence between the vector value and the state label, and the correspondence between the vector value and the state label is obtained by training and optimizing the multidimensional algorithm model based on the labeling result of the abnormal state training set; The state detection result of the unmanned vehicle output by the multi-dimensional algorithm model is obtained, including: The multi-dimensional algorithm model determines a state label corresponding to the target vector value based on the corresponding relationship, and determines a state detection result of the unmanned vehicle based on the state label.

3. The method for detecting the operating status of an unmanned vehicle according to claim 1, characterized in that: After inputting the multimodal feature data into the multidimensional algorithm model to obtain the state detection result of the unmanned vehicle output by the multidimensional algorithm model, the method further includes: When the state detection result of the unmanned vehicle is abnormal, review and confirm the on-site situation of the unmanned vehicle; If the unmanned vehicle is confirmed to be operating abnormally, an alarm message will be issued; If the unmanned vehicle is confirmed to be operating normally, the state detection result corresponding to the multimodal feature data is re-marked.

4. The method for detecting the operating status of an unmanned vehicle according to claim 1, characterized in that: The state information is preprocessed and sorted in chronological order to obtain multimodal feature data, including: Obtain an audio sequence according to the audio data, obtain an image sequence according to the image data, and obtain a video sequence according to the video data; Position-encoding the audio sequence, the image sequence and the video sequence in time order to obtain sequence information; Inputting the sequence information into a feature extraction model to obtain multimodal feature data output by the feature extraction model; The feature extraction model includes an audio feature extraction model, an image feature extraction model and a video feature extraction model; the audio feature extraction model is used to extract features from the audio sequence; the image feature extraction model is used to extract features from the image sequence; and the video feature extraction model is used to extract features from the video sequence.

5. The method for detecting the operating status of an unmanned vehicle according to claim 4, characterized in that: The step of inputting the multimodal feature data into a multidimensional algorithm model to obtain a state detection result of the unmanned vehicle output by the multidimensional algorithm model includes: The multi-dimensional algorithm model fuses various types of feature information in the multi-modal feature data to obtain fused data; The multi-dimensional algorithm model processes the fused data using a multi-head attention mechanism to focus on different features in the fused data to obtain a first intermediate result; The multi-dimensional algorithm model integrates the first intermediate result by setting a context layer to consider the time dependency of sequence information, thereby obtaining a state detection result of the unmanned vehicle.

6. The method for detecting the operating status of an unmanned vehicle according to any one of claims 1 to 5, characterized in that: The obtaining of the state information of the unmanned vehicle on the road section it passes through includes: Acquiring temperature information of the unmanned vehicle on the road section it passes through as temperature data through an infrared thermal imaging camera module, wherein the temperature data includes engine temperature and / or vehicle temperature distribution; Acquiring sound information of the unmanned vehicle on the road section it passes through as sound data through a sound array sensor, wherein the sound data includes engine sound and / or whole vehicle sound distribution; Image information of the driverless vehicle on the road section it passes through is acquired through a camera as image data.

7. A device for detecting the running status of an unmanned vehicle, characterized in that: The multi-dimensional data acquisition device is arranged on a road section where an unmanned vehicle passes. The running state detection device of the unmanned vehicle includes: A status information module, used to obtain status information of the unmanned vehicle on the road section it passes through; wherein the status information includes at least two of temperature data, audio data, image data and video data; A multimodal feature data module, used to pre-process the state information and sort it in chronological order to obtain multimodal feature data; A multi-dimensional algorithm model module, used to input the multi-modal feature data into the multi-dimensional algorithm model to obtain the state detection result of the unmanned vehicle output by the multi-dimensional algorithm model; Among them, the multi-dimensional algorithm model is used to obtain the target vector value according to the multi-modal feature data, and determine the state detection result of the unmanned vehicle according to the corresponding relationship between the vector value and the state label.

8. A multi-dimensional data acquisition device, characterized in that: The device comprises a sensor module, a memory, a processor, and a computer program stored in the memory and running on the processor; The sensor module is used to collect status information of the unmanned vehicle on the road section it passes through; when the processor executes the computer program, it implements the operating status detection method of the unmanned vehicle as described in any one of claims 1 to 6.

9. A system for detecting the operating status of an unmanned vehicle, characterized in that: It comprises a plurality of multi-dimensional data acquisition devices as described in claim 8, each of which is deployed at a different position of the road section where the unmanned vehicle passes.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the operating status detection method of an unmanned vehicle as described in any one of claims 1 to 6 is implemented.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the operating status detection method of an unmanned vehicle as described in any one of claims 1 to 6 is implemented.

Images