Fault Diagnosis System and Method for Unmanned Sweeping Vehicle Based on Multimodal Large Model; and Unmanned Sweeping Vehicle
By leveraging the reasoning capabilities of a multimodal large model framework and a large language model, the high cost and scalability issues of fault diagnosis for unmanned sweepers are resolved. This enables unified processing and flexible diagnosis of data from multiple modalities, thereby improving the accuracy and reliability of fault diagnosis.
Patent Information
- Application Number
- CN202611098733.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2026-07-23
- Publication Date
- 2026-11-10
- Estimated Expiration
- 2046-07-23
AI Technical Summary
Existing fault diagnosis technologies for unmanned sweepers require the design of dedicated models for each fault type, resulting in high development costs, poor system scalability, lack of cross-modal understanding capabilities, and difficulty in quickly responding to newly emerging fault types.
A unified framework based on a multimodal large model is adopted, which encodes multimodal data into a feature space that can be understood by a large language model through a multimodal encoder and a projection network. Fault diagnosis is performed by utilizing the reasoning ability of the large language model, including the combination of multimodal data acquisition, encoding, alignment training and projection network modules.
It enables unified processing of multiple modal data and fault types, reduces R&D costs, has flexible scalability, can quickly diagnose known and emerging faults, and improves the accuracy and reliability of fault diagnosis.
Smart Images

Figure CN122594760B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of fault diagnosis for unmanned vehicles, and in particular to a fault diagnosis system, diagnostic method, and unmanned sweeper based on a multimodal large model. Background Technology
[0002] With the rapid development of artificial intelligence and autonomous driving technologies, unmanned cleaning vehicles are being used more and more widely in urban sanitation, park cleaning, and other scenarios. The number of unmanned cleaning vehicles deployed is constantly increasing, and the deployment scenarios are becoming increasingly complex, covering a variety of environments such as non-standard roads, residential areas, and commercial plazas.
[0003] Unmanned cleaning vehicles face numerous challenges in actual operation: First, the operating environment is characterized by frequent movement of people and vehicles, leading to frequent environmental changes; second, the vehicle's own operating mechanism is complex, including multiple subsystems such as sweeping, vacuuming, and water spraying; furthermore, the vehicle is equipped with a variety of onboard sensors, including motor current sensors, inertial measurement units (IMUs), vibration sensors, noise sensors, surround-view cameras, and lidar. These factors result in a wide variety of malfunctions in unmanned cleaning vehicles, manifesting in diverse forms.
[0004] In terms of fault diagnosis, existing technologies mainly adopt the approach of designing dedicated models and algorithms for specific fault types. For example, Chinese patent application CN121859182A discloses a fault diagnosis method and device for unmanned sweepers. This method utilizes multi-source sensors on the unmanned sweeper, combined with an LSTM model, to diagnose specific faults in the automatic charging and water filling system through a set decision-making method.
[0005] However, existing technologies have the following shortcomings: High development costs: Dedicated diagnostic models need to be designed and trained for each type of fault. Different fault types and different combinations of modal data require the development of corresponding algorithms, resulting in long development cycles and high costs.
[0006] Poor system scalability: Existing methods cannot effectively support new types of faults that emerge after system deployment. It is necessary to collect data, design models, and train them again, making it difficult to respond quickly to actual operation and maintenance needs.
[0007] Lack of cross-modal understanding: Existing methods can usually only process data from a single or limited number of modalities, making it difficult to fully utilize the complementary information provided by the various sensors of the unmanned sweeper for comprehensive judgment.
[0008] The disclosure of the above background technical content is only for the purpose of assisting in understanding the concept and technical solution of this application, and does not necessarily provide technical instruction. Summary of the Invention
[0009] The purpose of this invention is to provide a fault diagnosis solution for unmanned sweepers that has a unified framework for processing multiple modal data, supports multiple fault types, and has flexible scalability.
[0010] To achieve the above objectives, the technical solution adopted by the present invention is as follows: A fault diagnosis system for unmanned sweepers based on a multimodal large model includes: The multimodal data acquisition module is configured to acquire multimodal data during the operation of the unmanned sweeper, including various types of data such as one-dimensional sensor time-series data, two-dimensional image data, and three-dimensional point cloud data. A multimodal encoder module, which includes multiple types of one-dimensional sequence encoders, image encoders, and point cloud encoders; A multimodal alignment training module utilizes a text encoder to train each encoder of the multimodal encoder module in two stages to encode the multimodal data into a feature space aligned to a large language model. The text encoder receives text description data adapted to the training state and encodes and outputs text embedding vectors. In the first stage, the similarity between the embedding vector output by a single encoder of the multimodal encoder module and the text embedding vector is calculated, and the corresponding single encoder is updated based on the similarity calculation results. In the second stage, the similarity between two or more embedding vectors output by multiple encoders of the multimodal encoder module and the text embedding vector is calculated, and the corresponding multiple encoders are updated based on the similarity calculation results. The projection network module is configured to project the embedding vectors output by each encoder trained in the multimodal encoder module onto an embedding representation that can be understood by the large language model. A large language model reasoning module is configured to receive the embedded representation and perform joint understanding and analysis based on the reasoning capabilities of the large language model to output the fault diagnosis results of the unmanned sweeper.
[0011] Furthermore, following any one or a combination of the aforementioned technical solutions, the first stage further includes updating the text encoder in reverse based on the similarity calculation results; The text encoder in the second stage receives text description data from two or more modal data, and also includes back-updating the text encoder based on the average of two or more similarity calculation results; Both the first and second stages aim to improve similarity as the goal of updating the encoder.
[0012] Furthermore, following any or a combination of the aforementioned technical solutions, the unmanned sweeper fault diagnosis system further includes an instruction construction module, which is configured to generate task instruction text according to a preset format. This text is used to guide the large language model inference module to perform fault diagnosis tasks. The task instruction text includes a general description of the multimodal data and diagnostic questions.
[0013] Furthermore, following any one or a combination of the aforementioned technical solutions, the projection network module is trained in the following manner: Acquire training samples, which include multimodal data samples, instruction texts, and corresponding standard fault diagnosis texts; The multimodal data samples are sequentially passed through the encoder of the multimodal encoder module and the projection network module to obtain a multimodal embedding representation; The multimodal embedding representation and the instruction text in the training samples are input into the large language model inference module to generate predicted fault diagnosis text; Calculate the cross-entropy loss between the predicted fault diagnosis text and the standard fault diagnosis text in the training samples, and update the parameters of the projection network module based on the gradient of the cross-entropy loss through backpropagation.
[0014] Furthermore, following any one or a combination of the aforementioned technical solutions, the training process of the projection network module is divided into two stages: During the first phase, the parameters of the multimodal encoder module and the large language model inference module are frozen, and the parameters of the projection network module are updated only based on the gradient backpropagation of the cross-entropy loss. In the second stage, all or part of the parameters of the large language model inference module are unfrozen and updated in coordination with the parameters of the projection network module.
[0015] Furthermore, following any one or a combination of the aforementioned technical solutions, the collaborative update of the parameters of the large language model inference module and the parameters of the projection network module includes: Calculate the cross-entropy loss: ,in, Represents the cross-entropy loss function. This indicates the text for predicting fault diagnosis. y This represents standard fault diagnosis text; Calculate cross-entropy loss L The gradient of the parameters of the inference module of the large language model: The trainable parameters of the large language model inference module are updated using this gradient. Calculate cross-entropy loss L Gradients of the projection network module parameters: ,in, This represents the gradient signal propagated back from the inference module of the large language model; The optimizer is used to simultaneously update the parameters of the large language model inference module. Parameters of the projection network module : ; ;in, η This is the learning rate.
[0016] Furthermore, following any or a combination of the aforementioned technical solutions, the multimodal encoder module includes a one-dimensional sequence encoder, which employs one of the following: a temporal Transformer network, a 1D convolutional network, a short-time Fourier transform, and wavelet decomposition. The one-dimensional sequence encoder is configured to encode the one-dimensional sensing temporal data to output a one-dimensional embedding vector. The one-dimensional sensing time-series data includes one or more of the following: motor current data of the unmanned sweeper, IMU acceleration and angular velocity data, vibration sensor data, and noise data; The similarity between the one-dimensional embedding vector and the text embedding vector is calculated using cosine similarity: = ,in, Represents a one-dimensional embedding vector. Represents a text embedding vector. Indicates the modulus length. This represents the similarity value between the one-dimensional embedding vector and the text embedding vector.
[0017] Furthermore, following any or a combination of the aforementioned technical solutions, the multimodal encoder module includes an image encoder, which employs either a ViT network or a CNN convolutional network. The image encoder is configured to encode the two-dimensional image data to output an image embedding vector. The two-dimensional image data includes single or multiple frames of environmental images captured by cameras installed on the sweeper. The similarity between the image embedding vector and the text embedding vector is calculated using cosine similarity: = ,in, Represents the image embedding vector. Represents a text embedding vector. Indicates the modulus length. This represents the similarity value between the image embedding vector and the text embedding vector.
[0018] Furthermore, following any or a combination of the aforementioned technical solutions, the multimodal encoder module includes a point cloud encoder, which employs one of Point-BERT, Point-MAE, and PointNet++. The point cloud encoder is configured to encode the three-dimensional point cloud data to output a point cloud embedding vector. The three-dimensional point cloud data includes single-moment point cloud data or multi-moment point cloud data sequences generated by radar installed on the sweeper. The similarity between the point cloud embedding vector and the text embedding vector is calculated using cosine similarity: = ,in, Represents the point cloud embedding vector. Represents a text embedding vector. Indicates the modulus length. This represents the similarity value between the point cloud embedding vector and the text embedding vector.
[0019] Furthermore, following any one or a combination of the aforementioned technical solutions, the text encoder adopts a CLIP text encoder; The single non-text modal data involved in the first stage corresponds to the same fault event as the text description data input to the text encoder, and the text description data contains a description of the single non-text modal data involved in the first stage. In the second stage, the multiple non-text modal data involved correspond to the same fault event as the text description data input to the text encoder, and the text description data also contains a joint description of the multiple non-text modal data involved in the second stage.
[0020] According to another aspect of the present invention, a fault diagnosis method for unmanned sweepers based on a multimodal large model is provided, comprising the following steps: A multimodal encoder module is configured, which includes multiple types such as a one-dimensional sequence encoder, an image encoder, and a point cloud encoder. The one-dimensional sequence encoder is configured to encode one-dimensional sensing time-series data to output a one-dimensional embedding vector, the image encoder is configured to encode two-dimensional image data to output an image embedding vector, and the point cloud encoder is configured to encode three-dimensional point cloud data to output a point cloud embedding vector. The text encoder is pre-trained using a text encoder in two stages. The text encoder is configured to receive text description data and encode and output text embedding vectors. In the first stage, the similarity between the embedding vector output by a single encoder of the multimodal encoder module and the text embedding vector is calculated, and the corresponding single encoder is updated based on the similarity calculation results. In the second stage, the similarity between two or more embedding vectors output by multiple encoders of the multimodal encoder module and the text embedding vector is calculated, and the corresponding multiple encoders are updated based on the similarity calculation results. Acquire multimodal data during the operation of the unmanned sweeper, including various types of data such as one-dimensional sensor time-series data, two-dimensional image data, and three-dimensional point cloud data; The multimodal data is input into the trained multimodal encoder module, and each encoder encodes and outputs the corresponding embedding vector. Project the embedding vectors output by each encoder onto an embedding representation that the large language model can understand; The embedded representation is jointly understood and analyzed based on the reasoning capabilities of a large language model to output the fault diagnosis results of the unmanned sweeper.
[0021] Furthermore, following any one or a combination of the aforementioned technical solutions, a projection network module is used to project the embedding vectors output by each encoder onto an embedding representation that the large language model can understand. The training method of the projection network module includes: Acquire training samples, which include multimodal data samples, instruction texts, and corresponding standard fault diagnosis texts; The multimodal data samples are sequentially passed through the encoder of the multimodal encoder module and the projection network module to obtain a multimodal embedding representation; The multimodal embedding representation and the instruction text in the training samples are input into the large language model for inference to generate predicted fault diagnosis text; Calculate the cross-entropy loss between the predicted fault diagnosis text and the standard fault diagnosis text in the training samples, and update the parameters of the projection network module based on the gradient of the cross-entropy loss through backpropagation.
[0022] Furthermore, based on any one or a combination of the aforementioned technical solutions, acquiring multimodal data during the operation of the unmanned sweeper includes: acquiring one or more of the following: motor current data, IMU acceleration and angular velocity data, vibration sensor data, and noise data of the unmanned sweeper; using a camera installed on the sweeper to collect single-frame or multi-frame environmental images; and using a radar installed on the sweeper to generate single-moment point cloud data or multi-moment point cloud data sequences. During the training process of the encoder, the text encoder receives text description data that is adapted to the training state to encode and output text embedding vectors. The similarity between the embedding vectors output by each encoder and the text embedding vector is calculated using cosine similarity: = , = , = ,in, This represents the similarity value between the one-dimensional embedding vector and the text embedding vector. This represents a one-dimensional embedding vector output by a one-dimensional sequence encoder. This represents the similarity value between the image embedding vector and the text embedding vector. This represents the image embedding vector output by the image encoder; This represents the similarity value between the point cloud embedding vector and the text embedding vector. This represents the point cloud embedding vector output by the point cloud encoder; Represents a text embedding vector. Indicates the modulus length.
[0023] According to another aspect of the present invention, an unmanned sweeper is provided, including a sweeper, an unmanned driving system, and a fault diagnosis system as described above.
[0024] The beneficial effects of the technical solution provided by this invention are as follows: a. This invention adopts a unified multimodal large model framework, eliminating the need to design dedicated models and algorithms for different fault types. Only one system is needed to process diagnostic tasks for multiple modal data and multiple fault types, significantly reducing R&D costs; b. Leveraging the powerful reasoning capabilities of large language models, this invention can not only diagnose known fault types, but also analyze and judge newly emerging faults after system deployment, possessing flexible diagnostic capabilities and good system scalability; c. By using a two-stage training method and projection network for a multimodal encoder, this invention can uniformly map multiple modal data, such as one-dimensional sensor time-series data, two-dimensional image data, and three-dimensional point cloud data, onto the feature space of a large language model, thereby achieving cross-modal joint understanding and comprehensive analysis, making full use of the complementary information from multiple sensors, and improving the accuracy and reliability of fault diagnosis. Attached Figure Description
[0025] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0026] Figure 1 A schematic block diagram of a fault diagnosis system for an unmanned sweeper vehicle based on a multimodal large model, provided as an exemplary embodiment of the present invention; Figure 2 A schematic diagram of the data input and encoding output states of a multimodal encoder provided as an exemplary embodiment of the present invention; Figure 3 A schematic diagram of the first training phase of a one-dimensional sequence encoder provided as an exemplary embodiment of the present invention; Figure 4 A schematic diagram of the first training phase of an image encoder provided as an exemplary embodiment of the present invention; Figure 5 A schematic diagram of the second training phase of a multimodal encoder provided as an exemplary embodiment of the present invention; Figure 6 A training schematic diagram of a projection network module provided as an exemplary embodiment of the present invention; Figure 7 A schematic diagram of the operating status of a fault diagnosis system for a trained encoder and projection network module, provided as an exemplary embodiment of the present invention; Figure 8 This is a flowchart illustrating a fault diagnosis method for unmanned sweepers based on a multimodal large model, provided as an exemplary embodiment of the present invention. Detailed Implementation
[0027] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.
[0028] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, apparatus, product, or device that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.
[0029] The present invention aims to provide a fault diagnosis system for unmanned sweepers based on a multimodal large model, which enables flexible diagnosis of multiple modal data and multiple fault types using a single model framework, thereby reducing development costs and improving the scalability of the system.
[0030] In one embodiment of the present invention, a fault diagnosis system for unmanned sweepers based on a multimodal large model is provided, such as... Figure 1 As shown, the fault diagnosis system includes: The multimodal data acquisition module is configured to acquire multimodal data during the operation of the unmanned sweeper, including various types of data such as one-dimensional sensor time-series data, two-dimensional image data, and three-dimensional point cloud data. A multimodal encoder module, which includes multiple types of one-dimensional sequence encoders, image encoders, and point cloud encoders; A multimodal alignment training module utilizes a text encoder to train each encoder of the multimodal encoder module in two stages to encode the multimodal data into a feature space aligned to a large language model. The text encoder receives text description data adapted to the training state and encodes and outputs text embedding vectors. In the first stage, the similarity between the embedding vector output by a single encoder of the multimodal encoder module and the text embedding vector is calculated, and the corresponding single encoder is updated based on the similarity calculation results. In the second stage, the similarity between two or more embedding vectors output by multiple encoders of the multimodal encoder module and the text embedding vector is calculated, and the corresponding multiple encoders are updated based on the similarity calculation results. The projection network module is configured to project the embedding vectors output by each encoder trained in the multimodal encoder module onto an embedding representation that can be understood by the large language model. The Large Language Model (LLM) reasoning module is configured to receive the embedded representation and perform joint understanding and analysis based on the reasoning capabilities of the Large Language Model (LLM) to output the fault diagnosis results of the unmanned sweeper.
[0031] As described above, the multimodal encoder module in this invention can be a free combination of a one-dimensional sequence encoder, an image encoder, and a point cloud encoder. The following example, using a combination of a one-dimensional sequence encoder, an image encoder, and a point cloud encoder, illustrates the working process of the fault diagnosis system: like Figure 2 As shown, a one-dimensional sequence encoder encodes one-dimensional sensing time-series data to output a one-dimensional embedding vector; the one-dimensional sensing time-series data mentioned here includes one or more of the following: motor current data, IMU acceleration and angular velocity data, vibration sensor data, and noise data of the unmanned sweeper; the one-dimensional sequence encoder can employ one of the following: temporal Transformer network, 1D convolutional network, short-time Fourier transform, and wavelet decomposition. like Figure 2 As shown, the image encoder encodes two-dimensional image data to output an image embedding vector; the two-dimensional image data mentioned here includes single or multiple frames of environmental images captured by a camera installed on the sweeper, and the image encoder can use either a ViT network or a CNN convolutional network. like Figure 2 As shown, the point cloud encoder encodes 3D point cloud data to output point cloud embedding vectors; the 3D point cloud data mentioned here includes single-time point cloud data or multi-time point cloud data sequences generated by the radar installed on the sweeper, and the point cloud encoder can use one of Point-BERT, Point-MAE, or PointNet++. See also Figure 2 The text encoder used for multimodal alignment training encodes the text description data to output a text embedding vector; the text encoder may be a CLIP text encoder; the single non-text modal data involved in the first stage corresponds to the same fault event as the text description data input to the text encoder, and the text description data contains a description of the single non-text modal data involved in the first stage; the multiple non-text modal data involved in the second stage correspond to the same fault event as the text description data input to the text encoder, and the text description data simultaneously contains a joint description of the multiple non-text modal data involved in the second stage.
[0032] Figure 3The first stage illustrates that the single non-text modal data involved is one-dimensional sensor time-series data. By calculating the similarity between the corresponding one-dimensional embedding vector and the text embedding vector, the corresponding one-dimensional sequence encoder is updated in reverse based on the similarity calculation result. Preferably, the text encoder is also updated in reverse at the same time. Specifically, for example, the one-dimensional sensor time-series data is a sequence of 1000 current values of a brush motor, and the text description data is "This is a sequence of current values of a brush motor, the circuit amplitude has increased." The text description data here is written according to the current actual fault during the training process, and the purpose is to enable the encoder to learn to understand the physical meaning represented by the one-dimensional sensor time-series data during the training phase. In the actual fault diagnosis phase outside the training phase, the text encoder is no longer needed.
[0033] Figure 4 The first stage illustrates that the single non-text modal data involved is two-dimensional image data. The similarity between the corresponding image embedding vector and the text embedding vector is calculated, and the corresponding image encoder is updated in reverse based on the similarity calculation result. Preferably, the text encoder is also updated in reverse simultaneously. Specifically, for example, the two-dimensional image data is a four-channel stitched image from a surround-view camera, and the text description data is: "This is a stitched image of four surround-view cameras. The top left is the front view, the top right is the rear view, the bottom left is the left view, and the bottom right is the right view. A large area of pedestrians appears in the front view." Similarly, this text description data is written during the training process based on the current actual fault, with the purpose of enabling the encoder to understand the physical meaning represented by the two-dimensional image data during the training phase. In the actual fault diagnosis phase outside the training phase, the text encoder is no longer needed.
[0034] The single non-text modal data involved in the first stage is three-dimensional point cloud data (not shown). Similarly, by calculating the similarity between the corresponding embedding vector and the text embedding vector, the corresponding image encoder is updated in reverse according to the similarity calculation result. Preferably, the text encoder is also updated in reverse at the same time. In one specific embodiment, cosine similarity is used to calculate the similarity between the embedding vectors output by each encoder (one-dimensional embedding vector, image embedding vector, point cloud embedding vector) and the text embedding vector: First, the formula for calculating the similarity between the one-dimensional embedding vector and the text embedding vector is as follows: = ,in, Represents a one-dimensional embedding vector. Represents a text embedding vector. Indicates the modulus length. This represents the similarity value between the one-dimensional embedding vector and the text embedding vector.
[0035] Secondly, the similarity calculation formula between the image embedding vector and the text embedding vector is as follows: = ,in, Represents the image embedding vector. Represents a text embedding vector. Indicates the modulus length. This represents the similarity value between the image embedding vector and the text embedding vector.
[0036] Furthermore, the similarity calculation formula between the point cloud embedding vector and the text embedding vector is as follows: = ,in, Represents the point cloud embedding vector. Represents a text embedding vector. Indicates the modulus length. This represents the similarity value between the point cloud embedding vector and the text embedding vector.
[0037] The second phase of the update process is as follows: Figure 5 As shown, a similarity matrix is calculated between the one-dimensional embedding vector output by the one-dimensional sequence encoder, the image embedding vector output by the image encoder, the point cloud embedding vector output by the point cloud encoder, and the text embedding vector. The one-dimensional sequence encoder, image encoder, and point cloud encoder are then updated in reverse based on the similarity matrix calculation results. Specifically, in the second stage, the text encoder receives text description data from two or more modalities. For example, when the sweeper brush is stuck by a bicycle on the side of the road, the one-dimensional sensing time-series data is motor current data, the two-dimensional image data is surround-view camera data, the three-dimensional point cloud data is radar data, and the text description data is "This is the current data from three interconnected data points. The current data continues to increase, and the forward-looking camera..." The algorithm calculates the similarity between the 1D embedding vector corresponding to the 1D sensing time-series data and the text embedding vector output by the text encoder, thereby updating the 1D sequence encoder; it also calculates the similarity between the image embedding vector corresponding to the 2D image data and the text embedding vector output by the text encoder, thereby updating the image encoder; and it calculates the similarity between the point cloud embedding vector corresponding to the 3D point cloud data and the text embedding vector output by the text encoder, thereby updating the point cloud encoder. Furthermore, it includes updating the text encoder in reverse based on the average of the above three similarity calculation results. Synchronously updating the text encoder is to avoid the fixed text encoder affecting the alignment effect.
[0038] Both the first and second stages aim to improve similarity as the goal of updating the encoder. Furthermore, the training of the first and / or second stages of each encoder can be performed iteratively multiple times.
[0039] In one specific embodiment, the unmanned sweeper fault diagnosis system further includes an instruction construction module, which is configured to generate task instruction text according to a preset format. This text is used to guide the large language model inference module to perform fault diagnosis tasks. The task instruction text includes a general description of the multimodal data and diagnostic questions.
[0040] After the multimodal encoders are trained, the projection network module projects the embedding vectors output by each encoder into an LLM space to obtain an embedding representation that the large language model can understand. This representation serves as the input to the large language model inference module (hereinafter referred to as LLM), along with the task instruction text output by the instruction construction module and manually labeled text descriptions, to train the projection network module. Figure 6 As shown, the task instruction text output by the instruction construction module is "This is the sweeper motor current data, vehicle surround view image, and point cloud data around the sweeper. What's wrong with the sweeper?" An example of the LLM output text is: "Increased current, a bicycle appears in front of the surround view image, there is data in front of the point cloud, the sweeper may be stuck by the bicycle, manual intervention is required." This output is compared with the manually labeled text description to update the parameters of the projection network module, ensuring that the embedded representation of its projection output more closely matches the LLM output with the manually labeled text description.
[0041] The specific training method for the projection network module is as follows: Acquire training samples, which include multimodal data samples, instruction texts, and corresponding standard fault diagnosis texts; The multimodal data samples are sequentially passed through the encoder of the multimodal encoder module and the projection network module to obtain a multimodal embedding representation; The multimodal embedding representation and the instruction text in the training samples are input into the large language model inference module to generate predicted fault diagnosis text; The cross-entropy loss between the predicted fault diagnosis text and the standard fault diagnosis text in the training samples is calculated using the following formula: ,in, Represents the cross-entropy loss function. This indicates the text for predicting fault diagnosis. y The standard fault diagnosis text is represented; and the parameters of the projection network module are updated according to the gradient backpropagation of the cross-entropy loss, so that the embedded representation of its projection output can make the LLM output closer to the manually labeled text description.
[0042] The training process for the projection network module here is also divided into two stages: During the first phase, the parameters of the multimodal encoder module and the large language model inference module are frozen, and the parameters of the projection network module are updated only based on the gradient backpropagation of the cross-entropy loss. In the second stage, all or part of the parameters of the large language model inference module are unfrozen and updated collaboratively with the parameters of the projection network module. The projection network module and the large language model inference module can be jointly trained in an end-to-end manner to fully adapt the multimodal embedding representation to the inference capability of the large language model, further improving the diagnostic effect. Specifically, the collaborative update method is as follows: Calculate the cross-entropy loss: ,in, Represents the cross-entropy loss function. This indicates the text for predicting fault diagnosis. y This represents standard fault diagnosis text; Calculate cross-entropy loss L The gradient of the parameters of the inference module of the large language model: The trainable parameters of the large language model inference module are updated using this gradient. Calculate cross-entropy loss L Gradients of the projection network module parameters: ,in, This represents the gradient signal propagated back from the inference module of the large language model; The optimizer is used to simultaneously update the parameters of the large language model inference module. Parameters of the projection network module : ;in, This represents the parameters of the large language model inference module at time t. , This represents the parameters of the large language model inference module at time t+1. ; ;in, This represents the parameters of the projection network module at time t. , This represents the parameters of the projection network module at time t+1. , η This is the learning rate.
[0043] After the projection network module is trained, the working process of the unmanned sweeper fault diagnosis system is as follows: Figure 7 As shown.
[0044] The unmanned sweeper fault diagnosis system provided in the above embodiments of the present invention does not require the design and training of different models for different fault types, nor does it require the implementation of different algorithms. Only one system is needed to solve the analysis of different modal data and different faults, resulting in low development costs. Furthermore, the two-stage training of the multimodal encoder and the training of the projection network can align and project normal and abnormal modal data into the feature space of the LLM. Based on the reasoning ability of the large model, it can also provide diagnostic results when different combinations of modal data and different fault types occur, demonstrating flexible diagnostic capabilities.
[0045] In one embodiment of the present invention, a fault diagnosis method for unmanned sweepers based on a multimodal large model is provided, such as... Figure 8 As shown, it includes the following steps: A multimodal encoder module is configured, which includes multiple types such as a one-dimensional sequence encoder, an image encoder, and a point cloud encoder. The one-dimensional sequence encoder is configured to encode one-dimensional sensing time-series data to output a one-dimensional embedding vector, the image encoder is configured to encode two-dimensional image data to output an image embedding vector, and the point cloud encoder is configured to encode three-dimensional point cloud data to output a point cloud embedding vector. The text encoder is pre-trained using a text encoder in two stages. The text encoder is configured to receive text description data and encode and output text embedding vectors. In the first stage, the similarity between the embedding vector output by a single encoder of the multimodal encoder module and the text embedding vector is calculated, and the corresponding single encoder is updated based on the similarity calculation results. In the second stage, the similarity between two or more embedding vectors output by multiple encoders of the multimodal encoder module and the text embedding vector is calculated, and the corresponding multiple encoders are updated based on the similarity calculation results. Acquire multimodal data during the operation of the unmanned sweeper, including various types of data such as one-dimensional sensor time-series data, two-dimensional image data, and three-dimensional point cloud data; The multimodal data is input into the trained multimodal encoder module, and each encoder encodes and outputs the corresponding embedding vector. Project the embedding vectors output by each encoder onto an embedding representation that the large language model can understand; The embedded representation is jointly understood and analyzed based on the reasoning capabilities of a large language model to output the fault diagnosis results of the unmanned sweeper.
[0046] Furthermore, a projection network module is used to project the embedding vectors output by each encoder onto an embedding representation that the large language model can understand. The training method for the projection network module includes: Acquire training samples, which include multimodal data samples, instruction texts, and corresponding standard fault diagnosis texts; The multimodal data samples are sequentially passed through the encoder of the multimodal encoder module and the projection network module to obtain a multimodal embedding representation; The multimodal embedding representation and the instruction text in the training samples are input into the large language model for inference to generate predicted fault diagnosis text; Calculate the cross-entropy loss between the predicted fault diagnosis text and the standard fault diagnosis text in the training samples, and update the parameters of the projection network module based on the gradient of the cross-entropy loss through backpropagation.
[0047] Furthermore, acquiring multimodal data during the operation of the unmanned sweeper includes: acquiring one or more of the following: motor current data, IMU acceleration and angular velocity data, vibration sensor data, and noise data; using cameras installed on the sweeper to collect single-frame or multi-frame environmental images; and using radar installed on the sweeper to generate single-time point cloud data or multi-time point cloud data sequences. During the training process of the encoder, the text encoder receives text description data that is adapted to the training state to encode and output text embedding vectors. The similarity between the embedding vectors output by each encoder and the text embedding vector is calculated using cosine similarity: = , = , = ,in, This represents the similarity value between the one-dimensional embedding vector and the text embedding vector. This represents a one-dimensional embedding vector output by a one-dimensional sequence encoder. This represents the similarity value between the image embedding vector and the text embedding vector. This represents the image embedding vector output by the image encoder; This represents the similarity value between the point cloud embedding vector and the text embedding vector. This represents the point cloud embedding vector output by the point cloud encoder; Represents a text embedding vector. Indicates the modulus length.
[0048] It should be noted that the fault diagnosis method for unmanned sweepers based on multimodal large models provided in this embodiment and the fault diagnosis system for unmanned sweepers based on multimodal large models provided in the above embodiments belong to the same inventive concept. Here, all the contents of the embodiment of the fault diagnosis system for unmanned sweepers based on multimodal large models are incorporated into the embodiment of the fault diagnosis method for unmanned sweepers based on multimodal large models by reference, and will not be repeated here.
[0049] Another embodiment of the present invention provides an unmanned sweeper, including a sweeper, an unmanned driving system, and a fault diagnosis system as described in the above embodiments, to perform the fault diagnosis method as described in the above embodiments.
[0050] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0051] The above description is only a specific embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.
Claims
1. A fault diagnosis system for unmanned sweeping vehicles based on a multimodal large model, characterized in that, include: The multimodal data acquisition module is configured to acquire multimodal data during the operation of the unmanned sweeper, including various types of data such as one-dimensional sensor time-series data, two-dimensional image data, and three-dimensional point cloud data. A multimodal encoder module, which includes multiple types of one-dimensional sequence encoders, image encoders, and point cloud encoders; A multimodal alignment training module utilizes a text encoder to train each encoder of the multimodal encoder module in two stages to encode the multimodal data into a feature space aligned to a large language model. The text encoder receives text description data adapted to the training state and encodes and outputs text embedding vectors. In the first stage, the similarity between the embedding vector output by a single encoder of the multimodal encoder module and the text embedding vector is calculated, and the corresponding single encoder is updated based on the similarity calculation results. In the second stage, the similarity between two or more embedding vectors output by multiple encoders of the multimodal encoder module and the text embedding vector is calculated, and the corresponding multiple encoders are updated based on the similarity calculation results. The projection network module is configured to project the embedding vectors output by each encoder trained in the multimodal encoder module onto an embedding representation that can be understood by the large language model. A large language model reasoning module is configured to receive the embedded representation and perform joint understanding and analysis based on the reasoning capabilities of the large language model to output the fault diagnosis results of the unmanned sweeper.
2. The fault diagnosis system for unmanned sweeping vehicles based on a multimodal large model according to claim 1, characterized in that, The first stage also includes updating the text encoder in reverse based on the similarity calculation results; The text encoder in the second stage receives text description data from two or more modal data, and also includes back-updating the text encoder based on the average of two or more similarity calculation results; Both the first and second stages aim to improve similarity as the goal of updating the encoder.
3. The fault diagnosis system for unmanned sweeping vehicles based on a multimodal large model according to claim 1, characterized in that, It also includes an instruction construction module, which is configured to generate task instruction text in a preset format, which is used to guide the large language model inference module to perform fault diagnosis tasks. The task instruction text includes a general description of the multimodal data and diagnostic questions.
4. The fault diagnosis system for unmanned sweeping vehicles based on a multimodal large model according to claim 1, characterized in that, The projection network module is trained in the following manner: Acquire training samples, which include multimodal data samples, instruction texts, and corresponding standard fault diagnosis texts; The multimodal data samples are sequentially passed through the encoder of the multimodal encoder module and the projection network module to obtain a multimodal embedding representation; The multimodal embedding representation and the instruction text in the training samples are input into the large language model inference module to generate predicted fault diagnosis text; Calculate the cross-entropy loss between the predicted fault diagnosis text and the standard fault diagnosis text in the training samples, and update the parameters of the projection network module based on the gradient of the cross-entropy loss through backpropagation.
5. The fault diagnosis system for unmanned sweeping vehicles based on a multimodal large model according to claim 4, characterized in that, The training process of the projection network module is divided into two stages: During the first phase, the parameters of the multimodal encoder module and the large language model inference module are frozen, and the parameters of the projection network module are updated only based on the gradient backpropagation of the cross-entropy loss. In the second stage, all or part of the parameters of the large language model inference module are unfrozen and updated in coordination with the parameters of the projection network module.
6. The fault diagnosis system for unmanned sweeping vehicles based on a multimodal large model according to claim 5, characterized in that, The collaborative update of the parameters of the large language model inference module and the projection network module includes: Calculate the cross-entropy loss: ,in, Represents the cross-entropy loss function. This indicates the text for predicting fault diagnosis. y This represents standard fault diagnosis text; Calculate cross-entropy loss L The gradient of the parameters of the inference module of the large language model: The trainable parameters of the large language model inference module are updated using this gradient. Calculate cross-entropy loss L Gradients of the projection network module parameters: ,in, This represents the gradient signal propagated back from the inference module of the large language model; The optimizer is used to simultaneously update the parameters of the large language model inference module. Parameters of the projection network module : ; ;in, η This is the learning rate.
7. The unmanned sweeper fault diagnosis system based on a multimodal large model according to claim 1, characterized in that, The multimodal encoder module includes a one-dimensional sequence encoder, which employs one of the following: temporal Transformer network, 1D convolutional network, short-time Fourier transform, and wavelet decomposition. The one-dimensional sequence encoder is configured to encode the one-dimensional sensing temporal data to output a one-dimensional embedding vector. The one-dimensional sensing time-series data includes one or more of the following: motor current data of the unmanned sweeper, IMU acceleration and angular velocity data, vibration sensor data, and noise data; The similarity between the one-dimensional embedding vector and the text embedding vector is calculated using cosine similarity: = ,in, Represents a one-dimensional embedding vector. Represents a text embedding vector. Indicates the modulus length. This represents the similarity value between the one-dimensional embedding vector and the text embedding vector.
8. The fault diagnosis system for unmanned sweeping vehicles based on a multimodal large model according to claim 1, characterized in that, The multimodal encoder module includes an image encoder, which employs either a ViT network or a CNN convolutional network. The image encoder is configured to encode the two-dimensional image data to output an image embedding vector. The two-dimensional image data includes single or multiple frames of environmental images captured by cameras installed on the sweeper. The similarity between the image embedding vector and the text embedding vector is calculated using cosine similarity: = ,in, Represents the image embedding vector. Represents a text embedding vector. Indicates the modulus length. This represents the similarity value between the image embedding vector and the text embedding vector.
9. The fault diagnosis system for unmanned sweeping vehicles based on a multimodal large model according to claim 1, characterized in that, The multimodal encoder module includes a point cloud encoder, which employs one of Point-BERT, Point-MAE, and PointNet++. The point cloud encoder is configured to encode the three-dimensional point cloud data to output a point cloud embedding vector. The three-dimensional point cloud data includes single-moment point cloud data or multi-moment point cloud data sequences generated by radar installed on the sweeper. The similarity between the point cloud embedding vector and the text embedding vector is calculated using cosine similarity: = ,in, Represents the point cloud embedding vector. Represents a text embedding vector. Indicates the modulus length. This represents the similarity value between the point cloud embedding vector and the text embedding vector.
10. The unmanned sweeper fault diagnosis system based on a multimodal large model according to any one of claims 1 to 9, characterized in that, The text encoder used is the CLIP text encoder. The single non-text modal data involved in the first stage corresponds to the same fault event as the text description data input to the text encoder, and the text description data contains a description of the single non-text modal data involved in the first stage. In the second stage, the multiple non-text modal data involved correspond to the same fault event as the text description data input to the text encoder, and the text description data also contains a joint description of the multiple non-text modal data involved in the second stage.
11. A fault diagnosis method for unmanned sweeping vehicles based on a multimodal large model, characterized in that, Includes the following steps: A multimodal encoder module is configured, which includes multiple types such as a one-dimensional sequence encoder, an image encoder, and a point cloud encoder. The one-dimensional sequence encoder is configured to encode one-dimensional sensing time-series data to output a one-dimensional embedding vector, the image encoder is configured to encode two-dimensional image data to output an image embedding vector, and the point cloud encoder is configured to encode three-dimensional point cloud data to output a point cloud embedding vector. The text encoder is pre-trained using a text encoder in two stages. The text encoder is configured to receive text description data and encode and output text embedding vectors. In the first stage, the similarity between the embedding vector output by a single encoder of the multimodal encoder module and the text embedding vector is calculated, and the corresponding single encoder is updated based on the similarity calculation results. In the second stage, the similarity between two or more embedding vectors output by multiple encoders of the multimodal encoder module and the text embedding vector is calculated, and the corresponding multiple encoders are updated based on the similarity calculation results. Acquire multimodal data during the operation of the unmanned sweeper, including various types of data such as one-dimensional sensor time-series data, two-dimensional image data, and three-dimensional point cloud data; The multimodal data is input into the trained multimodal encoder module, and each encoder encodes and outputs the corresponding embedding vector. Project the embedding vectors output by each encoder onto an embedding representation that the large language model can understand; The embedded representation is jointly understood and analyzed based on the reasoning capabilities of a large language model to output the fault diagnosis results of the unmanned sweeper.
12. The fault diagnosis method for unmanned sweeping vehicles based on a multimodal large model according to claim 11, characterized in that, The embedding vectors output by each encoder are projected onto an embedding representation that can be understood by a large language model using a projection network module. The training method for the projection network module includes: Acquire training samples, which include multimodal data samples, instruction texts, and corresponding standard fault diagnosis texts; The multimodal data samples are sequentially passed through the encoder of the multimodal encoder module and the projection network module to obtain a multimodal embedding representation; The multimodal embedding representation and the instruction text in the training samples are input into the large language model for inference to generate predicted fault diagnosis text; Calculate the cross-entropy loss between the predicted fault diagnosis text and the standard fault diagnosis text in the training samples, and update the parameters of the projection network module based on the gradient of the cross-entropy loss through backpropagation.
13. The fault diagnosis method for unmanned sweeping vehicles based on a multimodal large model according to claim 11, characterized in that, Acquiring multimodal data during the operation of the unmanned sweeper includes: acquiring one or more of the following: motor current data, IMU acceleration and angular velocity data, vibration sensor data, and noise data; using cameras installed on the sweeper to collect single-frame or multi-frame environmental images; and using radar installed on the sweeper to generate single-time point cloud data or multi-time point cloud data sequences. During the training process of the encoder, the text encoder receives text description data that is adapted to the training state to encode and output text embedding vectors. The similarity between the embedding vectors output by each encoder and the text embedding vector is calculated using cosine similarity: = , = , = ,in, This represents the similarity value between the one-dimensional embedding vector and the text embedding vector. This represents a one-dimensional embedding vector output by a one-dimensional sequence encoder. This represents the similarity value between the image embedding vector and the text embedding vector. This represents the image embedding vector output by the image encoder; This represents the similarity value between the point cloud embedding vector and the text embedding vector. This represents the point cloud embedding vector output by the point cloud encoder; Represents a text embedding vector. Indicates the modulus length.
14. An unmanned sweeping vehicle, characterized in that, It includes sweepers, unmanned driving systems, and fault diagnosis systems as described in any one of claims 1 to 10.
Citation Information
Patent Citations
Fault diagnosis method and device for unmanned sweeper
CN121859182A
Unmanned aerial vehicle engine fault diagnosis method and system, electronic equipment and storage medium
CN121980279A
