Vehicle condition monitoring method, device and equipment, medium and vehicle

Abstract

The invention relates to a vehicle condition monitoring method, device and equipment, a medium and a vehicle. The vehicle condition monitoring method comprises the following steps: acquiring multi-modal data of a vehicle; preprocessing the multi-modal data to obtain the preprocessed multi-modal data; inputting the preprocessed multi-modal data into a preset vehicle condition monitoring model, and monitoring the vehicle condition of the vehicle by the preset vehicle condition monitoring model based on the preprocessed multi-modal data to obtain a vehicle condition monitoring result and a vehicle condition suggestion result, the preset vehicle condition monitoring model comprises a first monitoring sub-model, a second monitoring sub-model, a third monitoring sub-model and a multi-model fusion layer, so that vehicle condition monitoring can be carried out through multi-modal data, the richness of the data is improved, meanwhile, the vehicle condition monitoring is carried out through the vehicle condition monitoring model obtained through multi-model fusion, and the vehicle condition monitoring efficiency is improved. The ability of the model to deal with complex conditions is improved, and the accuracy and efficiency of vehicle condition monitoring are further improved.

Description

Technical Field

[0001] The present disclosure relates to the technical field of vehicles, and in particular, to a vehicle condition monitoring method, device, equipment, medium and vehicle. Background Art

[0002] With the development of intelligent networked vehicles, vehicle condition monitoring and optimization have become an important part of vehicle intelligence. In recent years, intelligent algorithms based on deep learning have begun to be widely used in vehicle monitoring systems for vehicle condition monitoring.

[0003] The existing vehicle condition monitoring method is as follows: By acquiring sensor data collected by vehicle sensors, inputting the sensor data into a traditional machine learning model, and performing fault diagnosis or warning by the traditional machine learning model, and then determining the vehicle condition monitoring result.

[0004] However, in the existing vehicle condition monitoring process, the data used is single, and the traditional machine learning model has limitations in dealing with complex situations, resulting in inaccurate and low-efficiency vehicle condition monitoring. Summary of the Invention

[0005] In order to solve the above technical problems, the present disclosure provides a vehicle condition monitoring method, device, equipment, medium and vehicle.

[0006] The first aspect of the embodiments of the present disclosure provides a vehicle condition monitoring method, including:

[0007] Acquiring multi-modal data of a vehicle;

[0008] Preprocessing the multi-modal data to obtain preprocessed multi-modal data;

[0009] Inputting the preprocessed multi-modal data into a preset vehicle condition monitoring model, and monitoring the vehicle condition based on the preprocessed multi-modal data by the preset vehicle condition monitoring model to obtain a vehicle condition monitoring result and a vehicle condition suggestion result, where the preset vehicle condition monitoring model includes a first monitoring sub-model, a second monitoring sub-model, a third monitoring sub-model and a multi-model fusion layer.

[0010] The second aspect of the embodiments of the present disclosure provides a vehicle condition monitoring device, including:

[0011] A data acquisition module, configured to acquire multi-modal data of a vehicle;

[0012] A data processing module, configured to preprocess the multi-modal data to obtain preprocessed multi-modal data;

[0013] A vehicle condition monitoring module, configured to input the preprocessed multimodal data into a preset vehicle condition monitoring model. The preset vehicle condition monitoring model monitors the vehicle condition based on the preprocessed multimodal data to obtain a vehicle condition monitoring result and a vehicle condition suggestion result. The preset vehicle condition monitoring model includes a first monitoring sub-model, a second monitoring sub-model, a third monitoring sub-model, and a multi-model fusion layer.

[0014] The third aspect of the embodiments of the present disclosure provides an electronic device, including:

[0015] A processor;

[0016] A memory for storing executable instructions;

[0017] Wherein, the processor is configured to read the executable instructions from the memory and execute the executable instructions to implement the vehicle condition monitoring method provided in the first aspect above.

[0018] The fourth aspect of the embodiments of the present disclosure provides a computer-readable storage medium. The storage medium stores a computer program, which, when executed by a processor, causes the processor to implement the vehicle condition monitoring method provided in the first aspect above.

[0019] The fifth aspect of the embodiments of the present disclosure provides a vehicle, which includes the electronic device provided in the third aspect.

[0020] The technical solutions provided by the embodiments of the present disclosure have the following advantages compared with the prior art:

[0021] The vehicle condition monitoring method, device, equipment, medium, and vehicle provided by the embodiments of the present disclosure can obtain the multimodal data of the vehicle. After obtaining the multimodal data, the multimodal data is preprocessed to obtain the preprocessed multimodal data; the preprocessed multimodal data is input into a preset vehicle condition monitoring model, and the preset vehicle condition monitoring model monitors the vehicle condition based on the preprocessed multimodal data to obtain a vehicle condition monitoring result and a vehicle condition suggestion result. The preset vehicle condition monitoring model includes a first monitoring sub-model, a second monitoring sub-model, a third monitoring sub-model, and a multi-model fusion layer. Thus, vehicle condition monitoring can be performed through multimodal data, improving the richness of the data. At the same time, vehicle condition monitoring is performed through the preset vehicle condition monitoring model, that is, the multi-model fusion and collaboration method, improving the ability of the vehicle condition monitoring model to handle complex situations, and further improving the accuracy and efficiency of vehicle condition monitoring. Description of the Drawings

[0022] The accompanying drawings here are incorporated into the specification and form a part of this specification, showing embodiments consistent with the present disclosure and used together with the specification to explain the principles of the present disclosure.

[0023] To more clearly illustrate the technical solutions in the embodiments of the present disclosure or the prior art, the following will briefly introduce the accompanying drawings required for the description of the embodiments or the prior art. Obviously, for those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0024] Figure 1 is a flowchart of a vehicle condition monitoring method provided by an embodiment of the present disclosure;

[0025] Figure 2 is a flowchart of a method for monitoring vehicle conditions based on a preset vehicle condition monitoring model provided by an embodiment of the present disclosure;

[0026] Figure 3 is a flowchart of another vehicle condition monitoring method provided by an embodiment of the present disclosure;

[0027] Figure 4 is a schematic structural diagram of a vehicle condition monitoring device provided by an embodiment of the present disclosure;

[0028] Figure 5 is a schematic structural diagram of an electronic device provided by an embodiment of the present disclosure. Detailed implementation manners

[0029] In order to more clearly understand the above-mentioned objects, features, and advantages of the present disclosure, the following will further describe the solutions of the present disclosure. It should be noted that, without conflict, the embodiments of the present disclosure and the features in the embodiments can be combined with each other.

[0030] In the following description, many specific details are set forth to fully understand the present disclosure, but the present disclosure can also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only a part of the embodiments of the present disclosure, rather than all the embodiments.

[0031] It should be understood that the various steps recorded in the method embodiments of the present disclosure can be executed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this regard.

[0032] It should be noted that in this text, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprising", "including" or any other variant thereof are intended to cover non-exclusive inclusion, such that a process, method, article or device comprising a series of elements not only includes those elements, but also includes other elements not expressly listed, or elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "comprising a..." does not exclude the presence of additional identical elements in the process, method, article or device comprising the element.

[0033] It should be noted that the modification of "one" and "multiple" mentioned in this disclosure is illustrative rather than restrictive. Those skilled in the art should understand that unless otherwise clearly specified in the context, it should be understood as "one or more".

[0034] Generally, the existing vehicle condition monitoring method is as follows: by obtaining the sensor data collected by vehicle sensors, inputting the sensor data into a traditional machine learning model, and performing fault diagnosis or warning by the traditional machine learning model, and then determining the vehicle condition monitoring result.

[0035] However, due to the single data used in the existing vehicle condition monitoring process and the limitations of traditional machine learning models in dealing with complex situations, the problems of inaccurate vehicle condition monitoring and low efficiency are caused. In view of this problem, the embodiments of the present disclosure provide a vehicle condition monitoring method, which will be introduced below in combination with specific embodiments.

[0036] Figure 1 is a flowchart of a vehicle condition monitoring method provided by an embodiment of the present disclosure. This method can be executed by a vehicle condition monitoring device, which can be implemented in a software and / or hardware manner and can be configured in an electronic device, such as a server or a terminal. Among them, the terminal specifically includes an in-vehicle terminal, a computer, a tablet computer, etc.

[0037] As Figure 1 shown, the vehicle condition monitoring method provided in this embodiment includes the following steps.

[0038] S110. Obtain multi-modal data of the vehicle.

[0039] In the embodiments of the present disclosure, multi-modal data can be understood as data of different types or different modalities that affect the vehicle condition.

[0040] Multimodal data can include vehicle sensor data, external environment data of the vehicle, vehicle historical maintenance data, user behavior data, and so on.

[0041] Among them, vehicle sensor data can be understood as data collected by various real-time sensors of the vehicle.

[0042] Exemplarily, vehicle sensor data can specifically include engine temperature, battery voltage data, oil pressure data, tire pressure data, vibration sensor data, etc. Among them, engine temperature can be used to monitor the health of the engine, and abnormal temperature may lead to overheating or damage; battery voltage data can be used to monitor the health status of the battery, and a decrease in battery voltage may indicate a battery fault; oil pressure data can be used to judge whether the oil pressure is normal, and low oil pressure may cause engine damage; tire pressure data can be used to ensure normal tire pressure, and too low tire pressure will affect driving safety; vibration sensor data can be used to capture possible abnormal vibrations during vehicle operation, and is usually used to identify faults in the engine or transmission system.

[0043] External environment data of the vehicle can include driving conditions, etc. It is used to provide the background for vehicle condition analysis to avoid misjudgment.

[0044] Exemplarily, external environment data of the vehicle can specifically include weather information, road condition information, driving speed, etc.

[0045] Vehicle historical maintenance data can include record data related to vehicle maintenance and repair within a preset time period. It is used to reflect the "health track" of the vehicle and potential failure risks.

[0046] Exemplarily, vehicle historical maintenance data can specifically include the vehicle's historical fault records, repair time, repair type, repair items, maintenance cycle, etc.

[0047] User behavior data can be understood as the adoption situation of the user for system suggestions, user input data, user driving habits, etc.

[0048] Exemplarily, user behavior data can specifically include the user's feedback information on the vehicle state, the user's information on suggestion adoption, etc.

[0049] Specifically, the electronic device can, when receiving a vehicle condition monitoring instruction, or in response to reaching a preset time interval, acquire vehicle sensor data collected by vehicle sensors, simultaneously acquire external environment data of the vehicle, and obtain vehicle historical maintenance data and user behavior data from a preset database based on the identification information of the vehicle, and then obtain the multimodal data of the vehicle.

[0050] S120. Preprocess the multimodal data to obtain the preprocessed multimodal data.

[0051] In the embodiments of the present disclosure, preprocessing can be understood as processing multimodal data such as noise reduction and format unification. Exemplarily, preprocessing can include operations such as denoising processing, normalization processing, and timing processing.

[0052] Specifically, after the electronic device obtains the multimodal data, it performs preprocessing operations such as denoising processing, normalization processing, and timing processing on the multimodal data to obtain the preprocessed multimodal data.

[0053] S130. Input the preprocessed multimodal data into a preset vehicle condition monitoring model. The preset vehicle condition monitoring model monitors the vehicle condition based on the preprocessed multimodal data to obtain a vehicle condition monitoring result and a vehicle condition suggestion result. The preset vehicle condition monitoring model includes a first monitoring sub-model, a second monitoring sub-model, a third monitoring sub-model, and a multi-model fusion layer.

[0054] In the embodiments of the present disclosure, the vehicle condition monitoring result can be understood as a result used to represent the vehicle condition state. Exemplarily, the vehicle condition monitoring result can include normal vehicle condition, abnormal vehicle condition, warning, etc.

[0055] The vehicle condition suggestion result can be understood as a suggestion made according to the vehicle condition state for guiding or optimizing the vehicle state. Exemplarily, the vehicle condition monitoring result can include suggestions to check tire pressure, suggestions for in-station maintenance, suggestions to continue driving, no abnormalities, etc.

[0056] The preset vehicle condition monitoring model can be understood as a model that has been pre-trained for monitoring vehicle conditions.

[0057] Specifically, after the electronic device obtains the preprocessed multimodal data, it inputs the preprocessed multimodal data into the preset vehicle condition monitoring model. The first monitoring sub-model, the second monitoring sub-model, the third monitoring sub-model, and the multi-model fusion layer in the preset vehicle condition monitoring model perform fusion and collaborative processing to obtain a vehicle condition monitoring result and a vehicle condition suggestion result.

[0058] In the embodiments of the present disclosure, it is possible to obtain the multimodal data of the vehicle. After obtaining the multimodal data, preprocess the multimodal data to obtain the preprocessed multimodal data; input the preprocessed multimodal data into a preset vehicle condition monitoring model. The preset vehicle condition monitoring model monitors the vehicle condition based on the preprocessed multimodal data to obtain a vehicle condition monitoring result and a vehicle condition suggestion result. The preset vehicle condition monitoring model includes a first monitoring sub-model, a second monitoring sub-model, a third monitoring sub-model, and a multi-model fusion layer. Thus, it is possible to monitor the vehicle condition through multimodal data, improving the richness of the data. At the same time, by using the preset vehicle condition monitoring model, that is, the multi-model fusion and collaborative method for vehicle condition monitoring, the ability of the vehicle condition monitoring model to handle complex situations is improved, thereby improving the accuracy and efficiency of vehicle condition monitoring.

[0059] Based on the above embodiments of the present disclosure, by means of combining multi-modal data and fusing multiple models synergistically for vehicle condition monitoring, it is possible to comprehensively analyze and optimize the vehicle condition in a short time, improving the comprehensiveness, real-time performance, and accuracy of vehicle condition detection.

[0060] In the embodiments of the present disclosure, preprocessing the multi-modal data to obtain preprocessed multi-modal data may specifically include: denoising the multi-modal data; normalizing the denoised multi-modal data to obtain normalized data; and performing temporal processing on the normalized data to obtain preprocessed multi-modal data.

[0061] In the embodiments of the present disclosure, denoising can be understood as removing the noise in the data to ensure the accuracy of the data. Exemplarily, denoising may include missing value filling, outlier correction, duplicate data deletion, and so on.

[0062] Normalization can be understood as standardizing data with different dimensions to the same range so that the data is suitable for the model. Exemplarily, the Z-score can be used to standardize the data into zero mean and unit variance, or the min-max normalization method can be adopted to process the data into the interval (0, 1).

[0063] Temporal processing can be understood as aligning data with different time dimensions, aligning the data under the same time dimension to ensure the coherence of the data in the time dimension. Exemplarily, the sliding window technique can be adopted to slide a fixed-size window over the temporal data and aggregate or extract features from the data within the window to perform temporal processing on the data.

[0064] Specifically, after the electronic device obtains the multi-modal data, it denoises the multi-modal data, normalizes the denoised multi-modal data to obtain normalized data, and then performs temporal processing on the normalized data to obtain preprocessed multi-modal data. The preprocessed multi-modal data may include preprocessed vehicle sensor data, preprocessed vehicle exterior environment data, preprocessed vehicle historical maintenance data, preprocessed user behavior data, and so on.

[0065] In the embodiments of the present disclosure, after obtaining the multi-modal data, it is possible to preprocess the multi-modal data, remove the noise in the data, reduce the impact of outliers on the model, and at the same time unify data with different dimensions to the same dimension through normalization and temporal processing, eliminate the dimension difference, perform temporal processing on the data, facilitate the extraction of time-dependent features, improve the accuracy of the data, and lay a foundation for enhancing the accuracy and stability of subsequent model processing.

[0066] Figure 2 This is a flowchart of a method for vehicle condition monitoring based on a preset vehicle condition monitoring model provided by an embodiment of the present disclosure. As Figure 2 shown, the preprocessed multi-modal data is input into the preset vehicle condition monitoring model, and the preset vehicle condition monitoring model monitors the vehicle condition based on the preprocessed multi-modal data to obtain a vehicle condition monitoring result and a vehicle condition suggestion result. Specifically, the following steps may be included:

[0067] S210. Input the preprocessed vehicle sensor data into the first monitoring sub-model, and the first monitoring sub-model extracts features from the preprocessed vehicle sensor data to obtain target features.

[0068] In an embodiment of the present disclosure, the first monitoring sub-model may be a deepseek-R1 model.

[0069] The first monitoring sub-model may include a multi-head attention layer (Multi-Head Attention), a dynamic gating mechanism layer (Dynamic Gating Mechanism), a feed-forward neural network layer (Feed-Forward Network, FFN), and a layer normalization and residual connection layer.

[0070] Among them, the multi-head attention layer can be used to capture long-range dependencies and complex patterns in the input data. The multi-head attention layer includes multiple attention heads, and each attention head is used to capture relationships in different dimensions, improving the model's expressive ability through parallel computing.

[0071] The dynamic gating mechanism layer can be used to dynamically adjust the allocation of the model's computing resources according to the complexity of the input data and the task requirements. When processing a task, it can determine which sub-networks should be activated and dynamically activate some sub-networks, that is, dynamically adjust the computing depth and width of the model, thereby improving performance while reducing the computing cost.

[0072] The feed-forward neural network layer can be used to perform non-linear transformation processing on the output of the multi-head attention layer to further improve the model's expressive ability. The feed-forward neural network layer includes two linear layers, which are respectively responsible for feature extraction and feature reconstruction.

[0073] The layer normalization and residual connection layer can adopt layer normalization technology to accelerate model convergence and improve training stability. At the same time, normalization reduces the model's dependence on a specific input distribution and enhances the model's robustness; the input is directly passed to the output through the residual connection to alleviate the gradient disappearance problem and enhance the model depth.

[0074] In an embodiment of the present disclosure, the preprocessed vehicle sensor data is input into a first monitoring sub-model, and the first monitoring sub-model extracts features from the preprocessed vehicle sensor data to obtain target features. Specifically, it may include: inputting the preprocessed vehicle sensor data into a multi-head attention layer, and the multi-head attention layer extracts features from the preprocessed vehicle sensor data to obtain a first feature; inputting the first feature into a dynamic gating mechanism layer, and the dynamic gating mechanism layer determines a target expert sub-network in a feed-forward neural network layer based on the first feature; inputting the first feature into the feed-forward neural network layer, and the feed-forward neural network layer activates the target expert sub-network, and performs non-linear transformation and feature reconstruction processing on the first feature based on the target expert sub-network to obtain a second feature; inputting the second feature into a layer normalization and residual connection layer, and the layer normalization and residual connection layer outputs the target feature based on the second feature.

[0075] In an embodiment of the present disclosure, the first monitoring sub-model can support multi-modal data, including structured data such as sensor signal data and unstructured data such as text description data. For structured data, the Min-Max normalization algorithm is used for data normalization processing to map its value to the interval [0, 1]. For unstructured data, word embedding techniques (such as Word2Vec or BERT) are used to convert it into a vector form.

[0076] In some embodiments of the present disclosure, the vehicle sensor data may include preprocessed vehicle sensor signal data and corresponding text description information. Specifically, after the electronic device obtains the vehicle sensor data, it uploads the vehicle sensor data to the cloud, and the cloud performs mapping processing on the vehicle sensor data to obtain the text description information corresponding to the vehicle sensor data, receives the text description information sent by the cloud, and inputs the text description information and the preprocessed vehicle sensor signal data into the multi-head attention layer at the same time. The multi-head attention layer extracts features based on the preprocessed vehicle sensor signal data and the text description information to obtain a first feature. Thus, by inputting different modal information corresponding to the vehicle sensor data into the first monitoring sub-model for feature extraction, the accuracy of the obtained first feature is improved.

[0077] Specifically, after the electronic device obtains the preprocessed vehicle sensor data, it inputs the preprocessed vehicle sensor data into the multi-head attention layer. The multi-head attention layer extracts features from the preprocessed vehicle sensor data to obtain the first feature. The first feature is input into the dynamic gating mechanism layer. The dynamic gating mechanism layer determines the complexity of the input data and the task requirements based on the first feature, and determines the target expert sub-network to be activated in the feed-forward neural network layer based on the complexity of the input data and the task requirements. The first feature is input into the feed-forward neural network layer. The feed-forward neural network layer activates the target expert sub-network, and performs non-linear transformation and feature reconstruction processing on the first feature based on the target expert sub-network to obtain the second feature. Then, the second feature is input into the layer normalization and residual connection layer. The layer normalization and residual connection layer generates and outputs the target feature based on the second feature and the first feature.

[0078] In the embodiment of the present disclosure, the complexity of the input data and the service scenario can be determined through the dynamic gating mechanism layer, and then the expert sub-network can be dynamically adjusted, which improves the data processing efficiency while ensuring the accuracy of data processing.

[0079] S220. Input the target feature into the second monitoring sub-model. The second monitoring sub-model performs vehicle condition prediction based on the target feature and outputs the target prediction result.

[0080] In the embodiment of the present disclosure, the second monitoring sub-model can be a Support Vector Machine (SVM).

[0081] In the embodiment of the present disclosure, after the electronic device obtains the target feature output by the first monitoring sub-model and before inputting the target feature into the second monitoring sub-model, it normalizes the target feature to obtain the processed target feature. Thus, the processed target feature can meet the processing requirements of the second monitoring sub-model, and the robustness of the second monitoring sub-model is improved.

[0082] Further, inputting the target feature into the second monitoring sub-model, and the second monitoring sub-model performing vehicle condition prediction based on the target feature and outputting the target prediction result may specifically include: inputting the processed target feature into the second monitoring sub-model, and the second monitoring sub-model performing feature selection on the processed target feature based on a preset rule to obtain a third feature, and performing vehicle condition prediction based on the third feature and outputting the target prediction result.

[0083] In the embodiment of the present disclosure, the preset rule can be understood as a rule that is preset according to the knowledge in the field and statistical analysis data and is used for vehicle condition state feature selection.

[0084] The third feature is selected by the second monitoring sub-model based on a preset rule, and is the feature that can best reflect the vehicle state under the current working condition or application scenario, that is, the feature whose contribution degree to the vehicle state judgment is greater than the preset threshold.

[0085] The target prediction result can be understood as the prediction result of the vehicle condition.

[0086] In the embodiment of the present disclosure, before inputting the target feature into the second monitoring sub-model, the target feature can be normalized so that the processed target feature can meet the processing requirements of the second monitoring sub-model, improving the robustness of the second monitoring sub-model. Then, the normalized target feature is input into the second monitoring sub-model, and the second monitoring sub-model performs feature selection on the processed target feature based on a preset rule to obtain the third feature, so that the obtained third feature is the feature that can best reflect the vehicle state under the current working condition or application scenario, and further performs vehicle condition prediction based on the third feature, improving the accuracy of the obtained target prediction result.

[0087] S230. Input the preprocessed vehicle sensor data, preprocessed vehicle external environment data, preprocessed vehicle historical maintenance data, and preprocessed user behavior data into the third monitoring sub-model, and the third monitoring sub-model outputs vehicle action suggestions.

[0088] In the embodiment of the present disclosure, the third monitoring sub-model can be a reinforcement learning model based on the Actor-Critic framework.

[0089] The third monitoring sub-model can include an actor network and a critic network.

[0090] In the embodiment of the present disclosure, before inputting the preprocessed vehicle sensor data, preprocessed vehicle external environment data, preprocessed vehicle historical maintenance data, and preprocessed user behavior data into the third monitoring sub-model, and the third monitoring sub-model outputs vehicle action suggestions, the vehicle condition monitoring method can further include: extracting features from the preprocessed vehicle sensor data, preprocessed vehicle external environment data, preprocessed vehicle historical maintenance data, and preprocessed user behavior data to obtain sensor data features corresponding to the preprocessed vehicle sensor data, environment data features corresponding to the preprocessed vehicle external environment data, historical maintenance data features corresponding to the preprocessed vehicle historical maintenance data, and user behavior features corresponding to the preprocessed user behavior data.

[0091] Among them, the specific implementation manner of feature extraction for the preprocessed multi-modal data is similar to the existing data feature extraction method, and will not be elaborated here.

[0092] Further, the preprocessed vehicle sensor data, preprocessed external environment data, preprocessed vehicle historical maintenance data, and preprocessed user behavior data are input into the third monitoring sub-model, and the third monitoring sub-model outputs vehicle action suggestions. Specifically, the sensor data features, environment data features, historical maintenance data features, and user behavior features are input into the third monitoring sub-model. The third monitoring sub-model performs vectorization processing on the sensor data features, environment data features, historical maintenance data features, and user behavior features to obtain target feature vectors corresponding to the sensor data features, environment data features, historical maintenance data features, and user behavior features respectively. Each target feature vector is input into the third monitoring sub-model, and the third monitoring sub-model performs vehicle condition analysis based on the target feature vectors and outputs vehicle action suggestions.

[0093] Specifically, after obtaining the sensor data features, environment data features, historical maintenance data features, and user behavior features, the electronic device inputs the sensor data features, environment data features, historical maintenance data features, and user behavior features into the third monitoring sub-model. The third monitoring sub-model respectively performs vectorization processing on the sensor data features, environment data features, historical maintenance data features, and user behavior features to obtain Mu Bai feature vectors corresponding to each feature. Each target feature vector is input into the actor network in the third monitoring sub-model, and the first network outputs the action probability distribution corresponding to each target feature vector. Each target feature vector and the corresponding action are combined into a state-action pair, and the state-action pair is input into the critic network of the third monitoring sub-model, and the critic network outputs vehicle action suggestions.

[0094] In the embodiments of the present disclosure, when performing vehicle condition analysis, the third monitoring sub-model combines multi-modal and multi-dimensional data. It not only considers the impact of the vehicle's real-time sensor data and external environment data on the vehicle, but also combines the vehicle's historical maintenance data and user behavior data, providing a basis for determining vehicle action suggestions. Thereby, the accuracy of the obtained vehicle action suggestions is improved.

[0095] S240. Perform feature vector fusion processing on the feature vectors corresponding to the preprocessed vehicle sensor data, preprocessed external environment data, preprocessed vehicle historical maintenance data, and preprocessed user behavior data respectively to obtain a fused feature vector.

[0096] Specifically, the electronic device extracts features from the pre - processed vehicle sensor data, pre - processed external vehicle environment data, pre - processed vehicle historical maintenance data, and pre - processed user behavior data respectively to obtain multiple features. The extracted multiple features are encoded and normalized to obtain processed features. The processed features are vectorized to obtain feature vectors. Furthermore, the feature vectors are subjected to feature fusion processing to obtain fused feature vectors.

[0097] S250. Input the target prediction result, vehicle action suggestion, and fused feature vector into the multi - model fusion layer. The multi - model fusion layer performs vehicle condition analysis based on the target prediction result, vehicle action suggestion, and fused feature vector, and outputs a vehicle condition monitoring result and a vehicle condition suggestion result.

[0098] In the embodiment of the present disclosure, the network structure of the multi - model fusion layer generally adopts a lightweight multi - layer perceptron (MLP). The input includes the target prediction result output by the second monitoring sub - model, the action suggestion output by the reinforcement learning model, and the complete state vector, that is, the fused feature vector. The output is the final vehicle condition and vehicle condition suggestion. Its function is to unify the judgment results of multiple models, map different input data to a unified feature space, improve the system robustness, and convert complex vehicle condition recognition results into clear and executable intelligent suggestions to achieve a closed - loop of intelligent auxiliary decision - making. When needed, it can also support the personalized generation or matching of abnormal solution plans.

[0099] In the embodiment of the present disclosure, input the target prediction result, vehicle action suggestion, and fused feature vector into the multi - model fusion layer. The multi - model fusion layer performs vehicle condition analysis based on the target prediction result, vehicle action suggestion, and fused feature vector, and outputs a vehicle condition monitoring result and a vehicle condition suggestion result. Specifically, it may include: determining a first weight corresponding to the vehicle prediction result, a second weight corresponding to the vehicle action suggestion, and a third weight corresponding to the fused feature vector. The multi - model fusion layer outputs a vehicle condition monitoring result and a vehicle condition suggestion result based on the first weight, second weight, and third weight.

[0100] Specifically, the electronic device can determine the working condition and actual application scenario of the vehicle according to the vehicle sensor data and external vehicle environment data. According to the working condition and actual application scenario of the vehicle, determine a first weight corresponding to the vehicle prediction result, a second weight corresponding to the vehicle action suggestion, and a third weight corresponding to the fused feature vector. The multi - model fusion layer performs weighted fusion processing on the vehicle prediction result, vehicle action suggestion, and fused feature vector based on the first weight, second weight, and third weight, and finally outputs a vehicle condition monitoring result and a vehicle condition suggestion result.

[0101] For example, when the vehicle's operating condition is that known typical fault characteristics (such as low oil pressure, significant battery voltage fluctuations, etc.) are determined based on vehicle sensor data, the second monitoring sub-model is prioritized for rapid classification and judgment, and its weight will be increased to a dominant position, then the target prediction result output by the second monitoring sub-model will have a higher weight.

[0102] In complex environments with strong external interference such as extreme weather or severe sensor fluctuations, the system will tend to use the reinforcement learning model to make dominant judgments because of its strong adaptability to dynamic conditions. At this time, the vehicle action recommendations output by the third monitoring sub-model have a high weight.

[0103] In some typical complex working conditions (such as slope driving + engine temperature rise + road condition changes), the weights of the three input data in the multi-model fusion layer will tend to be balanced, and the overall stability and accuracy will be improved through fusion judgment.

[0104] As a result, the weight of the input data can be dynamically adjusted in combination with different working conditions and actual application scenarios, thereby improving the accuracy and flexibility of the vehicle condition monitoring results and vehicle condition recommendation results, thereby improving driving safety.

[0105] In the embodiments of the present disclosure, vehicle condition monitoring can be performed through a preset vehicle condition monitoring model, i.e., a multi-model fusion and collaborative approach, thereby improving the ability of the vehicle condition monitoring model to cope with complex situations, thereby improving the robustness of the preset vehicle condition monitoring model, as well as the accuracy and efficiency of vehicle condition monitoring.

[0106] In an embodiment of the present disclosure, before the preprocessed multimodal data is input into a preset vehicle condition monitoring model, and the preset vehicle condition monitoring model monitors the vehicle condition based on the preprocessed multimodal data to obtain a vehicle condition monitoring result and a vehicle condition recommendation result, the vehicle condition monitoring method may also include: obtaining a preset vehicle condition monitoring model.

[0107] Acquiring a preset vehicle condition monitoring model includes: acquiring a first monitoring sub-model, a second monitoring sub-model, and a third monitoring sub-model.

[0108] In the embodiment of the present disclosure, obtaining the first monitoring sub-model may specifically include: obtaining a first training data set; pre-training the first monitoring sub-model to be trained based on the first training data set to obtain a pre-trained first monitoring sub-model; performing self-supervised learning on the pre-trained first monitoring sub-model to obtain a first monitoring sub-model after self-supervised learning; performing parameter optimization on the first monitoring sub-model after self-supervised learning to obtain a trained first monitoring sub-model, i.e., the first monitoring sub-model in the embodiment of the present disclosure.

[0109] Among them, the first training data set is sensor data corresponding to different vehicle conditions of vehicles of different models in different usage scenarios.

[0110] Specifically, after the electronic device obtains the first training data set, it pre-trains the first monitoring sub-model to be trained based on the first training data set by using a multi-task learning strategy to obtain the pre-trained first monitoring sub-model; performs masking processing on each training data in the first training data set based on the masking modeling method and a preset masking ratio to obtain the masked training data; inputs the masked training data into the pre-trained first monitoring sub-model, and the pre-trained first monitoring sub-model predicts the masked part, and performs self-supervised learning on the pre-trained first monitoring sub-model according to the prediction result to obtain the first monitoring sub-model after self-supervised learning; after obtaining the first monitoring sub-model after self-supervised learning, optimizes the model parameters of the first monitoring sub-model after self-supervised learning based on the grid search method to obtain the first monitoring sub-model with training completed.

[0111] Exemplarily, when pre-training the first monitoring sub-model to be trained by using a multi-task learning strategy, the multiple related tasks may include Task 1: predicting the vehicle condition status (such as normal, abnormal, etc.); Task 2: predicting the change trend of the target sensor signal, etc. Specifically, the multiple related tasks can be set and adjusted according to actual needs, and are not limited herein.

[0112] In the embodiments of the present disclosure, the specific implementation manner of pre-training the first monitoring sub-model to be trained by using a multi-task learning strategy is similar to the existing implementation manner of model training by using a multi-task learning strategy, and will not be elaborated herein.

[0113] The specific implementation manner of performing self-supervised learning on the pre-trained first monitoring sub-model is similar to the existing implementation manner of performing self-supervised learning on a model, and will not be elaborated herein.

[0114] The specific implementation manner of optimizing the model parameters of the first monitoring sub-model after self-supervised learning based on the grid search method is similar to the existing implementation manner of adjusting model parameters based on the grid search method, and will not be elaborated herein.

[0115] Based on the above embodiments, the number of model parameters of the first monitoring sub-model with training completed can be reduced by pruning technology, improving the computing efficiency of the model.

[0116] In the embodiments of the present disclosure, the first monitoring sub-model can be pre-trained by using a multi-task learning strategy, enabling the first monitoring sub-model to share knowledge on multiple related tasks, enhancing its generalization ability, and outputting more representative features. The robustness of the model is improved through self-supervised learning. At the same time, the computing efficiency and inference accuracy of the model are improved through parameter optimization.

[0117] In the embodiments of the present disclosure, obtaining the second monitoring sub-model, that is, obtaining the trained second monitoring sub-model, may specifically include: obtaining the second training dataset; initializing the kernel function parameters and the penalty and reward coefficients in the second monitoring sub-model to be trained; inputting the second training dataset into the second monitoring sub-model to be trained respectively, and having the second monitoring sub-model to be trained perform vehicle condition prediction to output vehicle prediction results; adjusting the kernel function parameters and the penalty and reward coefficients based on the vehicle prediction results until convergence, so as to obtain the trained second monitoring sub-model, that is, the second monitoring sub-model in the embodiments of the present disclosure.

[0118] Among them, the second training dataset is the data features corresponding to the collected vehicle sensor data, and the annotation result is the vehicle state, that is, the vehicle condition prediction result (such as normal, abnormal, etc.).

[0119] Further, after obtaining the trained second monitoring sub-model, the parameters of the trained second monitoring sub-model can be fine-tuned based on the feedback data of the user. The specific implementation manner of specifically performing parameter fine-tuning is similar to the existing parameter fine-tuning manner for support vector machines, and will not be elaborated here.

[0120] In the embodiments of the present disclosure, the kernel function can be an RBF kernel function.

[0121] In the embodiments of the present disclosure, the parameters of the trained second monitoring sub-model can be fine-tuned based on the feedback data of the user, which improves the robustness of the second monitoring sub-model and the accuracy of vehicle condition monitoring.

[0122] In the embodiments of the present disclosure, obtaining the third monitoring sub-model, that is, obtaining the trained third monitoring sub-model, may specifically include: obtaining the third training dataset; training the third monitoring sub-model to be trained based on the third training dataset to obtain the trained third monitoring sub-model.

[0123] Among them, the third training dataset includes sensor data, user behavior data, vehicle historical maintenance data, and external vehicle environment data corresponding to different vehicle conditions of vehicles of different models in different usage scenarios.

[0124] The specific implementation manner of training the third monitoring sub-model to be trained based on the third training dataset is similar to the existing implementation manner of training a reinforcement learning model based on the Actor-Critic framework, and will not be elaborated here.

[0125] Further, after obtaining the trained third monitoring sub-model, the model parameters of the third monitoring sub-model are optimized based on the data of the user's adoption of vehicle condition suggestions.

[0126] Exemplarily, positive reward: the suggestion is adopted and the subsequent vehicle condition is improved (such as the oil pressure returns to normal), reward value: +1.

[0127] Negative reward: The suggestion is not adopted and the vehicle condition deteriorates (such as a malfunction caused by abnormal oil pressure), reward value: -1.

[0128] Additional reward: Considering the long-term effect (such as reducing maintenance costs), a delayed reward is designed. The delayed reward can be: if the suggestion avoids a serious malfunction, a high reward (such as +2) is given. Optimize the model parameters of the third monitoring sub-model according to the reward value.

[0129] In the embodiment of the present disclosure, the model parameters of the third monitoring sub-model can be optimized according to the suggestion adoption situation and user feedback, improving the accuracy and robustness of the third monitoring sub-model.

[0130] In the embodiment of the present disclosure, the electronic device can also regularly evaluate the performance of the preset vehicle condition monitoring model, obtain the actual operation data of the vehicle and the task requirements of the user, and continuously optimize the parameters of the preset vehicle condition monitoring model according to the actual operation data of the vehicle, the task requirements of the user, and the evaluation result. Thereby ensuring the long-term stability and performance of the preset vehicle condition monitoring model.

[0131] Figure 3 is a flowchart of another vehicle condition monitoring method provided by the embodiment of the present disclosure, as Figure 3 shown, the vehicle condition monitoring method may include the following steps:

[0132] S310. Obtain multi-modal data of the vehicle.

[0133] S320. Preprocess the multi-modal data to obtain preprocessed multi-modal data.

[0134] S330. Input the preprocessed multi-modal data into a preset vehicle condition monitoring model, and the preset vehicle condition monitoring model monitors the vehicle condition based on the preprocessed multi-modal data to obtain a vehicle condition monitoring result and a vehicle condition suggestion result. The preset vehicle condition monitoring model includes a first monitoring sub-model, a second monitoring sub-model, a third monitoring sub-model, and a multi-model fusion layer.

[0135] It should be noted that the specific implementation manners of steps S310 to S330 are similar to those of steps S110 to S130 in the above embodiment of the present disclosure, and will not be elaborated here.

[0136] S340. In response to the target operation of the user, display the vehicle condition monitoring result and the vehicle condition suggestion result, and / or display a three-dimensional vehicle model abnormal dynamic effect.

[0137] In the embodiment of the present disclosure, the target operation can be a click operation for entering the vehicle condition display interface.

[0138] In some embodiments of the present disclosure, in response to a target operation of a user, an electronic device displays the vehicle condition monitoring result and the vehicle condition suggestion result of a vehicle on a vehicle condition display interface. Meanwhile, when the vehicle condition monitoring result is an abnormal vehicle condition, based on the abnormal item information, a three-dimensional vehicle model abnormal animation is constructed and displayed.

[0139] Further, after displaying the vehicle condition monitoring result and the vehicle condition suggestion result, and / or displaying the three-dimensional vehicle model abnormal animation, in response to a sliding operation of the user on the vehicle condition display interface, page switching or abnormal item information switching is performed. Thus, based on the page switching or abnormal item information switching, it is convenient to view the previous abnormal item, the next abnormal item or all abnormal items, as well as the vehicle condition suggestions and vehicle condition guidance solutions corresponding to each abnormal item.

[0140] S350. Control an audio device to perform voice broadcast on the vehicle condition monitoring result and the vehicle condition suggestion result.

[0141] Specifically, after the electronic device displays the vehicle condition monitoring result and the vehicle condition suggestion result, and / or displays the three-dimensional vehicle model abnormal animation, it controls the audio device to perform voice broadcast on the vehicle condition monitoring result and the vehicle condition suggestion result, further facilitating the user to comprehensively understand the vehicle condition during driving and improving driving safety.

[0142] S360. When receiving a termination instruction from the user, stop the voice broadcast and / or control the display interface to exit.

[0143] In the embodiments of the present disclosure, the termination instruction may be a voice broadcast termination instruction or an interface display termination instruction.

[0144] Specifically, when the termination instruction is a voice broadcast termination instruction, the electronic device controls the audio device to stop the voice broadcast; when the termination instruction is an interface display termination instruction, the electronic device controls the display interface to exit.

[0145] In the embodiments of the present disclosure, after obtaining the vehicle condition monitoring result and the vehicle condition suggestion result, the vehicle condition monitoring result and the vehicle condition suggestion result can be displayed on the interface and / or voice broadcast, improving the user experience and providing a basis for the user to take timely measures.

[0146] In the embodiments of the present disclosure, when the electronic device obtains the vehicle condition monitoring result and the vehicle condition suggestion result and determines that the vehicle condition detection result is abnormal, it actively triggers a reminder event to remind the user.

[0147] In the embodiments of the present disclosure, when responding to the first boot of the vehicle machine, obtain an offline vehicle abnormal solution from the cloud; and cache the offline vehicle abnormal solution so as to perform vehicle condition guidance based on the offline vehicle abnormal solution when there is no network.

[0148] After the electronic device obtains the offline vehicle anomaly solution, it determines whether the offline vehicle anomaly solution conforms to the preset copywriting assembly rule; when it is determined that it does not conform, based on the preset copywriting assembly rule, the offline vehicle anomaly solution is assembled and cached.

[0149] The preset copywriting assembly rule can be understood as a rule that establishes a relationship between the vehicle condition advice and the vehicle condition status of the offline vehicle anomaly solution. For example, when a judgment condition is recognized, "if" is added before the judgment condition, thus facilitating the subsequent determination of the vehicle condition advice.

[0150] In the embodiment of the present disclosure, the offline vehicle anomaly solution can be obtained to facilitate vehicle condition guidance in the absence of a network, improving the timeliness of vehicle condition guidance.

[0151] Figure 4 It is a schematic structural diagram of a vehicle condition monitoring device provided by an embodiment of the present disclosure.

[0152] In the embodiment of the present disclosure, the vehicle condition monitoring device can be set in the electronic device and is understood as some functional modules in the above-mentioned electronic device. Specifically, the electronic device can be a server or a terminal. Among them, the terminal specifically includes an in-vehicle terminal, a computer, a tablet computer, etc., which are not limited here.

[0153] Such as Figure 4 As shown, the vehicle condition monitoring device 400 can include a data acquisition module 410, a data processing module 420, and a vehicle condition monitoring module 430.

[0154] The data acquisition module 410 can be used to acquire multi-modal data of the vehicle.

[0155] The data processing module 420 can be used to preprocess the multi-modal data to obtain the preprocessed multi-modal data.

[0156] The vehicle condition monitoring module 430 can be used to input the preprocessed multi-modal data into a preset vehicle condition monitoring model. The preset vehicle condition monitoring model monitors the vehicle condition based on the preprocessed multi-modal data to obtain a vehicle condition monitoring result and a vehicle condition advice result. The preset vehicle condition monitoring model includes a first monitoring sub-model, a second monitoring sub-model, a third monitoring sub-model, and a multi-model fusion layer.

[0157] In the embodiments of the present disclosure, multimodal data of a vehicle can be obtained. After the multimodal data is obtained, preprocessing is performed on the multimodal data to obtain preprocessed multimodal data; the preprocessed multimodal data is input into a preset vehicle condition monitoring model, and the preset vehicle condition monitoring model monitors the vehicle condition based on the preprocessed multimodal data to obtain a vehicle condition monitoring result and a vehicle condition suggestion result. The preset vehicle condition monitoring model includes a first monitoring sub-model, a second monitoring sub-model, a third monitoring sub-model, and a multi-model fusion layer. Thus, vehicle condition monitoring can be performed through multimodal data, improving the richness of the data. At the same time, vehicle condition monitoring is performed in the form of multi-model fusion and collaboration by the preset vehicle condition monitoring model, improving the ability of the vehicle condition monitoring model to handle complex situations, and further improving the accuracy and efficiency of vehicle condition monitoring.

[0158] In some embodiments of the present disclosure, the data processing module 420 can be specifically configured to perform denoising processing on the multimodal data;

[0159] Perform normalization processing on the denoised multimodal data to obtain normalized processing data;

[0160] Perform temporal processing on the normalized processing data to obtain preprocessed multimodal data.

[0161] In some embodiments of the present disclosure, the multimodal data includes vehicle sensor data, external vehicle environment data, vehicle historical maintenance data, and user behavior data.

[0162] The vehicle condition monitoring module 430 can be specifically configured to input the preprocessed vehicle sensor data into the first monitoring sub-model, and the first monitoring sub-model extracts features from the preprocessed vehicle sensor data to obtain target features;

[0163] Input the target features into the second monitoring sub-model, and the second monitoring sub-model performs vehicle condition prediction based on the target features and outputs a target prediction result;

[0164] Input the preprocessed vehicle sensor data, the preprocessed external vehicle environment data, the preprocessed vehicle historical maintenance data, and the preprocessed user behavior data into the third monitoring sub-model, and the third monitoring sub-model outputs vehicle action suggestions;

[0165] Perform feature vector fusion processing on the feature vectors corresponding to the preprocessed vehicle sensor data, the preprocessed external vehicle environment data, the preprocessed vehicle historical maintenance data, and the preprocessed user behavior data respectively to obtain a fused feature vector;

[0166] Input the target prediction result, vehicle action suggestion, and fused feature vector into the multi-model fusion layer. The multi-model fusion layer performs vehicle condition analysis based on the target prediction result, vehicle action suggestion, and fused feature vector, and outputs the vehicle condition monitoring result and vehicle condition suggestion result.

[0167] In some embodiments of the present disclosure, the first monitoring sub-model includes a multi-head attention layer, a dynamic gating mechanism layer, a feed-forward neural network layer, and a layer normalization and residual connection layer.

[0168] The vehicle condition monitoring module 430 may include a first processing unit, a second processing unit, and a third processing unit.

[0169] The first processing unit may be configured to input the preprocessed vehicle sensor data into the multi-head attention layer. The multi-head attention layer extracts features from the preprocessed vehicle sensor data to obtain the first feature.

[0170] Input the first feature into the dynamic gating mechanism layer. The dynamic gating mechanism layer determines the target expert sub-network in the feed-forward neural network layer based on the first feature.

[0171] Input the first feature into the feed-forward neural network layer. The feed-forward neural network layer activates the target expert sub-network and performs non-linear transformation and feature reconstruction processing on the first feature based on the target expert sub-network to obtain the second feature.

[0172] Input the second feature into the layer normalization and residual connection layer. The layer normalization and residual connection layer outputs the target feature based on the second feature.

[0173] The second processing unit may be configured to perform normalization processing on the target feature to obtain the processed target feature before inputting the target feature into the second monitoring sub-model. The second monitoring sub-model performs vehicle condition prediction based on the target feature and outputs the target prediction result.

[0174] The third processing unit may be configured to input the processed target feature into the second monitoring sub-model. The second monitoring sub-model performs feature selection on the processed target feature based on a preset rule to obtain the third feature, and performs vehicle condition prediction based on the third feature to output the target prediction result.

[0175] In some embodiments of the present disclosure, the vehicle condition monitoring module 430 may further include a fourth processing unit.

[0176] The fourth processing unit can be used to extract features from the pre - processed vehicle sensor data, pre - processed external vehicle environment data, pre - processed vehicle historical maintenance data, and pre - processed user behavior data before inputting them into the third monitoring sub - model, and the third monitoring sub - model outputs vehicle action suggestions, so as to obtain sensor data features corresponding to the pre - processed vehicle sensor data, environmental data features corresponding to the pre - processed external vehicle environment data, historical maintenance data features corresponding to the pre - processed vehicle historical maintenance data, and user behavior features corresponding to the pre - processed user behavior data.

[0177] The vehicle condition monitoring module 430 can also specifically be used to input the sensor data features, environmental data features, historical maintenance data features, and user behavior features into the third monitoring sub - model, and the third monitoring sub - model performs vectorization processing on the sensor data features, environmental data features, historical maintenance data features, and user behavior features to obtain target feature vectors corresponding to the sensor data features, environmental data features, historical maintenance data features, and user behavior features respectively;

[0178] Each target feature vector is input into the third monitoring sub - model, and the third monitoring sub - model performs vehicle condition analysis based on the target feature vector and outputs vehicle action suggestions.

[0179] In some embodiments of the present disclosure, the vehicle condition monitoring module 430 may further include a fifth processing unit.

[0180] The fifth processing unit can be used to determine the first weight corresponding to the target prediction result, the second weight corresponding to the vehicle action suggestion, and the third weight corresponding to the fusion feature vector, and the multi - model fusion layer outputs the vehicle condition monitoring result and the vehicle condition suggestion result based on the first weight, the second weight, and the third weight.

[0181] It should be noted that Figure 4 The vehicle condition monitoring device 400 shown can execute each step in the above - mentioned method embodiments and achieve each process and effect in the above - mentioned method embodiments, which will not be elaborated here.

[0182] Figure 5 is a schematic structural diagram of an electronic device provided by an embodiment of the present disclosure.

[0183] In the embodiments of the present disclosure, Figure 5 The electronic device shown can be a server or a terminal. Among them, the terminal specifically includes an in - vehicle terminal, a computer, a tablet computer, etc., which are not limited here.

[0184] Such as Figure 5As shown, the electronic device may include a processor 510 and a memory 520 storing computer program instructions.

[0185] Specifically, the processor 510 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present disclosure.

[0186] The memory 520 may include a mass storage for information or instructions. By way of example and not limitation, the memory 520 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive, or a combination of two or more of these. In a suitable case, the memory 520 may include a removable or non-removable (or fixed) medium. In a suitable case, the memory 520 may be internal or external to the integrated gateway device. In a particular embodiment, the memory 520 is a non-volatile solid state memory. In a particular embodiment, the memory 520 includes a read-only memory (ROM). In a suitable case, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically alterable ROM (EAROM), or a flash memory, or a combination of two or more of these.

[0187] The processor 510 reads and executes the computer program instructions stored in the memory 520 to perform the steps of the vehicle condition monitoring method provided by the embodiments of the present disclosure.

[0188] In one example, the electronic device may further include a transceiver 530 and a bus 540. Among them, as Figure 5 shown, the processor 510, the memory 520, and the transceiver 530 are connected through the bus 540 and complete communication with each other.

[0189] The bus 540 includes hardware, software, or both. By way of example and not limitation, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side BUS (FSB), a Hyper Transport (HT) interconnect, an Industrial Standard Architecture (ISA) bus, an InfiniBand interconnect, a Low Pin Count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus, or other suitable buses or a combination of two or more of these. Where appropriate, the bus 540 may include one or more buses.

[0190] Embodiments of the present disclosure also provide a computer-readable storage medium that may store a computer program. When the computer program is executed by a processor, the processor implements the vehicle condition monitoring method provided by the embodiments of the present disclosure.

[0191] The above storage medium may include, for example, a memory 520 storing computer program instructions that may be executed by a processor 510 of an electronic device to complete the vehicle condition monitoring method provided by the embodiments of the present disclosure. Optionally, the storage medium may be a non-transitory computer-readable storage medium. For example, the non-transitory computer-readable storage medium may be a ROM, a Random Access Memory (RAM), a Compact Disc ROM (CD-ROM), a magnetic tape, a floppy disk, and an optical data storage device, etc.

[0192] Embodiments of the present disclosure also provide a vehicle that includes an electronic device and can implement each of the processes and effects in the above embodiments of the present disclosure, which will not be elaborated herein.

[0193] Embodiments of the present disclosure also provide a computer program product, which includes a computer program or instructions. When the computer program or instructions are executed by a processor, the vehicle condition monitoring method provided by the embodiments of the present disclosure is implemented, and each process and effect in the above embodiments of the present disclosure can be achieved. Details are not described herein again.

[0194] The above are only specific implementation manners of the present disclosure, enabling those skilled in the art to understand or implement the present disclosure. Various modifications to these embodiments will be obvious to those skilled in the art. The general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to these embodiments described herein, but rather will be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A vehicle condition monitoring method, characterized in that, Including: Obtaining multi-modal data of a vehicle; Preprocessing the multi-modal data to obtain preprocessed multi-modal data; Inputting the preprocessed multi-modal data into a preset vehicle condition monitoring model, and monitoring the vehicle condition based on the preprocessed multi-modal data by the preset vehicle condition monitoring model to obtain a vehicle condition monitoring result and a vehicle condition suggestion result, where the preset vehicle condition monitoring model includes a first monitoring sub-model, a second monitoring sub-model, a third monitoring sub-model, and a multi-model fusion layer.

2. The method according to claim 1, characterized in that, The preprocessing the multi-modal data to obtain preprocessed multi-modal data includes: Performing denoising processing on the multi-modal data; Performing normalization processing on the denoised multi-modal data to obtain normalized processing data; Performing time series processing on the normalized processing data to obtain the preprocessed multi-modal data.

3. The method according to claim 1, wherein The multi-modal data includes vehicle sensor data, external vehicle environment data, vehicle historical maintenance data, and user behavior data; The inputting the preprocessed multi-modal data into a preset vehicle condition monitoring model, and monitoring the vehicle condition based on the preprocessed multi-modal data by the preset vehicle condition monitoring model to obtain a vehicle condition monitoring result and a vehicle condition suggestion result includes: Inputting the preprocessed vehicle sensor data into the first monitoring sub-model, and extracting features from the preprocessed vehicle sensor data by the first monitoring sub-model to obtain target features; Inputting the target features into the second monitoring sub-model, and performing vehicle condition prediction based on the target features by the second monitoring sub-model to output a target prediction result; Inputting the preprocessed vehicle sensor data, preprocessed external vehicle environment data, preprocessed vehicle historical maintenance data, and preprocessed user behavior data into the third monitoring sub-model, and outputting vehicle action suggestions by the third monitoring sub-model; Performing feature vector fusion processing on the feature vectors corresponding to the preprocessed vehicle sensor data, preprocessed external vehicle environment data, preprocessed vehicle historical maintenance data, and preprocessed user behavior data respectively to obtain a fused feature vector; Inputting the target prediction result, the vehicle action suggestions, and the fused feature vector into the multi-model fusion layer, and performing vehicle condition analysis based on the target prediction result, the vehicle action suggestions, and the fused feature vector by the multi-model fusion layer to output the vehicle condition monitoring result and the vehicle condition suggestion result.

4. The method according to claim 3, characterized in that, The first monitoring sub-model includes a multi-head attention layer, a dynamic gating mechanism layer, a feed-forward neural network layer, and a layer normalization and residual connection layer; The inputting the preprocessed vehicle sensor data into the first monitoring sub-model, and extracting features from the preprocessed vehicle sensor data by the first monitoring sub-model to obtain target features includes: Inputting the preprocessed vehicle sensor data into the multi-head attention layer, and extracting features from the preprocessed vehicle sensor data by the multi-head attention layer to obtain first features; Input the first feature into the dynamic gating mechanism layer, and determine the target expert sub-network in the feed-forward neural network layer based on the first feature by the dynamic gating mechanism layer; Input the first feature into the feed-forward neural network layer, activate the target expert sub-network by the feed-forward neural network layer, perform non-linear transformation and feature reconstruction processing on the first feature based on the target expert sub-network, and obtain a second feature; Input the second feature into the layer normalization and residual connection layer, and output the target feature based on the second feature by the layer normalization and residual connection layer; Before inputting the target feature into the second monitoring sub-model, and outputting a target prediction result by performing vehicle condition prediction on the target feature by the second monitoring sub-model, the method further includes: Perform normalization processing on the target feature to obtain the processed target feature; The step of inputting the target feature into the second monitoring sub-model, and outputting a target prediction result by performing vehicle condition prediction on the target feature by the second monitoring sub-model includes: Input the processed target feature into the second monitoring sub-model, perform feature selection on the processed target feature by the second monitoring sub-model based on a preset rule to obtain a third feature, perform vehicle condition prediction based on the third feature, and output the target prediction result.

5. The method according to claim 3, wherein Before inputting the preprocessed vehicle sensor data, preprocessed vehicle external environment data, preprocessed vehicle historical maintenance data, and preprocessed user behavior data into the third monitoring sub-model, and outputting a vehicle action suggestion by the third monitoring sub-model, the method further includes: Extract features from the preprocessed vehicle sensor data, the preprocessed vehicle external environment data, the preprocessed vehicle historical maintenance data, and the preprocessed user behavior data to obtain sensor data features corresponding to the preprocessed vehicle sensor data, environment data features corresponding to the preprocessed vehicle external environment data, historical maintenance data features corresponding to the preprocessed vehicle historical maintenance data, and user behavior features corresponding to the preprocessed user behavior data; The step of inputting the preprocessed vehicle sensor data, preprocessed vehicle external environment data, preprocessed vehicle historical maintenance data, and preprocessed user behavior data into the third monitoring sub-model, and outputting a vehicle action suggestion by the third monitoring sub-model includes: Input the sensor data features, the environment data features, the historical maintenance data features, and the user behavior features into the third monitoring sub-model, perform vectorization processing on the sensor data features, the environment data features, the historical maintenance data features, and the user behavior features by the third monitoring sub-model to obtain target feature vectors corresponding to the sensor data features, the environment data features, the historical maintenance data features, and the user behavior features respectively; Input each target feature vector into the third monitoring sub-model. The third monitoring sub-model performs vehicle condition analysis based on the target feature vector and outputs the vehicle action suggestions.

6. The method according to claim 3, characterized in that Input the target prediction result, the vehicle action suggestions, and the fused feature vector into the multi-model fusion layer. The multi-model fusion layer performs vehicle condition analysis based on the target prediction result, the vehicle action suggestions, and the fused feature vector and outputs the vehicle condition monitoring result and the vehicle condition suggestion result, including: Determine the first weight corresponding to the target prediction result, the second weight corresponding to the vehicle action suggestions, and the third weight corresponding to the fused feature vector. The multi-model fusion layer outputs the vehicle condition monitoring result and the vehicle condition suggestion result based on the first weight, the second weight, and the third weight.

7. A vehicle condition monitoring device, characterized in that, Including: A data acquisition module for acquiring multi-modal data of a vehicle; A data processing module for preprocessing the multi-modal data to obtain preprocessed multi-modal data; A vehicle condition monitoring module for inputting the preprocessed multi-modal data into a preset vehicle condition monitoring model. The preset vehicle condition monitoring model monitors the vehicle condition based on the preprocessed multi-modal data to obtain a vehicle condition monitoring result and a vehicle condition suggestion result. The preset vehicle condition monitoring model includes a first monitoring sub-model, a second monitoring sub-model, a third monitoring sub-model, and a multi-model fusion layer.

8. An electronic device, characterized in that, Including: A processor; A memory for storing executable instructions; Wherein, the processor is configured to read the executable instructions from the memory and execute the executable instructions to implement the vehicle condition monitoring method according to any one of claims 1-6 above.

9. A computer-readable storage medium, characterized in that, The storage medium stores a computer program. When the computer program is executed by a processor, the processor is caused to implement the vehicle condition monitoring method according to any one of claims 1-6 above.

10. A vehicle, characterized in that, Including the electronic device according to claim 8.

Images