Internet of vehicles platform operation and maintenance method, system and device based on multi-modal large model

Abstract

The invention relates to the technical field of Internet of Vehicles, and discloses an Internet of Vehicles platform operation and maintenance method, system and device based on a multi-modal large model. According to the invention, data acquisition is carried out on an Internet of Vehicles platform, multi-modal data determined by the Internet of Vehicles platform based on at least one data type is obtained, and the multi-modal data at least comprises a platform operation log and platform chart data; carrying out multi-modal fusion on the platform operation log and the platform chart data by adopting a multi-modal large model; and based on the fused data, determining a fault in the Internet of Vehicles platform and a cause of the fault. According to the method provided by the embodiment of the invention, the data of different modals can be fused by adopting the multi-modal large model in the operation and maintenance process of the Internet of Vehicles platform, and the fault and the fault generation reason in the platform are determined based on the fused data, so that the analysis dimension of fault diagnosis is improved, and the accuracy of platform operation and maintenance is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of Internet of Vehicles, and in particular to an Internet of Vehicles platform operation and maintenance method, system and device based on a multimodal large model. Background Art

[0002] With the rapid development of Internet of Vehicles technology, the Internet of Vehicles platform has become a key infrastructure for automakers and service providers. By collecting and analyzing a large amount of vehicle data, the Internet of Vehicles platform can provide users with a variety of services. As the scale and complexity of the Internet of Vehicles platform increase, the daily operation and maintenance of the platform becomes more and more complicated.

[0003] In related technologies, single-mode data is often used as the basis for the operation and maintenance of the Internet of Vehicles platform, which may lead to insufficient accuracy in data analysis and fault detection. How to improve the accuracy of the operation and maintenance of the Internet of Vehicles platform has become a problem that needs to be solved. Summary of the invention

[0004] In view of this, the present disclosure provides a vehicle networking platform operation and maintenance method, system and device based on a multimodal large model to solve the problem of how to improve the accuracy in the vehicle networking platform operation and maintenance process.

[0005] On the one hand, the present disclosure provides an operation and maintenance method for an Internet of Vehicles platform based on a multimodal big model, the method comprising: collecting data on the Internet of Vehicles platform, obtaining multimodal data determined by the Internet of Vehicles platform based on at least one data type, the multimodal data at least including platform operation logs and platform chart data; using the multimodal big model to perform multimodal fusion on the platform operation logs and platform chart data; and determining faults in the Internet of Vehicles platform and the causes of the faults based on the fused data.

[0006] On the other hand, the present disclosure also provides an Internet of Vehicles platform operation and maintenance device based on a multimodal large model, the device including: a collection module, used to collect data from the Internet of Vehicles platform, and obtain multimodal data determined by the Internet of Vehicles platform based on at least one data type, the multimodal data at least including platform operation logs and platform chart data; a fusion module, used to use the multimodal large model to perform multimodal fusion on the platform operation logs and platform chart data; a diagnosis module, used to determine faults in the Internet of Vehicles platform and the causes of the faults based on the fused data.

[0007] On the other hand, the present disclosure further provides a computer device, which includes: a memory and a processor, the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor implements the above-mentioned vehicle networking platform operation and maintenance method based on the multimodal large model by executing the computer instructions.

[0008] On the other hand, the present disclosure further provides a computer-readable storage medium, on which computer instructions are stored, and the computer instructions are used to enable a computer to implement the above-mentioned vehicle networking platform operation and maintenance method based on the multimodal large model.

[0009] On the other hand, the present disclosure further provides a computer program product, including computer instructions, which are used to enable a computer to execute the vehicle networking platform operation and maintenance method based on a multimodal large model of the above-mentioned first aspect or any corresponding embodiment.

[0010] Through the vehicle networking platform operation and maintenance method, system and device based on the multimodal big model of the above-mentioned embodiments of the present invention, data collection is performed on the vehicle networking platform, at least multimodal data including platform operation logs and platform chart data are obtained, and the multimodal big model is used to perform multimodal fusion on the multimodal data, and the faults and causes of the faults in the platform are determined based on the fused data. Therefore, the causes of the faults in the platform can be comprehensively analyzed from multiple data dimensions, and the analysis dimension of fault diagnosis in the platform is improved, thereby improving the accuracy of platform operation and maintenance.

[0011] In addition, the vehicle networking platform operation and maintenance method, system and device based on a multimodal large model provided by the embodiments of the present disclosure can also avoid the following problems: when using single modal data to analyze the cause of failure of the vehicle networking platform, the accuracy and comprehensiveness of data analysis and fault diagnosis are insufficient due to the inability to fully utilize the rich information of multimodal data. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the specific embodiments of the present disclosure or the technical solutions in the related technologies, the drawings required for use in the specific embodiments or the related technical descriptions will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0013] Figure 1 It is a flowchart of a vehicle networking platform operation and maintenance method based on a multimodal large model provided by an embodiment of the present disclosure;

[0014] Figure 2 It is a schematic diagram of the architecture of the vehicle networking platform operation and maintenance system based on the multimodal large model provided by the embodiment of the present disclosure;

[0015] Figure 3 It is a structural schematic diagram of a vehicle networking platform operation and maintenance device based on a multi-modal large model provided in an embodiment of the present disclosure;

[0016] Figure 4 It is another structural schematic diagram of the vehicle networking platform operation and maintenance device based on the multimodal large model provided in the embodiment of the present disclosure. DETAILED DESCRIPTION

[0017] In recent years, with the rapid development of Internet of Vehicles technology, the Internet of Vehicles platform has become an important infrastructure for automobile companies and service providers. The Internet of Vehicles platform provides users with a wealth of services by collecting and processing a large amount of vehicle data. However, as the scale and complexity of the platform increase, the daily operation and maintenance of the platform becomes more and more complicated. Related technologies often use single-modality data (such as text, images or audio) to perform data analysis and fault detection on the platform.

[0018] However, the related technology may have the following problems:

[0019] 1. Platform operation and maintenance based on single-modal data often rely on text logs for fault monitoring and analysis, and lack the ability to process multimodal data in other dimensions, resulting in insufficient accuracy and comprehensiveness of data analysis and fault diagnosis.

[0020] 2. Related technologies often rely on predefined rules and thresholds for fault detection, which limits the flexibility of detection and makes it difficult to cope with the complex and ever-changing Internet of Vehicles platform environment.

[0021] 3. Data of different modes are often scattered in different systems of the Internet of Vehicles platform, lacking effective integration and correlation analysis, forming data islands and limiting the comprehensive utilization and in-depth mining of data.

[0022] To solve at least one of the above problems, various embodiments of the present disclosure provide an operation and maintenance method for an Internet of Vehicles platform based on a multimodal big model, including: collecting data on the Internet of Vehicles platform, obtaining multimodal data determined by the Internet of Vehicles platform based on at least one data type, the multimodal data at least including platform operation logs and platform chart data; using the multimodal big model to perform multimodal fusion of the platform operation logs and platform chart data; and determining faults in the Internet of Vehicles platform and the causes of the faults based on the fused data.

[0023] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, rather than all the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present disclosure.

[0024] According to an embodiment of the present disclosure, a vehicle networking platform operation and maintenance method based on a multimodal large model is provided. Figure 1 is a flow chart of a vehicle networking platform operation and maintenance method based on a multimodal large model provided by an embodiment of the present disclosure, such as Figure 1As shown, the process includes the following steps:

[0025] Step S101, collect data from the Internet of Vehicles platform to obtain multimodal data determined by the Internet of Vehicles platform based on at least one data type, where the multimodal data at least includes a platform operation log and platform chart data.

[0026] In this embodiment, the data type in the Internet of Vehicles platform (hereinafter referred to as the platform) may include at least one of the following: text data, image data, video data, and audio data.

[0027] Here, the platform operation log as text data may include at least one of the following: operation log, error log, performance log, and the platform operation log may be used to record the platform's operation status, operation record, error information, etc. The platform chart data as image data may include at least one of the following: performance monitoring chart, traffic statistics chart, error distribution chart, and the platform chart data may be used to record the platform's operation status.

[0028] In a possible implementation of the above embodiment, collecting data on the platform and obtaining the platform operation log in the platform may include: using a log collection tool to collect and transmit log data in the platform, and storing the collected log data in a preset data structure.

[0029] The log collection tool and the preset data structure can be preset according to the actual application scenario or the experience of those skilled in the art, and can be tools in the existing technology or future development technology, which is not limited by the present disclosure. For example, the log collection tool can use Flume, which is a highly available, highly reliable, distributed system for massive log collection, aggregation and transmission. The preset data structure can use the Hadoop data warehouse.

[0030] Here, Flume is used as a log collection tool to adapt to the massive amount of data generated in the platform, thereby efficiently collecting and transmitting data.

[0031] Furthermore, collecting data from the platform and obtaining platform chart data in the platform may include: using an application programming interface (API) to collect chart data from the platform, and storing the chart data in a structured data form in a Hadoop data warehouse.

[0032] After acquiring the multimodal data, preprocessing is performed on the multimodal data, which may include: cleaning the data to remove noise and irrelevant information in the data; performing integrity checks on the data to ensure the integrity and consistency of the data; and converting data from different sources into a unified format for subsequent processing.

[0033] As an example, cleaning data to remove noise and irrelevant information in the data may include: removing redundant information in log data to fill in missing log entries; removing noise and interference information in image data to ensure image clarity and readability. Checking the integrity of data may include, but is not limited to: checking whether the timestamps in log data are continuous, and checking whether key indicators in chart data are missing.

[0034] After preprocessing the data, the multimodal data is formatted uniformly, the image data is converted into a standard image format, and the log data is converted into a structured data format; and

[0035] Convert data in different units in multimodal data into a unified unit, and convert timestamps in different formats into a unified time format.

[0036] Feature extraction of the unified-format multimodal data may include: extracting key features such as timestamp, error type, and operation type from log data, and extracting key indicators, trend lines, and other key features from image data.

[0037] Here, the feature extraction tool can be pre-set according to the actual application scenario or the experience of those skilled in the art, and can be an extraction tool in the existing technology or future development technology, which is not limited by the present disclosure. For example, the feature extraction tool can be a regular expression, natural language processing technology, image processing technology, etc.

[0038] Step S102: Use a multimodal large model to perform multimodal fusion on the platform operation log and the platform chart data.

[0039] In this embodiment, the multimodal large model may refer to a deep learning model in existing technology or future technology that can simultaneously process and understand multiple types of data (modalities). Multimodal fusion may refer to the process of comprehensively processing and uniformly expressing data from different modalities (such as text, images, videos, audio, etc.) so that the model can use multiple information sources to improve its ability to understand and reason.

[0040] Here, the multimodal large model may include but is not limited to: Contrastive Language-Image Pretraining (CLIP) model, Multitask Unified Model (MUM). As an example, the CLIP model may be used in the present disclosure to perform multimodal fusion of platform operation logs and platform chart data.

[0041] Among them, the CLIP model can be a multimodal model based on contrastive learning, which can convert multimodal data into a unified feature representation.

[0042] Step S103, based on the fused data, determine the faults in the Internet of Vehicles platform and the causes of the faults.

[0043] In this embodiment, a multi-modal large model is used to locate the fault location and the cause of the fault in the platform by analyzing the characteristics of the input fused data.

[0044] Here, the multimodal large model can locate the fault position of the platform in the image data by analyzing the spatial characteristics of the image data, and can diagnose the cause of the fault by linking and analyzing the characteristics of the image data and text data.

[0045] In the vehicle networking platform operation and maintenance method, system and device based on the multimodal big model of the above-mentioned embodiments of the present disclosure, data collection is performed on the vehicle networking platform, and at least multimodal data including platform operation logs and platform chart data is obtained. The multimodal big model is used to perform multimodal fusion on the multimodal data, and the faults and causes of the faults in the platform are determined based on the fused data. Therefore, the causes of the faults in the platform can be comprehensively analyzed from multiple data dimensions, and the analysis dimension of fault diagnosis in the platform is improved, thereby improving the accuracy of platform operation and maintenance.

[0046] In a specific embodiment, the multimodal large model is a fine-tuned model, and the fine-tuning of the multimodal large model is achieved based on the following steps:

[0047] The data loader of the initial multimodal large model is used to obtain multimodal historical data, and the historical text feature vector and the historical image feature vector corresponding to the multimodal historical data are determined;

[0048] Inputting the historical text feature vector and the historical image feature vector into the initial multimodal large model to determine the similarity between the historical text feature vector and the historical image feature vector;

[0049] Based on the similarity and the preset loss function of the initial multimodal large model, the preset optimizer is used to fine-tune the initial multimodal large model to determine the multimodal large model; wherein the preset loss function represents that the similarity between the relevant historical text feature vectors and the historical image feature vectors is the largest, and the similarity between the irrelevant historical text feature vectors and the historical image feature vectors is the smallest.

[0050] In this embodiment, the initial multimodal large model may refer to a pre-trained multimodal large model, which is trained by a preset multimodal data set to learn the association between different modalities, for example, the initial multimodal large model may refer to an initial CLIP model. The data loader may be a component for processing data in a deep learning and machine learning framework, which is used to load data from a storage medium (such as a file system, a database, etc.) and convert it into a format that can be processed by the model.

[0051] In a possible implementation of the above embodiment, using the data loader of the initial multimodal large model to obtain multimodal historical data and determining the historical text feature vector and the historical image feature vector corresponding to the multimodal historical data may include:

[0052] Load the initial CLIP model, use the data loader in the initial CLIP model to obtain multimodal historical data, use the image encoder in the initial CLIP model to convert the historical platform chart data into historical image feature vectors, and use the text encoder in the initial CLIP model to convert the historical platform operation log into historical text feature vectors.

[0053] Here, loading the initial CLIP model can be achieved by using the functions provided by the deep learning framework, which automatically handles the download and initialization of the model. Loading the model can include downloading the model weights, initializing the model structure, and loading the model into a suitable computing device. Loading the initial CLIP model can greatly reduce the training time and computing resource consumption, allowing the model to converge faster in the subsequent fine-tuning process.

[0054] The data loader in the initial CLIP model can be used to efficiently read data from disk and randomly shuffle the data during training to dynamically generate data batches, thereby improving the generalization ability of the model and reducing the risk of overfitting.

[0055] The image encoder in the initial CLIP model can be used to convert the input image into a feature vector of fixed length, and generate a representation of the image content by extracting the visual features of the image; the text encoder in the CLIP model can be used to convert the input text description into a feature vector of fixed length.

[0056] In a possible implementation of the above embodiment, before the data loader of the initial multimodal large model is used to acquire the multimodal historical data, the historical multimodal data may be preprocessed to convert the format of the multimodal data into a format acceptable to the initial CLIP model. The preprocessing may include: performing image scaling, cropping, and normalization on the historical platform chart data to ensure that the size and pixel value of the historical platform chart data meet the input requirements of the model; performing text segmentation, encoding, and padding on the historical platform operation log to convert the text into a token sequence that can be processed by the model.

[0057] Furthermore, inputting the historical text feature vector and the historical image feature vector into the initial multimodal macro model to determine the similarity between the historical text feature vector and the historical image feature vector may include:

[0058] The dot product or cosine similarity is used to determine the similarity between the historical text feature vector and the historical image feature vector to generate the true label, which can be used to guide the learning direction of the model.

[0059] Determine the similarity between the historical text feature vector and the historical image feature vector to generate the true label

[0060] Furthermore, based on the similarity and the preset loss function of the initial multimodal large model, the initial multimodal large model is fine-tuned using a preset optimizer to determine the multimodal large model, which may include:

[0061] Define the loss function and optimizer in the initial CLIP model, calculate the value of the loss function, average the losses of the historical text feature vector and the historical image feature vector, calculate the gradient through back propagation, and use the optimizer to update the parameters of the model until the initial CLIP model meets the convergence conditions, and determine the fine-tuned CLIP model.

[0062] As an example, a cross entropy loss function may be used as the loss function of the initial CLIP model, and an ADAM optimizer may be used as the optimizer of the initial CLIP model.

[0063] Among them, the cross entropy loss function can be a classification loss function, which can effectively measure the difference between the predicted probability distribution of the model and the true label distribution. The ADAM optimizer can be an adaptive learning rate optimizer that can dynamically adjust the learning rate according to the first-order and second-order moment estimates of the gradient, thereby accelerating the convergence process of the model.

[0064] Here, defining the loss function and optimizer in the initial CLIP model can provide guidance during the training process. By continuously minimizing the value of the loss function, the model can gradually learn the relationship between the image and the text, thereby achieving fine-tuning of the initial CLIP model.

[0065] In the vehicle networking platform operation and maintenance method, system and device based on a multimodal large model of the above-mentioned embodiments of the present invention, the initial CLIP model is fine-tuned using multimodal fusion data, so that the model can better understand the semantic relationship between images and texts, improve the complementarity of data in the vehicle networking platform, and improve the comprehensiveness and depth of analysis results during the operation and maintenance process; the use of pre-trained CLIP models can significantly improve the efficiency of feature extraction and reduce the time and computing resources required for fine-tuning; the data loading and preprocessing links ensure the quality and consistency of the input data, and through batch loading and random shuffling, the model can be better generalized, thereby reducing the risk of overfitting; through reasonable loss functions and optimizers, the model can more effectively learn the relationship between images and texts.

[0066] In a specific embodiment, a multimodal large model is used to perform multimodal fusion of platform operation logs and platform chart data, which is implemented based on the following steps:

[0067] The preset text encoder in the multimodal large model is used to convert the platform operation log in the multimodal data into a text feature vector;

[0068] The platform chart data in the multimodal data is converted into an image feature vector using a preset image encoder in the multimodal macro model;

[0069] The text feature vector and the image feature vector are mapped into the same feature space by contrastive learning.

[0070] In this embodiment, the contrastive learning method can be a self-supervised learning method, and the goal of contrastive learning is to maximize the similarity of positive sample pairs in the feature space and minimize the similarity between negative sample pairs. The positive sample pairs are composed of samples with similar features or semantics, and the negative sample pairs are composed of samples that are not similar in features or semantics.

[0071] Using a preset text encoder in the multimodal large model to convert the platform operation log in the multimodal data into a text feature vector can include: using a preset text conversion function in the fine-tuned CLIP model to determine the text feature vector corresponding to the platform operation log.

[0072] Using a preset image encoder in the multimodal large model to convert the platform chart data in the multimodal data into an image feature vector may include: using a preset image conversion function in the fine-tuned CLIP model to determine the image feature vector corresponding to the platform chart data.

[0073] Here, the preset text conversion function in the CLIP model may be an encode_text function, and the preset image conversion function in the CLIP model may be an encode_image function. The encode_text function may be used to call a preset text encoder to convert the platform operation log in the multimodal data into a text feature vector, and the encode_image function may be used to call a preset image encoder to convert the platform chart data in the multimodal data into an image feature vector.

[0074] The vehicle networking platform operation and maintenance method based on a multimodal large model in this implementation converts the platform operation log and chart data into text and image feature vectors respectively, so that the two different types of data can be represented uniformly, allowing the model to understand and analyze data from multiple dimensions, thereby improving the analysis dimension of the platform data; the contrastive learning method is used to map the text feature vector and the image feature vector to the same feature space, which helps to establish the semantic association between the image and the text, so that the model can more effectively discover the relationship between the log content and the chart data, thereby improving the accuracy of the analysis.

[0075] In a specific embodiment, based on the fused data, determining the faults and the causes of the faults in the Internet of Vehicles platform includes:

[0076] Using a multimodal large model, based on the cosine similarity between text feature vectors and image feature vectors, target faults that match the fault description in the platform operation log are identified in the platform chart data;

[0077] A multimodal large model is used to determine the causal relationship analysis results of the text feature vector and the image feature vector. Based on the causal relationship analysis results, the fault cause corresponding to the target fault is determined.

[0078] In this embodiment, cosine similarity may be an indicator for measuring the directional similarity of two vectors in a multidimensional space, and causal relationship analysis is a method for determining the causal relationship between variables, aiming to identify and verify the influence of a certain variable on another variable.

[0079] The cosine value between the text feature vector and the image feature vector is calculated using the following Euclidean dot product formula:

[0080] a·b=‖a‖‖b‖cosθ

[0081] Among them, ‖a‖ can represent the modulus of text feature vector a, that is, the length of vector a, ‖b‖ can represent the modulus of text feature vector b, that is, the length of vector b, θ can represent the angle between the two vectors, and a·b can represent the dot product of two vectors, which is used to describe the similarity of two vectors.

[0082] Furthermore, the cosine similarity between the text feature vector and the image feature vector can be calculated using the following formula:

[0083]

[0084] Among them, A·B represents the dot product of vectors A and B, which is calculated by multiplying the corresponding components and then summing them, that is, represents the square root of the sum of the squares of the components of vector A, used to represent the modulus ‖A‖, Represents the square root of the sum of the squares of the components of vector B, used to represent ‖B‖.

[0085] The vehicle networking platform operation and maintenance method based on a multimodal large model in this implementation can effectively measure the similarity between text feature vectors and image feature vectors through cosine similarity calculation, which can help quickly identify target faults that match the fault descriptions in the platform operation logs, and improve the accuracy and efficiency of fault detection; by analyzing the causal relationship between text feature vectors and image feature vectors, it can identify and verify the influence of different variables, help determine the direct cause of the fault, and reveal potential related factors, thereby providing an important basis for fault prevention.

[0086] In a specific embodiment, a multimodal large model is used to determine faults that match the fault description in the platform operation log in the platform chart data based on the cosine similarity between the text feature vector and the image feature vector, which is implemented based on the following steps:

[0087] Determine the cosine similarity between the text feature vector of the platform operation log and the image feature vector of the platform chart data, and when the cosine similarity is greater than a preset similarity threshold, determine similar target image data corresponding to the fault description of the platform operation log in the platform chart data;

[0088] A multimodal large model is used to analyze the spatial features in the target image data, locate the target fault in the target image data, and determine the target fault.

[0089] In this embodiment, after using the above formula to calculate the cosine similarity between the text feature vector and the image feature vector, the size relationship between the cosine similarity and the preset similarity threshold is determined; if the cosine similarity is greater than the preset similarity threshold, it means that the fault description in the platform operation log is semantically consistent with a target image data in the platform chart data, and it is determined that the target image data reflects the corresponding fault condition, then the target image data is located to determine the target fault.

[0090] As an example, the preset similarity threshold can be customized based on the specific situation. The preset similarity threshold can be set based on one of the following situations: historical data analysis, domain expert knowledge, real-time feedback information, and data distribution characteristics. Among them, by analyzing historical data, an effective similarity range can be found, so as to set a reasonable threshold, improve the accuracy of fault detection and reduce false alarms. Combining the experience and knowledge of domain experts to set the threshold ensures that the threshold setting is more in line with the actual situation and improves the reliability of the model.

[0091] In a possible implementation of the above embodiment, a multimodal large model is used to analyze spatial features in target image data, locate the target fault in the target image data, and determine the target fault. This can be achieved based on the following steps: computer vision technology is used to analyze image features in target image data and identify spatial patterns in the image data; a target detection algorithm is used to locate the fault in the extracted features, mark potential fault areas, and output bounding boxes and confidence scores to help determine the specific location of the fault; the text feature vector is combined with the spatial features in the image to further verify whether the identified fault area is consistent with the fault description in the platform operation log.

[0092] The vehicle networking platform operation and maintenance method based on the multimodal large model in this implementation combines text feature vectors with image feature vectors, and uses the advantages of the multimodal large model to more comprehensively understand and analyze fault information and improve the reliability of fault detection on the platform; through historical data analysis and the knowledge of domain experts, custom settings of preset similarity thresholds are made so that the system can adapt to different environments and data characteristics, thereby improving the practicality and adaptability of the model; through computer vision technology and target detection algorithms, the fault area is accurately located in the image, and a bounding box and confidence score are provided, making subsequent fault processing more efficient and reducing the need for manual intervention.

[0093] In a specific embodiment, a multimodal large model is used to determine the causal relationship analysis results of the text feature vector and the image feature vector, and based on the causal relationship analysis results, the fault cause corresponding to the target fault is determined, which is implemented based on the following steps:

[0094] Granger causality test is used to determine whether the text feature vector is the Granger cause of the image feature vector;

[0095] When the text feature vector is the Granger cause of the image feature vector, it is determined that the fault cause corresponding to the target fault of the image feature vector is related to the fault description corresponding to the text feature vector;

[0096] When the text feature vector is not the Granger cause of the image feature vector, it is determined that the fault cause corresponding to the target fault of the image feature vector is independent of the fault description corresponding to the text feature vector.

[0097] In this embodiment, the Granger causality test may be a statistical method for determining whether one time series can be used to predict another time series. Specifically, if the past values ​​of time series X can provide additional information about the future values ​​of time series Y, then X is said to have a Granger causal relationship with Y, which is used to characterize that changes in X may be statistically associated with changes in Y.

[0098] The Granger causality test is used to determine whether the text feature vector is the Granger cause of the image feature vector, which may include:

[0099] Using the Granger causality test, when it is determined that the historical values ​​of the time series corresponding to the text feature vector have predictive power for the future values ​​of the time series corresponding to the image feature vector, it is determined that the text feature vector is the Granger cause of the image feature vector.

[0100] When the text feature vector is the Granger cause of the image feature vector, determining that the fault cause corresponding to the target fault of the image feature vector is related to the fault description corresponding to the text feature vector may include:

[0101] When the text feature vector is the Granger cause of the image feature vector, there is a certain predictive relationship between the fault description in the characterization text and the fault state shown in the image, and it is determined that the target fault of the image feature vector is associated with the fault described by the text feature vector.

[0102] Accordingly, when the text feature vector is not the Granger cause of the image feature vector, determining that the fault cause corresponding to the target fault of the image feature vector is independent of the fault description corresponding to the text feature vector may include:

[0103] When the text feature vector is not the Granger cause of the image feature vector, it represents that there is no predictive relationship between the fault description in the text and the fault state shown in the image, and determines that the target fault of the image feature vector is irrelevant to the fault described by the text feature vector.

[0104] The vehicle networking platform operation and maintenance method based on a multimodal large model in this implementation can more accurately identify and locate faults and improve the accuracy of fault detection by analyzing the causal relationship between text and images. According to the analysis results, the maintenance and repair strategies can be optimized to solve potential faults in a targeted manner and improve the overall operation efficiency of the platform.

[0105] In a specific embodiment, a Granger causality test is used to determine whether a text feature vector is a Granger cause of an image feature vector, including:

[0106] Determine the time series data of the text feature vector as the first data, and determine the time series data of the image feature vector as the second data;

[0107] Based on the first data and the second data, determining a first formula and a second formula, the first formula being used to represent that a current value of the second data is determined by a historical value of the first data and a historical value of the second data, and the second formula being used to represent that a current value of the first data is determined by a historical value of the first data and a historical value of the second data;

[0108] The first formula and the second formula are used to determine whether the text feature vector is the Granger cause of the image feature vector.

[0109] In this embodiment, determining the time series data of the text feature vector as the first data and determining the time series data of the image feature vector as the second data may include: determining the time series data of the text feature vector as the first data X t , determine the time series data of the image feature vector as the second data Y t .

[0110] Here, the time series data of the text feature vector can be used to represent the log information recorded at different timestamps, and the time series of the image feature vector can be used to represent the visual representation of the fault at different time points.

[0111] Determining the first formula and the second formula based on the first data and the second data may include:

[0112] The first formula is expressed as follows:

[0113]

[0114] Among them, Y t Can represent the current value of the second data, X t-i can represent the historical value of the first data, α i Can be expressed with X t-i The coefficient of correlation, used to represent X t-i Y t The degree of influence, Y t-i can represent the historical value of the second data, β i Can be expressed with Y t-i The coefficient of correlation is used to represent Y t-i Y t The influence degree of , m can represent the number of lag periods considered, that is, the number of time points of historical data considered, μ 1t The error term can be represented by , which can be assumed to be white noise with zero mean.

[0115] The second formula is expressed as follows:

[0116]

[0117] Among them, X t Can represent the current value of the first data, Y t-i can represent the historical value of the second data, γ i Can be expressed with Y t-i The coefficient of correlation is used to represent Y t-i X t The degree of influence, X t-i can represent the historical value of the first data, δ i Can be expressed with X t-i The coefficient of correlation, used to represent X t-i X t The influence degree of , m can represent the number of lag periods considered, that is, the number of time points of historical data considered, μ 2t The error term can be represented by , which can be assumed to be white noise with zero mean.

[0118] For the first formula, the corresponding null hypothesis H 0 It can be H 0 :α 1 =α 2 =…=α q =0, for the second formula, the corresponding null hypothesis H 0 It can be H 0 :δ 1 =δ 2 =…=δ q =0.

[0119] Here, the null hypothesis of the first formula states that all t-i The correlation coefficient α i are all equal to 0, used to represent X t-i Y t There is no significant effect. If the null hypothesis is true, it means that the historical value of the first data cannot be used to predict the current value of the second data. The null hypothesis of the second formula indicates that all t-i The correlation coefficient δ i are all equal to 0, used to represent X t-i X t There is no significant effect. If the null hypothesis is true, it means that the historical value of the first data cannot be used to predict the current value of the first data.

[0120] The above null hypothesis can be used to perform Granger causality test. If the first formula and / or the second formula rejects the corresponding null hypothesis, it can be characterized that the corresponding variable plays an important role in predicting the target variable.

[0121] Specifically, if the coefficient α of the historical value of the first data in the first formula is iis statistically significantly different from zero, and the coefficient γ of the historical value of the second data in the second formula i If the first data is statistically significant to zero, it is determined that the first data is the cause of the change in the second data.

[0122] If the coefficient γ of the historical value of the second data in the second formula i is statistically significantly different from zero, and the coefficient α of the historical value of the first data in the second formula i If the second data is statistically significant to zero, it is determined that the second data is the cause of the change in the first data.

[0123] If the coefficient α of the historical value of the first data in the first formula i is statistically significantly different from zero, and the coefficient γ of the historical value of the second data in the second formula i If the value is statistically significantly different from zero, it is determined that there is a bidirectional causal relationship between the second data and the first data.

[0124] If the coefficient α of the historical value of the first data in the first formula i is statistically significantly zero, and the coefficient γ of the historical value of the second data in the second formula i If the value is statistically significant to zero, it is determined that there is no causal relationship between the second data and the first data.

[0125] Among them, if it is significantly different from zero, it means that the parameter has a substantial impact on the model results, that is, the parameter is important in predicting the dependent variable. If it is significantly zero, it means that the parameter has no substantial impact on the model results, that is, it can be considered unimportant in the model.

[0126] In a possible implementation of the above embodiment, when it is determined that the first data is the cause of the change in the second data, the fault description in the characterization operation log can predict the fault state reflected in the image data. When it is determined that the second data is the cause of the change in the first data, the fault manifestation in the characterization image data may affect the description of the operation log. When it is determined that the second data has a bidirectional causal relationship with the first data, the characterization image data may affect the recording of the fault description, and the description in the log may in turn affect the analysis of the image data. When there is no causal relationship between the second data and the first data, there is no direct causal relationship between the characterization fault description and the image data.

[0127] In the vehicle networking platform operation and maintenance method based on a multimodal large model in this implementation, when it is determined that the text feature vector (fault description) is the cause of the image feature vector (fault manifestation), the system can effectively use the information in the operation log to predict the fault state reflected in the image data. This helps to identify potential problems in advance, thereby optimizing maintenance and management strategies; if the image feature vector is confirmed to be the cause of the text feature vector, it means that the fault manifestation in the image has an impact on the log description. This can promote the improvement of fault information recording, making the log more targeted and accurate; when it is confirmed that there is a two-way causal relationship, it indicates that the fault description and image data affect each other, and the interaction between the two can be comprehensively considered in fault analysis, which helps to improve the overall level of intelligence of fault management.

[0128] In an optional application scenario, Figure 2 is a schematic diagram of the architecture of the vehicle networking platform operation and maintenance system based on the multimodal large model provided by the embodiment of the present disclosure, such as Figure 2 As shown, where:

[0129] The data acquisition module is used to obtain operation logs and chart data. The data preprocessing module is used to clean, convert formats and extract features of multimodal data. The multimodal data fusion module is used to load multimodal models and data, define optimizers to perform training cycles on multimodal data, and determine the fine-tuned multimodal large model. The fault detection and classification module is used to obtain input multimodal data, perform multimodal fusion based on multimodal data, and perform fault detection and classification. The fault location and diagnosis module is used to locate and diagnose faults on multimodal data.

[0130] Figure 3 is a structural diagram of a vehicle networking platform operation and maintenance device based on a multimodal large model provided by an embodiment of the present disclosure, such as Figure 3 As shown, the device can be applied to intelligent electronic devices such as servers and computers; the device includes: a collection module 301, a fusion module 302 and a diagnosis module 303;

[0131] The acquisition module 301 is used to collect data from the Internet of Vehicles platform, and obtain multimodal data determined by the Internet of Vehicles platform based on at least one data type, where the multimodal data at least includes platform operation logs and platform chart data;

[0132] A fusion module 302 is used to perform multimodal fusion of platform operation logs and platform chart data using a multimodal large model;

[0133] The diagnosis module 303 is used to determine the faults and causes of the faults in the Internet of Vehicles platform based on the fused data.

[0134] In a specific embodiment, the multimodal large model is a fine-tuned model, and the apparatus further includes a fine-tuning module 304, wherein:

[0135] A fine-tuning module 304 is used to obtain multimodal historical data using a data loader of the initial multimodal large model, and determine a historical text feature vector and a historical image feature vector corresponding to the multimodal historical data;

[0136] Inputting the historical text feature vector and the historical image feature vector into the initial multimodal large model to determine the similarity between the historical text feature vector and the historical image feature vector;

[0137] Based on the similarity and the preset loss function of the initial multimodal large model, the preset optimizer is used to fine-tune the initial multimodal large model to determine the multimodal large model; wherein the preset loss function represents that the similarity between the relevant historical text feature vectors and the historical image feature vectors is the largest, and the similarity between the irrelevant historical text feature vectors and the historical image feature vectors is the smallest.

[0138] In a specific embodiment, the fusion module 302 is used to convert the platform operation log in the multimodal data into a text feature vector using a preset text encoder in the multimodal macro model;

[0139] The platform chart data in the multimodal data is converted into an image feature vector using a preset image encoder in the multimodal macro model;

[0140] The text feature vector and the image feature vector are mapped into the same feature space by contrastive learning.

[0141] In a specific embodiment, the diagnosis module 303 is used to determine the target fault that matches the fault description in the platform operation log in the platform chart data based on the cosine similarity between the text feature vector and the image feature vector using the multimodal large model;

[0142] A multimodal large model is used to determine the causal relationship analysis results of the text feature vector and the image feature vector. Based on the causal relationship analysis results, the fault cause corresponding to the target fault is determined.

[0143] In a specific embodiment, the diagnosis module 303 is used to determine the cosine similarity between the text feature vector of the platform operation log and the image feature vector of the platform chart data, and when the cosine similarity is greater than a preset similarity threshold, determine similar target image data corresponding to the fault description of the platform operation log in the platform chart data;

[0144] A multimodal large model is used to analyze the spatial features in the target image data, locate the target fault in the target image data, and determine the target fault.

[0145] In a specific embodiment, the diagnosis module 303 is used to use Granger causality test to determine whether the text feature vector is the Granger cause of the image feature vector;

[0146] When the text feature vector is the Granger cause of the image feature vector, it is determined that the fault cause corresponding to the target fault of the image feature vector is related to the fault description corresponding to the text feature vector;

[0147] When the text feature vector is not the Granger cause of the image feature vector, it is determined that the fault cause corresponding to the target fault of the image feature vector is independent of the fault description corresponding to the text feature vector.

[0148] In a specific embodiment, the diagnosis module 303 is used to determine a first formula and a second formula based on the first data and the second data, the first formula is used to characterize that the current value of the second data is determined by the historical value of the first data and the historical value of the second data, and the second formula is used to characterize that the current value of the first data is determined by the historical value of the first data and the historical value of the second data;

[0149] The first formula and the second formula are used to determine whether the text feature vector is the Granger cause of the image feature vector.

[0150] It should be noted that: the vehicle networking platform operation and maintenance device based on the multimodal large model provided in the above embodiment only uses the division of the above program modules as an example to illustrate when implementing the corresponding vehicle networking platform operation and maintenance method based on the multimodal large model. In actual applications, the above processing can be assigned to different program modules as needed, that is, the internal structure of the device can be divided into different program modules to complete all or part of the above-described processing. In addition, the device provided in the above embodiment and the corresponding Figure 1 The embodiments of the method shown belong to the same concept, and the specific implementation process is detailed in the method embodiments, which will not be repeated here.

[0151] The present disclosure also provides a computer device having the above Figure 3 The Internet of Vehicles platform operation and maintenance device based on a multimodal large model is shown.

[0152] See also Figure 4 , Figure 4 is another structural schematic diagram of the vehicle networking platform operation and maintenance device based on the multimodal large model provided by the embodiment of the present disclosure, such as Figure 4As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 4 A processor 10 is taken as an example.

[0153] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0154] The memory 20 stores instructions executable by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.

[0155] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0156] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0157] The computer device also includes an input device 30 and an output device 40. The processor 10, the memory 20, the input device 30 and the output device 40 may be connected via a bus or other means. Figure 4 The example of connecting through bus is taken in the following.

[0158] The input device 30 can receive input digital or character information, and generate key signal input related to the user settings and function control of the computer device, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator bar, one or more mouse buttons, a trackball, a joystick, etc. The output device 40 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The above-mentioned display device includes but is not limited to a liquid crystal display, a light emitting diode, a display and a plasma display. In some optional embodiments, the display device can be a touch screen.

[0159] The computer device further comprises a communication interface 30 for the computer device to communicate with other devices or a communication network.

[0160] The embodiments of the present disclosure also provide a computer-readable storage medium. The above-mentioned method according to the embodiments of the present disclosure can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium and downloaded through a network, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state drive, etc.; further, the storage medium can also include a combination of the above-mentioned types of memory. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0161] A part of the present disclosure may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present disclosure through the operation of the computer. Those skilled in the art should understand that the existence of computer program instructions in computer-readable media includes, but is not limited to, source files, executable files, installation package files, etc., and accordingly, the way in which computer program instructions are executed by a computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to the computer.

[0162] Although the embodiments of the present disclosure have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present disclosure, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. The vehicle networking platform operation and maintenance method based on a multi-modal large model is characterized by: The method comprises: Collecting data from the Internet of Vehicles platform to obtain multimodal data determined by the Internet of Vehicles platform based on at least one data type, wherein the multimodal data at least includes a platform operation log and platform chart data; Using a multimodal large model to perform multimodal fusion on the platform operation log and the platform chart data; Based on the fused data, the fault in the Internet of Vehicles platform and the cause of the fault are determined.

2. The method according to claim 1, characterized in that The multimodal large model is a fine-tuned model, and fine-tuning the multimodal large model is achieved based on the following steps: Using the data loader of the initial multimodal large model to acquire multimodal historical data, and determining the historical text feature vector and the historical image feature vector corresponding to the multimodal historical data; Inputting the historical text feature vector and the historical image feature vector into the initial multimodal large model, and determining the similarity between the historical text feature vector and the historical image feature vector; Based on the similarity and the preset loss function of the initial multimodal large model, the initial multimodal large model is fine-tuned using a preset optimizer to determine the multimodal large model; wherein the preset loss function represents that the similarity between the relevant historical text feature vectors and the historical image feature vectors is the largest, and the similarity between the irrelevant historical text feature vectors and the historical image feature vectors is the smallest.

3. The method according to claim 1, characterized in that The multimodal large model is used to perform multimodal fusion of the platform operation log and the platform chart data, which is achieved based on the following steps: The preset text encoder in the multimodal large model is used to convert the platform operation log in the multimodal data into a text feature vector; Using a preset image encoder in the multimodal macro model, converting platform chart data in the multimodal data into an image feature vector; The text feature vector and the image feature vector are mapped into the same feature space by using a contrastive learning method.

4. The method according to claim 3, characterized in that The determining, based on the fused data, a fault in the Internet of Vehicles platform and a cause of the fault, includes: Using a multimodal large model, based on the cosine similarity between text feature vectors and image feature vectors, target faults that match the fault description in the platform operation log are identified in the platform chart data; The multimodal large model is used to determine a causal relationship analysis result between the text feature vector and the image feature vector, and based on the causal relationship analysis result, a fault cause corresponding to the target fault is determined.

5. The method according to claim 4, characterized in that Using a multimodal large model, based on the cosine similarity between text feature vectors and image feature vectors, faults that match the fault description in the platform operation log are identified in the platform chart data, based on the following steps: Determine the cosine similarity between the text feature vector of the platform operation log and the image feature vector of the platform chart data, and when the cosine similarity is greater than a preset similarity threshold, determine similar target image data corresponding to the fault description of the platform operation log in the platform chart data; A multimodal large model is used to analyze the spatial features in the target image data, locate the target fault in the target image data, and determine the target fault.

6. The method according to claim 4, characterized in that The multimodal large model is used to determine the causal relationship analysis result between the text feature vector and the image feature vector, and based on the causal relationship analysis result, the fault cause corresponding to the target fault is determined, which is achieved based on the following steps: Granger causality test is used to determine whether the text feature vector is the Granger cause of the image feature vector; When the text feature vector is a Granger cause of the image feature vector, determining that a fault cause corresponding to a target fault of the image feature vector is related to a fault description corresponding to the text feature vector; When the text feature vector is not the Granger cause of the image feature vector, it is determined that the fault cause corresponding to the target fault of the image feature vector is independent of the fault description corresponding to the text feature vector.

7. The method according to claim 5, characterized in that The Granger causality test is used to determine whether the text feature vector is the Granger cause of the image feature vector, including: Determine the time series data of the text feature vector as the first data, and determine the time series data of the image feature vector as the second data; Based on the first data and the second data, determining a first formula and a second formula, the first formula being used to represent that a current value of the second data is determined by a historical value of the first data and a historical value of the second data, and the second formula being used to represent that a current value of the first data is determined by a historical value of the first data and a historical value of the second data; The first formula and the second formula are used to determine whether the text feature vector is the Granger cause of the image feature vector.

8. The vehicle networking platform operation and maintenance device based on a multi-modal large model is characterized by: The device comprises: A collection module, used to collect data from the Internet of Vehicles platform, and obtain multimodal data determined by the Internet of Vehicles platform based on at least one data type, wherein the multimodal data at least includes a platform operation log and platform chart data; A fusion module, used for performing multimodal fusion of the platform operation log and the platform chart data using a multimodal large model; The diagnosis module is used to determine the fault in the Internet of Vehicles platform and the cause of the fault based on the fused data.

9. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the vehicle networking platform operation and maintenance method based on a multimodal large model as described in any one of claims 1 to 7 by executing the computer instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the vehicle networking platform operation and maintenance method based on a multimodal large model as described in any one of claims 1 to 7.

Images