Electric vehicle charging pile monitoring method and device based on cross-modal intelligent sensing and storage medium

Abstract

The invention discloses an electric vehicle charging pile monitoring method and device based on cross-modal intelligent sensing and a storage medium in the technical field of charging pile fault early warning, and aims to solve the problems of long early warning time delay, incomplete monitoring, low efficiency and insufficient fire prevention and control in charging pile monitoring. According to the method, image data acquired by a charging pile sensor and an external camera are integrated, a long-short-term memory network and a YOLOv7 model are adopted to dynamically monitor three states of a charging pile before charging, during charging and after charging, and the conditions of electrical abnormity, temperature exceeding, environment abnormity, charging abnormity after charging and the like are focused on. Multi-dimensional and cross-modal operation monitoring and risk early warning are provided, and safe and efficient operation of the charging pile is ensured; and meanwhile, the system can automatically upload related data and image information to a cloud end when an abnormal condition is detected, so that intelligent remote monitoring is realized.

Description

Technical Field

[0001] The present invention relates to a method, device and storage medium for monitoring an electric vehicle charging pile based on cross-modal intelligent sensing, and belongs to the technical field of charging pile fault early warning. Background Art

[0002] With the popularity and development of shared electric bikes, charging stations, as a key supporting infrastructure, are becoming increasingly widespread. However, charging stations can experience various faults or anomalies during operation, such as overloads, short circuits, leakage, and fires. These issues pose a serious threat to user safety and property. Therefore, effectively monitoring and providing early warnings for charging station operation, and promptly detecting and addressing faults or anomalies, have become pressing technical challenges.

[0003] Even when the charging station is not charging, overcharging the battery pack or failing to shut off the current can cause the battery temperature to gradually rise, ultimately leading to a fire. Therefore, monitoring the operating status of the charging station is essential. While faults and anomalies during charging often receive attention, monitoring during the pre- and post-charging states is equally important. Before charging, loose battery connectors or a malfunction in the battery management system can lead to poor contact, potentially causing overheating. After charging, if the battery management system fails to shut off the current or the temperature rises too high, fires, explosions, and other safety incidents can occur. Therefore, a comprehensive charging station monitoring system must cover the entire charging cycle, including all states before, during, and after charging, to ensure the stability and safety of the charging station.

[0004] Current electric vehicle charging pile monitoring methods on the market suffer from extended warning times, incomplete monitoring, low efficiency, and insufficient fire prevention and control. Traditional methods rely primarily on centralized data processing and simple decision-making rules, making them inadequate for complex and diverse fault prediction needs. To improve the efficiency and accuracy of monitoring charging pile operating conditions and fault warnings, a new approach is being developed. Summary of the Invention

[0005] The purpose of the present invention is to overcome the deficiencies in the prior art and to solve the problems of the existing electric vehicle charging pile monitoring method, such as prolonged warning time, incomplete monitoring, low efficiency and insufficient fire prevention and control.

[0006] In order to solve the above technical problems, the present invention is implemented by adopting the following technical solutions:

[0007] In a first aspect, a method for monitoring a vehicle charging pile based on cross-modal intelligent sensing is provided, comprising the following steps:

[0008] Acquire real-time operating data inside the charging pile and real-time image data outside the charging pile; and pre-process the real-time operating data and real-time image data; wherein the real-time operating data includes current, voltage, and temperature inside the charging pile;

[0009] Perform weighted fusion on the operating data at the current moment and the previous period respectively to obtain the weighted fusion actual value at the current moment and the weighted fusion actual value in the previous period;

[0010] Build and train long short-term memory deep learning models based on charging and non-charging conditions;

[0011] According to the current charging state, the weighted fusion actual value in the period before the current moment is input into the trained long short-term memory deep learning model to predict the weighted fusion prediction value at the current moment;

[0012] Build and train the YOLOv7 model, input the preprocessed real-time image data into the trained YOLOv7 model, and output the fire risk detection status;

[0013] The sliding window technology is used to calculate the mean and standard deviation of historical data deviation, and two fault warning thresholds are set according to the current charging status;

[0014] When the difference between the predicted value after weighted fusion and the actual value exceeds the fault warning threshold of the corresponding charging state or a fire risk is detected, it is determined that the operating state is abnormal.

[0015] Furthermore, the construction of the long short-term memory deep learning model includes constructing a long short-term memory deep learning model basic framework, and the long short-term memory deep learning model basic framework includes an input layer, a hidden layer, a fully connected layer and an output layer;

[0016] The input layer is used to receive time series data; wherein, the time series data is obtained by weighted fusion of real-time operation data features, and the features after weighted fusion of real-time operation data are obtained by weighted fusion of real-time operation data. Expressed as:

[0017]

[0018] Current in real-time operation data, The voltage and is the temperature in the real-time operation data, and their weights are current The weight of ,Voltage The weight of ,temperature The weight of ;

[0019] For the current time t, the features after weighted fusion of real-time running data Arrange the data to get the time series data, which is represented as:

[0020] X=[ F t - 1 , F t - 2 , … , F t - n ]

[0021] in is a vector of dimension [n,1], where n is the window size;

[0022] The hidden layer includes multiple LSTM units;

[0023] A single LSTM unit includes an input gate, a forget gate, and an output gate;

[0024] The forget gate is used to control the information discarded in the memory unit state. The formula of the forget gate is:

[0025]

[0026] in, It is the Forget Gate Momentary status, is the weight matrix of the forget gate, is the bias vector of the forget gate, yes The hidden layer state at time t, It is a sigmoid function with an output value range of 0 to 1;

[0027] The fully connected layer connects the output of the hidden layer to the output layer to generate the final prediction result; when the output of the hidden layer is assumed to be The formula for the fully connected layer is:

[0028] ;

[0029] is the weight matrix of the fully connected layer, is the weight matrix of the input features, is the bias vector of the fully connected layer, is the output of the fully connected layer, The predicted value probability after weighted fusion of real-time running data.

[0030] Furthermore, the input gate is used to update the information in the memory cell state. The input gate formula is:

[0031] ;

[0032] ;

[0033] ;

[0034] For Input the activation value of the gate at any moment, represents the weight matrix of the input gate, represents the bias vector of the input gate; is The new memory content of the moment, is the weight matrix of the memory unit, is the bias vector of the memory cell, is The state of the memory unit at the moment, is The state of the memory unit at the moment, yes Activation function.

[0035] Furthermore, the output gate is used to determine the hidden state output of the memory cell state. The output gate formula is:

[0036] ;

[0037] is the activation value of the output gate, is the weight matrix of the output gate, is the bias vector for the output gate.

[0038] Furthermore, the constructing and training of the long short-term memory deep learning model according to the charging and non-charging states respectively includes:

[0039] Two long short-term memory deep learning models, Model 1 and Model 2, are constructed and trained; wherein, Model 1 corresponds to the non-charging state, and Model 2 corresponds to the charging state.

[0040] Furthermore, the YOLOv7 model is used to detect real-time image data outside the charging pile;

[0041] The real-time image data includes: fire and smoke;

[0042] It also includes: identifying the three states of the indicator light on the surface of the charging pile socket; among them, the three states include flashing green light, flashing red light and no indicator light detected; among them, the flashing green light means charging is in progress, the flashing red light means charging is complete, and the no indicator light detected means the charging is not in progress.

[0043] Furthermore, the setting of two fault warning thresholds includes:

[0044] The deviation function is defined by using the overall deviation between the features after weighted fusion of real-time running data and the model prediction value , the formula is:

[0045]

[0046] is the feature after weighted fusion of real-time running data at the current time t, It is the probability of the predicted value after weighted fusion of real-time running data;

[0047] Calculate the average deviation using a sliding window and standard deviation , the formula is:

[0048]

[0049]

[0050] is the size of the sliding window;

[0051] The fault warning threshold The calculation formula is:

[0052]

[0053] k is an adjustable parameter, and the value range of k is 1.5-3.

[0054] Furthermore, the setting of two fault warning thresholds according to the current charging state includes:

[0055] Set fault warning thresholds according to the non-charging state and the charging state of .

[0056] In a second aspect, a vehicle charging pile monitoring device based on cross-modal intelligent sensing is provided, comprising:

[0057] The data acquisition and preprocessing module is used to collect real-time operation data through sensors on the charging pile and obtain real-time image data using an external camera; and preprocess the real-time operation data and real-time image data;

[0058] Model training module, used to build and train long short-term memory deep learning models and YOLOv7 models to monitor the external environment of charging piles;

[0059] The monitoring module is used to analyze the real-time operation data and real-time image data of the charging pile to monitor the operation status of the charging pile according to the current charging status.

[0060] In a third aspect, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the electric vehicle charging pile monitoring method based on cross-modal intelligent sensing as described in any one of the first aspects is implemented.

[0061] Compared with the prior art, the present invention has at least the following beneficial effects:

[0062] This invention significantly improves the comprehensiveness of charging pile monitoring by comprehensively analyzing cross-modal, multi-dimensional data. Compared with traditional methods, this invention uses an LSTM deep learning model to predict real-time operating data, thereby more effectively and accurately identifying potential abnormal conditions. At the same time, through the monitoring of real-time image data, the monitoring capability of the external environment is enhanced. This invention utilizes not only the faster response to abnormal conditions, but also reduces the frequency and cost of manual inspections through remote monitoring, thereby improving the operational efficiency of charging piles. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 The figure shows a module diagram of the charging pile monitoring system proposed in the first embodiment of the present invention;

[0064] Figure 2 The figure shows the LSTM model training flow chart proposed in the first embodiment of the present invention;

[0065] Figure 3 The figure shows a flow chart of the monitoring module proposed in the first embodiment of the present invention. DETAILED DESCRIPTION

[0066] The technical solution of the present invention is described in detail below through the accompanying drawings and specific embodiments. It should be understood that the embodiments of the present application and the specific features in the embodiments are detailed descriptions of the technical solution of the present application, rather than limitations on the technical solution of the present application. Unless there is a conflict, the embodiments of the present application and the technical features in the embodiments can be combined with each other.

[0067] The term "and / or" in this document simply describes a relationship between related objects, indicating that three possible relationships exist. For example, "A and / or B" can mean: A exists alone, A and B exist simultaneously, or B exists alone. Additionally, the character " / " in this document generally indicates an "or" relationship between the related objects.

[0068] Example 1:

[0069] like Figure 1As shown, this embodiment provides a method for monitoring a vehicle charging pile based on cross-modal intelligent sensing, comprising the following steps:

[0070] Acquire real-time operation data inside the charging pile and real-time image data outside the charging pile; and pre-process the real-time operation data and real-time image data;

[0071] The data acquisition and preprocessing module collects current, voltage, and temperature data through sensors on the charging pile to monitor the internal operating status of the charging pile, and obtains image data through an external camera to monitor the external environment of the charging pile in real time. The data is then preprocessed for higher-quality data analysis.

[0072] The model training module includes a long short-term memory (LSTM) deep learning model and a YOLOv7 deep learning model. The LSTM model uses the weighted fusion of current, voltage, and temperature data from the collection points as input time series for model training. It outputs a weighted fusion prediction of current, voltage, and temperature for the next time step and sets a warning fault threshold based on historical data. The YOLOv7 model uses the surveillance video image data from the collection points as input for model training and outputs fire and smoke detection results. It also adds the ability to monitor socket indicator lights in real time to assist in monitoring the operating status of charging piles.

[0073] The monitoring module performs a comprehensive analysis based on real-time data and image data. When the error between the predicted value and the actual value exceeds a set threshold or a fire risk is detected, the system triggers an early warning and sends the early warning information and related image data to the cloud.

[0074] The data acquisition section samples the voltage, current, and temperature data of the charging pile every 5 seconds, thereby continuously acquiring monitoring signals generated within the charging pile during the cycle time. Sensor technology is used to collect current, voltage, and temperature data. The current sensor is used to monitor the current changes of the charging pile in real time and detect abnormal fluctuations in the power load, such as overload, short circuit, and other potential faults. The voltage sensor is used to monitor the stability of the voltage. Excessive voltage fluctuations can affect the operation of the equipment and even cause damage to the equipment. The temperature sensor detects temperature changes in the charging pile and its surrounding environment, especially heat dissipation failures that may be caused by long-term high-temperature operation, which in turn may cause safety issues such as fire. The camera is then used to continuously collect image data of the external surrounding environment of the charging pile, and intelligent visual monitoring technology is used to identify early abnormal conditions such as fire and smoke.

[0075] The preprocessing part uses a variety of technologies to improve data quality and ensure the accuracy of subsequent analysis. First, the noise filtering algorithm is used to denoise the sensor data such as current, voltage and temperature to eliminate noise signals caused by environmental factors, electromagnetic interference, etc. Next, the sensor data is standardized or normalized to unify data of different dimensions to the same scale, which facilitates efficient processing and analysis of the model. In addition, for image data, denoising algorithms and image binarization techniques are used to filter out noise and enhance key features, thereby providing clearer and more effective input data for deep learning models. Through these preprocessing steps, irrelevant information and outliers in the original data are successfully eliminated, and valuable data is extracted, thereby greatly improving the accuracy and quality of data analysis.

[0076] Subsequently, a long short-term memory deep learning model and a YOLOv7 model were built and trained, and fault warning thresholds were set based on historical data;

[0077] The LSTM model training process in the model training module is as follows Figure 3 As shown, specifically:

[0078] Obtain the internal data of the charging pile under two operating states - "not charging" and "charging", which are used to train model 1 and model 2 respectively. The data set of the "not charging" state is The data set for the “charging” state is The data was normalized and divided into datasets, and data augmentation was performed to address the problem of missing datasets. 70% of the dataset was divided into a training set for model training, and 30% was divided into a test set for verifying the accuracy of model predictions.

[0079] Build the basic framework of the LSTM deep learning model, which includes the input layer, hidden layer, fully connected layer, and output layer, and initialize the hyperparameters to obtain the initialized LSTM deep learning model;

[0080] Define the range of hyperparameters and use particle swarm optimization algorithm to search for the best hyperparameter combination;

[0081] Training LSTM model: Build and train the LSTM model using the optimal hyperparameter combination;

[0082] Verify model prediction accuracy: Evaluate the model's loss on the test set to determine the model's predictive ability;

[0083] The overall deviation between the actual value of the relevant characteristic variable and the model predicted value is used to define the deviation function, and then the average deviation value is calculated using a sliding window to determine the warning threshold;

[0084] The long short-term memory deep learning model is specifically characterized by: including an input layer, a hidden layer, a fully connected layer and an output layer;

[0085] The input layer is used to receive time series data; wherein, the time series data is obtained by weighted fusion of real-time operation data features, and the features after weighted fusion of real-time operation data are obtained by weighted fusion of real-time operation data. Expressed as:

[0086]

[0087] Current in real-time operation data, The voltage and is the temperature in the real-time operation data, and their weights are current The weight of ,Voltage The weight of ,temperature The weight of ;

[0088] For the current time t, the features after weighted fusion of real-time running data Arrange the data to get the time series data, which is represented as:

[0089] X=[ F t - 1 , F t - 2 , … , F t - n ]

[0090] in is a vector of dimension [n,1], where n is the window size.

[0091] The hidden layer includes multiple LSTM units;

[0092] A single LSTM unit includes an input gate, a forget gate, and an output gate;

[0093] The forget gate is used to control the information discarded in the memory unit state. The formula of the forget gate is:

[0094]

[0095] in, It is the Forget Gate Momentary status, is the weight matrix of the forget gate, is the bias vector of the forget gate, yes The hidden layer state at time t, It is a sigmoid function with an output value range of 0 to 1.

[0096] The input gate is used to update the information in the memory cell state. The input gate formula is:

[0097] ;

[0098] ;

[0099] ;

[0100] For Input the activation value of the gate at any moment, represents the weight matrix of the input gate, represents the bias vector of the input gate; is The new memory content of the moment, is the weight matrix of the memory unit, is the bias vector of the memory cell, is The state of the memory unit at the moment, is The state of the memory unit at the moment, yes Activation function.

[0101] Furthermore, the output gate is used to determine the hidden state output of the memory cell state. The output gate formula is:

[0102] ;

[0103] is the activation value of the output gate, is the weight matrix of the output gate, is the bias vector for the output gate.

[0104] The fully connected layer connects the output of the hidden layer to the output layer to generate the final prediction result; when the output of the hidden layer is assumed to be The formula for the fully connected layer is:

[0105] ;

[0106] is the weight matrix of the fully connected layer, is the weight matrix of the input features, is the bias vector of the fully connected layer, is the output of the fully connected layer, The predicted value probability after weighted fusion of real-time running data.

[0107] The YOLOv7 model is specifically used to detect real-time image data outside the charging pile;

[0108] The real-time image data includes: fire and smoke;

[0109] It also includes: identifying the three states of the indicator light on the surface of the charging pile socket; among them, the three states include flashing green light, flashing red light and no indicator light detected; among them, the flashing green light means charging is in progress, the flashing red light means charging is complete, and the no indicator light detected means the charging is not in progress.

[0110] By comparing with the fault warning threshold, if the fault warning threshold is exceeded, it is determined that the operating status is abnormal, and the real-time operating data and real-time image data are sent to the cloud;

[0111] Among them, the long short-term memory deep learning model performs anomaly prediction based on the real-time operation data after weighted fusion; the YOLOv7 model is based on real-time image data; the real-time operation data includes the current, voltage and temperature inside the charging pile.

[0112] Specifically, the fault warning threshold in the model training module is calculated as follows:

[0113] First, we use the overall deviation between the actual value of the relevant feature variable and the model predicted value to define the deviation function. The formula is as follows:

[0114] ;

[0115] in, is the actual value of the weighted fusion feature variable at the current time t, is the weighted fusion eigenvalue predicted by the model, Represents the deviation at that moment, calculated using the mean square error (MSE);

[0116] Then the sliding window is used to calculate the average deviation and standard deviation , the formula is as follows:

[0117]

[0118]

[0119] in is the sliding window size, is the mean within the window, is the standard deviation within the window. Finally, according to the above mean and standard deviation To calculate the warning threshold:

[0120]

[0121] Among them, k is an adjustable parameter, usually selected between 1.5 and 3. is the calculated fault warning threshold; specifically, two thresholds are calculated based on the data of different operating states of the charging pile.

[0122] Data set for the "Not Charging" state , calculate the warning threshold , the formula is as follows:

[0123]

[0124] "Charging" status dataset , calculate the warning threshold , the formula is as follows:

[0125]

[0126] in 、 are the sliding mean and standard deviation of the deviation in the “not charging” state; 、 are the sliding mean and standard deviation of the deviation in the "charging" state; and Can be used for fault warning in "not charging" and "charging" states respectively;

[0127] In this embodiment, if Figure 2 As shown, the YOLOv7 model fire and smoke detection training process in the model training module is:

[0128] Obtain image data of the external environment of the charging pile, perform normalization and data enhancement processing, and divide the dataset into a 7:3 ratio, with 70% used as the training set and 30% used as the test set;

[0129] Build the basic framework of the YOLOv7 model;

[0130] Set the hyperparameter range and search for the best hyperparameters through cross-validation or optimization algorithms;

[0131] Train the YOLOv7 model using optimal hyperparameters and optimize weights to detect fire and smoke;

[0132] Verify the model accuracy and evaluate the precision, recall and average precision on the test set;

[0133] Set the classification threshold based on the test results;

[0134] The final output result is: fire or smoke is detected or fire or smoke is not detected.

[0135] The training process of YOLOv7 in the model training module to detect the status of the charging pile socket indicator light is as follows:

[0136] Data preprocessing: Obtain image data of the charging pile's external environment, perform normalization and data augmentation, and divide the dataset into a 7:3 ratio, with 70% used for training and 30% for testing.

[0137] Build the YOLOv7 model framework: According to the detection task requirements, build the basic framework of the YOLOv7 model, design the network structure and initialize the model.

[0138] Hyperparameter tuning: Set the hyperparameter range and search for the optimal hyperparameters through cross-validation or optimization algorithms (such as grid search and Bayesian optimization).

[0139] Train the YOLOv7 model: Use the optimal hyperparameters to train the YOLOv7 model and optimize the weights to detect the status of the charging station socket indicator light and identify three states: flashing green light, flashing red light, and no indicator light detected.

[0140] Model validation and evaluation: Evaluate the model's precision, recall, and average precision on the test set to verify the accuracy of the model.

[0141] Set the classification threshold: According to the detection results and the confidence of the model output, set the classification threshold to determine whether the target state is detected.

[0142] The final output result is: according to the detected category and confidence level, the status of the current socket indicator light is output, and the results are: "flashing green light", "flashing red light", "no indicator light detected";

[0143] Through these steps, the system can determine and output the status of the charging pile socket indicator light.

[0144] The monitoring module process is as follows Figure 3 As shown:

[0145] S1: Data Collection and Preprocessing: Sensors on the charging pile collect current, voltage, and temperature data every 5 seconds to monitor the internal operating status of the charging pile. Simultaneously, an external camera continuously captures images of the surrounding area of ​​the charging pile to identify early abnormalities such as fire and smoke. All collected data is filtered and standardized through a preprocessing module to ensure data quality and improve the accuracy of subsequent analysis.

[0146] S2: The YOLO module detects the indicator light of the charging pile socket. If the detected indicator light is "flashing green" or the current and voltage data collected by the sensor fluctuate, it means that the socket is charging the electric vehicle, and jump to step S3; otherwise, it means that the charging pile socket is in the "after charging" or "before charging" state, and jump to step S4;

[0147] S3: Set the LSTM warning threshold T to be equal to the threshold T2 after model 2 training, and jump to S5;

[0148] S4: Set the LSTM warning threshold T to be equal to the threshold T1 after model 1 training, and jump to S5;

[0149] S5: Model prediction and detection: The preprocessed sensor data (current, voltage, temperature) is weighted and fused. The resulting time series data is used as input. The LSTM deep learning model is used to perform time series prediction and output the predicted value of the weighted fusion data for the next time step. The YOLO model is also used to detect the presence of fire or smoke.

[0150] S6: Comprehensive Analysis: The monitoring module combines sensor data and image data for cross-modal analysis. First, analysis is performed based on sensor data, with the data sampling frequency set to once every 5 seconds. When the deviation Dt between the real-time monitoring value Tn and the predicted value Tp exceeds the set warning threshold T, the process jumps to step S8. Image data is also introduced for cross-modal fusion analysis. If abnormal conditions such as fire and smoke are detected through image recognition technology, the process also jumps to step S8. Otherwise, after saving the charging data and its analysis results, the process jumps to step S1 for re-monitoring.

[0151] S7: Warning Triggering and Information Transmission: Once a warning is triggered, the warning information and related image data are sent to the cloud for remote monitoring. Operations and maintenance personnel can respond promptly and take necessary measures to address any faults or safety hazards. After saving the charging data and its analysis results, the process proceeds to step S1 for the next monitoring and analysis.

[0152] Example 2:

[0153] like Figure 1 As shown, a vehicle charging pile monitoring device based on cross-modal intelligent sensing is provided, comprising: a data acquisition and preprocessing module, a model training module and a monitoring module;

[0154] The data acquisition and preprocessing module collects current, voltage, and temperature data from sensors on the charging pile, while also capturing image data using an external camera to monitor the charging pile's operating status and external environment in real time. This cross-modal sensing approach combines internal device operating status information with external environmental information, including information about smoke, flames, and whether the vehicle is still charging, providing more comprehensive monitoring. All data is then preprocessed. The model training module includes a long short-term memory (LSTM) network and a YOLOv7 deep learning model. The LSTM model is trained on a weighted fusion of time series data such as current, voltage, and temperature to predict the probability of future abnormalities and set fault warning thresholds based on historical data. The YOLOv7 model is also used to train image data to monitor abnormalities in the charging pile's external environment, enhancing fire risk prevention and control capabilities. It also adds the ability to monitor outlet indicator lights in real time to assist the monitoring module. The monitoring module comprehensively analyzes the internal real-time data and external image data of the charging pile to monitor the operating status of the charging pile. The charging pile operation mainly includes three operating states: "before charging", "charging" and "after charging". In the "before charging" and "charging" states, it focuses on monitoring current abnormalities, voltage instability, temperature exceeding the standard and environmental abnormalities; in the "after charging" state, it focuses on monitoring abnormal conditions such as incomplete charging due to battery failure or loose connection, and charging completed but the system did not automatically stop. The monitoring module realizes comprehensive monitoring of the operating status of the charging pile, and sends relevant information and image data to the cloud when the operating status is abnormal. The system improves the comprehensiveness and accuracy of charging pile monitoring, enhances user experience and the safety and stability of charging piles.

[0155] The data acquisition part of the data acquisition and preprocessing module performs timed sampling every 5 seconds through the current sensor, voltage sensor and temperature sensor installed on the charging pile, thereby continuously monitoring the internal operating status of the charging pile. At the same time, in this embodiment, an external camera is used to continuously collect image data of the external environment of the charging pile. The current sensor is used to detect the current changes of the charging pile in real time and identify abnormal current fluctuations, such as potential faults such as overload or short circuit; the voltage sensor is used to monitor the stability of the voltage to avoid unstable operation or even damage to the equipment due to excessive voltage fluctuations; the temperature sensor monitors the temperature of the interior of the charging pile and its surrounding environment to prevent safety problems such as poor heat dissipation or fire caused by high temperature. At the same time, the present invention uses an external camera to continuously collect image data of the external environment of the charging pile, identifies early signs of fire such as flames and smoke through intelligent visual monitoring technology, and uses image data to achieve cross-monitoring with sensor data, further enhancing the safety monitoring capabilities of the system.

[0156] The preprocessing portion of the data acquisition and preprocessing module utilizes multiple techniques to optimize data quality. First, a noise filtering algorithm is used to remove noise signals generated by the environment or electromagnetic interference from sensor data such as current, voltage, and temperature. Subsequently, the sensor data is standardized or normalized to unify data of different dimensions, enabling efficient processing by the deep learning model. Furthermore, denoising and binarization techniques are applied to image data to reduce noise interference and highlight key features, providing clear and reliable input data for subsequent deep learning analysis.

[0157] The LSTM deep learning model in the model training module includes:

[0158] Build the basic framework of the LSTM deep learning model, which includes the input layer, hidden layer, fully connected layer, and output layer, and initialize the parameters to obtain the initialized LSTM deep learning model;

[0159] Historical monitoring data for charging piles was obtained, normalized, and divided into datasets. Since the system needed to monitor both "charging" and "non-charging" states, two corresponding models were trained: Model 1 was used to train data on charging piles operating in the "non-charging" state, and Model 2 was used to train data on charging piles operating in the "charging" state. 70% of the data was allocated as a training set for model training, and 30% was allocated as a test set to verify the accuracy of the model's predictions.

[0160] The YOLOv7 model training in the model training module includes: using the monitoring video image data of the collection point as input, training the fire and smoke detection model, and outputting the fire and smoke detection results. After preprocessing the images captured by the surveillance camera, the trained YOLOv7 model is used to detect the fire and smoke in the image in real time. During the training process, the YOLOv7 model learns to identify these dangerous features by inputting a large amount of labeled fire and smoke image data, ensuring that early abnormal phenomena such as fire and smoke can be quickly and accurately identified in real-time monitoring. At the same time, the model training of real-time monitoring of socket indicator lights is added to assist in monitoring the operating status of the charging pile.

[0161] Specifically, the monitoring module primarily monitors the charging pile's three operating states: "before charging," "charging," and "after charging." This monitoring is achieved primarily by utilizing data from the charging pile's internal sensors: charging current, voltage, and temperature, as well as image data captured by an external camera. When the charging state is "before charging" or "after charging," the model is trained using data from the charging pile's "uncharging" state to detect abnormalities such as battery failure or loose connections causing the charging to fail to reach the expected level; abnormalities such as charging completion but the system not automatically stopping charging due to other reasons; and battery failures due to battery anomalies or excessive temperatures. When the charging state is "charging," the model is trained using data from the charging pile's "charging" state to detect abnormalities such as charging interruptions due to current overload, short circuit, or excessive current fluctuation; abnormalities such as charging damage due to voltage instability; and abnormalities such as charging pile or battery temperatures exceeding safe ranges, causing malfunctions or safety hazards. When an abnormal operating state is detected, the relevant information and image data are transmitted to the cloud.

[0162] Example 3:

[0163] A computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the electric vehicle charging pile monitoring method based on cross-modal intelligent sensing as described in Example 1 is implemented.

[0164] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0165] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for monitoring electric vehicle charging piles based on cross-modal intelligent sensing, characterized by: The following steps are involved: Acquire real-time operating data inside the charging pile and real-time image data outside the charging pile; and pre-process the real-time operating data and real-time image data; wherein the real-time operating data includes current, voltage, and temperature inside the charging pile; Perform weighted fusion on the operating data at the current moment and the previous period respectively to obtain the weighted fusion actual value at the current moment and the weighted fusion actual value in the previous period; Build and train long short-term memory deep learning models based on charging and non-charging conditions; According to the current charging state, the weighted fusion actual value in the period before the current moment is input into the trained long short-term memory deep learning model to predict the weighted fusion prediction value at the current moment; Build and train the YOLOv7 model, input the preprocessed real-time image data into the trained YOLOv7 model, and output the fire risk detection status; The sliding window technology is used to calculate the mean and standard deviation of historical data deviation, and two fault warning thresholds are set according to the current charging status; When the difference between the predicted value after weighted fusion and the actual value exceeds the fault warning threshold of the corresponding charging state or a fire risk is detected, it is determined that the operating state is abnormal.

2. The electric vehicle charging pile monitoring method based on cross-modal intelligent sensing according to claim 1 is characterized in that: The construction of the long short-term memory deep learning model includes constructing a long short-term memory deep learning model basic framework, and the long short-term memory deep learning model basic framework includes an input layer, a hidden layer, a fully connected layer and an output layer; The input layer is used to receive time series data; wherein, the time series data is obtained by weighted fusion of real-time operation data features, and the features after weighted fusion of real-time operation data are obtained by weighted fusion of real-time operation data. Expressed as: ； Current in real-time operation data, The voltage and is the temperature in the real-time operation data, and their weights are current The weight of ,Voltage The weight of ,temperature The weight of ; For the current time t, the features after weighted fusion of real-time running data Arrange the data to get the time series data, which is represented as: ； in is a vector of dimension [n,1], where n is the window size; The hidden layer includes multiple LSTM units; A single LSTM unit includes an input gate, a forget gate, and an output gate; The forget gate is used to control the information discarded in the memory unit state. The formula of the forget gate is: ； in, It is the Forget Gate Momentary status, is the weight matrix of the forget gate, is the bias vector of the forget gate, yes The hidden layer state at time t, It is a sigmoid function with an output value range of 0 to 1; The fully connected layer connects the output of the hidden layer to the output layer to generate the final prediction result; when the output of the hidden layer is assumed to be The formula for the fully connected layer is: ； is the weight matrix of the fully connected layer, is the weight matrix of the input features, is the bias vector of the fully connected layer, is the output of the fully connected layer, The predicted value probability after weighted fusion of real-time running data.

3. The electric vehicle charging pile monitoring method based on cross-modal intelligent sensing according to claim 2 is characterized in that: The input gate is used to update the information in the memory cell state. The input gate formula is: ； ； ； For Input the activation value of the gate at any moment, represents the weight matrix of the input gate, represents the bias vector of the input gate; is The new memory content of the moment, is the weight matrix of the memory unit, is the bias vector of the memory cell, is The state of the memory unit at the moment, is The state of the memory unit at the moment, yes Activation function.

4. The electric vehicle charging pile monitoring method based on cross-modal intelligent sensing according to claim 2 is characterized in that: The output gate is used to determine the hidden state output of the memory unit state. The output gate formula is: ； is the activation value of the output gate, is the weight matrix of the output gate, is the bias vector for the output gate.

5. The electric vehicle charging pile monitoring method based on cross-modal intelligent sensing according to claim 1 is characterized in that: The constructing and training of the long short-term memory deep learning model according to the charging and non-charging states includes: Two long short-term memory deep learning models, Model 1 and Model 2, are constructed and trained; wherein, Model 1 corresponds to the non-charging state, and Model 2 corresponds to the charging state.

6. The electric vehicle charging pile monitoring method based on cross-modal intelligent sensing according to claim 1 is characterized in that: The YOLOv7 model is used to detect real-time image data outside the charging pile; The real-time image data includes: fire and smoke; It also includes: identifying the three states of the indicator light on the surface of the charging pile socket; among them, the three states include flashing green light, flashing red light and no indicator light detected; among them, the flashing green light means charging is in progress, the flashing red light means charging is complete, and the no indicator light detected means the charging is not in progress.

7. The electric vehicle charging pile monitoring method based on cross-modal intelligent sensing according to claim 1 is characterized in that: The setting of two fault warning thresholds includes: The deviation function is defined by using the overall deviation between the features after weighted fusion of real-time running data and the model prediction value , the formula is: ； is the feature after weighted fusion of real-time running data at the current time t, It is the probability of the predicted value after weighted fusion of real-time running data; Calculate the average deviation using a sliding window and standard deviation , the formula is: ； ； is the size of the sliding window; The fault warning threshold The calculation formula is: ； k is an adjustable parameter, and the value range of k is 1.5-3.

8. The electric vehicle charging pile monitoring method based on cross-modal intelligent sensing according to claim 7 is characterized in that: The setting of two fault warning thresholds according to the current charging state includes: Set fault warning thresholds according to the non-charging state and the charging state of .

9. A monitoring device for electric vehicle charging piles based on cross-modal intelligent sensing, characterized in that: include: The data acquisition and preprocessing module is used to collect real-time operation data through sensors on the charging pile and obtain real-time image data using an external camera; And pre-process the real-time operation data and real-time image data; Model training module, used to build and train long short-term memory deep learning models and YOLOv7 models to monitor the external environment of charging piles; The monitoring module is used to analyze the real-time operation data and real-time image data of the charging pile to monitor the operation status of the charging pile according to the current charging status.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the electric vehicle charging pile monitoring method based on cross-modal intelligent sensing according to any one of claims 1 to 8 is implemented.

Images