A tram charging pile monitoring method and device based on cross-modal intelligent sensing and a storage medium

Abstract

This invention discloses a method, device, and storage medium for monitoring electric vehicle charging piles based on cross-modal intelligent sensing in the field of charging pile fault early warning technology. It aims to solve problems such as prolonged early warning time, incomplete monitoring, low efficiency, and insufficient fire prevention in charging pile monitoring. This method integrates image data acquired by charging pile sensors and external cameras, and uses a long short-term memory network and a YOLOv7 model to dynamically monitor the charging pile in three states: "before charging," "during charging," and "after charging." It focuses on electrical anomalies, excessive temperature, environmental anomalies, and charging anomalies after charging is completed, providing multi-dimensional, cross-modal operational monitoring and risk early warning to ensure the safe and efficient operation of charging piles. Simultaneously, the system can automatically upload relevant data and image information to the cloud when an anomaly is detected, achieving intelligent remote monitoring.

Description

Technical Field

[0001] This invention relates to a method, device, and storage medium for monitoring electric vehicle charging piles based on cross-modal intelligent sensing, belonging to the field of charging pile fault early warning technology. Background Technology

[0002] With the popularization and development of shared electric bicycles, charging stations, as an important supporting facility, are being used more and more widely. However, various malfunctions or abnormalities may occur during the operation of charging stations, such as overload, short circuit, leakage, and fire, which pose serious threats to the safety and property of users. Therefore, how to effectively monitor and provide early warning of the operation of charging stations, and promptly detect and handle malfunctions or abnormalities, has become an urgent technical problem to be solved.

[0003] Even when the charging station is not operational during charging, overcharging or failure to cut off current in time can cause the battery temperature to gradually rise and eventually lead to a fire. Therefore, monitoring the operational status of charging stations is essential. While faults and anomalies during the "charging" phase often receive attention, monitoring before and after charging is equally crucial. Before charging, loose battery interfaces or battery management system malfunctions can lead to poor contact and overheating. After charging, failure of the battery management system to cut off current in time or excessive temperature can also cause fires, explosions, and other safety accidents. Therefore, a comprehensive charging station monitoring system must cover the entire charging cycle, including all states before, during, and after charging, to ensure the stability and safety of the charging station.

[0004] Current electric vehicle charging station monitoring methods generally suffer from problems such as prolonged warning times, incomplete monitoring, low efficiency, and inadequate fire prevention. Traditional methods mainly rely on centralized data processing and simple judgment rules, which are insufficient to meet the complex and diverse needs of fault prediction. To improve the efficiency and accuracy of monitoring the operating status of charging stations and providing fault warnings, [further solutions are needed]. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the prior art and to solve the problems of prolonged early warning time, incomplete monitoring, low efficiency, and insufficient fire prevention in existing electric vehicle charging pile monitoring methods.

[0006] To solve the above-mentioned technical problems, the present invention is implemented using the following technical solution:

[0007] Firstly, a method for monitoring electric vehicle charging piles based on cross-modal intelligent sensing is provided, comprising the following steps:

[0008] The system acquires real-time operating data inside the charging pile and real-time image data outside the charging pile; and preprocesses the real-time operating data and real-time image data; wherein, the real-time operating data includes the current, voltage and temperature inside the charging pile;

[0009] We perform weighted fusion on the current time and the previous period of operation data to obtain the weighted fusion actual value at the current time and the weighted fusion actual value in the previous period of operation.

[0010] Long Short-Term Memory (LSTM) deep learning models are constructed and trained separately during charging and non-charging processes.

[0011] Based on the current charging state, the weighted fusion actual value of the previous period is input into the trained long short-term memory deep learning model to predict the weighted fusion prediction value of the current moment.

[0012] Build and train a YOLOv7 model, input preprocessed real-time image data into the trained YOLOv7 model, and output the fire risk detection status.

[0013] The mean and standard deviation of historical data deviation are calculated using sliding window technology, and two fault warning thresholds are set according to the current charging status.

[0014] When the difference between the weighted and fused predicted value and the actual value exceeds the fault warning threshold for the corresponding charging state or when a fire risk is detected, the operation is deemed abnormal.

[0015] Furthermore, the construction of the long short-term memory deep learning model includes building a basic framework for the long short-term memory deep learning model, which includes an input layer, a hidden layer, a fully connected layer, and an output layer.

[0016] The input layer is used to receive time series data; wherein, the time series data is obtained through weighted fusion of real-time running data features, and the weighted fusion of real-time running data features... Represented as:

[0017]

[0018] For the current in real-time running data, For real-time running data voltage and For the temperature in the real-time running data, their weights are respectively current. The weight is ,Voltage The weight is ,temperature The weight is ;

[0019] For the current time t, the features of the weighted and fused real-time running data The data is arranged to obtain time series data, which is represented as follows:

[0020] X=[ F t - 1 , F t - 2 , … , F t - n ]

[0021] in It is a vector of dimension [n, 1], where n is the window size;

[0022] The hidden layer includes multiple LSTM units;

[0023] A single LSTM unit includes an input gate, a forget gate, and an output gate;

[0024] The forget gate is used to control the information discarded from the state of the memory unit. The formula for the forget gate is:

[0025]

[0026] in, Is it the Gate of Oblivion? Current state It is the weight matrix of the forget gate. It is the bias vector of the forget gate. yes The hidden layer state at any given time. It is the sigmoid function whose output value ranges from 0 to 1;

[0027] The fully connected layer connects the output of the hidden layer to the output layer to generate the final prediction result; assuming the output of the hidden layer is... The formula for a fully connected layer is:

[0028] ;

[0029] It is the weight matrix of the fully connected layer. It is the weight matrix of the input features. It is the bias vector of the fully connected layer. It is the output of the fully connected layer. The probability of the predicted value after weighted fusion of real-time running data.

[0030] Furthermore, the input gate is used to update the information in the memory cell state, and the input gate formula is:

[0031] ;

[0032] ;

[0033] ;

[0034] In order to be in Enter the activation value of the gate at any time. This represents the weight matrix of the input gate. This represents the bias vector of the input gate; Is New memories of the moment It is the weight matrix of the memory units. It is the bias vector of the memory unit. Is The state of the memory unit at any given moment. Is The state of the memory unit at any given moment. yes Activation function.

[0035] Furthermore, the output gate is used to determine the hidden state output of the memory cell, and the output gate formula is:

[0036] ;

[0037] It is the activation value of the output gate. It is the weight matrix of the output gate. It is the bias vector of the output gate.

[0038] Furthermore, the step of constructing and training the long short-term memory deep learning model separately during charging and non-charging includes:

[0039] Two long short-term memory deep learning models, Model 1 and Model 2, are constructed and trained; Model 1 corresponds to the non-charging state, and Model 2 corresponds to the charging state.

[0040] Furthermore, the YOLOv7 model is used to detect real-time image data outside the charging pile;

[0041] The real-time image data includes: firelight and smoke;

[0042] It also includes: identifying three states of the indicator lights on the surface of the charging pile socket; the three states include flashing green light, flashing red light, and no indicator light detected; flashing green light indicates that charging is in progress, flashing red light indicates that charging is complete, and no indicator light detected indicates that charging is not in progress.

[0043] Furthermore, the setting of the two fault warning thresholds includes:

[0044] A deviation function is defined by utilizing the overall deviation between the weighted fusion of features from real-time running data and the model's predicted values. The formula is:

[0045]

[0046] It is a weighted and fused feature of the real-time running data at the current time t. It is the probability of the predicted value after weighted fusion of real-time running data;

[0047] Calculate the average deviation using a sliding window. and standard deviation The formula is:

[0048]

[0049]

[0050] It is the size of the sliding window;

[0051] The fault warning threshold The calculation formula is:

[0052]

[0053] k is an adjustable parameter, and its value ranges from 1.5 to 3.

[0054] Furthermore, the setting of two fault warning thresholds based on the current charging status includes:

[0055] Fault warning thresholds are set separately for non-charging and charging states. of .

[0056] Secondly, a monitoring device for electric vehicle charging piles based on cross-modal intelligent sensing is provided, comprising:

[0057] The data acquisition and preprocessing module is used to collect real-time operating data through sensors on the charging pile, acquire real-time image data using an external camera, and preprocess the real-time operating data and real-time image data.

[0058] The model training module is used to build and train long short-term memory deep learning models and YOLOv7 models to monitor the external environment of charging piles.

[0059] The monitoring module is used to analyze the real-time operation data and real-time image data of the charging pile to monitor the operation status of the charging pile according to the current charging status.

[0060] Thirdly, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the electric vehicle charging pile monitoring method based on cross-modal intelligent sensing as described in any one of the first aspects.

[0061] Compared with the prior art, the present invention has at least the following beneficial effects:

[0062] This invention significantly improves the comprehensiveness of charging pile monitoring by comprehensively analyzing cross-modal, multi-dimensional data. Compared to traditional methods, this invention utilizes an LSTM deep learning model to predict real-time operational data, thereby more effectively and accurately identifying potential abnormal states. Simultaneously, by monitoring real-time image data, it enhances the monitoring capability of the external environment. This invention not only improves the response speed to abnormal states but also reduces the frequency and cost of manual inspections through remote monitoring, thus improving the operational efficiency of charging piles. Attached Figure Description

[0063] Figure 1 The diagram shown is a module diagram of the charging pile monitoring system proposed in Embodiment 1 of the present invention;

[0064] Figure 2 The diagram shown is a flowchart of the LSTM model training process proposed in Embodiment 1 of the present invention.

[0065] Figure 3 The diagram shown is a flowchart of the monitoring module proposed in Embodiment 1 of the present invention. Detailed Implementation

[0066] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the embodiments and specific features in the embodiments are detailed descriptions of the technical solution of the present application, rather than limitations thereof. In the absence of conflict, the embodiments and technical features in the embodiments can be combined with each other.

[0067] In this article, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0068] Example 1:

[0069] like Figure 1As shown, this embodiment provides a method for monitoring electric vehicle charging piles based on cross-modal intelligent sensing, including the following steps:

[0070] Acquire real-time operating data inside the charging pile and real-time image data outside the charging pile; and preprocess the real-time operating data and real-time image data;

[0071] The data acquisition and preprocessing module collects current, voltage, and temperature data through sensors on the charging pile to monitor the internal operating status of the charging pile, and acquires image data through an external camera to monitor the external environment of the charging pile in real time. The data is then preprocessed for higher quality data analysis.

[0072] The model training module includes a Long Short-Term Memory (LSTM) deep learning model and a YOLOv7 deep learning model. The LSTM model uses the weighted fusion of current, voltage, and temperature data from the collection points as input time series for model training, and outputs the weighted fusion prediction values ​​of current, voltage, and temperature for the next time step. It also sets early warning fault thresholds based on historical data. The YOLOv7 model uses monitoring video image data from the collection points as input for model training, and outputs the detection results of fire and smoke. It also adds the ability to monitor socket indicator lights in real time to assist in monitoring the operating status of the charging pile.

[0073] The monitoring module performs comprehensive analysis based on real-time data and image data. When the error between the predicted value and the actual value exceeds a set threshold or a fire risk is detected, the system triggers an early warning and sends the warning information and relevant image data to the cloud.

[0074] The data acquisition section samples the voltage, current, and temperature data of the charging pile every 5 seconds, continuously acquiring monitoring signals generated internally within the charging pile during the cycle. Sensor technology is used to collect current, voltage, and temperature data. Current sensors monitor real-time current changes in the charging pile, detecting abnormal fluctuations in the electrical load, such as overloads, short circuits, and other potential faults. Voltage sensors monitor voltage stability; excessive voltage fluctuations can affect equipment operation and even cause damage. Temperature sensors detect temperature changes in the charging pile and its surrounding environment, especially potential heat dissipation failures caused by prolonged high-temperature operation, which could lead to fires and other safety issues. Cameras continuously collect image data of the external environment of the charging pile, using intelligent visual monitoring technology to identify early anomalies such as fires and smoke.

[0075] The preprocessing section employs multiple techniques to improve data quality and ensure the accuracy of subsequent analysis. First, noise filtering algorithms are used to denoise sensor data such as current, voltage, and temperature, eliminating noise signals caused by environmental factors and electromagnetic interference. Next, the sensor data is standardized or normalized to unify data of different dimensions to the same scale, facilitating efficient model processing and analysis. Furthermore, for image data, denoising algorithms and image binarization techniques are used to filter out noise and enhance key features, thus providing clearer and more effective input data for deep learning models. Through these preprocessing steps, irrelevant information and outliers in the original data are successfully removed, extracting valuable data and significantly improving the accuracy and quality of data analysis.

[0076] Subsequently, a long short-term memory deep learning model and a YOLOv7 model were constructed and trained, and fault warning thresholds were set based on historical data.

[0077] The LSTM model training process in the model training module is as follows: Figure 3 As shown, specifically:

[0078] We acquired internal data on the charging pile's operation in two states—"not charging" and "charging"—to train Model 1 and Model 2, respectively. The dataset for the "not charging" state is as follows: The dataset in the "charging" state is The data is normalized and the dataset is divided. Data augmentation is added to address the issue of insufficient dataset. Specifically, 70% of the dataset is divided into a training set for model training and 30% into a test set for validating the accuracy of the model's predictions.

[0079] The basic framework for building an LSTM deep learning model is constructed, which includes an input layer, hidden layers, fully connected layers, and an output layer. The hyperparameters are then initialized to obtain the initialized LSTM deep learning model.

[0080] Define the range of hyperparameters and use the particle swarm optimization algorithm to search for the optimal combination of hyperparameters;

[0081] Training the LSTM model: Construct and train the LSTM model using the optimal combination of hyperparameters;

[0082] Validate model prediction accuracy: Evaluate the model's loss on the test set to determine the model's predictive ability;

[0083] By utilizing the overall deviation between the actual values ​​of relevant feature variables and the model predictions, a deviation function is defined, and then the average deviation is calculated using a sliding window to determine the warning threshold.

[0084] The long short-term memory deep learning model is specifically characterized by an input layer, a hidden layer, a fully connected layer, and an output layer.

[0085] The input layer is used to receive time series data; wherein, the time series data is obtained through weighted fusion of real-time running data features, and the weighted fusion of real-time running data features... Represented as:

[0086]

[0087] For the current in real-time running data, For real-time running data voltage and For the temperature in the real-time running data, their weights are respectively current. The weight is ,Voltage The weight is ,temperature The weight is ;

[0088] For the current time t, the features of the weighted and fused real-time running data The data is arranged to obtain time series data, which is represented as follows:

[0089] X=[ F t - 1 , F t - 2 , … , F t - n ]

[0090] in It is a vector of dimension [n, 1], where n is the window size.

[0091] The hidden layer includes multiple LSTM units;

[0092] A single LSTM unit includes an input gate, a forget gate, and an output gate;

[0093] The forget gate is used to control the information discarded from the state of the memory unit. The formula for the forget gate is:

[0094]

[0095] in, Is it the Gate of Oblivion? Current state It is the weight matrix of the forget gate. It is the bias vector of the forget gate. yes The hidden layer state at any given time. It is the sigmoid function whose output value ranges from 0 to 1.

[0096] The input gate is used to update the information in the memory cell state, and the input gate formula is:

[0097] ;

[0098] ;

[0099] ;

[0100] In order to be in Enter the activation value of the gate at any time. This represents the weight matrix of the input gate. This represents the bias vector of the input gate; Is New memories of the moment It is the weight matrix of the memory units. It is the bias vector of the memory unit. Is The state of the memory unit at any given moment. Is The state of the memory unit at any given moment. yes Activation function.

[0101] Furthermore, the output gate is used to determine the hidden state output of the memory cell, and the output gate formula is:

[0102] ;

[0103] It is the activation value of the output gate. It is the weight matrix of the output gate. It is the bias vector of the output gate.

[0104] The fully connected layer connects the output of the hidden layer to the output layer to generate the final prediction result; assuming the output of the hidden layer is... The formula for a fully connected layer is:

[0105] ;

[0106] It is the weight matrix of the fully connected layer. It is the weight matrix of the input features. It is the bias vector of the fully connected layer. It is the output of the fully connected layer. The probability of the predicted value after weighted fusion of real-time running data.

[0107] The YOLOv7 model is specifically used to detect real-time image data outside the charging pile.

[0108] The real-time image data includes: firelight and smoke;

[0109] It also includes: identifying three states of the indicator lights on the surface of the charging pile socket; the three states include flashing green light, flashing red light, and no indicator light detected; flashing green light indicates that charging is in progress, flashing red light indicates that charging is complete, and no indicator light detected indicates that charging is not in progress.

[0110] By comparing with the fault warning threshold, when the fault warning threshold is exceeded, it is determined that the operation status is abnormal, and the real-time operation data and real-time image data are sent to the cloud.

[0111] The long short-term memory deep learning model predicts anomalies based on weighted and fused real-time operating data; the YOLOv7 model is based on real-time image data; and the real-time operating data includes the current, voltage, and temperature inside the charging pile.

[0112] Specifically, the fault warning threshold in the model training module is calculated as follows:

[0113] First, using the overall deviation between the actual values ​​of relevant feature variables and the model predictions, we define a deviation function, as shown in the following formula:

[0114] ;

[0115] in, It is the actual value of the weighted fusion feature variable at the current time t. These are the weighted fused feature values ​​predicted by the model. The deviation at that moment is represented by the mean squared error (MSE).

[0116] The average deviation was then calculated using a sliding window. and standard deviation The formula is as follows:

[0117]

[0118]

[0119] in It's the size of the sliding window. It is the mean within the window. This is the standard deviation within the window. Finally, based on the above mean... and standard deviation To calculate the warning threshold:

[0120]

[0121] Where k is an adjustable parameter, typically chosen between 1.5 and 3. The calculated fault warning threshold is used to determine the two thresholds based on data from different operating states of the charging pile.

[0122] Data set in the "not charging" state Calculate the warning threshold The formula is as follows:

[0123]

[0124] Data set in the "charging" state Calculate the warning threshold The formula is as follows:

[0125]

[0126] in , These are the moving mean and standard deviation of the deviation in the "not charging" state, respectively; where... , These are the moving mean and standard deviation of the deviation during the "charging" state, respectively; thus and It can be used for fault warnings in "not charging" and "charging" states, respectively;

[0127] In this embodiment, as Figure 2 As shown, the training process for the YOLOv7 model in the model training module to detect fire and smoke is as follows:

[0128] Image data of the external environment of the charging pile is acquired, normalized and data augmented, and the dataset is divided in a 7:3 ratio, with 70% used for training and 30% for testing.

[0129] The basic framework for building YOLOv7 models;

[0130] Define the range of hyperparameters and search for the optimal hyperparameters through cross-validation or optimization algorithms;

[0131] The YOLOv7 model was trained using optimal hyperparameters, and the weights were optimized to detect fire and smoke.

[0132] Validate model accuracy and evaluate precision, recall, and mean precision on the test set;

[0133] Set classification thresholds based on the test results;

[0134] The final output will be either: fire or smoke detected or no fire or smoke detected.

[0135] The training process for YOLOv7 in the model training module to detect the status of the charging pile socket indicator light is as follows:

[0136] Data preprocessing: Image data of the external environment of the charging pile is acquired, and normalization and data augmentation are performed. The dataset is divided in a 7:3 ratio, with 70% used for the training set and 30% for the test set.

[0137] Building the YOLOv7 model framework: Based on the requirements of the detection task, build the basic framework of the YOLOv7 model, design the network structure, and initialize the model.

[0138] Hyperparameter tuning: Set the range of hyperparameters and search for the best hyperparameters through cross-validation or optimization algorithms (such as grid search, Bayesian optimization, etc.).

[0139] Training the YOLOv7 model: The YOLOv7 model is trained using optimal hyperparameters to optimize weights for detecting the status of the charging station socket indicator lights, identifying three states: flashing green light, flashing red light, and no indicator light detected.

[0140] Model validation and evaluation: Evaluate the model's precision, recall, and mean precision on the test set to validate the model's accuracy.

[0141] Set classification threshold: Based on the detection results and the confidence level of the model output, set a classification threshold to determine whether the target state has been detected.

[0142] The final output is as follows: Based on the detected category and confidence level, the current status of the socket indicator light is output, and the results are: "Flashing green light", "Flashing red light", "No indicator light detected";

[0143] Through these steps, the system can determine and output the status of the charging pile socket indicator light.

[0144] The monitoring module process is as follows: Figure 3 As shown:

[0145] S1: Data Acquisition and Preprocessing: Sensors on the charging pile collect current, voltage, and temperature data every 5 seconds to monitor the internal operating status of the charging pile. Simultaneously, an external camera continuously captures images of the surrounding environment to identify early anomalies such as flames and smoke. All collected data undergoes noise filtering and standardization by a preprocessing module to ensure data quality and improve the accuracy of subsequent analysis.

[0146] S2: The YOLO module detects the indicator light status of the charging pile socket. If the detected indicator light is "flashing green" or the current and voltage data collected by the sensor fluctuates, it means that the socket is charging the electric vehicle, and proceeds to step S3; otherwise, it means that the charging pile socket is in the "after charging" or "before charging" state, and proceeds to step S4.

[0147] S3: Set the warning threshold T of LSTM to equal the threshold T2 after training Model 2, and then jump to S5;

[0148] S4: Set the warning threshold T of LSTM to equal the threshold T1 after training Model 1, and then jump to S5.

[0149] S5: Model Prediction and Detection: The preprocessed sensor data (current, voltage, temperature) is weighted and fused to obtain time series data as input. The time series is then predicted using an LSTM deep learning model, and the weighted fused data prediction value for the next time step is output. At the same time, the YOLO model is used to detect whether there is fire or smoke.

[0150] S6: Comprehensive Analysis: The monitoring module combines sensor data and image data for cross-modal analysis. First, it analyzes the sensor data, setting the data sampling frequency to once every 5 seconds. When the deviation Dt between the real-time monitored value Tn and the predicted value Tp exceeds the set warning threshold T, it proceeds to step S8. Simultaneously, it introduces image data for cross-modal fusion analysis. If abnormal conditions such as fire or smoke are detected through image recognition technology, it also proceeds to step S8. Otherwise, after saving the charging data and its analysis results, it proceeds to step S1 to re-monitor.

[0151] S7: Warning Triggering and Information Transmission: After a warning is triggered, the warning information and related image data are sent to the cloud for remote monitoring. Maintenance personnel can respond promptly and take necessary measures to address faults or safety hazards. After saving the charging data and its analysis results, proceed to step S1 for the next monitoring and analysis.

[0152] Example 2:

[0153] like Figure 1 As shown, a monitoring device for electric vehicle charging piles based on cross-modal intelligent sensing is provided, including: a data acquisition and preprocessing module, a model training module and a monitoring module;

[0154] The data acquisition and preprocessing module collects current, voltage, and temperature data through sensors on the charging pile, and simultaneously acquires image data using an external camera to monitor the charging pile's operating status and external environment in real time. This cross-modal sensing method combines internal operating status information with external environmental data, including smoke, fire, and whether the electric vehicle is still charging, providing more comprehensive monitoring. All data is then preprocessed. The model training module includes a Long Short-Term Memory (LSTM) network and a YOLOv7 deep learning model. The LSTM model is used to train a weighted fusion of time-series data (current, voltage, and temperature) to predict the probability of future anomalies and set fault warning thresholds based on historical data. The YOLOv7 model is used to train the image data to monitor anomalies in the charging pile's external environment, enhancing fire risk prevention capabilities. It also adds the ability to monitor socket indicator lights in real time to assist the monitoring module. The monitoring module comprehensively analyzes the charging pile's internal real-time data and external image data to monitor its operational status. The charging pile's operation mainly includes three states: "before charging," "during charging," and "after charging." In the "before charging" and "during charging" states, the module focuses on monitoring for issues such as abnormal current, unstable voltage, excessive temperature, and environmental anomalies. In the "after charging" state, the module focuses on monitoring for abnormal situations such as incomplete charging due to battery failure or loose connections, or charging completion without automatic system shutdown. The monitoring module achieves comprehensive monitoring of the charging pile's operational status and sends relevant information and image data to the cloud when abnormalities occur. This system improves the comprehensiveness and accuracy of charging pile monitoring, enhancing user experience and the safety and stability of the charging pile.

[0155] The data acquisition and preprocessing module's data acquisition section uses current sensors, voltage sensors, and temperature sensors installed on the charging pile to perform timed sampling every 5 seconds, thereby continuously monitoring the charging pile's internal operating status. Simultaneously, this embodiment utilizes an external camera to continuously acquire image data of the charging pile's external environment. The current sensor is used to detect real-time current changes in the charging pile and identify abnormal current fluctuations, such as potential faults like overload or short circuits; the voltage sensor monitors voltage stability to prevent unstable equipment operation or even damage due to excessive voltage fluctuations; the temperature sensor monitors the temperature of the charging pile's internal environment and its surroundings to prevent safety issues such as poor heat dissipation or fires caused by high temperatures. Furthermore, this invention utilizes an external camera to continuously acquire image data of the charging pile's external environment, identifying early signs of fire such as flames and smoke through intelligent visual monitoring technology, and cross-monitoring the image data with sensor data to further enhance the system's safety monitoring capabilities.

[0156] The preprocessing section of the data acquisition and preprocessing module utilizes various techniques to optimize data quality. First, for sensor data such as current, voltage, and temperature, noise filtering algorithms are used to remove noise signals generated by the environment or electromagnetic interference. Then, the sensor data is standardized or normalized to unify data of different dimensions, enabling efficient processing by deep learning models. Furthermore, for image data, denoising and binarization techniques are applied to reduce noise interference and highlight key features, thereby providing clear and reliable input data for subsequent deep learning analysis.

[0157] The LSTM deep learning model in the model training module includes:

[0158] The basic framework for building an LSTM deep learning model is constructed, which includes an input layer, hidden layers, fully connected layers, and an output layer. The parameters are then initialized to obtain the initialized LSTM deep learning model.

[0159] Historical monitoring data of charging piles is acquired, and the data is normalized and divided into datasets. Since the system needs to monitor both "charging" and "not charging" states, two corresponding models are trained. Model 1 is used to train on charging pile operation data in the "not charging" state, and Model 2 is used to train on charging pile operation data in the "charging" state. 70% of the dataset is allocated to the training set for model training, and 30% is allocated to the test set to verify the accuracy of the model predictions.

[0160] The YOLOv7 model training module includes: using monitoring video image data from the acquisition points as input to train a model for detecting fire and smoke, and outputting the detection results for fire and smoke. After preprocessing the images captured by the surveillance cameras, the trained YOLOv7 model is used to detect fire and smoke in the images in real time. During the training process, the YOLOv7 model learns to recognize these hazardous features by inputting a large amount of labeled fire and smoke image data, ensuring that it can quickly and accurately identify early anomalies such as fire and smoke in real-time monitoring. Simultaneously, model training for real-time monitoring of socket indicator lights is also added to assist in monitoring the operating status of charging piles.

[0161] Specifically, the monitoring module primarily monitors three operating states of the charging pile: "before charging," "during charging," and "after charging." It utilizes data from three sensors inside the charging pile—current, voltage, and temperature—as well as image data collected by an external camera. When the charging state is "before charging" or "after charging," the module uses operating data from when the charging pile is "not charging" for model training to detect abnormal situations such as battery failure or loose connections preventing the expected charging level from being reached; abnormal situations where charging is completed but the system does not automatically stop charging; and abnormal situations where battery failure occurs due to battery malfunction or excessive temperature. When the charging state is "during charging," the module uses operating data from when the charging pile is "charging" for model training to detect abnormal charging interruptions due to current overload, short circuits, or excessive current fluctuations; abnormal charging damage due to unstable voltage; and abnormal situations where the charging pile or battery temperature exceeds safe limits, leading to malfunctions or safety hazards. When an abnormal operating state occurs, relevant information and image data will be sent to the cloud.

[0162] Example 3:

[0163] A computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the electric vehicle charging pile monitoring method based on cross-modal intelligent sensing as described in Embodiment 1.

[0164] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0165] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for monitoring electric vehicle charging piles based on cross-modal intelligent sensing, characterized in that: Includes the following steps: The system acquires real-time operating data inside the charging pile and real-time image data outside the charging pile; and preprocesses the real-time operating data and real-time image data; wherein, the real-time operating data includes the current, voltage and temperature inside the charging pile; We perform weighted fusion on the current time and the previous period of operation data to obtain the weighted fusion actual value at the current time and the weighted fusion actual value in the previous period of operation. Long Short-Term Memory (LSTM) deep learning models are constructed and trained separately during charging and non-charging processes. Based on the current charging state, the weighted fusion actual value of the previous period is input into the trained long short-term memory deep learning model to predict the weighted fusion prediction value of the current moment. Build and train a YOLOv7 model, input preprocessed real-time image data into the trained YOLOv7 model, and output the fire risk detection status. The mean and standard deviation of historical data deviation are calculated using sliding window technology, and two fault warning thresholds are set according to the current charging status. When the difference between the weighted and fused predicted value and the actual value exceeds the fault warning threshold for the corresponding charging state or when a fire risk is detected, the operation is deemed abnormal.

2. The method for monitoring electric vehicle charging piles based on cross-modal intelligent sensing according to claim 1, characterized in that, The construction of the long short-term memory deep learning model includes building a basic framework for the long short-term memory deep learning model, which includes an input layer, a hidden layer, a fully connected layer, and an output layer. The input layer is used to receive time series data; wherein, the time series data is obtained through weighted fusion of real-time running data features, and the weighted fusion of real-time running data features... Represented as: ； For the current in real-time running data, For real-time running data voltage and For the temperature in the real-time running data, their weights are respectively current. The weight is ,Voltage The weight is ,temperature The weight is ; For the current time t, the features of the weighted and fused real-time running data The data is arranged to obtain time series data, which is represented as follows: ； in It is a vector of dimension [n, 1], where n is the window size; The hidden layer includes multiple LSTM units; A single LSTM unit includes an input gate, a forget gate, and an output gate; The forget gate is used to control the information discarded from the state of the memory unit. The formula for the forget gate is: ； in, Is it the Gate of Oblivion? Current state It is the weight matrix of the forget gate. It is the bias vector of the forget gate. yes The hidden layer state at any given time. It is the sigmoid function whose output value ranges from 0 to 1; The fully connected layer connects the output of the hidden layer to the output layer to generate the final prediction result; assuming the output of the hidden layer is... The formula for a fully connected layer is: ； It is the weight matrix of the fully connected layer. It is the weight matrix of the input features. It is the bias vector of the fully connected layer. It is the output of the fully connected layer. The probability of the predicted value after weighted fusion of real-time running data.

3. The method for monitoring electric vehicle charging piles based on cross-modal intelligent sensing according to claim 2, characterized in that, The input gate is used to update the information in the memory cell state, and the input gate formula is: ； ； ； In order to be in Enter the activation value of the gate at any time. This represents the weight matrix of the input gate. This represents the bias vector of the input gate; Is New memories of the moment It is the weight matrix of the memory units. It is the bias vector of the memory unit. Is The state of the memory unit at any given moment. Is The state of the memory unit at any given moment. yes Activation function.

4. The method for monitoring electric vehicle charging piles based on cross-modal intelligent sensing according to claim 2, characterized in that, The output gate is used to determine the hidden state output of the memory cell. The output gate formula is: ； It is the activation value of the output gate. It is the weight matrix of the output gate. It is the bias vector of the output gate.

5. The method for monitoring electric vehicle charging piles based on cross-modal intelligent sensing according to claim 1, characterized in that, The process of constructing and training long short-term memory deep learning models based on charging and non-charging states includes: Two long short-term memory deep learning models, Model 1 and Model 2, are constructed and trained; Model 1 corresponds to the non-charging state, and Model 2 corresponds to the charging state.

6. The method for monitoring electric vehicle charging piles based on cross-modal intelligent sensing according to claim 1, characterized in that, The YOLOv7 model is used to detect real-time image data outside the charging pile; The real-time image data includes: firelight and smoke; It also includes: identifying three states of the indicator lights on the surface of the charging pile socket; the three states include flashing green light, flashing red light, and no indicator light detected; flashing green light indicates that charging is in progress, flashing red light indicates that charging is complete, and no indicator light detected indicates that charging is not in progress.

7. The method for monitoring electric vehicle charging piles based on cross-modal intelligent sensing according to claim 1, characterized in that, The two types of fault warning thresholds include: A deviation function is defined by utilizing the overall deviation between the weighted fusion of features from real-time running data and the model's predicted values. The formula is: ； It is a weighted and fused feature of the real-time running data at the current time t. It is the probability of the predicted value after weighted fusion of real-time running data; Calculate the average deviation using a sliding window. and standard deviation The formula is: ； ； It is the size of the sliding window; The fault warning threshold The calculation formula is: ； k is an adjustable parameter, and its value ranges from 1.5 to 3.

8. The method for monitoring electric vehicle charging piles based on cross-modal intelligent sensing according to claim 7, characterized in that, The two fault warning thresholds set according to the current charging status include: Fault warning thresholds are set separately for non-charging and charging states. of .

9. A monitoring device for electric vehicle charging piles based on cross-modal intelligent sensing, characterized in that, include: The data acquisition and preprocessing module is used to collect real-time operating data through sensors on the charging pile and to acquire real-time image data using an external camera. It also preprocesses real-time running data and real-time image data; The model training module is used to build and train long short-term memory deep learning models and YOLOv7 models to monitor the external environment of charging piles. The monitoring module is used to analyze the real-time operation data and real-time image data of the charging pile to monitor the operation status of the charging pile according to the current charging status.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the electric vehicle charging pile monitoring method based on cross-modal intelligent sensing as described in any one of claims 1-8.

Images