A charging station anomaly detection method based on multi-modal feature fusion

Abstract

The application discloses a charging station anomaly detection method based on multi-modal feature fusion, and relates to the technical field of charging facility operation monitoring, and comprises the following steps: S1, collecting visual and equipment operation data; S2, preprocessing and space-time alignment, and dividing a visual region of interest; S3, converting the equipment operation data into an operation state feature map and associating the visual data; S4, inputting an improved DINOv2 network, generating multi-modal visual features through part alignment and cross-modal masking reconstruction; S5, part corresponding fusion; S6, inputting a PatchCore algorithm, generating a candidate result through time sequence Patch trajectory matching and a normal feature memory bank; and S7, outputting an abnormal result and alarm information after multi-modal consistency verification. The application adopts the improved DINOv2 network and the PatchCore algorithm, and realizes precise detection of charging station anomalies.

Description

Technical Field

[0001] This invention relates to the field of charging facility operation monitoring technology, and in particular to a method for detecting anomalies in charging stations based on multimodal feature fusion. Background Technology

[0002] With the continuous growth of new energy vehicle ownership and the number of public charging infrastructures, the operational safety and equipment anomaly monitoring of charging stations have received widespread attention. Existing charging station safety monitoring systems typically deploy multiple high-definition cameras and edge computing devices within the station, using deep learning models to analyze real-time video data, identifying flames, smoke, personnel anomalies, charging gun malfunctions, and environmental hygiene incidents, and uploading alarm information and key video clips to a central control platform. Regarding the charging equipment's own operating status, existing technologies also use fixed thresholds for parameters such as voltage, current, power, and temperature to determine equipment anomalies. However, existing intelligent video monitoring systems primarily detect the station environment and visible safety events. For anomalies such as overheating, abnormal contact, mechanical loosening, and cable damage within the charging equipment and its components, the following problems still exist: Charging equipment anomalies typically manifest simultaneously as changes in appearance, localized temperature rise, electrical fluctuations, and changes in sound vibration. A single data source is insufficient to fully reflect the abnormal state, easily leading to missed detections of early overheating, component loosening, and contact anomalies. Visible light images, infrared thermal images, and equipment operating data have different sampling frequencies and spatial locations, making it difficult for existing methods to accurately map different modal information to the same equipment component, easily resulting in feature mismatches. Traditional visual feature extraction methods suffer from poor stability when there is vehicle obstruction, environmental reflection, changes in lighting, or missing data, and abnormal features are easily masked by background interference. Existing patch anomaly detection methods mostly rely on static matching of local features at a single moment with normal samples, failing to fully consider the continuous development of anomalies over time, making it difficult to distinguish between instantaneous operational fluctuations and persistent equipment anomalies. Furthermore, single detection results lack multimodal consistency verification, easily misidentifying short-term obstruction, environmental noise, and normal load changes as anomalies, affecting the accuracy and reliability of alarm results.

[0003] Therefore, how to provide a charging station anomaly detection method based on multimodal feature fusion is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0004] One objective of this invention is to propose a charging station anomaly detection method based on multimodal feature fusion. This invention employs a combination of an improved DINOv2 network and the PatchCore algorithm. It constructs standardized multimodal visual samples by spatiotemporally aligning visible light images, infrared thermal images, electrical operation data, and sound vibration data. The improved DINOv2 network introduces a component alignment mechanism and cross-modal occlusion reconstruction, accurately mapping different modal features to the same charging equipment component, and utilizing other modal information to complete missing or interfered patch features. Furthermore, it identifies continuous changes in anomalies through temporal patch trajectory matching combined with a normal feature memory, and uses multimodal consistency verification to eliminate false anomaly responses, achieving accurate detection, location, and alarm of charging equipment anomalies. This method boasts advantages such as high detection accuracy, low false alarm rate, strong anti-interference capability, and applicability to online monitoring.

[0005] An anomaly detection method for charging stations based on multimodal feature fusion according to an embodiment of the present invention includes the following steps: S1. Collect visual perception data and equipment operation data during the operation of the charging station to generate a multimodal raw dataset of the charging station; S2. Perform data preprocessing and spatiotemporal alignment on the original multimodal dataset of the charging station to determine the visual region of interest corresponding to the charging equipment and generate multimodal aligned data; S3. Convert the device operation data in the multimodal alignment data into an operation status feature map, and associate the operation status feature map with the visual perception data to generate a standardized multimodal visual sample set; S4. Input the standardized multimodal visual sample set into the improved DINOv2 network, and improve the DINOv2 network by setting a component alignment mechanism and cross-modal occlusion reconstruction; based on the component alignment mechanism, arrange the patch features of different modalities according to their corresponding components, and restore the occluded patch features through cross-modal occlusion reconstruction to generate multimodal visual features. S5. Based on the correspondence between the components of the charging device, the multimodal visual features are fused to generate multimodal fused visual features of the charging device. S6. Input the multimodal fusion visual features of the charging device into the PatchCore algorithm, associate the Patch features of the corresponding Patch positions in continuous time, obtain the trajectory matching deviation based on temporal Patch trajectory matching, and perform feature matching between the Patch features and the normal feature memory to generate candidate anomaly detection results. S7. Perform multimodal consistency verification on the candidate anomaly detection results, and generate the final anomaly detection results and anomaly alarm information for the charging station.

[0006] Optionally, S1 specifically includes: Obtain the charging device identifiers of each charging device in the charging station and establish the correspondence between the collection location and the charging device identifiers; Collect visible light image data and infrared thermal image data during the operation of the charging station, add the collection time, collection location and charging equipment identification, and generate visual perception data; Collect electrical operation data and sound vibration data during the operation of the charging station, add the collection time, collection location and charging equipment identification, and generate equipment operation data; The visual perception data and equipment operation data are correlated based on the collection time and charging equipment identification, and a data index is established according to the charging equipment identification and collection time to generate a multimodal raw dataset of the charging station.

[0007] Optionally, S2 specifically includes: Read the visual perception data and equipment operation data from the multimodal raw dataset of the charging station, and extract the collection time, collection location and charging equipment identifier corresponding to the visual perception data and equipment operation data; Visual perception data is denoised, sized, and distorted; equipment operation data is processed for missing values, outliers are removed, and sampling frequency is standardized. Based on the acquisition time, the visual perception data and equipment operation data are corrected for time deviation and resampled to form a time-corresponding data sequence. Based on the collection location and charging device identification, the visual perception data and device operation data are mapped to the corresponding charging devices; Visible light image data and infrared thermal image data are extracted from visual perception data. The outline of the charging device in the visible light image data and the boundary of the temperature region in the infrared thermal image data are extracted, and spatial registration is performed based on the outline of the charging device and the boundary of the temperature region. Based on the outline of the charging device and the location of its components, the visual region of interest is divided. The visual perception data that completes time correspondence, device mapping and spatial registration is correlated with the device operation data to generate multimodal alignment data.

[0008] Optionally, S3 specifically includes: Electrical operation data and sound vibration data are extracted from the equipment operation data in the multimodal aligned data, and then continuously segmented according to a preset time window; Extract the numerical change sequence of electrical operation data within each time window, normalize the numerical change sequence, and arrange it into a two-dimensional matrix according to the acquisition time to generate an electrical time series feature map. Short-time Fourier transform is performed on the sound vibration data within each time window, and the obtained spectral amplitudes are arranged according to time and frequency to generate a sound vibration time-frequency diagram. The electrical timing feature map and the sound vibration time-frequency map are standardized in size, and the channels are combined according to the acquisition time and the charging equipment identification to generate the operating status feature map; The operating status feature map is associated with the visual perception data based on the collection time and the charging device identification. The samples are divided and standardized according to the visual region of interest to generate a standardized multimodal visual sample set.

[0009] Optionally, the improved DINOv2 network includes a multimodal embedding module, a component alignment module, an occlusion reconstruction module, a feature encoding module, and a feature output module. The component alignment module embeds a component alignment mechanism, and the occlusion reconstruction module sets up cross-modal occlusion reconstruction. The multimodal embedding module divides the visual perception data and running state feature map in the standardized multimodal visual sample set into multiple patches, converts each patch into a patch vector, maps the patch vectors of different modalities to the same feature dimension through a linear projection layer, and superimposes the codes corresponding to the patch position, acquisition time, charging device identifier and modality category to generate multimodal patch features. The component alignment module determines the charging device component corresponding to the multimodal patch feature based on the visual region of interest, divides the multimodal patch features corresponding to the same charging device component and the acquisition time into component feature groups, and arranges the multimodal patch features in each component feature group based on the component alignment mechanism to generate a component aligned patch sequence. The occlusion reconstruction module selects target patch features from the component-aligned patch sequence, replaces the target patch features with occlusion markers, and generates an occlusion patch sequence. The feature encoding module performs multi-layer feature encoding on the masked patch sequence input to multiple sequentially connected Transformer encoding layers to generate masked patch encoded features. The occlusion reconstruction module is based on cross-modal occlusion reconstruction. It extracts the remaining modal coding features in the occlusion patch coding features that belong to the same component feature group as the target patch features, maps the remaining modal coding features to the modal feature space to which the target patch features belong, generates the target patch reconstruction features, and replaces the corresponding occlusion markers with the target patch reconstruction features to generate multi-layer patch coding features. The feature output module restores and arranges the multi-layer patch-encoded features according to the visual region of interest and modal category, retains the correspondence between the multi-layer patch-encoded features and the charging device identifier and the acquisition time, and generates multi-modal visual features.

[0010] Optionally, the component alignment mechanism specifically includes: In the component alignment module, the patch position, acquisition time, charging device identifier and modality category corresponding to each multimodal patch feature are read, and the region boundary of the visual region of interest is obtained. The spatial coverage area corresponding to each multimodal patch feature is determined based on the patch location. The ratio of the overlap area between the spatial coverage area and each visual region of interest to the spatial coverage area is calculated to generate the region overlap ratio. The region of interest with the largest overlap ratio and reaching the preset region matching value is identified as the region of interest to which the multimodal patch feature belongs, and the multimodal patch feature with the largest overlap ratio but not reaching the preset region matching value is marked as the background patch feature. Multimodal patch features other than background patch features are grouped according to the charging device identifier, acquisition time, and region of interest. Multimodal patch features with the same charging device identifier, acquired at the same time, and belonging to the same region of interest are grouped into the same component feature group. Establish the modality arrangement order according to modality category, and arrange the multimodal patch features of the same modality category according to the spatial order of the patch position in the visual region of interest; when the part feature group lacks a multimodal patch feature of the corresponding modality category, write a missing mark in the corresponding arrangement position; The completed component feature groups are bound to the charging device identifier, acquisition time, and visual region of interest to generate a component alignment index. Then, the multimodal patch features and gap markers are connected according to the component alignment index to generate a component alignment patch sequence.

[0011] Optionally, S5 specifically includes: Read multimodal visual features and their corresponding charging device identifiers, acquisition times, regions of interest, and modal categories; Based on the charging device identification, the time of data collection, and the region of visual interest, the multimodal visual features belonging to the same charging device component are divided into component visual feature groups. The feature dimensions of the multimodal visual features in each component's visual feature group are unified, and they are arranged according to modality category and patch position to generate a component multimodal feature sequence. Extract the effective modal features corresponding to the same patch position in the multimodal feature sequence of the component, perform a dimension-wise average of the effective modal features to generate common features, and perform a dimension-wise subtraction between each effective modal feature and the common features to generate difference features; By concatenating and linearly mapping the common features with the individual differences, component fusion features are generated. The correspondence between components of the charging device is determined based on the spatial location of each visual region of interest in the charging device. The fusion features of each component are arranged according to the correspondence, and the arranged component fusion features are spliced ​​together to generate multimodal fusion visual features of the charging device.

[0012] Optionally, S6 specifically includes: The multimodal fusion visual features of the charging device corresponding to the normal charging process are obtained. The normal Patch feature set is divided according to the charging device identifier, visual region of interest and Patch position. Core set sampling is performed on each normal Patch feature set to obtain representative Patch features. A normal feature memory is constructed based on the representative Patch features. Based on the acquisition time, normal patch features with the same visual region of interest and patch position in each normal patch feature set are associated, the feature change between adjacent normal patch features is calculated, and the normal patch features are combined with the corresponding feature change to generate a normal temporal patch trajectory set. Read the multimodal fusion visual features of the charging device to be detected, extract the features of the patch to be detected according to the patch position, associate the features of the patch to be detected with the same visual region of interest and patch position in consecutive time according to the acquisition time, calculate the feature change between adjacent features of the patch to be detected, and combine the features of the patch to be detected with the corresponding feature change to generate a temporal patch trajectory. Based on time-series patch trajectory matching, the time-series patch trajectory is matched with the normal time-series patch trajectory set to generate trajectory matching deviation; Each patch feature to be detected is matched with the representative patch features in the normal feature memory, and a local matching bias is generated based on the minimum feature distance. The trajectory matching deviation and local matching deviation are normalized and added together to determine the abnormal response value at each patch position. The abnormal response values ​​are then restored and arranged according to the patch position to generate a pixel-level abnormal response map. Anomaly regions are extracted from the pixel-level anomaly response map. Anomaly components are determined based on the region of interest to which the anomaly region belongs. Feature aggregation is performed on the features of the patch to be detected covered by the anomaly region to generate anomaly region features. Obtain an anomaly type feature library based on the anomaly region features of labeled anomaly samples, perform feature matching between the anomaly region features and the anomaly type feature library, and determine the anomaly category corresponding to the minimum feature distance as the anomaly type; The anomaly level is determined based on the preset range of the abnormal response value within the abnormal area, and the maximum abnormal response value within the abnormal area is normalized to the recognition confidence level. The charging device identifier, acquisition time, visual region of interest, abnormal component, anomaly type, anomaly level, and recognition confidence level corresponding to the abnormal area are combined to generate candidate anomaly detection results.

[0013] Optionally, the time-series patch-based trajectory matching specifically includes: The features of each patch to be detected in the temporal patch trajectory and the corresponding feature change amount are combined into trajectory nodes according to the acquisition time, and the trajectory nodes are connected in the order of acquisition time. Based on the charging device identifier, the visual region of interest, and the patch location, the corresponding normal time-series patch trajectory is selected from the normal time-series patch trajectory set to generate a candidate normal time-series patch trajectory set. The time boundaries of the time-series patch trajectory and each candidate normal time-series patch trajectory are unified, and the missing trajectory nodes are interpolated and filled in according to the acquisition time to obtain the time-aligned trajectory corresponding to each candidate normal time-series patch trajectory. The differences in patch features and the differences in feature change between each trajectory node in the temporal patch trajectory and each trajectory node in the time-aligned trajectory are calculated sequentially. The differences in patch features and the differences in feature change are normalized and then added one by one to generate a trajectory matching matrix. Establish monotonic matching paths in the trajectory matching matrix according to the increasing direction of the acquisition time, accumulate the difference value corresponding to the position passed by each monotonic matching path, and determine the monotonic matching path with the smallest accumulated difference value as the target matching path. Extract the difference values ​​of each corresponding trajectory node in the target matching path, calculate the absolute change of the difference values ​​between adjacent corresponding trajectory nodes, calculate the mean difference value and the mean absolute change of the difference value respectively, add the mean difference value and the mean absolute change of the difference value to generate a single trajectory matching result; The matching results of individual tracks corresponding to the time-series patch tracks and each normal time-series patch track in the candidate normal time-series patch track set are obtained respectively, and the single track matching result with the smallest value is determined as the track matching deviation.

[0014] Optionally, S7 specifically includes: Read the candidate anomaly detection results, and extract the visual perception data and device operation data corresponding to the anomaly region from the multimodal alignment data based on the charging device identifier, acquisition time, and visual region of interest; Visible light image data and infrared thermal image data are extracted from visual perception data, and the image changes and temperature changes of the abnormal area at continuous acquisition time are calculated, respectively; electrical operation data and sound vibration data are extracted from equipment operation data, and the electrical changes and sound vibration changes at the corresponding acquisition time are calculated, respectively. Obtain visual perception data and equipment operation data corresponding to historical normal charging processes, and determine the corresponding normal change intervals based on the distribution of image changes, temperature changes, electrical changes and sound vibration changes in data with the same charging equipment identification and visual region of interest. Obtain a preset anomaly verification table, which records the correspondence between different anomaly types and target data categories and support quantity thresholds; query the preset anomaly verification table based on the anomaly types in the candidate anomaly detection results to determine the target data categories and support quantity thresholds for participating in multimodal consistency verification; The change amount corresponding to the target data category is compared with the corresponding normal change range. When the change amount exceeds the corresponding normal change range, the corresponding anomaly support flag is set to one. When the change amount is within the corresponding normal change range, the corresponding anomaly support flag is set to zero. The number of anomaly support labels with a value of one in the target data category is counted. When the number of anomaly support labels reaches the support number threshold, the candidate anomaly detection result is retained; when the number of anomaly support labels does not reach the support number threshold, the candidate anomaly detection result is marked as a pseudo-anomaly. The candidate anomaly detection results are merged according to the abnormal component and the time of data collection to generate the final anomaly detection result of the charging station. Anomaly alarm information is generated based on the abnormal component, anomaly type, anomaly level and identification confidence level.

[0015] The beneficial effects of this invention are: This invention addresses the issues of inconsistent sampling frequencies, spatial locations, and incomplete single-modal information in charging station anomaly detection by improving the DINOv2 network and the PatchCore algorithm through cascading integration. It employs unified time-base calibration, device identifier mapping, and visual region of interest (ROI) segmentation to achieve spatiotemporal alignment of visible light images, infrared thermal images, electrical operation data, and acoustic vibration data. Furthermore, it converts device operation data into operational status feature maps, constructing standardized multimodal visual samples. A component alignment mechanism is introduced in the feature extraction stage, accurately mapping different modal features to the same charging device component based on the overlap ratio between the patch spatial coverage and the ROI, thus reducing cross-modal positional errors. This method addresses mismatches and background interference, while simultaneously recovering missing or interfered target patch features by utilizing the modal coding features of other components through cross-modal occlusion reconstruction. This improves feature integrity under complex lighting, partial occlusion, and data loss conditions. In the anomaly detection stage, a normal feature memory is constructed, and temporal patch trajectory matching is introduced. By combining trajectory matching deviation and local matching deviation, an anomaly response map is generated to identify the continuous development process of anomalies over time. Finally, based on the anomaly type, the corresponding data category is selected for multimodal consistency verification, eliminating false anomalies caused by instantaneous load fluctuations, environmental reflections, and noise interference. This achieves accurate output of abnormal components, anomaly types, anomaly levels, and recognition confidence of the charging equipment. Attached Figure Description

[0016] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is an overall flowchart of a charging station anomaly detection method based on multimodal feature fusion proposed in this invention; Figure 2 This is a diagram of the improved DINOv2 network structure proposed in this invention; Figure 3 This is a structural diagram of the PatchCore algorithm based on temporal Patch trajectory matching proposed in this invention. Detailed Implementation

[0017] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0018] refer to Figures 1-3 A charging station anomaly detection method based on multimodal feature fusion includes the following steps: S1. Collect visual perception data and equipment operation data during the operation of the charging station to generate a multimodal raw dataset of the charging station; S2. Perform data preprocessing and spatiotemporal alignment on the original multimodal dataset of the charging station to determine the visual region of interest corresponding to the charging equipment and generate multimodal aligned data; S3. Convert the device operation data in the multimodal alignment data into an operation status feature map, and associate the operation status feature map with the visual perception data to generate a standardized multimodal visual sample set; S4. Input the standardized multimodal visual sample set into the improved DINOv2 network, and improve the DINOv2 network by setting a component alignment mechanism and cross-modal occlusion reconstruction; based on the component alignment mechanism, arrange the patch features of different modalities according to their corresponding components, and restore the occluded patch features through cross-modal occlusion reconstruction to generate multimodal visual features. S5. Based on the correspondence between the components of the charging device, the multimodal visual features are fused to generate multimodal fused visual features of the charging device. S6. Input the multimodal fusion visual features of the charging device into the PatchCore algorithm, associate the Patch features of the corresponding Patch positions in continuous time, obtain the trajectory matching deviation based on temporal Patch trajectory matching, and perform feature matching between the Patch features and the normal feature memory to generate candidate anomaly detection results. S7. Perform multimodal consistency verification on the candidate anomaly detection results, and generate the final anomaly detection results and anomaly alarm information for the charging station.

[0019] In this embodiment, S1 specifically refers to: Obtain the charging device identifiers of each charging device in the charging station and establish the correspondence between the collection location and the charging device identifiers; Collect visible light image data and infrared thermal image data during the operation of the charging station, add the collection time, collection location and charging equipment identification, and generate visual perception data; Collect electrical operation data and sound vibration data during the operation of the charging station, add the collection time, collection location and charging equipment identification, and generate equipment operation data; The visual perception data and equipment operation data are correlated based on the collection time and charging equipment identification, and a data index is established according to the charging equipment identification and collection time to generate a multimodal raw dataset of the charging station. In the specific implementation process, multiple visible light cameras, infrared thermal imagers, and edge computing devices are deployed at the charging stations. The multiple visible light cameras are arranged according to the spatial distribution of charging spaces and charging equipment, with each visible light camera covering one or more charging spaces. The infrared thermal imagers are set in front of or above the side of the charging equipment, and the edge computing devices are connected to the visible light cameras, infrared thermal imagers, and equipment operation data acquisition interfaces, respectively.

[0020] Each charging device is assigned a unique identifier, consisting of a site number and a device number. A mapping relationship between the acquisition location and the charging device identifier is established based on the installation coordinates, field of view, and coverage area of ​​each acquisition device. The resolution of visible light image data is set to 1920×1080 pixels, and the acquisition frequency is set to 25 frames / second; the resolution of infrared thermal image data is set to 640×512 pixels, and the acquisition frequency is set to 10 frames / second. The image frames, acquisition time, acquisition location, and charging device identifier are combined to generate visual perception data. Camera channel identifiers are configured for each visible light camera and infrared thermal imager, and these identifiers are bound to the acquisition location, the covered charging parking space, and the charging device identifier. Visible light video data is synchronously written to a network hard disk recorder for local storage. Edge computing devices extract corresponding image frames according to a preset acquisition frequency and associate the image frames with the camera channel identifier, acquisition time, and charging device identifier.

[0021] Electrical operation data is collected at a sampling frequency of 100Hz, and sound vibration data is collected at a sampling frequency of 12.8kHz. All acquisition devices are connected to a unified clock source to control the time error within 10ms. A composite index is formed by the charging device identifier and the acquisition time, and visual perception data and equipment operation data with a time difference of no more than 40ms are classified into the same data record. The associated data records are stored sequentially according to the composite index to generate the multimodal raw dataset of the charging station.

[0022] In this embodiment, S2 specifically refers to: Read the visual perception data and equipment operation data from the multimodal raw dataset of the charging station, and extract the collection time, collection location and charging equipment identifier corresponding to the visual perception data and equipment operation data; Visual perception data is denoised, sized, and distorted; equipment operation data is processed for missing values, outliers are removed, and sampling frequency is standardized. Based on the acquisition time, the visual perception data and equipment operation data are corrected for time deviation and resampled to form a time-corresponding data sequence. Based on the collection location and charging device identification, the visual perception data and device operation data are mapped to the corresponding charging devices; Visible light image data and infrared thermal image data are extracted from visual perception data. The outline of the charging device in the visible light image data and the boundary of the temperature region in the infrared thermal image data are extracted, and spatial registration is performed based on the outline of the charging device and the boundary of the temperature region. The visual region of interest is divided according to the outline of the charging device and the position of its components. The visual perception data that completes time correspondence, device mapping and spatial registration is associated with the device operation data to generate multimodal alignment data. In the specific implementation process, visible light image data and infrared thermal image data were uniformly adjusted to 640×512 pixels, and image noise was removed using 3×3 median filtering. Distortion correction was performed based on camera intrinsic parameters and distortion coefficients. For equipment operation data with no more than 5 consecutive missing sampling points, linear interpolation was used to fill in the gaps. Data deviating from the mean of the corresponding time window by more than 3 times the standard deviation was identified as outliers and replaced with adjacent valid values. A time reference was established with a uniform time interval of 100ms. Electrical operation data was uniformly resampled to 100Hz, and corresponding time periods were extracted according to each reference time. For sound vibration data, the original sampling sequence of 12.8kHz was retained, and corresponding sound vibration data segments were extracted according to each reference time. For visual perception data, the image frame with the smallest time difference from the reference time was selected, and the maximum allowable time difference was set to 50ms.

[0023] A calibration board is used to obtain corresponding feature points between visible light image data and infrared thermal image data. A homography matrix is ​​calculated based on at least eight sets of corresponding feature points, and the infrared thermal image data is projected onto the pixel coordinate system of the visible light image data. A visual region of interest (ROI) is defined based on the charging device outline and pre-calibrated component positions. Spatial registration is completed when the intersection-union ratio (IUU) between the projected region and the corresponding component region is not less than 0.80. Visual perception data and device operation data are correlated according to the charging device identifier and a unified reference time to generate multimodal alignment data.

[0024] In this embodiment, S3 specifically refers to: Electrical operation data and sound vibration data are extracted from the equipment operation data in the multimodal aligned data, and then continuously segmented according to a preset time window; Extract the numerical change sequence of electrical operation data within each time window, normalize the numerical change sequence, and arrange it into a two-dimensional matrix according to the acquisition time to generate an electrical time series feature map. Short-time Fourier transform is performed on the sound vibration data within each time window, and the obtained spectral amplitudes are arranged according to time and frequency to generate a sound vibration time-frequency diagram. The electrical timing feature map and the sound vibration time-frequency map are standardized in size, and the channels are combined according to the acquisition time and the charging equipment identification to generate the operating status feature map; The operating status feature map is associated with the visual perception data based on the collection time and the charging device identification. The samples are divided and standardized according to the visual region of interest to generate a standardized multimodal visual sample set. In the specific implementation process, the preset time window length was set to 2 seconds, and the overlap rate of adjacent time windows was set to 50%. A numerical change matrix was constructed according to the variable category of electrical operation data as rows and the acquisition time as columns. The maximum and minimum values ​​in the training data were used to normalize each variable to 0-1, generating an electrical time series feature map. The sound vibration data underwent short-time Fourier transform using a Hanning window with a window length of 1024 sampling points, a frame shift of 256 sampling points, and a fast Fourier transform number of 1024. A logarithmic transform was performed on the spectral amplitude to generate a sound vibration time-frequency diagram.

[0025] The electrical timing feature map and the sound vibration time-frequency map were uniformly adjusted to 224×224 pixels using bilinear interpolation. These were then combined according to the electrical feature channel and the sound vibration feature channel to generate an operational status feature map. Visual perception data was selected based on the charging device identifier and the center time of the time window, and cropped according to the visual region of interest. The cropped visual perception data was then uniformly adjusted to 224×224 pixels. Mean normalization was performed on both the operational status feature map and the visual perception data, and the acquisition time, charging device identifier, and visual region of interest number were recorded to generate a standardized multimodal visual sample set.

[0026] In this embodiment, the improved DINOv2 network includes a multimodal embedding module, a component alignment module, an occlusion reconstruction module, a feature encoding module, and a feature output module. The component alignment module embeds a component alignment mechanism, and the occlusion reconstruction module is configured with cross-modal occlusion reconstruction. The multimodal embedding module divides the visual perception data and running state feature map in the standardized multimodal visual sample set into multiple patches, converts each patch into a patch vector, maps the patch vectors of different modalities to the same feature dimension through a linear projection layer, and superimposes the codes corresponding to the patch position, acquisition time, charging device identifier and modality category to generate multimodal patch features. The component alignment module determines the charging device component corresponding to the multimodal patch feature based on the visual region of interest, divides the multimodal patch features corresponding to the same charging device component and the acquisition time into component feature groups, and arranges the multimodal patch features in each component feature group based on the component alignment mechanism to generate a component aligned patch sequence. The occlusion reconstruction module selects target patch features from the component-aligned patch sequence, replaces the target patch features with occlusion markers, and generates an occlusion patch sequence. The feature encoding module performs multi-layer feature encoding on the masked patch sequence input to multiple sequentially connected Transformer encoding layers to generate masked patch encoded features. The occlusion reconstruction module is based on cross-modal occlusion reconstruction. It extracts the remaining modal coding features in the occlusion patch coding features that belong to the same component feature group as the target patch features, maps the remaining modal coding features to the modal feature space to which the target patch features belong, generates the target patch reconstruction features, and replaces the corresponding occlusion markers with the target patch reconstruction features to generate multi-layer patch coding features. The feature output module restores and arranges the multi-layer patch-coded features according to the visual region of interest and modal category, retains the correspondence between the multi-layer patch-coded features and the charging device identifier and the acquisition time, and generates multi-modal visual features. In the specific implementation process, the DINOv2 network was improved by retaining the original DINOv2 network's patch embedding and Transformer encoding framework, and adding a multimodal embedding module, a component alignment module, and an occlusion reconstruction module. Visible light images, infrared thermal images, and operational status feature maps were uniformly adjusted to 224×224 pixels, and divided into 256 patches of 14×14 pixels. Each patch was expanded in row-major order and mapped to a 384-dimensional patch vector through a linear projection layer corresponding to the modality. Position codes were generated based on the patch's row and column positions, and acquisition time codes were generated at 100ms time intervals. Charging device identification codes and modality category codes were obtained through a trainable embedding table. These codes were then added dimension-by-dimensionally to the patch vector to generate multimodal patch features.

[0027] The component alignment module reads the component number corresponding to the visual region of interest, groups patches belonging to the same charging device component in different modalities into the same component feature group, and arranges them uniformly according to modal category and spatial location. Locations lacking corresponding modal features are written into a 384-dimensional trainable gap vector, generating a component alignment patch sequence. This process can confine surface texture, temperature distribution, and operational state changes to the same component range, reducing erroneous associations caused by cross-modal positional offsets.

[0028] The occlusion reconstruction module randomly selects 30% of the patch features from each part feature group as the target patch features. Only one modality is occluded at the same patch location, and the occlusion is replaced with a 384-dimensional trainable occlusion vector, generating an occlusion patch sequence. This occlusion patch sequence is input into 12 Transformer encoding layers, each with 6 self-attention units, to generate occlusion patch encoded features. The remaining modality encoded features from the same part feature group are extracted, and their similarity to the context features of the occlusion location is calculated. The normalized similarity is used as a weight for weighted summation, and then passed through the linear mapping layer corresponding to the target modality to generate the target patch reconstruction features. The occlusion location is replaced with the target patch reconstruction features, and the patches are restored according to the original modality category and patch location, generating multi-layer patch encoded features and multi-modal visual features. During training, the mean absolute difference between the reconstructed features and the target patch features before occlusion is used as a reconstruction constraint, and the cosine difference between different modality features of the same part is used as an alignment constraint, with weights set to 0.7 and 0.3, respectively. Compared to the unmodified DINOv2 network, this structure can establish component-level cross-modal correspondences and utilize other modalities to complete key features when some modalities are occluded or temporarily missing, thereby improving the completeness and stability of abnormal feature extraction.

[0029] In this embodiment, the component alignment mechanism is specifically as follows: In the component alignment module, the patch position, acquisition time, charging device identifier and modality category corresponding to each multimodal patch feature are read, and the region boundary of the visual region of interest is obtained. The spatial coverage area corresponding to each multimodal patch feature is determined based on the patch location. The ratio of the overlap area between the spatial coverage area and each visual region of interest to the spatial coverage area is calculated to generate the region overlap ratio. The region of interest with the largest overlap ratio and reaching the preset region matching value is identified as the region of interest to which the multimodal patch feature belongs, and the multimodal patch feature with the largest overlap ratio but not reaching the preset region matching value is marked as the background patch feature. Multimodal patch features other than background patch features are grouped according to the charging device identifier, acquisition time, and region of interest. Multimodal patch features with the same charging device identifier, acquired at the same time, and belonging to the same region of interest are grouped into the same component feature group. Establish the modality arrangement order according to modality category, and arrange the multimodal patch features of the same modality category according to the spatial order of the patch position in the visual region of interest; when the part feature group lacks a multimodal patch feature of the corresponding modality category, write a missing mark in the corresponding arrangement position; The completed component feature groups are bound to the charging device identifier, acquisition time, and visual region of interest to generate a component alignment index. Then, the multimodal patch features and gap markers are connected according to the component alignment index to generate a component alignment patch sequence. In the specific implementation process, the component alignment mechanism first reads the row and column numbers of the patch corresponding to each multimodal patch feature, and determines the rectangular spatial coverage area corresponding to each multimodal patch feature based on the side length of a single patch and its position in the original input data. For 224×224 pixel input data, after dividing the data into 14×14 pixel patches, each patch corresponds to a 14×14 pixel region in the original input data. The boundary of each visual region of interest is represented as the coordinates of the upper left corner and the lower right corner. The width and height of the intersection between the patch spatial coverage area and each visual region of interest are calculated respectively. The two are multiplied to obtain the overlap area, and then the overlap area is divided by the area of ​​the patch spatial coverage area to generate the region overlap ratio.

[0030] The preset region matching value is set to 0.60. When any patch overlaps with multiple regions of interest (ROIs), the ROI with the largest overlap ratio is selected as the ROI to which the patch belongs. When the largest overlap ratio is less than 0.60, the corresponding multimodal patch feature is marked as a background patch feature and does not participate in the subsequent component-level feature arrangement. Subsequently, a three-level grouping index is established according to the charging device identifier, acquisition time, and ROI number. Multimodal patch features with the same three index values ​​are grouped into the same component feature group, thereby ensuring that the visible light feature, infrared thermal feature, and operating status feature corresponding to the same component at the same time are in the same processing unit.

[0031] A fixed modal arrangement order is established according to the visible light mode, infrared thermal mode, and operating state mode. Within each mode, the patches are spatially arranged in order of row number priority and column number increment. When any component feature group lacks a feature corresponding to a certain mode or patch position, a trainable missing vector of dimension 384 is written at the corresponding position, and the missing state is recorded as one, while other valid feature positions are recorded as zero. The charging device identifier, acquisition time, visual region of interest number, modal category, and patch position are sequentially combined to form a component alignment index. Then, the valid multimodal patch features and missing vectors are connected according to the component alignment index to generate a component alignment patch sequence. This mechanism can avoid position mismatch between different modalities due to different imaging viewpoints, resolutions, and sampling methods, and establish a correspondence between texture changes, temperature changes, and operating state changes of the same charging device component at a unified position. At the same time, background patch culling reduces interference from irrelevant areas, providing a stable component-level feature foundation for subsequent cross-modal occlusion reconstruction.

[0032] In this embodiment, S5 specifically refers to: Read multimodal visual features and their corresponding charging device identifiers, acquisition times, regions of interest, and modal categories; Based on the charging device identification, the time of data collection, and the region of visual interest, the multimodal visual features belonging to the same charging device component are divided into component visual feature groups. The feature dimensions of the multimodal visual features in each component's visual feature group are unified, and they are arranged according to modality category and patch position to generate a component multimodal feature sequence. Extract the effective modal features corresponding to the same patch position in the multimodal feature sequence of the component, perform a dimension-wise average of the effective modal features to generate common features, and perform a dimension-wise subtraction between each effective modal feature and the common features to generate difference features; By concatenating and linearly mapping the common features with the individual differences, component fusion features are generated. The correspondence between components of the charging device is determined based on the spatial position of each region of interest in the charging device. The fusion features of each component are arranged according to the correspondence, and the arranged component fusion features are spliced ​​together to generate multimodal fusion visual features of the charging device. In the specific implementation process, the visible light features, infrared thermal features, and operating status features in the visual feature groups of each component are uniformly mapped to 384 dimensions through a linear projection layer, and arranged in the order of visible light mode, infrared thermal mode, and operating status mode. Within each mode, a correspondence is established according to the order of Patch row number priority and column number increment. For the same Patch location, the valid modal features without missing markers are extracted. The values ​​of each valid modal feature in the same dimension are added together and divided by the number of valid modes to generate 384-dimensional common features. Each valid modal feature is then subtracted from the common features dimension by dimension to generate the corresponding difference features.

[0033] The differential features corresponding to the three modalities are set to fixed positions. When a corresponding modality is missing, a 384-dimensional zero vector is written at the corresponding position. The common features and the three differential features are concatenated in sequence to form a 1536-dimensional concatenated feature, which is then compressed to 384 dimensions through a linear mapping layer as the component fusion patch feature for the corresponding patch position. The component fusion patch features are arranged according to the patch position to generate a component fusion feature sequence.

[0034] The component arrangement order is determined based on the position of the center point of each visual region of interest in the image coordinate system of the charging device. The fusion feature sequences of each component are connected according to the arrangement order, and the visual region of interest and patch position index are retained. The fusion feature sequences of each component are averaged and spliced ​​one dimension at a time, and compressed into 768-dimensional device-level fusion features through a linear mapping layer. The component fusion feature sequences are combined with the device-level fusion features to generate multimodal fusion visual features of the charging device.

[0035] In this embodiment, S6 specifically refers to: The multimodal fusion visual features of the charging device corresponding to the normal charging process are obtained. The normal Patch feature set is divided according to the charging device identifier, visual region of interest and Patch position. Core set sampling is performed on each normal Patch feature set to obtain representative Patch features. A normal feature memory is constructed based on the representative Patch features. Based on the acquisition time, normal patch features with the same visual region of interest and patch position in each normal patch feature set are associated, the feature change between adjacent normal patch features is calculated, and the normal patch features are combined with the corresponding feature change to generate a normal temporal patch trajectory set. Read the multimodal fusion visual features of the charging device to be detected, extract the features of the patch to be detected according to the patch position, associate the features of the patch to be detected with the same visual region of interest and patch position in consecutive time according to the acquisition time, calculate the feature change between adjacent features of the patch to be detected, and combine the features of the patch to be detected with the corresponding feature change to generate a temporal patch trajectory. Based on time-series patch trajectory matching, the time-series patch trajectory is matched with the normal time-series patch trajectory set to generate trajectory matching deviation; Each patch feature to be detected is matched with the representative patch features in the normal feature memory, and a local matching bias is generated based on the minimum feature distance. The trajectory matching deviation and local matching deviation are normalized and added together to determine the abnormal response value at each patch position. The abnormal response values ​​are then restored and arranged according to the patch position to generate a pixel-level abnormal response map. Anomaly regions are extracted from the pixel-level anomaly response map. Anomaly components are determined based on the region of interest to which the anomaly region belongs. Feature aggregation is performed on the features of the patch to be detected covered by the anomaly region to generate anomaly region features. Obtain an anomaly type feature library based on the anomaly region features of labeled anomaly samples, perform feature matching between the anomaly region features and the anomaly type feature library, and determine the anomaly category corresponding to the minimum feature distance as the anomaly type; The anomaly level is determined based on the preset range of the anomaly response value within the anomaly area, and the maximum anomaly response value within the anomaly area is normalized to the recognition confidence level. The charging device identifier, acquisition time, visual region of interest, abnormal component, anomaly type, anomaly level, and recognition confidence level corresponding to the anomaly area are combined to generate candidate anomaly detection results. In the specific implementation process, historical data from 30 consecutive days of normal operation without fault alarms was selected as normal charging process data. A normal patch feature set was established based on the charging equipment identifier, visual region of interest (ROI), and patch location. A core set sampling method was used to retain 10% of the normal patch features from each set, with a maximum of 2000 representative patch features retained for each ROI, thus constructing a normal feature memory. Patch features with the same ROI and patch location were arranged according to the acquisition time, forming a trajectory every 10 consecutive time points. The patch features at adjacent time points were subtracted dimension-wise and the absolute value was taken to generate feature change. The patch features and feature change were concatenated to form the normal temporal patch trajectory and the temporal patch trajectory to be detected.

[0036] The trajectory of the temporal patch to be detected is matched with the trajectory of the normal temporal patch to obtain the trajectory matching deviation. Simultaneously, the Euclidean distance between the features of the patch to be detected and the representative features of the normal feature memory is calculated, and the minimum Euclidean distance is taken as the local matching deviation. The two types of deviations are normalized to 0-1 respectively and then added to generate anomaly response values. Anomaly response matrices are then reconstructed according to the patch positions, and pixel-level anomaly response maps are generated through bilinear interpolation. Continuous regions with anomaly response values ​​not lower than 0.60 are identified as anomalous regions. Isolated regions with an area less than 32 pixels are removed, and the features of the patch to be detected covered by the anomalous regions are averaged dimension-wise to generate anomalous region features.

[0037] Calculate the Euclidean distance between the abnormal region features and the representative features of each category in the abnormal type feature library, and determine the abnormal category corresponding to the minimum distance as the abnormal type; when the maximum abnormal response value is not less than 0.60 and less than 1.00, it is determined as a level 3 abnormality; when the maximum abnormal response value is not less than 1.00 and less than 1.40, it is determined as a level 2 abnormality; when the maximum abnormal response value is not less than 1.40, it is determined as a level 1 abnormality. Subtract 0.60 from the maximum abnormal response value and divide by 1.40, and limit the resulting value to the range of 0 to 1 to generate the identification confidence score, and finally form the candidate abnormality detection result.

[0038] In this embodiment, the time-series patch trajectory matching specifically refers to: The features of each patch to be detected in the temporal patch trajectory and the corresponding feature change amount are combined into trajectory nodes according to the acquisition time, and the trajectory nodes are connected in the order of acquisition time. Based on the charging device identifier, the visual region of interest, and the patch location, the corresponding normal time-series patch trajectory is selected from the normal time-series patch trajectory set to generate a candidate normal time-series patch trajectory set. The time boundaries of the time-series patch trajectory and each candidate normal time-series patch trajectory are unified, and the missing trajectory nodes are interpolated and filled in according to the acquisition time to obtain the time-aligned trajectory corresponding to each candidate normal time-series patch trajectory. The differences in patch features and the differences in feature change between each trajectory node in the temporal patch trajectory and each trajectory node in the time-aligned trajectory are calculated sequentially. The differences in patch features and the differences in feature change are normalized and then added one by one to generate a trajectory matching matrix. Establish monotonic matching paths in the trajectory matching matrix according to the increasing direction of the acquisition time, accumulate the difference value corresponding to the position passed by each monotonic matching path, and determine the monotonic matching path with the smallest accumulated difference value as the target matching path. Extract the difference values ​​of each corresponding trajectory node in the target matching path, calculate the absolute change of the difference values ​​between adjacent corresponding trajectory nodes, calculate the mean difference value and the mean absolute change of the difference value respectively, add the mean difference value and the mean absolute change of the difference value to generate a single trajectory matching result; The matching results of individual trajectories corresponding to each normal time-series patch trajectory in the candidate normal time-series patch trajectory set are obtained respectively, and the single trajectory matching result with the smallest value is determined as the trajectory matching deviation. In the specific implementation process, the features of each patch to be detected are subtracted dimension-wise from the features of adjacent time points, and the absolute value is taken to generate the corresponding feature change. Then, the features of the patch to be detected and the feature change are concatenated into trajectory nodes according to the acquisition time. Each temporal patch trajectory contains 10 consecutive trajectory nodes. Normal temporal patch trajectories are selected based on the charging device identifier, visual region of interest, and patch location, with the number of candidate normal temporal patch trajectories set to 20. When a trajectory node is missing, linear interpolation is performed using the corresponding dimension values ​​of the two trajectory nodes before and after the missing time point. If the number of missing nodes exceeds 20% of the total number of trajectory nodes, the corresponding normal temporal patch trajectory is discarded.

[0039] Calculate the Euclidean distance of the patch features and the Euclidean distance of the feature change between the target trajectory node and the normal trajectory node. Normalize both distances to 0-1 based on the maximum and minimum values ​​in the normal trajectory data, and then sum them one by one to form the trajectory matching matrix. Establish a monotonic matching path along the increasing direction of the acquisition time, allowing movement of one position to the right, down, or lower right each time. Using the first matrix position as the starting point and the last matrix position as the ending point, accumulate the difference values ​​of the positions traversed by the path, and determine the path with the smallest accumulated difference value as the target matching path.

[0040] All difference values ​​in the target matching path are extracted, the mean of the difference values ​​is calculated, and the mean of the absolute values ​​of the differences between adjacent difference values ​​is calculated. The two results are added together to generate a single trajectory matching result. The single trajectory matching results of the temporal patch trajectory and 20 candidate normal temporal patch trajectories are calculated separately, and the minimum value is selected as the trajectory matching deviation. This matching method allows for slight time scaling or sampling offset between the normal charging process and the process to be detected, which can avoid misjudgment caused by comparing patch features at only a single moment and highlight the changing characteristics of anomalies that develop continuously over time.

[0041] In this embodiment, S7 specifically refers to: Read the candidate anomaly detection results, and extract the visual perception data and device operation data corresponding to the anomaly region from the multimodal alignment data based on the charging device identifier, acquisition time, and visual region of interest; Visible light image data and infrared thermal image data are extracted from visual perception data, and the image changes and temperature changes of the abnormal area at continuous acquisition time are calculated, respectively; electrical operation data and sound vibration data are extracted from equipment operation data, and the electrical changes and sound vibration changes at the corresponding acquisition time are calculated, respectively. Obtain visual perception data and equipment operation data corresponding to historical normal charging processes, and determine the corresponding normal change intervals based on the distribution of image changes, temperature changes, electrical changes and sound vibration changes in data with the same charging equipment identification and visual region of interest. Obtain a preset anomaly verification table, which records the correspondence between different anomaly types and target data categories and support quantity thresholds; query the preset anomaly verification table based on the anomaly types in the candidate anomaly detection results to determine the target data categories and support quantity thresholds for participating in multimodal consistency verification; The change amount corresponding to the target data category is compared with the corresponding normal change range. When the change amount exceeds the corresponding normal change range, the corresponding anomaly support flag is set to one. When the change amount is within the corresponding normal change range, the corresponding anomaly support flag is set to zero. The number of anomaly support labels with a value of one in the target data category is counted. When the number of anomaly support labels reaches the support number threshold, the candidate anomaly detection result is retained; when the number of anomaly support labels does not reach the support number threshold, the candidate anomaly detection result is marked as a pseudo-anomaly. The candidate anomaly detection results are merged according to the abnormal component and the time of data collection to generate the final anomaly detection result of the charging station. Anomaly alarm information is generated based on the abnormal component, anomaly type, anomaly level and identification confidence level. In the specific implementation process, based on the charging device identifier, acquisition time, and visual region of interest in the candidate anomaly detection results, data from five consecutive acquisition times before and after the anomaly occurred are extracted. The absolute value of the difference in grayscale values ​​of corresponding pixels in adjacent visible light images is averaged to generate image change. The absolute value of the average temperature difference between adjacent infrared thermal images in the anomaly area is taken as temperature change. The average value of the rate of change of voltage, current, and power at adjacent times is taken as electrical change. The rate of change of sound vibration spectrum energy is taken as sound vibration change. Based on historical normal data of the same charging device and visual region of interest, the mean and standard deviation of the four types of changes are calculated respectively, and the range between the mean and three times the standard deviation is determined as the normal change interval.

[0042] A preset anomaly verification table records the anomaly type, target data category, and supported quantity threshold. For example, overheating anomalies correspond to infrared thermal image data and electrical operation data, with a supported quantity threshold of 1; loose component anomalies correspond to visible light image data, sound vibration data, and electrical operation data, with a supported quantity threshold of 2; and surface damage anomalies correspond to visible light image data, with a supported quantity threshold of 1. Data categories exceeding the normal variation range are marked as "one," otherwise marked as "zero." When the number of "one" markers reaches the supported quantity threshold, the candidate anomaly detection result is retained; otherwise, it is marked as a false anomaly. The retained results of the same anomalous component within three consecutive acquisition times are merged, and the anomalous component, anomaly type, anomaly level, and identification confidence level are output to generate the final anomaly detection result and anomaly alarm information for the charging station.

[0043] When generating abnormal alarm information, the charging equipment identifier is converted into the corresponding alarm site and equipment name, and associated with the camera channel that triggered the detection, the time of the abnormality, the screenshot of the abnormal area, and key video clips before and after the abnormality. The push recipients are determined according to the abnormality level: Level 1 abnormalities are immediately pushed to on-duty personnel and safety management personnel; Level 2 abnormalities are pushed to on-duty personnel and equipment maintenance personnel; and Level 3 abnormalities are written into the daily inspection record and pushed to on-duty personnel. After the edge computing device completes continuous detection and confirmation, it uploads the abnormal alarm information and key video clips to the central control server. Abnormal alarm information includes alarm time, alarm site, camera channel, abnormal equipment, abnormal component, abnormality type, abnormality level, screenshot of the abnormal area, key video clips, and processing status. For abnormal alarm information generated consecutively within a preset silent time for the same charging equipment, the same abnormal component, and the same abnormal type, subsequent abnormal alarm information is merged into the first abnormal alarm record, and the abnormal response value and identification confidence level are updated to avoid repeatedly pushing the same alarm within a short period.

[0044] Example 1: To verify the feasibility of this invention in practice, it was applied to an anomaly detection scenario for charging equipment at a large public charging station. This charging station has 36 DC and AC charging devices, which operate in open or semi-open environments and are susceptible to changes in lighting, rain reflections, vehicle obstruction, environmental noise, and load fluctuations. In actual operation, anomalies such as overheating of the charging gun interface, abnormal contact of the wiring terminals, loose cooling fans, damaged equipment casings, and surface damage to charging cables exhibit different forms. Some anomalies are only manifested as a slow increase in local temperature or changes in sound vibration, while others can only be observed in visible light images. Traditional methods relying on fixed electrical thresholds or single image recognition are prone to missed detections and false alarms, and it is difficult to accurately determine the component causing the anomaly. The charging station's existing multi-target intelligent video security monitoring system already has functions for detecting flames, smoke, personnel behavior, charging gun repositioning, and environmental hygiene. This invention, as the anomaly detection function for charging equipment, is integrated into the existing edge computing equipment and central control platform, sharing the station's camera channels, internal network, network hard disk recorder, and alarm push interface. While performing the original video security event recognition, the edge computing device performs multimodal anomaly detection on visual perception data and device operation data, and uploads the abnormal alarm information of the charging device to the central control platform in the original alarm information format.

[0045] Each charging device in the charging station is equipped with a unique charging device identifier. Visible light cameras and infrared thermal imagers are installed directly in front of and above the sides of each charging device. These cameras simultaneously access voltage, current, and power data from the charging controller, as well as sound and vibration data collected from within the device. The visible light image resolution is set to 1920×1080 pixels, the infrared thermal image resolution to 640×512 pixels, the electrical operation data sampling frequency to 100Hz, and the sound and vibration data sampling frequency to 12.8kHz. All data types are time-stamped using a unified clock and associated according to the charging device identifier. Data with a time difference of no more than 40ms are grouped into the same multimodal data set.

[0046] The collected data underwent noise reduction, missing value imputation, outlier removal, and time resampling. The infrared thermal image was then projected onto the visible light image coordinate system. Based on the outline of the charging device, regions of visual interest (ROIs) were defined, including the charging gun interface, display area, heat dissipation area, housing area, wiring area, and charging cable area, ensuring that data from different modalities could be mapped to specific charging device components. Device operation data was segmented into 2-second time windows, with an overlap rate of 50% between adjacent time windows. Electrical operation data was converted into an electrical time-series feature map, and acoustic vibration data underwent a short-time Fourier transform to generate an acoustic vibration time-frequency map. These data, along with the visible light and infrared thermal images, formed a multimodal sample.

[0047] Multimodal samples are input into the improved DINOv2 network, and visible light images, infrared thermal images, and operational status feature maps are divided into 14×14 pixel patches. A component alignment mechanism is used to arrange the features of different modal patches belonging to the same charging device component. Patches with an overlap ratio of 0.60 are assigned to the corresponding visual region of interest, while the remaining patches are discarded as background features. During training, 30% of the target patch features are randomly occluded, and the target patch features are recovered using the encoded features of other modalities within the same component. This enhances the model's feature extraction capability under conditions of image occlusion, missing thermal images, or short-term fluctuations in operational data.

[0048] The normal feature memory is built using historical data from continuous normal operation without maintenance alarms. Each visual region of interest (ROI) retains no more than 2000 representative patch features through core set sampling. During detection, features with the same ROI and patch location within 10 consecutive time windows are concatenated to form a temporal patch trajectory. Simultaneously, the local matching deviation between the detected patch feature and the normal feature memory, as well as the trajectory matching deviation between the detected temporal patch trajectory and the normal temporal patch trajectory, are calculated. These two types of deviations are normalized and then fused. Continuous regions with anomaly response values ​​reaching 0.60 are identified as candidate anomaly regions. Consistency verification is then performed using visible light, temperature, electrical, and acoustic vibration data to filter out false anomalies caused by short-term vehicle occlusion, environmental reflection, and instantaneous load changes.

[0049] This embodiment generated a total of 18,000 valid samples, including 14,400 normal samples and 3,600 abnormal samples. Abnormal samples included 900 cases of overheating charging gun interfaces, 720 cases of loose heat dissipation components, 680 cases of damaged equipment housings, 760 cases of abnormal terminal contact, and 540 cases of surface damage to charging cables. Abnormal samples were confirmed by maintenance records, manual verification results, and offline controllable equipment test results. To verify the detection effect, under the same sample, hardware platform, and data partitioning conditions, this invention was compared with traditional electrical thresholding methods, single visible light convolutional neural network methods, the original DINOv2 and PatchCore combination method, and a multimodal method without component alignment and temporal patch trajectory matching. The results are shown in Table 1.

[0050] Table 1. Comparison of the overall performance of different anomaly detection methods for charging equipment

[0051] As shown in Table 1, the method of this invention achieves a relatively balanced result in terms of detection accuracy, anomaly recall, false alarm rate, cross-multiplication ratio of anomaly regions, and detection time. The traditional electrical threshold method has a detection accuracy of only 82.6% and an anomaly recall rate of 78.4%. This is mainly because the method can only identify obvious anomalies such as voltage, current, or temperature exceeding a fixed threshold, and it is difficult to detect early anomalies such as local overheating, slight loosening, and surface damage in a timely manner. Although a single visible light convolutional neural network improves the detection accuracy to 85.9%, it is easily affected by changes in lighting, vehicle obstruction, and surface reflections, resulting in a false alarm rate of 8.1%. The original DINOv2 combined with PatchCore method can extract richer visual features, achieving a detection accuracy of 90.8%. However, due to the lack of component-level correspondence between different modal features and the fact that it only matches patch features at a single moment, the anomaly recall rate and localization accuracy are still limited. A detection accuracy of 93.4% was achieved using a multimodal method that did not employ component alignment and temporal patch trajectory matching, indicating that the introduction of infrared thermal, electrical, and acoustic vibration data can effectively supplement single visual information.

[0052] This invention establishes the correspondence between different modal features of the same component through a component alignment mechanism, and supplements missing or interfered features using cross-modal occlusion reconstruction. Simultaneously, it identifies continuous changes in abnormal features through temporal patch trajectory matching, further improving the detection accuracy to 96.7%, the anomaly recall rate to 95.2%, and reducing the false alarm rate to 2.6%. The cross-union ratio of the abnormal region reaches 0.84, indicating that this invention can more accurately determine the location of anomalies. In this embodiment, the average detection time for a single set of data is 91ms. After continuous acquisition time confirmation, alarm information generation, and central control uploading, the average response time for anomaly alarms is controlled within 4s, which meets the response requirements of the charging station real-time monitoring system.

[0053] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A charging station anomaly detection method based on multimodal feature fusion, characterized in that, Includes the following steps: S1. Collect visual perception data and equipment operation data during the operation of the charging station to generate a multimodal raw dataset of the charging station; S2. Perform data preprocessing and spatiotemporal alignment on the original multimodal dataset of the charging station to determine the visual region of interest corresponding to the charging equipment and generate multimodal aligned data; S3. Convert the device operation data in the multimodal alignment data into an operation status feature map, and associate the operation status feature map with the visual perception data to generate a standardized multimodal visual sample set; S4. Input the standardized multimodal visual sample set into the improved DINOv2 network, and improve the DINOv2 network by setting a component alignment mechanism and cross-modal occlusion reconstruction; based on the component alignment mechanism, arrange the patch features of different modalities according to their corresponding components, and restore the occluded patch features through cross-modal occlusion reconstruction to generate multimodal visual features. S5. Based on the correspondence between the components of the charging device, the multimodal visual features are fused to generate multimodal fused visual features of the charging device. S6. Input the multimodal fusion visual features of the charging device into the PatchCore algorithm, associate the Patch features of the corresponding Patch positions in continuous time, obtain the trajectory matching deviation based on temporal Patch trajectory matching, and perform feature matching between the Patch features and the normal feature memory to generate candidate anomaly detection results. S7. Perform multimodal consistency verification on the candidate anomaly detection results, and generate the final anomaly detection results and anomaly alarm information for the charging station.

2. The charging station anomaly detection method based on multimodal feature fusion according to claim 1, characterized in that, Specifically, S1 is: Obtain the charging device identifiers of each charging device in the charging station and establish the correspondence between the collection location and the charging device identifiers; Collect visible light image data and infrared thermal image data during the operation of the charging station, add the collection time, collection location and charging equipment identification, and generate visual perception data; Collect electrical operation data and sound vibration data during the operation of the charging station, add the collection time, collection location and charging equipment identification, and generate equipment operation data; The visual perception data and equipment operation data are correlated based on the collection time and charging equipment identification, and a data index is established according to the charging equipment identification and collection time to generate a multimodal raw dataset of the charging station.

3. The charging station anomaly detection method based on multimodal feature fusion according to claim 1, characterized in that, Specifically, S2 is: Read the visual perception data and equipment operation data from the multimodal raw dataset of the charging station, and extract the collection time, collection location and charging equipment identifier corresponding to the visual perception data and equipment operation data; Visual perception data is denoised, sized, and distorted; equipment operation data is processed for missing values, outliers are removed, and sampling frequency is standardized. Based on the acquisition time, the visual perception data and equipment operation data are corrected for time deviation and resampled to form a time-corresponding data sequence. Based on the collection location and charging device identification, the visual perception data and device operation data are mapped to the corresponding charging devices; Visible light image data and infrared thermal image data are extracted from visual perception data. The outline of the charging device in the visible light image data and the boundary of the temperature region in the infrared thermal image data are extracted, and spatial registration is performed based on the outline of the charging device and the boundary of the temperature region. Based on the outline of the charging device and the location of its components, the visual region of interest is divided. The visual perception data that completes time correspondence, device mapping and spatial registration is correlated with the device operation data to generate multimodal alignment data.

4. The charging station anomaly detection method based on multimodal feature fusion according to claim 1, characterized in that, Specifically, S3 is: Electrical operation data and sound vibration data are extracted from the equipment operation data in the multimodal aligned data, and then continuously segmented according to a preset time window; Extract the numerical change sequence of electrical operation data within each time window, normalize the numerical change sequence, and arrange it into a two-dimensional matrix according to the acquisition time to generate an electrical time series feature map. Short-time Fourier transform is performed on the sound vibration data within each time window, and the obtained spectral amplitudes are arranged according to time and frequency to generate a sound vibration time-frequency diagram. The electrical timing feature map and the sound vibration time-frequency map are standardized in size, and the channels are combined according to the acquisition time and the charging equipment identification to generate the operating status feature map; The operating status feature map is associated with the visual perception data based on the collection time and the charging device identification. The samples are divided and standardized according to the visual region of interest to generate a standardized multimodal visual sample set.

5. The charging station anomaly detection method based on multimodal feature fusion according to claim 1, characterized in that, The improved DINOv2 network includes a multimodal embedding module, a component alignment module, an occlusion reconstruction module, a feature encoding module, and a feature output module. The component alignment module embeds a component alignment mechanism, and the occlusion reconstruction module sets up cross-modal occlusion reconstruction. The multimodal embedding module divides the visual perception data and running state feature map in the standardized multimodal visual sample set into multiple patches, converts each patch into a patch vector, maps the patch vectors of different modalities to the same feature dimension through a linear projection layer, and superimposes the codes corresponding to the patch position, acquisition time, charging device identifier and modality category to generate multimodal patch features. The component alignment module determines the charging device component corresponding to the multimodal patch feature based on the visual region of interest, divides the multimodal patch features corresponding to the same charging device component and the acquisition time into component feature groups, and arranges the multimodal patch features in each component feature group based on the component alignment mechanism to generate a component aligned patch sequence. The occlusion reconstruction module selects target patch features from the component-aligned patch sequence, replaces the target patch features with occlusion markers, and generates an occlusion patch sequence. The feature encoding module performs multi-layer feature encoding on the masked patch sequence input to multiple sequentially connected Transformer encoding layers to generate masked patch encoded features. The occlusion reconstruction module is based on cross-modal occlusion reconstruction. It extracts the remaining modal coding features in the occlusion patch coding features that belong to the same component feature group as the target patch features, maps the remaining modal coding features to the modal feature space to which the target patch features belong, generates the target patch reconstruction features, and replaces the corresponding occlusion markers with the target patch reconstruction features to generate multi-layer patch coding features. The feature output module restores and arranges the multi-layer patch-encoded features according to the visual region of interest and modal category, retains the correspondence between the multi-layer patch-encoded features and the charging device identifier and the acquisition time, and generates multi-modal visual features.

6. The charging station anomaly detection method based on multimodal feature fusion according to claim 5, characterized in that, The component alignment mechanism is specifically as follows: In the component alignment module, the patch position, acquisition time, charging device identifier and modality category corresponding to each multimodal patch feature are read, and the region boundary of the visual region of interest is obtained. The spatial coverage area corresponding to each multimodal patch feature is determined based on the patch location. The ratio of the overlap area between the spatial coverage area and each visual region of interest to the spatial coverage area is calculated to generate the region overlap ratio. The region of interest with the largest overlap ratio and reaching the preset region matching value is identified as the region of interest to which the multimodal patch feature belongs, and the multimodal patch feature with the largest overlap ratio but not reaching the preset region matching value is marked as the background patch feature. Multimodal patch features other than background patch features are grouped according to the charging device identifier, acquisition time, and region of interest. Multimodal patch features with the same charging device identifier, acquired at the same time, and belonging to the same region of interest are grouped into the same component feature group. Establish the modality arrangement order according to modality category, and arrange the multimodal patch features of the same modality category according to the spatial order of the patch position in the visual region of interest; when the part feature group lacks a multimodal patch feature of the corresponding modality category, write a missing mark in the corresponding arrangement position; The completed component feature groups are bound to the charging device identifier, acquisition time, and visual region of interest to generate a component alignment index. Then, the multimodal patch features and gap markers are connected according to the component alignment index to generate a component alignment patch sequence.

7. The charging station anomaly detection method based on multimodal feature fusion according to claim 1, characterized in that, Specifically, S5 is: Read multimodal visual features and their corresponding charging device identifiers, acquisition times, regions of interest, and modal categories; Based on the charging device identification, the time of data collection, and the region of visual interest, the multimodal visual features belonging to the same charging device component are divided into component visual feature groups; The feature dimensions of the multimodal visual features in each component's visual feature group are unified, and they are arranged according to modality category and patch position to generate a component multimodal feature sequence. Extract the effective modal features corresponding to the same patch position in the multimodal feature sequence of the component, perform a dimension-wise average of the effective modal features to generate common features, and perform a dimension-wise subtraction between each effective modal feature and the common features to generate difference features; By concatenating and linearly mapping the common features with the individual differences, component fusion features are generated. The correspondence between components of the charging device is determined based on the spatial location of each visual region of interest in the charging device. The fusion features of each component are arranged according to the correspondence, and the arranged component fusion features are spliced ​​together to generate multimodal fusion visual features of the charging device.

8. The charging station anomaly detection method based on multimodal feature fusion according to claim 1, characterized in that, Specifically, S6 is: The multimodal fusion visual features of the charging device corresponding to the normal charging process are obtained. The normal Patch feature set is divided according to the charging device identifier, visual region of interest and Patch position. Core set sampling is performed on each normal Patch feature set to obtain representative Patch features. A normal feature memory is constructed based on the representative Patch features. Based on the acquisition time, normal patch features with the same visual region of interest and patch position in each normal patch feature set are associated, the feature change between adjacent normal patch features is calculated, and the normal patch features are combined with the corresponding feature change to generate a normal temporal patch trajectory set. Read the multimodal fusion visual features of the charging device to be detected, extract the features of the patch to be detected according to the patch position, associate the features of the patch to be detected with the same visual region of interest and patch position in consecutive time according to the acquisition time, calculate the feature change between adjacent features of the patch to be detected, and combine the features of the patch to be detected with the corresponding feature change to generate a temporal patch trajectory. Based on time-series patch trajectory matching, the time-series patch trajectory is matched with the normal time-series patch trajectory set to generate trajectory matching deviation; Each patch feature to be detected is matched with the representative patch features in the normal feature memory, and a local matching bias is generated based on the minimum feature distance. The trajectory matching deviation and local matching deviation are normalized and added together to determine the abnormal response value at each patch position. The abnormal response values ​​are then restored and arranged according to the patch position to generate a pixel-level abnormal response map. Anomaly regions are extracted from the pixel-level anomaly response map. Anomaly components are determined based on the region of interest to which the anomaly region belongs. Feature aggregation is performed on the features of the patch to be detected covered by the anomaly region to generate anomaly region features. Obtain an anomaly type feature library based on the anomaly region features of labeled anomaly samples, perform feature matching between the anomaly region features and the anomaly type feature library, and determine the anomaly category corresponding to the minimum feature distance as the anomaly type; The anomaly level is determined based on the preset range of the abnormal response value within the abnormal area, and the maximum abnormal response value within the abnormal area is normalized to the recognition confidence level. The charging device identifier, acquisition time, visual region of interest, abnormal component, anomaly type, anomaly level, and recognition confidence level corresponding to the abnormal area are combined to generate candidate anomaly detection results.

9. The charging station anomaly detection method based on multimodal feature fusion according to claim 8, characterized in that, The time-series Patch trajectory matching specifically refers to: The features of each patch to be detected in the temporal patch trajectory and the corresponding feature change amount are combined into trajectory nodes according to the acquisition time, and the trajectory nodes are connected in the order of acquisition time. Based on the charging device identifier, the visual region of interest, and the patch location, the corresponding normal time-series patch trajectory is selected from the normal time-series patch trajectory set to generate a candidate normal time-series patch trajectory set. The time boundaries of the time-series patch trajectory and each candidate normal time-series patch trajectory are unified, and the missing trajectory nodes are interpolated and filled in according to the acquisition time to obtain the time-aligned trajectory corresponding to each candidate normal time-series patch trajectory. The differences in patch features and the differences in feature change between each trajectory node in the temporal patch trajectory and each trajectory node in the time-aligned trajectory are calculated sequentially. The differences in patch features and the differences in feature change are normalized and then added one by one to generate a trajectory matching matrix. Establish monotonic matching paths in the trajectory matching matrix according to the increasing direction of the acquisition time, accumulate the difference value corresponding to the position passed by each monotonic matching path, and determine the monotonic matching path with the smallest accumulated difference value as the target matching path. Extract the difference values ​​of each corresponding trajectory node in the target matching path, calculate the absolute change of the difference values ​​between adjacent corresponding trajectory nodes, calculate the mean difference value and the mean absolute change of the difference value respectively, add the mean difference value and the mean absolute change of the difference value to generate a single trajectory matching result; The matching results of individual tracks corresponding to the time-series patch tracks and each normal time-series patch track in the candidate normal time-series patch track set are obtained respectively, and the single track matching result with the smallest value is determined as the track matching deviation.

10. The charging station anomaly detection method based on multimodal feature fusion according to claim 1, characterized in that, Specifically, S7 is: Read the candidate anomaly detection results, and extract the visual perception data and device operation data corresponding to the anomaly region from the multimodal alignment data based on the charging device identifier, acquisition time, and visual region of interest; Visible light image data and infrared thermal image data are extracted from visual perception data, and the image changes and temperature changes of the abnormal area at continuous acquisition time are calculated respectively. Electrical operation data and sound vibration data are extracted from the equipment operation data, and the electrical changes and sound vibration changes at the corresponding acquisition time are calculated respectively. Obtain visual perception data and equipment operation data corresponding to historical normal charging processes, and determine the corresponding normal change intervals based on the distribution of image changes, temperature changes, electrical changes and sound vibration changes in data with the same charging equipment identification and visual region of interest. Obtain a preset anomaly verification table, which records the correspondence between different anomaly types and target data categories and support quantity thresholds; Based on the anomaly types in the candidate anomaly detection results, query the preset anomaly verification table to determine the target data categories and support quantity thresholds for participating in multimodal consistency verification; The change amount corresponding to the target data category is compared with the corresponding normal change range. When the change amount exceeds the corresponding normal change range, the corresponding anomaly support flag is set to one. When the change amount is within the corresponding normal change range, the corresponding anomaly support flag is set to zero. The number of anomaly support tags with a value of one in the target data category is counted. When the number of anomaly support tags reaches the support count threshold, the candidate anomaly detection results are retained. When the number of anomaly support labels does not reach the support number threshold, the candidate anomaly detection result is labeled as a pseudo-anomaly; The candidate anomaly detection results are merged according to the abnormal component and the time of data collection to generate the final anomaly detection result of the charging station. Anomaly alarm information is generated based on the abnormal component, anomaly type, anomaly level and identification confidence level.

Images