Lottery store violation detection method and system based on multi-modal data fusion

Abstract

The invention provides a lottery store violation detection method and system based on multi-modal data fusion, and relates to the technical field of intelligent monitoring, and the method comprises the steps: detecting a current abnormal event, carrying out time sequence perception target detection on monitoring video data, and recognizing violation electric equipment in a video image frame based on adaptive feature enhancement and multi-target tracking. The method comprises the following steps: acquiring time sequence data of current abnormity and illegal electric equipment detection, constructing a time sequence incidence relation of current abnormity, equipment detection and an electric state based on a dynamic causal network, calculating a time-varying weight and instantaneous causal intensity by utilizing conditional entropy increment, acquiring optimal time lag in combination with eigenvector conversion and a dynamic programming algorithm, and determining the current abnormity and illegal electric equipment detection according to the optimal time lag. And weighting the integral of the instantaneous causal intensity and the exponential function of the optimal time delay to obtain an event matching score, and distinguishing a temporary power utilization event and a continuous illegal power utilization event based on the event matching score.

Description

Technical Field

[0001] The present invention relates to intelligent monitoring technology, and in particular to a lottery store violation detection method and system based on multimodal data fusion. Background Art

[0002] Some lottery stores have engaged in illegal electricity use, such as charging electric vehicles in the store and illegally using high-power electrical appliances. This illegal use of electricity not only increases the potential safety hazards of electricity use, but also violates relevant lottery management regulations. Currently, the detection of illegal electricity use in lottery stores has the following problems: Traditional power monitoring methods can only obtain power load data and cannot accurately determine whether the anomaly is caused by illegal power-using equipment; a single video surveillance system cannot automatically identify and continuously track illegal power-using equipment, and is prone to missed detections and false detections; there is a lack of effective correlation analysis methods between power data and video data, making it difficult to establish a corresponding relationship between illegal power-using equipment and current anomalies; there is a lack of quantitative indicators for the determination of illegal power usage incidents, and it is impossible to distinguish between temporary and continuous illegal power usage. Summary of the Invention

[0003] The embodiments of the present invention provide a lottery store violation detection method and system based on multimodal data fusion, which can solve the problems in the prior art.

[0004] According to a first aspect of the embodiments of the present invention, Provides a lottery store violation detection method based on multimodal data fusion, including: Obtain electricity monitoring data and surveillance video data from lottery stores; Detect abnormal current events based on the power monitoring data and record the corresponding current detection timestamp; perform time-series-aware target detection on the monitoring video data, identify illegal power-consuming devices in the video image frames based on adaptive feature enhancement and multi-target tracking, and record illegal power-consuming events and corresponding video timestamps based on device clustering and spatiotemporal trajectory features; A time dimension correlation analysis is performed on the abnormal current events and illegal power-consuming equipment, including: obtaining time series data of current anomalies and illegal power-consuming equipment detection, constructing a time series correlation relationship between current anomalies, equipment detection and power consumption status based on a dynamic causal network, using conditional entropy increment to calculate time-varying weights and instantaneous causal strength, combining eigenvector conversion and dynamic programming algorithm to obtain the optimal time lag, weighting the integral of the instantaneous causal strength and the exponential function of the optimal time lag to obtain an event matching score, and distinguishing temporary power consumption events from continuous illegal power consumption events based on the event matching score; when the continuous illegal power consumption event is detected, generating violation alarm information.

[0005] The target detection for the surveillance video data with time-series perception includes: Preprocessing the surveillance video data to obtain a standardized video frame sequence; Identify socket areas in standardized video frame sequences; detect illegal electrical devices and construct spatiotemporal trajectories containing location coordinates, velocity, acceleration, motion direction angle, and timestamps; Calculate an adaptive distance metric between the illegal electrical device and the socket area, where the adaptive distance metric is composed of the minimum Euclidean distance between the current location of the device and the location point set of the socket area and a speed influence factor; When multiple illegal electrical devices are detected, the device concentration degree of the socket area is calculated. The device concentration degree is the weighted sum of the type weight of each illegal electrical device and its corresponding Gaussian distance response value, which is calculated using the adaptive distance metric and Gaussian kernel parameters. constructing a violation behavior feature vector based on the spatiotemporal trajectory, the adaptive distance metric, and the device aggregation, the violation behavior feature vector including the cumulative residence time, the number of entry and exit events, the motion characteristics, and the maximum aggregation value; The violation behavior feature vector is normalized and weighted to obtain a violation risk score, and the illegal electrical equipment is identified and the corresponding timestamp is recorded.

[0006] Steps to identify outlet zones and detect offending electrical devices include: The backbone neural network is used to extract multi-level features of video images and obtain feature map sequences of different scales; Constructing an adaptive feature enhancement module, including: calculating channel attention weights through global average pooling and multi-layer perceptron; calculating spatial attention maps through convolution of max pooling and average pooling features; using the product of channel attention weights and spatial attention maps as the feature enhancement coefficient to enhance the feature map sequence; using adaptive weights to fuse the enhanced features, and performing socket region detection based on the fused features through a region proposal network; A time-aware multi-target detection framework is constructed, including: performing three-dimensional convolution feature extraction on multiple consecutive frames of images, calculating the time-series attention coefficient between adjacent frames, and obtaining time-series enhancement features; extracting optical flow features between adjacent frames, splicing the optical flow features with the time-series enhancement features, and obtaining motion features through the convolution layer; performing multi-target tracking through appearance feature extraction and motion prediction, and outputting the detection results of illegal electrical equipment.

[0007] Scene partitioning of surveillance video data and regional adaptive adjustment of violation risk scores based on scene partitioning include: The lottery store is divided into a ticket display area, a customer waiting area, and an equipment operation area. Static and dynamic features are extracted from each area. Static features include socket distribution density and ticket vending equipment layout characteristics; dynamic features include the degree of crowd concentration, equipment movement trajectory, and power load changes. Adaptive weights are calculated based on regional characteristics and time period characteristics, and static and dynamic features are hierarchically integrated. Differentiated violation determinations are implemented for each area: ticket display area: detect temporary electricity usage; customer waiting area: detect mobile charging equipment and collective illegal electricity usage; equipment operation area: detect abnormal working status of ticket vending equipment and access to illegal electricity equipment.

[0008] Performing a time dimension correlation analysis on the abnormal current event and the illegal power-consuming equipment includes: obtaining time series data of current anomaly and illegal power-consuming equipment detection, constructing a time series correlation relationship between current anomaly, equipment detection and power consumption status based on a dynamic causal network, calculating time-varying weights and instantaneous causal strength using conditional entropy increment, combining eigenvector conversion and dynamic programming algorithm to obtain the optimal time lag, weighting the integral of the instantaneous causal strength with the exponential function of the optimal time lag to obtain an event matching score, and distinguishing temporary power consumption events from persistent illegal power consumption events based on the event matching score, including: Acquire abnormal current event data and illegal electric device detection data, wherein the abnormal current event data includes a current detection timestamp of the abnormal current event, and the illegal electric device detection data includes a video timestamp of the illegal electric device; Constructing a dynamic causal network, the dynamic causal network including a current node, a non-compliant electrical equipment detection node, and an equipment power status node; calculating time-varying edge weights between nodes in the dynamic causal network based on a conditional entropy increment, the conditional entropy increment representing the intensity of information flow between nodes; Calculating the instantaneous causal strength between nodes in the dynamic causal network, the instantaneous causal strength is calculated by the mutual information between the node at the current moment and the node at the next moment and the conditional probability of the node at the current moment, and integrating the instantaneous causal strength within a preset time window to obtain a cumulative causal effect; The temporal features of abnormal current events and illegal electrical equipment detection events are converted into time-frequency domain feature vectors. An improved dynamic programming algorithm is used to construct a cumulative cost matrix. The improved dynamic programming algorithm includes a time-varying weight coefficient for information flow intensity and a direction-sensitive jump connection that is adaptive to feature similarity. The optimal time-delay relationship is obtained based on the minimum path. The cumulative causal effect and the exponential function of the optimal time lag relationship are weighted to obtain an event matching score; when the event matching score is greater than a first preset threshold and the duration of the cumulative causal effect is greater than a second preset threshold, it is determined to be a continuous illegal electricity use event.

[0009] The calculation of cumulative causal effect includes: Obtain node characteristics at the current moment and the next moment, wherein the node characteristics include abnormal current amplitude, duration, and device status information; The probability distribution of node features is calculated using the kernel density estimation method, where the bandwidth of the kernel function is determined by an adaptive method based on k-nearest neighbors, and the number of k-nearest neighbors is dynamically adjusted according to the sample density; The probability distribution of the node features is used to calculate the joint probability distribution of the node features at adjacent moments, and the mutual information and conditional probability are calculated based on the joint probability distribution and the marginal probability distribution; the instantaneous causal strength is calculated based on the mutual information and conditional probability: ; Among them, I(X t ;X t+1 ) is the feature vector X at the current moment t and the next moment eigenvector X t+1 The mutual information between t+1 |X t ) is the feature vector X at the known current moment t Under the condition of t+1 Conditional probability of occurrence; When integrating the instantaneous causal strength, an adaptive time window mechanism is adopted. The length of the time window is adjusted with the rate of change of the causal strength. When the rate of change exceeds the preset adjustment threshold, the time window is shortened, and when the rate of change is lower than the preset adjustment threshold, the time window is increased. The cumulative causal effect is obtained according to the integration result, and the cumulative causal effect represents the degree of continuous influence between adjacent time nodes.

[0010] The time series features of abnormal current events and illegal electrical equipment detection events are converted into time-frequency domain feature vectors. An improved dynamic programming algorithm is used to construct a cumulative cost matrix. The improved dynamic programming algorithm includes the introduction of a time-varying weight coefficient for information flow intensity and a direction-sensitive jump connection with adaptive feature similarity. The optimal time-delay relationship is obtained based on the minimum path, including: Extract the temporal characteristics of abnormal current events and illegal electrical equipment detection events, including the current amplitude change rate, duration, and fluctuation characteristics in the time domain, and the main frequency component, spectrum energy distribution, and frequency change trend in the frequency domain. Standardize the time and frequency domain characteristics to obtain time-frequency domain feature vectors, and construct current event feature sequences and device event feature sequences. An improved dynamic programming algorithm is used to construct the cumulative cost matrix of current event feature sequences and device event feature sequences. The improvements include: introducing the intensity of information flow between nodes as a time-varying weight coefficient into the cost calculation, setting direction-sensitive jump connections in the cumulative cost matrix calculation process, and adaptively adjusting the jump step size based on feature similarity; An initial time-lag relationship is obtained based on the minimum path of the cumulative cost matrix, a time-lag consistency index is calculated, and the time window for feature extraction is segmentedly adjusted according to the time-lag consistency index. It is iteratively optimized until the time-lag consistency index meets the preset optimization threshold requirement, thereby obtaining the optimal time-lag relationship.

[0011] According to a second aspect of the embodiments of the present invention, Provides a lottery store violation detection system based on multimodal data fusion, including: The first unit is used to obtain the power monitoring data and surveillance video data of the lottery store; The second unit is configured to detect abnormal current events based on the power monitoring data and record corresponding current detection timestamps; perform time-series-aware target detection on the monitoring video data, identify illegal power-consuming devices in video image frames based on adaptive feature enhancement and multi-target tracking, and record illegal power-consuming events and corresponding video timestamps based on device aggregation and spatiotemporal trajectory features; The third unit is used to perform time dimension correlation analysis on the abnormal current events and illegal power-consuming equipment, including: obtaining time series data of current anomalies and illegal power-consuming equipment detection, constructing a time series correlation relationship between current anomalies, equipment detection and power consumption status based on a dynamic causal network, using conditional entropy increment to calculate time-varying weights and instantaneous causal strength, combining eigenvector conversion and dynamic programming algorithm to obtain the optimal time lag, weighting the integral of the instantaneous causal strength with the exponential function of the optimal time lag to obtain an event matching score, distinguishing temporary power consumption events from continuous illegal power consumption events based on the event matching score; when the continuous illegal power consumption event is detected, generating violation alarm information.

[0012] According to a third aspect of the embodiments of the present invention, An electronic device is provided, comprising: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0013] According to a fourth aspect of the embodiments of the present invention, A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the method described above is implemented.

[0014] The beneficial effects of the present invention include: By combining multimodal analysis methods with power monitoring and video surveillance data, the system accurately identifies illegal electricity usage. A time-aware target detection framework based on adaptive feature enhancement effectively extracts spatiotemporal features from videos. Device clustering and adaptive distance metrics accurately characterize the interaction between illegal electrical devices and socket areas. Furthermore, a differentiated judgment strategy based on scene partitioning improves the accuracy of illegal detection in different functional areas.

[0015] An innovative temporal correlation analysis method based on a dynamic causal network was constructed. By calculating the time-varying weights between nodes using conditional entropy increments, the intensity of information flow was accurately characterized. Kernel density estimation and an adaptive time window mechanism were used to calculate the cumulative causal effects, effectively capturing the characteristics of persistent violations. Furthermore, an improved dynamic programming algorithm was used to obtain the optimal time-lag relationship, achieving a precise correspondence between current anomalies and detection events of violating equipment.

[0016] A hierarchical feature fusion strategy and adaptive optimization mechanism are employed. Regional characteristics are considered in the hierarchical fusion of static and dynamic features, improving the pertinence of violation determinations. A consistency indicator is introduced during the time-lag relationship optimization process for iterative adjustment, enhancing the reliability of time series correlations. Ultimately, event matching scores are used to effectively distinguish between temporary and persistent violations of electricity use, providing a reliable basis for violation alerts. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 This is a flow chart of a lottery store violation detection method based on multimodal data fusion according to an embodiment of the present invention; Figure 2 A comparison chart of multi-target tracking performance. DETAILED DESCRIPTION

[0018] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0019] The following specific embodiments are used to describe the technical solution of the present invention in detail. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0020] Figure 1 FIG. 1 is a flow chart of a lottery store violation detection method based on multimodal data fusion according to an embodiment of the present invention. Figure 1 As shown, the method includes: Obtain electricity monitoring data and surveillance video data from lottery stores; Detect abnormal current events based on the power monitoring data and record the corresponding current detection timestamp; perform time-series-aware target detection on the monitoring video data, identify illegal power-consuming devices in the video image frames based on adaptive feature enhancement and multi-target tracking, and record illegal power-consuming events and corresponding video timestamps based on device clustering and spatiotemporal trajectory features; Performing a time-dimensional correlation analysis on the abnormal current events and illegal power-consuming equipment, including: obtaining time series data of current anomalies and illegal power-consuming equipment detection, constructing a time-series correlation relationship between current anomalies, equipment detection, and power consumption status based on a dynamic causal network, calculating time-varying weights and instantaneous causal strength using conditional entropy increments, combining eigenvector conversion with a dynamic programming algorithm to obtain an optimal time lag, weighting the integral of the instantaneous causal strength with an exponential function of the optimal time lag to obtain an event matching score, and distinguishing temporary power consumption events from persistent illegal power consumption events based on the event matching score; When the persistent illegal electricity usage event is detected, illegal alarm information is generated.

[0021] Optional, The target detection for the surveillance video data with time-series perception includes: Preprocessing the surveillance video data to obtain a standardized video frame sequence; Identify socket areas in standardized video frame sequences; detect illegal electrical devices and construct spatiotemporal trajectories containing location coordinates, velocity, acceleration, motion direction angle, and timestamps; Calculate an adaptive distance metric between the illegal electrical device and the socket area, where the adaptive distance metric is composed of the minimum Euclidean distance between the current location of the device and the location point set of the socket area and a speed influence factor; When multiple illegal electrical devices are detected, the device concentration degree of the socket area is calculated. The device concentration degree is the weighted sum of the type weight of each illegal electrical device and its corresponding Gaussian distance response value, which is calculated using the adaptive distance metric and Gaussian kernel parameters. constructing a violation behavior feature vector based on the spatiotemporal trajectory, the adaptive distance metric, and the device aggregation, the violation behavior feature vector including the cumulative residence time, the number of entry and exit events, the motion characteristics, and the maximum aggregation value; The violation behavior feature vector is normalized and weighted to obtain a violation risk score, and the illegal electrical equipment is identified and the corresponding timestamp is recorded.

[0022] For example, bilinear interpolation is used to adjust the video frames to a uniform size of 1920×1080 pixels, histogram equalization is performed on each frame to enhance the image contrast, Gaussian filtering is used to eliminate noise, and finally brightness normalization is performed to obtain a standardized video frame sequence. When identifying the socket area in the standardized video frame sequence, a socket template feature library is set, which contains feature descriptions of common socket types. A sliding window is used to perform feature matching in the image, with the window size set to 64×64 pixels and a step size of 32 pixels. When the feature matching degree exceeds 0.85, the area is marked as a candidate socket area. Non-maximum suppression is performed on the candidate area, and the detection frame with the highest confidence is retained as the socket area.

[0023] When detecting illegal electrical devices and constructing spatiotemporal trajectories, we first establish a feature template library for illegal electrical devices, including the appearance features of common illegal devices such as battery chargers and mobile charging devices. Target detection is performed in each frame of the image to obtain the location coordinates of the device. The motion parameters of the device are calculated by associating targets between adjacent frames. Specifically, at a sampling rate of 25 frames per second, the velocity and acceleration are calculated every 5 frames. The velocity is the displacement of two adjacent sampling points divided by the time interval, and the acceleration is the rate of change of the velocity. The direction of motion angle is calculated from the position of adjacent sampling points. These parameters and the corresponding timestamps are combined to form a spatiotemporal trajectory.

[0024] When calculating the adaptive distance metric, the Euclidean distance between the device's current location and all points in the socket area is first calculated, and the minimum value is selected as the base distance. The speed impact factor is dynamically adjusted based on the device's speed. When the speed is less than 0.5 m / s, the impact factor is 0.8; when the speed is between 0.5 and 2 m / s, the impact factor increases linearly to 1.2; when the speed is greater than 2 m / s, the impact factor remains at 1.2. The base distance is multiplied by the speed impact factor to obtain the adaptive distance metric.

[0025] When multiple illegal electrical devices are detected, the device clustering degree of the outlet area is calculated. First, weights are assigned based on device type. For example, a battery charger has a weight of 0.8, while a mobile charger has a weight of 0.6. The Gaussian kernel parameter σ is set to 2 meters, and the adaptive distance metric is substituted into the Gaussian function to calculate the distance response value. The device clustering degree is calculated by multiplying the type weight of each device by the corresponding distance response value and summing the results.

[0026] When constructing the violation feature vector, the cumulative dwell time counts the total time a device spends within a 3-meter radius of the outlet area. The number of entry and exit events records the number of times a device enters and leaves this radius. Movement features include average speed, maximum acceleration, and frequency of direction changes. The maximum clustering value is the peak value of the device clustering during the observation period.

[0027] When normalizing feature vectors, we use the minimum-maximum normalization method to map each dimension to a range of 0-1. When calculating the weighted violation risk score, we set a weight of 0.4 for cumulative dwell time, 0.2 for entry and exit events, 0.2 for movement features, and 0.2 for maximum aggregation. If the violation risk score exceeds 0.75, the device is identified as violating the electrical regulations and the current timestamp is recorded.

[0028] Through the above technical solution, in the time-series perception target detection of monitoring video data, adaptive distance measurement and device clustering are used to accurately characterize the behavioral characteristics of illegal electrical equipment, and the weighted calculation of multi-dimensional feature vectors is combined to realize violation risk assessment.

[0029] Optional, Steps to identify outlet zones and detect offending electrical devices include: The backbone neural network is used to extract multi-level features of video images and obtain feature map sequences of different scales; Constructing an adaptive feature enhancement module, including: calculating channel attention weights through global average pooling and multi-layer perceptron; calculating spatial attention maps through convolution of max pooling and average pooling features; using the product of channel attention weights and spatial attention maps as the feature enhancement coefficient to enhance the feature map sequence; using adaptive weights to fuse the enhanced features, and performing socket region detection based on the fused features through a region proposal network; A time-aware multi-target detection framework is constructed, including: performing three-dimensional convolution feature extraction on multiple consecutive frames of images, calculating the time-series attention coefficient between adjacent frames, and obtaining time-series enhancement features; extracting optical flow features between adjacent frames, splicing the optical flow features with the time-series enhancement features, and obtaining motion features through the convolution layer; performing multi-target tracking through appearance feature extraction and motion prediction, and outputting the detection results of illegal electrical equipment.

[0030] For example, the backbone network uses the ResNet-50 architecture, extracting features through layer-by-layer convolution operations. The input image size is 1920×1080 pixels. After five stages of feature extraction, feature maps of sizes 240×135, 120×67, 60×33, 30×16, and 15×8 are obtained, with the number of channels in each feature map being 256, 512, 1024, 2048, and 4096, respectively.

[0031] When constructing the adaptive feature enhancement module, a combination of channel attention and spatial attention is employed. For channel attention, a global average pooling operation is performed on each feature map, compressing each channel into a single value to obtain a channel descriptor. Taking a 240×135×256 feature map as an example, pooling yields a 1×1×256 channel descriptor. This descriptor is processed by a multilayer perceptron (MLP), which consists of two fully connected layers with a Reluctant Unit (ReLU) activation function in between. The number of neurons in the hidden layer is 1 / 16 of the number of channels, and the number of neurons in the output layer is equal to the number of channels. The resulting channel attention weights range from 0 to 1. To calculate the spatial attention map, the feature map is first subjected to max pooling and average pooling, respectively, to obtain two feature maps of the same size. Taking a 240×135×256 feature map as an example, pooling yields two 240×135×1 feature maps. These two feature maps are concatenated along the channel dimension and passed through a 7×7 convolutional layer with a sigmoid activation function, outputting a 240×135×1 spatial attention map, also with values ranging from 0 to 1. The channel attention weights are multiplied by the spatial attention map to obtain a feature enhancement coefficient, which is applied to each layer's feature map for selective feature enhancement. The enhanced features are fused using adaptive weights learned through a 1×1 convolutional layer to ensure that important features receive higher weights. The fused features are processed by a region proposal network to generate candidate boxes for the socket region. The region proposal network uses nine anchor boxes of varying scales and sizes, and uses two parallel branches to predict the candidate box's category score and bounding box coordinate offset. A non-maximum suppression threshold of 0.7 is set, and the top 20 scoring candidate boxes are retained as the socket region detection results.

[0032] When building a time-aware multi-target detection framework, we first perform 3D convolution feature extraction on multiple consecutive image frames. Sixteen consecutive image frames are selected, and a 3×3×3 3D convolution kernel with a stride of (1,1,1) is used to extract spatiotemporal features. The temporal attention coefficient between adjacent frames is calculated by calculating feature similarity, with frames with higher similarity corresponding to higher attention coefficients. The attention coefficient is applied to the feature map to obtain temporal enhancement features. Optical flow feature extraction uses the PWC-Net network to calculate the pixel displacement field between adjacent frames. For two consecutive frames, a two-channel optical flow field with the same image size is output, with horizontal and vertical motion vectors corresponding to one channel each. The optical flow features and temporal enhancement features are concatenated in the channel dimension and processed through a 3×3 convolution layer to obtain motion features.

[0033] The multi-target tracking module consists of two parts: surface feature extraction and motion prediction. Surface features are extracted using a deep residual network with a feature dimension of 512. Motion prediction is implemented using a Kalman filter, and the state vector contains target position, size, and velocity information. During tracking, the current detection result is correlated with the existing trajectory by calculating the similarity, with a similarity threshold set to 0.5. For newly appearing targets, a new trajectory is created when they are detected in three consecutive frames; for disappearing targets, the trajectory is terminated when they are not detected in five consecutive frames. The final output is the detection results of the illegal electrical equipment, including the device type, location coordinates, and tracking ID.

[0034] For training, the dataset was split into training, validation, and test sets in a 7:2:1 ratio. The training data was augmented with random cropping, rotation, and brightness and contrast adjustments. For video sequences, random frame sampling and temporal perturbations were also introduced to enhance the model's robustness to temporal variations. Model training was performed in a phased strategy. In the first phase, the backbone network was initialized with the weights of a ResNet-50 pre-trained on ImageNet. Fine-tuning was performed using the SGD optimizer with momentum, with an initial learning rate of 0.01 and a momentum factor of 0.9. The training batch size was 32, and training was repeated for 100 epochs. The loss function included a focal loss for object detection, a smooth L1 loss for bounding box regression, and an attention weight regularization term for the feature enhancement module. The second phase optimized temporal feature learning. Optical flow features were extracted using the pre-trained PWC-Net model, and the temporal attention module was trained using a teacher-student architecture. The ReID model in the multi-object tracking module was trained using a triplet loss.

[0035] Existing target detection technologies primarily rely on single features and static image analysis, lacking the full utilization of temporal information, leading to missed detections and false detections in complex scenarios. To address the particularities of detecting illegal electrical equipment in lottery stores, this application introduces an adaptive feature enhancement module that highlights key areas and features through a combination of channel attention and spatial attention; constructs a temporally aware multi-target detection framework that fully utilizes the temporal information in video sequences; and combines optical flow features with depth features to enhance the system's ability to perceive moving targets.

[0036] like Figure 2The multi-target tracking performance comparison chart shows the occlusion duration (in frames), ranging from 0 to 45 frames; the vertical axis shows the tracking recovery rate (percentage), ranging from 0% to 100%. The three curves represent the performance of different tracking methods: the present invention (square markers), DeepSORT (circle markers), and SORT (triangle markers). It can be seen that when the occlusion duration is short (5-10 frames), all three methods maintain a high tracking recovery rate, but the present invention performs best, reaching over 98.3%. As the occlusion duration increases, the performance of all three methods decreases, but the performance degradation of the present invention is significantly slower than that of the other two methods. When the occlusion lasts for 20 frames, the present invention still maintains a 93.5% recovery rate, while DeepSORT and SORT drop to 77.3% and 68.9%, respectively. Under long-term occlusion (45 frames), the present invention's tracking recovery rate is 73.6%, significantly higher than DeepSORT's 43.7% and SORT's 29.5%, showing a particularly significant advantage. This result fully demonstrates the excellent tracking robustness of the proposed temporal-aware multi-target detection framework, particularly when dealing with long-term occlusion scenarios. By effectively combining 3D convolutional feature extraction, temporal attention mechanisms, and optical flow feature analysis, this paper successfully addresses the performance bottleneck of traditional methods in occlusion scenarios, providing reliable technical support for practical applications such as detecting illegal electricity usage in lottery stores.

[0037] Optional, Scene partitioning of surveillance video data and regional adaptive adjustment of violation risk scores based on scene partitioning include: The lottery store is divided into a ticket display area, a customer waiting area, and an equipment operation area. Static and dynamic features are extracted from each area. Static features include socket distribution density and ticket vending equipment layout characteristics; dynamic features include the degree of crowd concentration, equipment movement trajectory, and power load changes. Adaptive weights are calculated based on regional characteristics and time period characteristics, and static and dynamic features are hierarchically integrated. Differentiated violation determinations are implemented for each area: ticket display area: detect temporary electricity usage; customer waiting area: detect mobile charging equipment and collective illegal electricity usage; equipment operation area: detect abnormal working status of ticket vending equipment and access to illegal electricity equipment.

[0038] For example, a lottery store is divided into three functional areas: ticket display, customer waiting area, and equipment operation area. This area division is defined using spatial coordinates, establishing a two-dimensional coordinate system with the store entrance as the origin. Areas are defined based on the layout of the store's facilities.

[0039] For static features, socket density is expressed as the number of sockets per square meter. A grid-based statistical method is used to divide each area into 0.5m x 0.5m grid cells, and the number of sockets within each grid is counted. Ticket vending machine layout features are characterized by the spacing between devices and their orientation angles. Relative position encoding is used to calculate the distance and orientation relationship between device center points.

[0040] For dynamic features, the concentration of people is quantified by density and dwell time within the area. A head-and-shoulder detection algorithm is used to count people and analyze dwell time. During peak hours, the density in the customer waiting area can reach 0.8 people per square meter, with an average dwell time of 15 minutes; the density in the ticket display area is approximately 0.4 people per square meter, with an average dwell time of 5 minutes; and the density in the equipment operation area is approximately 0.3 people per square meter, with an average dwell time of 8 minutes. Device movement trajectory features are extracted using trajectory encoding methods. Detected movable devices are continuously tracked, recording their position sequences and speed changes. For example, mobile charging devices are recorded, with their position coordinate sequences and dwell point information, and their movement frequency and cumulative distance traveled are calculated. In the customer waiting area, the average speed of mobile charging devices is 0.2 meters per second, and the cumulative distance traveled typically ranges from 5 to 20 meters. The power load variation feature is captured using micro-current sensors, which monitor current fluctuations in each circuit in real time. When current fluctuations outside the normal range are detected, the amplitude and duration of the fluctuation are recorded.

[0041] When calculating adaptive weights based on regional and time-of-day characteristics, regional functional attributes and time-of-day differences are taken into account. Regional functional attribute weights are set based on the importance of each area: the equipment operation area has a weight of 0.5, the customer waiting area has a weight of 0.3, and the ticket display area has a weight of 0.2. Time-of-day feature weights are dynamically adjusted based on the level of business activity during different time periods, categorized as peak, off-peak, and low-peak periods. Peak periods have a weight of 0.5, off-peak periods have a weight of 0.3, and low-peak periods have a weight of 0.2. Static and dynamic features are hierarchically integrated using a multi-level weighted combination approach. The first level integrates various indicators within static features, assigning a weight of 0.6 to socket density and a weight of 0.4 to ticket vending machine layout, to produce a comprehensive score for static features. The second level integrates various indicators within dynamic features, assigning a weight of 0.3 to personnel concentration, 0.4 to equipment movement trajectories, and 0.3 to power load variations, to produce a comprehensive score for dynamic features. The third level integrates the comprehensive scores of static and dynamic features, and adjusts the weight ratio according to the characteristics of the time period. During peak periods, the weight of dynamic features is increased to 0.7, and the weight of static features is reduced to 0.3; the opposite is true during off-peak periods.

[0042] When performing differentiated violation assessments for each area, temporary power usage detection in the ticket display area focuses on short-term current fluctuations and mobile device lingering. The threshold for determining temporary power usage is set as follows: current fluctuations outside the normal range lasting longer than 30 seconds and less than 5 minutes, and unauthorized devices being detected within 1 meter of an outlet. When these conditions are met, the system issues a temporary power usage warning, with a violation risk score between 0.6 and 0.8. In customer waiting areas, mobile charging device detection focuses on device type identification and prolonged lingering. The system can identify 10 common mobile charging devices, including mobile phone power banks and laptop power adapters. If such devices are detected near an outlet for more than 10 minutes, the system identifies them as mobile charging activity, with a violation risk score of 0.7. Mass illegal power usage is determined when three or more mobile charging devices are detected within a 15-minute window; in this case, the violation risk score increases to 0.9. Violation determination in the equipment operation area involves two aspects: Detection of abnormal operating conditions for ticket vending equipment is based on analysis of the equipment's power load characteristic curves, establishing a library of power usage patterns for normal operating conditions. When the real-time monitored power usage pattern deviates from the standard pattern by more than 20% for more than five minutes, it is considered an abnormal operating state, with a violation risk score of 0.75. Detection of illegal power device access is achieved by identifying the type of unauthorized device and its duration. If the system detects that a high-power unauthorized device, such as an electric vehicle charger, has been connected and in use for more than 10 minutes, the violation risk score reaches 0.95, and the system immediately generates a high-priority alarm.

[0043] This application uses a differentiated violation judgment mechanism based on scene partitioning, which can monitor possible illegal electricity usage behaviors in each functional area in a targeted manner and improve the accuracy and efficiency of detection.

[0044] Optional, Performing a time dimension correlation analysis on the abnormal current event and the illegal power-consuming equipment includes: obtaining time series data of current anomaly and illegal power-consuming equipment detection, constructing a time series correlation relationship between current anomaly, equipment detection and power consumption status based on a dynamic causal network, calculating time-varying weights and instantaneous causal strength using conditional entropy increment, combining eigenvector conversion and dynamic programming algorithm to obtain the optimal time lag, weighting the integral of the instantaneous causal strength with the exponential function of the optimal time lag to obtain an event matching score, and distinguishing temporary power consumption events from persistent illegal power consumption events based on the event matching score, including: Acquire abnormal current event data and illegal electric device detection data, wherein the abnormal current event data includes a current detection timestamp of the abnormal current event, and the illegal electric device detection data includes a video timestamp of the illegal electric device; Constructing a dynamic causal network, the dynamic causal network including a current node, a non-compliant electrical equipment detection node, and an equipment power status node; calculating time-varying edge weights between nodes in the dynamic causal network based on a conditional entropy increment, the conditional entropy increment representing the intensity of information flow between nodes; Calculating the instantaneous causal strength between nodes in the dynamic causal network, the instantaneous causal strength is calculated by the mutual information between the node at the current moment and the node at the next moment and the conditional probability of the node at the current moment, and integrating the instantaneous causal strength within a preset time window to obtain a cumulative causal effect; The temporal features of abnormal current events and illegal electrical equipment detection events are converted into time-frequency domain feature vectors. An improved dynamic programming algorithm is used to construct a cumulative cost matrix. The improved dynamic programming algorithm includes a time-varying weight coefficient for information flow intensity and a direction-sensitive jump connection that is adaptive to feature similarity. The optimal time-delay relationship is obtained based on the minimum path. The cumulative causal effect and the exponential function of the optimal time lag relationship are weighted to obtain an event matching score; when the event matching score is greater than a first preset threshold and the duration of the cumulative causal effect is greater than a second preset threshold, it is determined to be a continuous illegal electricity usage event; otherwise, it is determined to be a temporary electricity usage event.

[0045] For example, abnormal current event data and illegal electrical equipment detection data are obtained. Abnormal current event data is collected by intelligent current sensors installed in lottery store distribution boxes, with a sampling frequency set to 10Hz. The current value and corresponding timestamp are recorded. When the current value exceeds the preset normal range (e.g., ±20% of the rated current) for more than 3 seconds, the system generates a current abnormality event record containing information such as the abnormality start timestamp, the abnormal current value, and the duration of the abnormality. Illegal electrical equipment detection data comes from the video analysis system.

[0046] When constructing a dynamic causal network, three types of nodes are defined: the current node E represents the current sensor's measurement value; the illegal device detection node D represents the device type and location identified by the video detection system; and the device power status node S represents the inferred actual device power usage. In terms of time series, a dynamic network is constructed with T consecutive time points, with an interval of 0.1 seconds between each time point. Node connections include those within the same time point and those between different time points. The former represents spatial correlation, while the latter represents temporal correlation. The time-varying edge weights between nodes in the dynamic causal network are calculated based on the conditional entropy increment. The conditional entropy increment quantifies the intensity of information flow between nodes. It is calculated by statistically analyzing the reduction in uncertainty when observing the information of node B, given the information of node A, for any two nodes A and B. In practical implementation, node data is discretized, dividing continuous variables such as current values and device locations into multiple intervals. Current values are divided into intervals of 0.5 amperes, and device locations are divided into intervals of 0.5 meters. The conditional entropy increment is calculated by statistically analyzing the discretized conditional probability distribution. For example, in a certain detection, when the current node E increased from 3.2 amperes to 5.8 amperes, the conditional entropy increment of the illegal device detection node D reached 0.85, indicating that there is a strong information flow between the two.

[0047] When calculating the instantaneous causal strength between nodes in a dynamic causal network, the mutual information between the current node and the next node and the conditional probability of the node at the current moment are used. This is achieved by constructing a sliding time window with a window size of 5 seconds (i.e., 50 sampling points) and a step size of 0.5 seconds. Within each window, the temporal dependencies between nodes are calculated. For example, the instantaneous causal strength of an abnormal current event 0.3 seconds ago on the current device detection result is calculated. The resulting value ranges from 0 to 1, with larger values indicating stronger causal relationships. The instantaneous causal strength is accumulated within the 5-second window to obtain the cumulative causal effect, which is used to characterize the long-term correlation between events.

[0048] The short-time Fourier transform (SFT) method is used to convert the temporal features of abnormal current events and illegal electrical equipment detection events into time-frequency domain feature vectors. For abnormal current events, features such as current change rate, amplitude fluctuation frequency, and duration are extracted; for equipment detection events, features such as device movement speed, position change pattern, and device type are extracted. The feature vector dimension is set to 64, and feature normalization is performed to ensure consistency in magnitude across all dimensions. A modified dynamic programming algorithm is used to construct a cumulative cost matrix, which is used to calculate the optimal time-lag relationship between events. This algorithm incorporates two improvements over standard dynamic time warping (DTW): first, a time-varying weight coefficient for information flow intensity is introduced. The weight value is dynamically adjusted based on the conditional entropy increment, resulting in a lower path cost when the information flow intensity is high; second, direction-sensitive jump connections are added to adapt to feature similarity, allowing for non-continuous matching along the time axis. The jump step size is adaptively adjusted based on feature similarity, with a maximum jump step size of 3. By backtracing the path with the minimum cumulative cost, the optimal time-lag relationship between abnormal current events and equipment detection events is obtained, with typical time-lag values ranging from 0.2 seconds to 2.5 seconds.

[0049] The event matching score is calculated by weighting the cumulative causal effect with an exponential decay function of the optimal time lag relationship. The smaller the time lag, the closer the exponential function value is to 1; the larger the time lag, the closer the exponential function value is to 0. The weighting is set according to the application scenario, with typical values of 0.6 to 0.4, with the former representing the cumulative causal effect and the latter representing the time lag relationship. The resulting event matching score ranges from 0 to 1.

[0050] The criteria for distinguishing between temporary electricity usage events and persistent illegal electricity usage events are as follows: when the event matching score is greater than the first preset threshold and the duration of the cumulative causal effect is greater than the second preset threshold, it is determined to be a persistent illegal electricity usage event; otherwise, it is determined to be a temporary electricity usage event.

[0051] Existing technologies for detecting illegal electricity use are primarily based on video recognition or current monitoring, both of which have certain drawbacks. This application constructs a dynamic causal network to establish a temporal correlation model between current anomalies and device detection events. It introduces conditional entropy increments to calculate time-varying weights, accurately quantifying the intensity of information flow between events. It employs an improved dynamic programming algorithm to address the time alignment problem. Furthermore, it improves the accuracy of event matching through a comprehensive evaluation of cumulative causal effects and time lag relationships.

[0052] Table 1 shows the performance comparison of the current anomaly and illegal power device correlation performance between the present invention and three other existing methods (the traditional DTW algorithm, the fixed time window method, and the correlation coefficient-based method) in the current anomaly and illegal power device correlation analysis task. As can be seen, the event correlation accuracy of the present invention reaches 92.7%, demonstrating that it can more accurately associate current anomaly events with corresponding illegal power device detection events. Time lag estimation accuracy: The mean square error of the present invention's time lag estimation is only 0.28 seconds, significantly lower than that of other methods, demonstrating that it can more accurately determine the time delay relationship between current anomalies and device detection events. Illegal power usage identification performance: The present invention achieves an 89.5% identification rate for illegal power usage while maintaining a low false alarm rate (4.8%) and false negative rate (5.7%), which is crucial for accurately distinguishing normal power usage from illegal power usage in practical applications. Temporary / persistent event differentiation: The present invention achieves an F1 score of 0.91 for distinguishing temporary and persistent illegal power usage events, significantly higher than other methods, demonstrating its stronger ability to identify event types in complex scenarios. It's worth noting that the computational complexity of this method is slightly higher than that of the traditional DTW algorithm (1.32 vs. 1.00). This is due to the introduction of conditional entropy increment calculation and an improved dynamic programming algorithm. However, considering the significant performance improvement, this moderately increased computational cost is reasonable and acceptable. The data in this table fully demonstrates the technical advantages of this method in the application scenario of illegal electricity usage detection in lottery stores, especially its ability to accurately analyze the correlation between current anomalies and illegal electrical equipment in complex environments with low false positives.

[0053] Table 1 Comparison of performance correlation between abnormal current and illegal electrical equipment

[0054] Optional, The calculation of cumulative causal effect includes: Obtain node characteristics at the current moment and the next moment, wherein the node characteristics include abnormal current amplitude, duration, and device status information; The probability distribution of node features is calculated using the kernel density estimation method, where the bandwidth of the kernel function is determined by an adaptive method based on k-nearest neighbors, and the number of k-nearest neighbors is dynamically adjusted according to the sample density; The probability distribution of the node features is used to calculate the joint probability distribution of the node features at adjacent moments, and the mutual information and conditional probability are calculated based on the joint probability distribution and the marginal probability distribution; the instantaneous causal strength is calculated based on the mutual information and conditional probability: ; Among them, I(X t ;X t+1 ) is the feature vector X at the current moment t and the next moment eigenvector Xt+1 The mutual information between t+1 |X t ) is the feature vector X at the known current moment t Under the condition of t+1 Conditional probability of occurrence; When integrating the instantaneous causal strength, an adaptive time window mechanism is adopted. The length of the time window is adjusted with the rate of change of the causal strength. When the rate of change exceeds the preset adjustment threshold, the time window is shortened, and when the rate of change is lower than the preset adjustment threshold, the time window is increased. The cumulative causal effect is obtained according to the integration result, and the cumulative causal effect represents the degree of continuous influence between adjacent time nodes.

[0055] For example, node features are first acquired at the current moment and the next moment. Node features include the abnormal current amplitude, duration, and device status information. The abnormal current amplitude refers to the deviation of the current value from the normal baseline value and is collected in real time by a current sensor with a sampling rate of 10Hz. The duration records the duration of the abnormal current state. Device status information includes device type identification, operating mode flag, and location coordinates and is acquired through a video analysis system with an update frequency of 25 frames per second. Node features are represented as feature vectors, which contain multiple dimensions, each corresponding to a characteristic attribute.

[0056] When using kernel density estimation to calculate the probability distribution of node features, the Gaussian kernel function is selected as the base kernel function. This method places a smoothed Gaussian distribution around each data point and superimposes these distributions to estimate the overall probability density. The kernel function's bandwidth determines the smoothness of the estimate; a too large bandwidth results in oversmoothing, while a too small bandwidth results in excessive noise. This application uses an adaptive k-nearest-neighbor method to determine the bandwidth. Specifically, for each point in the feature space, the distance to its kth nearest neighbor is calculated as the local bandwidth. The value of k is not fixed but dynamically adjusted based on sample density. Smaller k values (e.g., k=5) are used in areas with high sample density, while larger k values (e.g., k=15) are used in areas with low sample density to balance the bias and variance of the estimate. Sample density is obtained through a preliminary global bandwidth estimate. The initial bandwidth is set using the Silverman rule. For d-dimensional data, the bandwidth calculation takes into account the number of samples n and the standard deviation of the data. Discrete variables in device status information, such as device type identification, are processed using a discrete kernel function.

[0057] To calculate the joint probability distribution of node features at adjacent moments using the probability distribution of node features, a multivariate kernel density estimation method is used. Specifically, the feature vectors at the current time t and the next time t+1 are concatenated into a higher-dimensional joint feature vector, and kernel density estimation is then applied to this high-dimensional space. For high-dimensional data, to mitigate the curse of dimensionality, a variable grouping strategy is employed: variables with strong correlations are grouped together. For example, current-related features are grouped together, while device status-related features are grouped together. Probability distributions are estimated for each group before being combined. The marginal probability distribution is obtained by integrating or summing the joint probability distribution over the corresponding dimension. For example, the marginal distribution of the feature vector at the current time is obtained by summing the joint distribution over all possible values at the next time. To improve computational efficiency, a gridding method is used to divide the feature space into a fixed-size grid, and probability values are calculated at each grid point. Mutual information and conditional probabilities are calculated based on the joint and marginal probability distributions. Mutual information measures the amount of information shared between two random variables and is calculated as the logarithmic expectation of the ratio of the product of the joint distribution and the marginal distribution. For example, when the abnormal current amplitude suddenly increases from 2 amps to 4 amps, the mutual information value reaches 0.78, indicating a strong correlation between the features at the two moments. Conditional probability represents the probability of a feature occurring at the next moment given the features at the current moment. It is calculated by dividing the joint probability by the marginal probability at the current moment.

[0058] An adaptive time window mechanism is used to integrate instantaneous causal strength. The base time window length is set to 5 seconds, with 50 sampling points within the window. The time window length is not fixed but dynamically adjusted based on the rate of change of causal strength. The rate of change is calculated by averaging the difference in instantaneous causal strength between three consecutive sampling points. When the rate of change exceeds a preset adjustment threshold (e.g., 0.05 / second), the system assumes that the causal relationship is changing rapidly and reduces the time window to 2.5 seconds to capture the change more precisely. When the rate of change falls below the preset adjustment threshold, the time window is increased to 10 seconds to obtain a more stable estimate. Changes in window length are smoothly transitioned to avoid sudden changes.

[0059] The integration process uses the trapezoidal rule to weight the sum of instantaneous causal strengths within a time window, with weights decaying over time, with more recent moments receiving greater weights. For example, in the electric scooter charging scenario, the cumulative causal effect calculated within a 5-second window is 3.25, indicating a strong, sustained impact; whereas in the mobile phone temporary charging scenario, the cumulative causal effect within the same window is 1.28, indicating a relatively weak impact. A larger cumulative causal effect value indicates a more persistent and significant impact of the current event on subsequent events.

[0060] Existing causal relationship analysis techniques primarily use statistical models with fixed parameters, making them difficult to adapt to complex scenarios such as current anomalies and illegal electrical equipment detection. This application introduces adaptive bandwidth kernel density estimation based on k-nearest neighbors, enabling probability distribution estimates to automatically adjust their smoothness based on data density. It also proposes a new method for calculating instantaneous causal strength, comprehensively considering two complementary metrics: mutual information and conditional probability. Finally, it designs an adaptive time window mechanism, enabling the integration process to dynamically adjust the window size based on the rate of change of the causal relationship.

[0061] Optional, The time series features of abnormal current events and illegal electrical equipment detection events are converted into time-frequency domain feature vectors. An improved dynamic programming algorithm is used to construct a cumulative cost matrix. The improved dynamic programming algorithm includes the introduction of a time-varying weight coefficient for information flow intensity and a direction-sensitive jump connection with adaptive feature similarity. The optimal time-delay relationship is obtained based on the minimum path, including: Extract the temporal characteristics of abnormal current events and illegal electrical equipment detection events, including the current amplitude change rate, duration, and fluctuation characteristics in the time domain, and the main frequency component, spectrum energy distribution, and frequency change trend in the frequency domain. Standardize the time and frequency domain characteristics to obtain time-frequency domain feature vectors, and construct current event feature sequences and device event feature sequences. An improved dynamic programming algorithm is used to construct the cumulative cost matrix of current event feature sequences and device event feature sequences. The improvements include: introducing the intensity of information flow between nodes as a time-varying weight coefficient into the cost calculation, setting direction-sensitive jump connections in the cumulative cost matrix calculation process, and adaptively adjusting the jump step size based on feature similarity; An initial time-lag relationship is obtained based on the minimum path of the cumulative cost matrix, a time-lag consistency index is calculated, and the time window for feature extraction is segmentedly adjusted according to the time-lag consistency index. It is iteratively optimized until the time-lag consistency index meets the preset optimization threshold requirement, thereby obtaining the optimal time-lag relationship.

[0062] For example, for abnormal current events, the extracted time-domain features include the current amplitude change rate, duration, and fluctuation characteristics. The current amplitude change rate is calculated by dividing the difference in current values between adjacent sampling points by the sampling interval. The duration records the duration of the abnormal current state with millisecond accuracy. The fluctuation characteristics are characterized by calculating the standard deviation of the current values and the peak-to-valley ratio. When a normal electrical appliance is operating, the standard deviation of current fluctuation is typically less than 0.3 amps, while unstable electrical equipment can reach over 0.8 amps. Frequency-domain features are obtained by performing a short-time Fourier transform on the time-domain current data, with a window length of 2 seconds and a 50% overlap rate. The extracted frequency-domain features include the dominant frequency component, spectral energy distribution, and frequency variation trend. The dominant frequency component refers to the frequency point with the highest energy in the spectrum. Standard electrical appliances typically exhibit a fixed dominant frequency characteristic. For example, lottery ticket printing equipment typically has a dominant frequency component of approximately 5 Hz when operating. The spectral energy distribution is characterized by calculating the energy proportion of different frequency bands, which are divided into three intervals: low frequency (0-10 Hz), mid-frequency (10-30 Hz), and high frequency (30-50 Hz). The frequency change trend is represented by the change slope of the main frequency component in the continuous time window. During the charging process of the electric vehicle, the main frequency component usually shows a downward trend in the first 10 minutes, with a slope of about -0.2Hz / minute.

[0063] For detection events of illegal electrical devices, time and frequency domain features are also extracted. Time domain features include device position change rate, target confidence, and size change features. The position change rate indicates the rate of change of the device center coordinates over time, measured in pixels per second. The target confidence is the probability value output by the detection algorithm, ranging from 0 to 1. The size change feature records the change in the area of the device's bounding box. Frequency domain features focus on the periodic characteristics of device detection results and are obtained by performing spectral analysis on the detection confidence sequence.

[0064] The time-domain and frequency-domain features are standardized using the Z-score method, ensuring that the mean of each dimension is 0 and the standard deviation is 1. The standardized features are combined into a time-frequency domain feature vector with 64 dimensions, including 32 dimensions for time-domain features and 32 dimensions for frequency-domain features. Current event feature sequences and device event feature sequences are constructed based on timestamps, with sequence length dynamically adjusted based on the monitoring duration.

[0065] When constructing the cumulative cost matrix using an improved dynamic programming algorithm, a basic distance metric is first defined. The distance between feature vectors is weighted Euclidean distance. Weights for different feature dimensions are set based on feature importance, which is pre-evaluated using a feature selection algorithm. For example, the weight for the current amplitude change rate dimension is set to 0.2, the weight for the dominant frequency component dimension is set to 0.15, and the weights for other dimensions range from 0.01 to 0.1. The improved dynamic programming algorithm incorporates two factors based on standard dynamic time warping (DTW). First, the intensity of information flow between nodes is incorporated into the cost calculation as a time-varying weight coefficient. Information flow intensity is obtained through conditional entropy increment and describes the causal relationship between two nodes. A weight adjustment factor is introduced into the cumulative cost calculation. When the information flow intensity is high, the corresponding path cost decreases; otherwise, it increases. Specifically, the original feature distance is multiplied by the inverse function of the information flow intensity. When the flow intensity is 0.8, the distance is reduced to 0.6 times the original value; when the flow intensity is 0.2, the distance is increased to 1.4 times the original value. This makes the algorithm more inclined to select paths with strong causal relationships. Second, direction-sensitive skip connections are implemented during the calculation of the cumulative cost matrix. The standard DTW algorithm requires path continuity and only allows local movement in three directions (right, up, and right-up). The improved algorithm adds skip connections, allowing for the skipping of some points under certain conditions to account for noise and missing points in the event sequence. The skip connections are set to be direction-sensitive, with a maximum skip step size of 3 in the horizontal direction (current event axis) and 2 in the vertical direction (device event axis), reflecting the differences in sampling rate and reliability between current events and device events. The skip step size is adaptively adjusted based on feature similarity, controlled by a set similarity threshold. When the feature similarity between adjacent points is greater than 0.85, the maximum step size is allowed; when the similarity is between 0.7 and 0.85, medium step size is allowed; and when the similarity is below 0.7, only the minimum step size is allowed. This adaptive mechanism ensures that large jumps can be made in areas with good signal quality to improve efficiency, while small step sizes are used in areas with poor signal quality to ensure accuracy.

[0066] The initial time lag relationship is obtained based on the minimum path of the cumulative cost matrix. Starting from the lower right corner of the matrix, the path with the minimum cumulative cost is traced back to the upper left corner. The corresponding relationship along the path is recorded, that is, the time correspondence between the current event and the device event. The initial time lag relationship is defined as the difference between the timestamps of the two events. The mean, median, and standard deviation of this sequence are calculated to obtain the statistical characteristics of the time lag.

[0067] The lag consistency index is calculated by dividing the standard deviation of the lag series by the average lag. This index reflects the stability of the lag estimate; smaller values indicate more consistent lag estimates. The time window for feature extraction is segmented based on the lag consistency index. When the index value is greater than 0.3, the original sequence is divided into multiple subsegments, each with a length of 1 / 3 to 1 / 2 the original. The optimal lag within each subsegment is calculated. Feature extraction parameters are also adjusted, such as increasing the weight of frequency domain features or adjusting the extraction window length for time domain features.

[0068] The iterative optimization process sets a maximum number of iterations to 5. After each iteration, a new time-lag consistency index is calculated. Optimization stops when the index value is less than the preset optimization threshold of 0.15 or the maximum number of iterations is reached. The result with the smallest time-lag consistency index is ultimately selected as the optimal time-lag relationship. In the electric vehicle charging scenario test, after 3 iterations, the time-lag consistency index dropped from the initial 0.28 to 0.12, and the optimal time-lag relationship was determined to be 0.85 seconds, indicating a 0.85-second delay between the occurrence of the current anomaly event and the confirmation of the device detection event.

[0069] This application solves the problem of timing alignment of abnormal current events and illegal power equipment detection events by constructing a comprehensive time-frequency domain feature vector, introducing causal strength-driven dynamic programming, designing direction-sensitive adaptive jump connections, and iterative optimization based on time-delay consistency. It realizes accurate correlation analysis in complex scenarios, overcomes the impact of noise and missing time series data, and improves the reliability of identifying illegal power usage behaviors.

[0070] Optionally, based on spatiotemporal trajectories, adaptive distance metrics, and device aggregation, basic features are extracted, including cumulative residence time, number of entry and exit events, motion features, and maximum aggregation; a behavior pattern dictionary is established, and the basic feature sequences of historical violation samples are clustered and analyzed to extract feature vectors of typical violation patterns and store the corresponding risk levels; the dynamic time warping distance between the current basic feature sequence and the typical pattern in the behavior pattern dictionary is calculated, and the K patterns with the highest similarity are selected as references, and the similarity score is used as a supplementary dimension of the violation behavior feature vector. Specifically, the density clustering algorithm DBSCAN is used to cluster the basic feature sequences, where the neighborhood radius is set to 0.5 and the minimum number of samples is set to 4. After normalization of the features, N typical violation pattern clusters are obtained; the fast dynamic time warping algorithm is used to calculate the sequence similarity, and an appropriate initial window size is set. The basic similarity is calculated using the Euclidean distance, and the optimal path distance is obtained recursively; based on the distance measurement results, the most similar patterns are weighted, and the smaller the distance, the greater the weight. The risk level of each pattern is multiplied by the corresponding weight and the sum is obtained to obtain the similarity score, which is added as a new feature dimension to the original basic feature vector to obtain an enhanced violation behavior feature vector.

[0071] The time-series-aware multi-target detection framework also includes a suspected minor identification module, which includes: extracting human key points and constructing height assessment features based on the time-series enhancement features, including: estimating relative height using the pixel ratio of human key points to fixed reference objects in the scene; extracting carried item features and behavioral features based on the motion features, including: detecting student-specific items such as backpacks, and analyzing hand-reaching height, operation proficiency, and waiting and pausing behaviors during the lottery purchase process; calculating a suspected minor risk score, including: assigning weight coefficients to relative height features, carried item features, and behavioral features, respectively, wherein the highest score is obtained when the relative height is below a first preset threshold, a secondary score is added when student-specific items such as backpacks are detected, and a behavioral score is added when the hand-reaching height is too high, multiple pauses and waiting, or the operation is inconsistent during the operation; when the suspected minor risk score exceeds the preset threshold, the lottery purchase record is marked as pending verification, and a verification task is generated including the lottery purchase time, scene image, and feature description; the verification task is pushed to the manual review platform for secondary confirmation, and the review results are fed back for optimization and adjustment of feature weights.

[0072] Specifically, the method for identifying suspected minors based on video surveillance data achieves intelligent early warning of illegal lottery purchasing behavior through feature extraction, risk assessment, and manual review. For height assessment feature extraction, the system uses lottery counters and door frames as calibration reference points during store deployment, recording their actual heights. Real-time analysis extracts 17 key points, including the top of the head, neck, shoulders, hips, and ankles, and calculates the ratio of the target's pixel height to the reference height to obtain relative height. A higher risk score is assigned when the target's relative height is less than 160 cm. For item feature extraction, an object detection model is used to identify typical student items, such as backpacks and student ID holders. The model input is a 320x320 pixel video frame, and a detection confidence level above 0.6 is considered a valid recognition result. Detection of backpacks is assigned a feature score of 0.3, while detection of other characteristic student items is assigned a feature score of 0.1-0.2.

[0073] During the behavioral feature extraction phase, key action sequences during the lottery purchase process are analyzed. The positions of key hand points are calculated to determine whether the operator's posture is abnormal. A hand position higher than the shoulder indicates difficulty reaching for the target and is recorded as an unnatural posture. Within a 30-second analysis window, pauses and waiting behaviors are counted. More than three pauses, or a single pause exceeding five seconds, are considered hesitant. The operation sequence returned by the lottery vending terminal is analyzed for operational consistency. Intervals greater than four seconds or repeated key presses are considered unskilled. A behavioral feature score of 0.1-0.3 is assigned based on the number and severity of abnormal behaviors. The risk assessment phase weights the height feature score, the item feature score, and the behavioral feature score to generate an overall risk score. If the risk score exceeds 0.7, an alert is generated. The alert contains information such as the purchase time, the target person's image, the estimated height, the detected feature item category and confidence level, and a description of the abnormal behavior. During the manual review phase, a dedicated person reviews the alert task and confirms or rejects the system's determination. The review results are used to dynamically adjust feature weights: for approved alerts, the corresponding feature weight increases by 0.05; for rejected alerts, the corresponding feature weight decreases by 0.05. The weight adjustment range is limited to within plus or minus 30% of the original value to ensure system stability.

[0074] The present invention combines multi-dimensional features such as height assessment, identification of carried items, and behavior analysis to achieve intelligent early warning of suspected lottery purchasing behavior by minors. It can complete suspicious target screening without facial recognition, reducing computational complexity, improving on-site identification efficiency, and continuously optimizing model performance through a manual review and feedback mechanism, providing effective technical support for preventing minors from purchasing lottery tickets.

[0075] According to a second aspect of the embodiments of the present invention, Provides a lottery store violation detection system based on multimodal data fusion, including: The first unit is used to obtain the power monitoring data and surveillance video data of the lottery store; The second unit is configured to detect abnormal current events based on the power monitoring data and record corresponding current detection timestamps; perform time-series-aware target detection on the monitoring video data, identify illegal power-consuming devices in video image frames based on adaptive feature enhancement and multi-target tracking, and record illegal power-consuming events and corresponding video timestamps based on device aggregation and spatiotemporal trajectory features; The third unit is used to perform time dimension correlation analysis on the abnormal current events and illegal power-consuming equipment, including: obtaining time series data of current anomalies and illegal power-consuming equipment detection, constructing a time series correlation relationship between current anomalies, equipment detection and power consumption status based on a dynamic causal network, using conditional entropy increment to calculate time-varying weights and instantaneous causal strength, combining eigenvector conversion and dynamic programming algorithm to obtain the optimal time lag, weighting the integral of the instantaneous causal strength with the exponential function of the optimal time lag to obtain an event matching score, distinguishing temporary power consumption events from continuous illegal power consumption events based on the event matching score; when the continuous illegal power consumption event is detected, generating violation alarm information.

[0076] According to a third aspect of the embodiments of the present invention, An electronic device is provided, comprising: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0077] According to a fourth aspect of the embodiments of the present invention, A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the method described above is implemented.

[0078] The present invention may be a method, an apparatus, a system and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present invention.

[0079] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A lottery store violation detection method based on multimodal data fusion, characterized by: include: Obtain electricity monitoring data and surveillance video data from lottery stores; Detecting abnormal current events according to the power monitoring data and recording corresponding current detection timestamps; Performing time-series-aware target detection on the surveillance video data, identifying illegal electrical devices in the video image frames based on adaptive feature enhancement and multi-target tracking, and recording illegal electricity use events and corresponding video timestamps based on device clustering and spatiotemporal trajectory features; Performing a time-dimensional correlation analysis on the abnormal current events and illegal power-consuming equipment, including: obtaining time series data of current anomalies and illegal power-consuming equipment detection, constructing a time-series correlation relationship between current anomalies, equipment detection, and power consumption status based on a dynamic causal network, calculating time-varying weights and instantaneous causal strength using conditional entropy increments, combining eigenvector conversion with a dynamic programming algorithm to obtain an optimal time lag, weighting the integral of the instantaneous causal strength with an exponential function of the optimal time lag to obtain an event matching score, and distinguishing temporary power consumption events from persistent illegal power consumption events based on the event matching score; When the persistent illegal electricity usage event is detected, illegal alarm information is generated.

2. The method according to claim 1, characterized in that The target detection for the surveillance video data with time-series perception includes: Preprocessing the surveillance video data to obtain a standardized video frame sequence; Identify socket areas in standardized video frame sequences; detect illegal electrical devices and construct spatiotemporal trajectories containing location coordinates, velocity, acceleration, motion direction angle, and timestamps; Calculate an adaptive distance metric between the illegal electrical device and the socket area, where the adaptive distance metric is composed of the minimum Euclidean distance between the current location of the device and the location point set of the socket area and a speed influence factor; When multiple illegal electrical devices are detected, the device concentration degree of the socket area is calculated. The device concentration degree is the weighted sum of the type weight of each illegal electrical device and its corresponding Gaussian distance response value, which is calculated using the adaptive distance metric and Gaussian kernel parameters. constructing a violation behavior feature vector based on the spatiotemporal trajectory, the adaptive distance metric, and the device aggregation, the violation behavior feature vector including the cumulative residence time, the number of entry and exit events, the motion characteristics, and the maximum aggregation value; The violation behavior feature vector is normalized and weighted to obtain a violation risk score, and the illegal electrical equipment is identified and the corresponding timestamp is recorded.

3. The method according to claim 2, characterized in that Steps to identify outlet zones and detect offending electrical devices include: The backbone neural network is used to extract multi-level features of video images and obtain feature map sequences of different scales; Constructing an adaptive feature enhancement module, including: calculating channel attention weights through global average pooling and multi-layer perceptron; calculating spatial attention maps through convolution of max pooling and average pooling features; using the product of channel attention weights and spatial attention maps as the feature enhancement coefficient to enhance the feature map sequence; using adaptive weights to fuse the enhanced features, and performing socket region detection based on the fused features through a region proposal network; A time-aware multi-target detection framework is constructed, including: performing three-dimensional convolution feature extraction on multiple consecutive frames of images, calculating the time-series attention coefficient between adjacent frames, and obtaining time-series enhancement features; extracting optical flow features between adjacent frames, splicing the optical flow features with the time-series enhancement features, and obtaining motion features through the convolution layer; performing multi-target tracking through appearance feature extraction and motion prediction, and outputting the detection results of illegal electrical equipment.

4. The method according to claim 2, characterized in that Scene partitioning of surveillance video data and regional adaptive adjustment of violation risk scores based on scene partitioning include: The lottery store is divided into a ticket display area, a customer waiting area, and an equipment operation area. Static and dynamic features are extracted from each area. Static features include socket distribution density and ticket vending equipment layout characteristics; dynamic features include the degree of crowd concentration, equipment movement trajectory, and power load changes. Adaptive weights are calculated based on regional characteristics and time period characteristics, and static and dynamic features are hierarchically integrated. Differentiated violation determinations are implemented for each area: ticket display area: detect temporary electricity usage; customer waiting area: detect mobile charging equipment and collective illegal electricity usage; equipment operation area: detect abnormal working status of ticket vending equipment and access to illegal electricity equipment.

5. The method according to claim 1, wherein Performing a time dimension correlation analysis on the abnormal current event and the illegal power-consuming equipment includes: obtaining time series data of current anomaly and illegal power-consuming equipment detection, constructing a time series correlation relationship between current anomaly, equipment detection and power consumption status based on a dynamic causal network, calculating time-varying weights and instantaneous causal strength using conditional entropy increment, combining eigenvector conversion and dynamic programming algorithm to obtain the optimal time lag, weighting the integral of the instantaneous causal strength with the exponential function of the optimal time lag to obtain an event matching score, and distinguishing temporary power consumption events from persistent illegal power consumption events based on the event matching score, including: Acquire abnormal current event data and illegal electric device detection data, wherein the abnormal current event data includes a current detection timestamp of the abnormal current event, and the illegal electric device detection data includes a video timestamp of the illegal electric device; Constructing a dynamic causal network, the dynamic causal network including a current node, a non-compliant electrical equipment detection node, and an equipment power status node; calculating time-varying edge weights between nodes in the dynamic causal network based on a conditional entropy increment, the conditional entropy increment representing the intensity of information flow between nodes; Calculating the instantaneous causal strength between nodes in the dynamic causal network, the instantaneous causal strength is calculated by the mutual information between the node at the current moment and the node at the next moment and the conditional probability of the node at the current moment, and integrating the instantaneous causal strength within a preset time window to obtain a cumulative causal effect; The temporal features of abnormal current events and illegal electrical equipment detection events are converted into time-frequency domain feature vectors. An improved dynamic programming algorithm is used to construct a cumulative cost matrix. The improved dynamic programming algorithm includes a time-varying weight coefficient for information flow intensity and a direction-sensitive jump connection that is adaptive to feature similarity. The optimal time-delay relationship is obtained based on the minimum path. The cumulative causal effect and the exponential function of the optimal time lag relationship are weighted to obtain an event matching score; when the event matching score is greater than a first preset threshold and the duration of the cumulative causal effect is greater than a second preset threshold, it is determined to be a continuous illegal electricity use event.

6. The method according to claim 5, characterized in that The calculation of cumulative causal effect includes: Obtain node characteristics at the current moment and the next moment, wherein the node characteristics include abnormal current amplitude, duration, and device status information; The probability distribution of node features is calculated using the kernel density estimation method, where the bandwidth of the kernel function is determined by an adaptive method based on k-nearest neighbors, and the number of k-nearest neighbors is dynamically adjusted according to the sample density; The probability distribution of the node features is used to calculate the joint probability distribution of the node features at adjacent moments, and the mutual information and conditional probability are calculated based on the joint probability distribution and the marginal probability distribution; the instantaneous causal strength is calculated based on the mutual information and conditional probability: ； Among them, I(X t ;X t+1 ) is the feature vector X at the current moment t and the next moment eigenvector X t+1 The mutual information between t+1 |X t ) is the feature vector X at the known current moment t Under the condition of t+1 Conditional probability of occurrence; When integrating the instantaneous causal strength, an adaptive time window mechanism is adopted. The length of the time window is adjusted with the rate of change of the causal strength. When the rate of change exceeds the preset adjustment threshold, the time window is shortened, and when the rate of change is lower than the preset adjustment threshold, the time window is increased. The cumulative causal effect is obtained according to the integration result, and the cumulative causal effect represents the degree of continuous influence between adjacent time nodes.

7. The method according to claim 5, characterized in that The time series features of abnormal current events and illegal electrical equipment detection events are converted into time-frequency domain feature vectors. An improved dynamic programming algorithm is used to construct a cumulative cost matrix. The improved dynamic programming algorithm includes the introduction of a time-varying weight coefficient for information flow intensity and a direction-sensitive jump connection with adaptive feature similarity. The optimal time-delay relationship is obtained based on the minimum path, including: Extract the temporal characteristics of abnormal current events and illegal electrical equipment detection events, including the current amplitude change rate, duration, and fluctuation characteristics in the time domain, and the main frequency component, spectrum energy distribution, and frequency change trend in the frequency domain. Standardize the time and frequency domain characteristics to obtain time-frequency domain feature vectors, and construct current event feature sequences and device event feature sequences. An improved dynamic programming algorithm is used to construct the cumulative cost matrix of current event feature sequences and device event feature sequences. The improvements include: introducing the intensity of information flow between nodes as a time-varying weight coefficient into the cost calculation, setting direction-sensitive jump connections in the cumulative cost matrix calculation process, and adaptively adjusting the jump step size based on feature similarity; An initial time-lag relationship is obtained based on the minimum path of the cumulative cost matrix, a time-lag consistency index is calculated, and the time window for feature extraction is segmentedly adjusted according to the time-lag consistency index. It is iteratively optimized until the time-lag consistency index meets the preset optimization threshold requirement, thereby obtaining the optimal time-lag relationship.

8. A lottery store violation detection system based on multimodal data fusion, used to implement the method according to any one of claims 1 to 7, characterized in that: include: The first unit is used to obtain the power monitoring data and surveillance video data of the lottery store; a second unit, configured to detect abnormal current events according to the power monitoring data and record corresponding current detection timestamps; Performing time-series-aware target detection on the surveillance video data, identifying illegal electrical devices in the video image frames based on adaptive feature enhancement and multi-target tracking, and recording illegal electricity use events and corresponding video timestamps based on device clustering and spatiotemporal trajectory features; The third unit is used to perform time-dimensional correlation analysis on the abnormal current events and illegal power-consuming equipment, including: obtaining time series data of current anomalies and illegal power-consuming equipment detection, constructing a time series correlation relationship between current anomalies, equipment detection and power consumption status based on a dynamic causal network, calculating time-varying weights and instantaneous causal strength using conditional entropy increments, combining eigenvector conversion and dynamic programming algorithms to obtain the optimal time lag, weighting the integral of the instantaneous causal strength with the exponential function of the optimal time lag to obtain an event matching score, and distinguishing temporary power consumption events from persistent illegal power consumption events based on the event matching score; When the persistent illegal electricity usage event is detected, illegal alarm information is generated.

9. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images