Intelligent warehousing monitoring system and method based on artificial intelligence

Abstract

The invention relates to the technical field of intelligent monitoring systems, in particular to a storage intelligent monitoring system and method based on artificial intelligence, and the system comprises a multi-source data collection module which synchronously collects video data, environment sensing data and equipment operation data in a storage scene, guarantees the time consistency of multi-source data through a timestamp synchronization mechanism, and transmits the data to a server; the acquisition frequency is dynamically adjusted according to data types; the target detection and positioning module is used for realizing real-time detection and position positioning of personnel, forklifts, AGVs and goods targets in a warehouse based on the preprocessed video data, adopting an improved YOLOv8 model and then introducing an attention mechanism to improve the recognition precision in a target shielding scene; according to the invention, the multi-source data acquisition module is used as a sensing antenna, a comprehensive, synchronous and clean data source is constructed, a foundation is laid for subsequent analysis, and multi-dimensional data can be efficiently acquired without manual inspection.

Description

Technical Field

[0001] This invention relates to the field of intelligent monitoring system technology, and in particular to an intelligent monitoring system and method for warehouses based on artificial intelligence. Background Technology

[0002] As a core hub in the supply chain connecting production and distribution, the operational quality of modern warehousing directly determines a company's market responsiveness and overall efficiency. With the rapid development of e-commerce retail and intelligent manufacturing, warehousing scenarios are characterized by a surge in the variety of goods, more complex operational processes, and maximized space utilization. Existing warehouse monitoring methods are mainly divided into two categories: traditional manual monitoring and basic intelligent monitoring. Both have significant drawbacks and are difficult to meet the needs of modern warehousing. Traditional manual monitoring relies on regular inspections by patrol personnel and fixed camera monitoring. Not only do labor costs rise sharply with the expansion of warehouse scale, but it also suffers from inherent problems such as response delays, high missed detection rates, and subjective judgment biases. Although basic intelligent monitoring introduces technologies such as video recognition, it mostly adopts a single data collection and analysis mode, without establishing a correlation mechanism between video data and environmental parameters (temperature, humidity, dust concentration) and equipment operating status (forklift load, rack stress). This results in significant bias in anomaly detection and a high false alarm rate. Summary of the Invention

[0003] In response to the technical problems mentioned in the background section, this invention provides an intelligent warehouse monitoring system and method based on artificial intelligence.

[0004] The technical solution adopted in this invention is: an intelligent warehouse monitoring system based on artificial intelligence, comprising: Multi-source data acquisition module: Synchronously collects video data, environmental sensor data and equipment operation data in the warehouse scenario, and uses a timestamp synchronization mechanism to ensure the time consistency of multi-source data. The acquisition frequency is dynamically adjusted according to the data type. Target detection and localization module: Based on preprocessed video data, it realizes real-time detection and location of personnel, forklifts, AGVs and goods in the warehouse. It adopts an improved YOLOv8 model and further improves the recognition accuracy in target occlusion scenarios by introducing an attention mechanism. Multimodal attention fusion module: Deeply fuses preprocessed environmental / device sensing data with output target information, dynamically assigns weights to different data through an attention mechanism, and highlights key information for anomaly detection; Anomaly detection module: Based on the fused features of the output, an improved isolated forest model is constructed to detect warehouse anomalies; Target Behavior Analysis Module: Further analyzes the temporal correlation of target behaviors based on the anomalies initially identified by the anomaly detection module that are related to the target. Environment and Equipment Status Assessment Module: Based on the environmental or equipment anomalies initially identified by the anomaly detection module, this module combines time-series information from multimodal fusion features to quantitatively assess environmental stability and equipment health status, providing a basis for anomaly tracing. Anomaly Cause Tracing Module: Based on causal inference theory, and combined with the analysis results of the target behavior analysis module and the environment and equipment status assessment module, an anomaly cause-effect graph is constructed to locate the root cause of the anomaly; Intelligent Decision and Strategy Generation Module: Based on the root causes of anomalies located by the anomaly causal tracing module and the analysis results of each module, a reinforcement learning decision model is constructed to generate targeted handling strategies; Execution and Feedback Optimization Module: This module transforms the decision-making strategies generated by the intelligent decision-making and strategy generation module into specific execution instructions, controls the actions of execution equipment within the warehouse, and collects feedback data after strategy execution for model optimization, forming a closed-loop system.

[0005] In one embodiment, the multi-source data acquisition module specifically includes the following: Data types and acquisition devices: Video data is acquired through high-definition network cameras deployed in warehouse aisles, shelving areas, and entrances / exits; environmental data is acquired through distributed sensor nodes, including temperature and humidity sensors, dust concentration sensors, and gas sensors; equipment data is acquired through the OBD interfaces of forklifts and AGV mobile devices and shelving stress sensors, including equipment operating speed, load weight, battery level, and real-time shelving stress. Data preprocessing: Differentiated preprocessing methods are adopted based on the noise characteristics of different types of data; Video data denoising: Gaussian filtering is used to remove image noise, as shown in the following formula: ; in, The grayscale value of the pixel after filtering; The current pixel coordinates; , σ represents the pixel coordinates of the center of the convolution kernel; σ is the standard deviation of the Gaussian function. The original pixel grayscale value; * indicates a convolution operation; Sensor data standardization: Due to the large difference in dimensions between environmental and device data, Z-score standardization is used to eliminate the influence of dimensions. The formula is as follows: ; in, The data is standardized. This is the raw sensor data; This is the mean of this type of data; This represents the standard deviation of this type of data; Data time synchronization: The timestamp alignment method based on the network time protocol is adopted. All acquisition devices are synchronized to the clock of the warehouse monitoring center. The time error is controlled within 50ms. The synchronized data is stored in the format of "device ID, timestamp and data value".

[0006] In one embodiment, the target detection and localization module specifically includes the following: Object detection model: An improved YOLOv8 neck network is added, incorporating a coordinate attention module to enhance the model's ability to extract spatial location information of objects. The model loss function adopts CIoU loss, as shown in the following formula: ; in, The intersection-union ratio (IUU) of the predicted bounding box and the ground truth bounding box reflects the degree of overlap between the two. Center point of the prediction box Center point of the real frame The Euclidean distance; It is the diagonal length of the smallest bounding rectangle that simultaneously contains both the predicted bounding box and the ground truth bounding box; This is the balance coefficient; A parameter used to measure aspect ratio consistency; Target location localization: Based on the detected target bounding box, calculate the real-time position of the target in the warehouse coordinate system; First, the conversion relationship between pixel coordinates and world coordinates is obtained through camera calibration, using the following formula: ; in, This is the camera intrinsic parameter matrix; It is a rotation matrix; (X,Y,Z) is the translation vector; (X,Y,Z) are the coordinates of the target in the world coordinate system. This module outputs information such as target type, real-time world coordinates (X,Y,Z), and target size, with positioning error controlled within ±10cm.

[0007] In one embodiment, the multimodal attention fusion module specifically includes the following: Feature extraction: Feature extraction is performed on each modality of data separately; Video target features are obtained by improving the output feature map of YOLOv8, with a dimension of D1=512; Environmental sensor data, including temperature and humidity (T), dust concentration (P), and combustible gas concentration (Q), are extracted using a 3-layer fully connected network with a dimension of D2=64. Device data are extracted using a 1D-CNN to extract temporal features with a dimension of D3=64. Attention fusion model: Construct a two-branch attention mechanism, including a modal attention branch and a temporal attention branch; Modal attention weight calculation: Considering the differences in the contribution of different modalities to anomaly detection, the attention weight for each modality is calculated using the following formula: ; in, For the first Attention weights for each modality =1 corresponds to video target features. =2 corresponds to environmental characteristics. =3 corresponds to the device characteristics; For the first Feature vectors of each modality; S is the average vector of the three modal features; S(·) is the similarity calculation function; Temporal attention weight calculation: Considering the temporal correlation of warehouse data, an LSTM network is used to extract temporal features and calculate temporal weights, as shown in the following formula: ; ; in, for time The hidden state; for The modal features at each time step are weighted by modal weights; for Always hide your status; for Time-order weights at specific moments; , Here are the weight matrix and bias terms; T is the time window length; Final fusion feature calculation: Combining modality weights and temporal weights, the final fusion features are obtained, as shown in the following formula: ; in, for The final fused feature at each moment has a dimension of D = 512 + 64 + 64 = 640.

[0008] In one embodiment, the anomaly detection module specifically includes the following: Anomaly detection model: Improve the isolated tree construction process of isolated forest by prioritizing the selection of dimensions with lower density in the fusion features when randomly selecting splitting attributes; The feature dimension density Density(d) is defined as the reciprocal of the standard deviation of the feature values ​​in that dimension, and the split attribute selection probability P(d) is positively correlated with Density(d). Anomaly score calculation: The anomaly score is obtained by calculating the path length of the sample across all isolated trees in the isolated forest, using the following formula: ; The closer the score is to 1, the more likely the sample is to be abnormal; the closer it is to 0, the more normal the sample is. in, For the sample Abnormal scores; This represents the number of training samples; For the sample Path length in a single isolated tree; for The average value across all isolated trees; As a correction factor, when When the value is greater than 2, c(n) = 2H(n-1) - (2(n-1)) / n, where H(n) is the nth harmonic number (H(n) = ln(n) + γ, where γ ≈ 0.5772 is Euler's constant). Preliminary anomaly type determination: A preset anomaly scoring threshold θ is used. When s(x,n)≥θ, it is determined to be an anomaly, and the modal information in the fusion features is used for preliminary classification.

[0009] In one embodiment, the target behavior analysis module specifically includes the following: Behavioral feature extraction: Based on the output real-time coordinates of the target, calculate the target's motion features, including velocity V, acceleration A, and motion trajectory similarity; Velocity and acceleration are calculated as follows: ; ; in, for Target speed at all times; , , for Target coordinates at any given time; For time intervals; for Target acceleration at any moment; Behavior classification model: Construct a 3D-CNN+Transformer model, with the input being the motion features of the target in 16 consecutive frames and the video frame features, and the output being the behavior category; The model uses cross-entropy loss for classification, as shown in the following formula: ; in, Cross-entropy loss; The number of samples; Number of behavior categories; For the sample Category The true label; For the sample Category The predicted probability.

[0010] In one embodiment, the environment and equipment status assessment module specifically includes the following: Environmental stability assessment: The fuzzy comprehensive evaluation method is used to construct an environmental factor evaluation matrix, as shown in the following formula: ; Where R is a 3×3 evaluation matrix; r ij The membership degree of the i-th environmental factor belonging to the j-th evaluation level is calculated using the membership function. Combining environmental factor weight vector W env The environmental stability score E is obtained; E=W env ·R·[1,2,3] T T represents the real-time temperature and humidity data inside the warehouse; the larger the E value, the more unstable the environment. Equipment health status assessment: For key equipment such as forklifts and racks, an equipment health index is constructed using the following formula: ; ∈[0,1], the closer HI is to 1, the better the device health status; where, , , As weight; for Constant shelf stress, , These are the maximum and minimum allowable stresses for the shelving, respectively. for Forklift load at all times This is the rated maximum load for the forklift; for Forklift speed at all times Speed ​​limits for forklifts within the warehouse; When the environmental stability score E≥2.5 or the equipment health index HI≤0.3, it is determined to be an environmental or equipment anomaly, and a detailed assessment report is output.

[0011] In one embodiment, the anomaly causality tracing module specifically includes the following: Cause-effect graph construction: Using abnormal events as result nodes and target behavior, environmental state, and device state as cause nodes, a directed acyclic graph is constructed, and the causal strength between nodes is calculated using Do-Calculus. Causality strength calculation: Causality strength is calculated using the intervention probability, using the following formula: ; in, This represents an abnormal event; I represents the cause node; do(·) represents the intervention operation, indicating that I should be forcibly set to 1 or 0; Root cause localization: Sort all cause nodes by causal strength, select the two nodes with the largest absolute value of C(I,O) as the root cause of the anomaly, and combine the time sequence information to determine the causal order.

[0012] In one embodiment, the intelligent decision-making and strategy generation module specifically includes the following: RL Model Construction: Define the state space S as the fused feature F final(t) The set of anomaly types and root cause information; the action space A represents the executable measures; the reward function R is a quantitative indicator of the decision-making effect, as shown in the following formula: ; in, Rewards are given for resolving anomalies. Penalties for handling costs; In response to the delay penalty; Policy generation: The deep deterministic policy gradient algorithm is used to train the RL model and output the optimal action sequence, as shown in the following formula: ; in, For state The optimal strategy under these circumstances; For state Next action The optimal action value function; The execution and feedback optimization module specifically includes the following: Command execution: Through industrial Ethernet and Modbus protocol, decision-making strategies are converted into standardized control commands, with execution delay controlled within 1 second; Feedback data collection: Collect multi-source data within 5 minutes after execution, including whether the anomaly has been eliminated, changes in environmental / equipment status, and handling costs, as feedback samples; Model optimization: An online learning mechanism is used, adding feedback samples to the training set to update the parameters of the models in the multimodal attention fusion module, anomaly detection module, and intelligent decision-making and policy generation module, as shown in the following formula: ; in, These are the updated model parameters; The parameters were before the update; The learning rate; This represents the gradient of the loss function with respect to the parameters. This is a feedback sample.

[0013] In one embodiment, the specific steps include: Step 1: Deploy and start the multi-source data acquisition equipment; High-definition network cameras are installed in key locations such as storage aisles, shelving areas, and entrances and exits. Environmental sensors for temperature, humidity, dust concentration, and gas are distributed throughout the storage space. Data acquisition interfaces are connected to mobile devices such as forklifts and AGVs. Stress detection equipment is also installed in key parts of the shelving. Start all data acquisition devices and enable the time synchronization function based on the network time protocol to ensure that the video data captured by the camera, the environmental data collected by various sensors, and the equipment operation data are consistent in time. The data will be automatically stored in the system database in the format of "device identifier-time stamp-data content". Step 2: Activate the target detection and localization function; Activate the improved target detection model in the system background. The model has preset parameters for warehouse scenarios and does not require manual adjustment. The system will automatically analyze the pre-processed video data, identify targets such as people, forklifts, AGVs, and goods in the scene in real time, and calculate the specific location of these targets in the physical space of the warehouse through the coordinate transformation relationship calibrated by the camera. Finally, it will output core information such as target type, real-time location, and size. Step 3: Initiate multimodal data fusion analysis; The system automatically calls the multimodal attention fusion module, which can complete the feature extraction of various types of data without manual intervention, and extract key feature information from video target data, environmental sensing data and device operation data respectively; The module will dynamically adjust the importance of different data through a dual-branch attention mechanism. In scenarios such as cargo accumulation, it will automatically increase the weight of video data and in scenarios such as abnormal temperature and humidity, it will automatically increase the weight of environmental data. At the same time, it will highlight the data features of key time nodes before and after the anomaly occurs, and finally generate comprehensive feature data that integrates multi-dimensional information. Step 4: Activate the anomaly detection function; Once the improved isolated forest anomaly detection model is activated, the system will automatically perform anomaly analysis using the fused features generated in step 3 as input. The model quantifies the degree of anomaly by calculating the path length of data samples in an isolated tree. When the anomaly score reaches the preset standard, it is determined to be an abnormal event. Combined with the modal information in the fused features, the model initially identifies the anomaly type and issues a prompt on the system interface. Step 5: Conduct in-depth analysis of target behavior; For the abnormal events related to personnel and equipment initially identified in step 4, the system automatically activates the target behavior analysis function; By combining the target's real-time location change data, the target's motion state and behavior pattern are analyzed, and the behavior characteristics are correlated with the anomaly detection results to eliminate misjudgments caused by temporary occlusion of the target and accurately confirm the attributes of abnormal behavior. Step 6: Conduct an environmental and equipment status assessment; In response to the environmental or equipment anomalies initially identified in step 4, the system automatically activates the status assessment module; Fuzzy comprehensive evaluation method is used to assess the stability of environmental factors such as temperature, humidity and dust concentration. By constructing a health index, the health status of equipment such as forklift load and rack stress is assessed, and a detailed assessment report is generated to clarify the degree of environmental instability or the level of equipment health hazards. Step 7: Activate the exception root cause tracing function; The system automatically constructs a causal relationship diagram that includes factors such as target behavior, environmental state, and equipment state, based on confirmed abnormal events. By calculating the causal strength, the degree of correlation between each factor and the abnormal event is quantified, and after ranking, the root cause and indirect cause of the abnormality are determined, thus clarifying the causal logic. Step 8: Generate and confirm the handling strategy; Based on the root cause of the anomaly and the evaluation results, the system invokes a reinforcement learning decision-making model to automatically generate targeted response strategies. The strategy will cover specific actions to be taken. Staff can view the strategy and confirm its execution on the system interface. In case of emergency, the system supports the automatic execution of key actions. Step 9: Execute the strategy and complete feedback optimization; Based on the confirmed handling strategy, the system sends control commands to the execution equipment such as alarms, air conditioners, access control systems, and equipment controllers through the industrial network to ensure accurate execution of actions; After the strategy is executed, the system automatically collects multi-source feedback data over a period of time, including whether the anomaly has been eliminated, changes in the environment and equipment status, and handling costs. This data is used as samples to update the model training set, automatically optimizing the parameters of data fusion, anomaly detection, and decision-making models, thereby improving the accuracy and adaptability of the system in subsequent operation.

[0014] The beneficial effects of this invention are as follows: Compared with existing technologies, the multi-source data acquisition module acts as a sensing probe, constructing a comprehensive, synchronous, and clean data source, laying the foundation for subsequent analysis, and efficiently acquiring multi-dimensional data without manual inspection. The target detection and positioning module breaks through the limitation of seeing objects but not their locations, upgrading video recognition from qualitative judgment to quantitative positioning, accurately tracking the dynamics of goods, equipment, and personnel, and significantly reducing the workload of monitoring personnel. The multimodal attention fusion module, as a core innovation, transforms simple superposition of multi-source data into intelligent collaboration through dynamic weight allocation, improving the efficiency and comprehensiveness of anomaly analysis. The anomaly detection module optimizes the model to balance detection speed and accuracy, reducing false alarms and missed detections, and building a reliable early warning barrier for warehouse safety. The target behavior analysis module deepens the control dimension, extending from static positioning to dynamic prediction, accurately identifying violations and providing early warnings, improving the foresight of risk prevention and control. The environmental and equipment status assessment module transforms fuzzy states into quantitative indicators, realizing refined management of the environment and equipment, and capturing hidden risks in advance. The anomaly tracing module overcomes existing technological limitations, quickly pinpointing the root cause of anomalies and providing rectification directions. This reduces the recurrence of similar problems from the source, building a closed-loop risk management system. The intelligent decision-making and strategy generation module achieves seamless integration of anomaly detection and automated handling, dynamically generating optimal solutions based on the scenario, significantly improving handling efficiency and reducing the need for manual intervention. The execution and feedback optimization module endows the system with self-evolution capabilities, adapting to changes in the scenario through online learning, ensuring long-term stable and efficient operation. Attached Figure Description

[0015] Figure 1 This is a system block diagram of the present invention; Detailed Implementation

[0016] In the description of this invention, it should be noted that the terms "front", "up", "down", "left", "right", "vertical", "horizontal", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0017] To address the problems existing in the background technology, this application proposes the following technical solution: an intelligent warehouse monitoring system and method based on artificial intelligence; The warehouse intelligent monitoring system specifically includes the following: Multi-source data acquisition module: The core function of this module is to simultaneously acquire video data, environmental sensor data, and equipment operation data in the warehouse setting, providing a comprehensive data source for subsequent analysis. A timestamp synchronization mechanism ensures the temporal consistency of the multi-source data, and the acquisition frequency is dynamically adjusted according to the data type.

[0018] Data types and acquisition devices: Video data is acquired through high-definition network cameras (1920×1080 resolution, 25fps) deployed in warehouse aisles, shelving areas, and entrances / exits; environmental data is acquired through distributed sensor nodes, including temperature and humidity sensors (accuracy ±0.1℃, ±1%RH), dust concentration sensors (measurement range 0-10mg / m³), and gas sensors (for flammable storage scenarios, detecting combustible gas concentration); equipment data is acquired through the OBD interfaces of mobile devices such as forklifts and AGVs, and shelving stress sensors (measurement range 0-500MPa), including equipment operating speed, load weight, battery level, and real-time shelving stress.

[0019] Data preprocessing: Differentiated preprocessing methods are adopted to address the noise characteristics of different types of data.

[0020] Video data denoising: Gaussian filtering is used to remove image noise, as shown in the following formula: ; in, The grayscale value of the pixel after filtering; The current pixel coordinates; , σ represents the center pixel coordinates of the convolution kernel; σ is the standard deviation of the Gaussian function (values ​​range from 0.8 to 1.2, dynamically adjusted according to the warehouse lighting intensity, with a larger value used when the lighting is weak). The value represents the original pixel grayscale value; * indicates a convolution operation.

[0021] Sensor data standardization: Due to the large difference in dimensions between environmental and device data, Z-score standardization is used to eliminate the influence of dimensions. The formula is as follows: ; in, The data is standardized. This is the raw sensor data; This is the mean of this type of data; This represents the standard deviation of this type of data.

[0022] Data time synchronization: The network time protocol (NTP)-based timestamp alignment method is adopted. All acquisition devices are synchronized to the warehouse monitoring center clock, and the time error is controlled within 50ms. The synchronized data is stored in the format of "device ID, timestamp and data value".

[0023] The above technical solution is explained as follows: The multi-source data acquisition module, acting as the sensing antenna of the entire monitoring system, completely changes the fragmented and lagging status quo of traditional warehouse data acquisition. In terms of efficiency improvement, this module, through the NTP timestamp synchronization mechanism, controls the time error of video, environmental, and equipment data within 50ms, avoiding analytical biases caused by data asynchrony and saving significant data calibration time for subsequent fusion processing. Addressing actual warehouse needs, the combined acquisition method of distributed sensor nodes and device interfaces eliminates the need for manual inspection and recording, making it particularly suitable for full-coverage monitoring of large automated warehouses. Regarding risk control, differentiated preprocessing methods effectively filter noise; for example, Gaussian filtering reduces video image interference, and Z-score standardization eliminates differences in sensor data dimensions, ensuring data authenticity and reliability, providing accurate input for anomaly detection, and avoiding missed detections due to data distortion. In terms of resource optimization, the dynamic adjustment of acquisition frequency reduces acquisition density during non-critical periods, minimizing storage resource consumption and extending the lifespan of sensor equipment. Its unique value lies in constructing a comprehensive, synchronous, and clean data source, laying a solid foundation for subsequent intelligent analysis.

[0024] Target Detection and Localization Module: Based on preprocessed video data, this module enables real-time detection and location of targets such as personnel, forklifts, AGVs, and goods within the warehouse, providing target information for subsequent behavior analysis and anomaly detection. An improved YOLOv8 model is employed, and an attention mechanism is introduced to enhance recognition accuracy in target occlusion scenarios.

[0025] Object detection model: An improved YOLOv8 neck network is introduced by adding a coordinate attention (CA) module to enhance the model's ability to extract spatial location information of objects. The model loss function adopts CIoU loss, as shown in the following formula: ; in, The intersection-union ratio (IUU) of the predicted bounding box and the ground truth bounding box reflects the degree of overlap between the two. Center point of the prediction box Center point of the real frame The Euclidean distance; It is the diagonal length of the smallest bounding rectangle that simultaneously contains both the predicted bounding box and the ground truth bounding box; Balance coefficient ( = / ( + )); The parameter used to measure aspect ratio consistency is (v=4 / π²(arctan(wgt / hgt)-arctan(w / h))², where w and h are the predicted bounding box width and height, and wgt and hgt are the actual bounding box width and height).

[0026] Target location localization: Based on the detected target bounding box, calculate the real-time position of the target in the warehouse coordinate system. First, obtain the conversion relationship between pixel coordinates and world coordinates through camera calibration, using the following formula: ; in, This is the camera intrinsic parameter matrix (composed of parameters such as focal length and pixel size, obtained through Zhang Zhengyou calibration method); It is a rotation matrix; (X,Y,Z) is the translation vector; (X,Y,Z) is the target's coordinates in the world coordinate system (with the warehouse entrance as the origin). The axis is the direction of the channel. The axis is the direction of the shelf. (The axis is in the height direction).

[0027] This module outputs information such as target type (personnel / forklift / goods, etc.), real-time world coordinates (X,Y,Z), and target size, with positioning error controlled within ±10cm.

[0028] The above technical solutions are explained as follows: The target detection and localization module focuses on the dynamic tracking of core warehouse elements, solving the pain points of traditional monitoring. In terms of efficiency improvement, the improved YOLOv8 model, by introducing a coordinate attention module, significantly improves the accuracy of identifying occluded targets, eliminating the need for repeated manual video checks and greatly reducing the workload of monitoring personnel. Based on camera calibration and position conversion technology, the target positioning error is controlled within ±10cm, enabling real-time coordinate tracking of goods and equipment. Warehouse managers can clearly understand asset distribution through the system, avoiding the time-consuming and labor-intensive problem of manual inventory. Regarding risk control, the accurate target classification and localization capabilities can quickly identify violations such as "personnel entering the forklift operating area" and "goods exceeding shelf boundaries," issuing warnings 3-5 seconds earlier than manual monitoring, buying time for safe handling. In terms of resource optimization, clear target location information can help optimize forklift travel paths and reduce unnecessary mileage.

[0029] Multimodal attention fusion module: Breaking the limitations of isolated analysis of multi-source data in existing technologies, it deeply integrates preprocessed environmental / device sensing data with the output target information, and dynamically allocates the weight of different data through an attention mechanism to highlight key information for anomaly judgment.

[0030] Feature extraction: Feature extraction was performed on each modality of data. Video target features were obtained by improving the output feature map of YOLOv8, with a dimension of D1=512; environmental sensor data (temperature and humidity T, dust concentration P, combustible gas concentration Q) were extracted using a 3-layer fully connected network, with a dimension of D2=64; equipment data (forklift speed V, load W, rack stress S) were extracted using a 1D-CNN, with a dimension of D3=64.

[0031] Attention fusion model: Construct a two-branch attention mechanism, including modal attention branch and temporal attention branch.

[0032] Modal attention weight calculation: Considering the differences in the contribution of different modalities to anomaly detection, the attention weight for each modality is calculated using the following formula: ; in, For the first Attention weights for each modality ( =1 corresponds to video target features. =2 corresponds to environmental characteristics. =3 Corresponding device characteristics); For the first Feature vectors of each modality; S(a,b) is the average vector of the three modal features; S(·) is the similarity calculation function, which uses cosine similarity (S(a,b)=a·b / (||a||·||b||)).

[0033] Temporal attention weight calculation: Considering the temporal correlation of warehouse data, an LSTM network is used to extract temporal features and calculate temporal weights, as shown in the following formula: ; ; in, for time The hidden state; for The modal features at each time step are weighted by modal weights; for Always hide your status; for Time-order weights at specific moments; , Here are the weight matrix and bias terms; T is the length of the time window (taken as 10s, corresponding to 250 frames of video data).

[0034] Final fusion feature calculation: Combining modality weights and temporal weights, the final fusion features are obtained, as shown in the following formula: ; in, for The final fused feature at any given time has a dimension of D = 512 + 64 + 64 = 640. This fused feature includes both target spatial information and environmental and device state information, while highlighting key modal and key temporal features, providing comprehensive and accurate input for subsequent anomaly detection.

[0035] The above technical solutions are explained as follows: Regarding the efficiency improvement of the multimodal attention fusion module, the dual-branch attention mechanism can dynamically focus on key data. For example, it automatically increases the weight of video target features when there are abnormalities in cargo stacking, and strengthens the contribution of environmental data when there are abnormalities in temperature and humidity. This avoids redundant calculations for balanced processing of all data, thus improving the analysis speed of subsequent anomaly detection. For complex warehousing scenarios, the fused feature vector integrates multi-dimensional information on space, environment, and equipment, eliminating the need to call multiple analysis models separately, simplifying the processing flow and improving system response efficiency. In terms of risk control, the deep fusion of multi-dimensional data avoids the one-sidedness of single-data detection. For example, video alone cannot determine forklift overloading, but by combining equipment load data and location information, complex risks such as "forklift overloading" can be accurately identified.

[0036] Anomaly Detection Module: Based on the output fusion features, an improved isolated forest model is constructed to detect warehouse anomalies. This model solves the problem of low detection accuracy of traditional isolated forests for high-dimensional data by introducing density information of fusion features.

[0037] Anomaly detection model: Improve the isolated tree construction process of isolated forest by prioritizing the selection of dimensions with lower density in the fusion features (i.e., dimensions where anomalous features are more likely to appear) when randomly selecting splitting attributes.

[0038] Define the feature dimension density Density(d) as the reciprocal of the standard deviation of the eigenvalues ​​in that dimension (Density(d) = 1 / σ). d The splitting attribute selection probability P(d) is positively correlated with Density(d).

[0039] Anomaly score calculation: The anomaly score is obtained by calculating the path length of the sample across all isolated trees in the isolated forest, using the following formula: ; The closer the score is to 1, the more likely the sample is to be abnormal; the closer it is to 0, the more normal the sample is.

[0040] in, For the sample Abnormal scores; This represents the number of training samples; For the sample The path length in a single isolated tree (i.e., the number of edges from the root node to the leaf node). for The average value across all isolated trees; As a correction factor, when When the value is greater than 2, c(n) = 2H(n-1) - (2(n-1)) / n, where H(n) is the nth harmonic number (H(n) = ln(n) + γ, where γ ≈ 0.5772 is Euler's constant).

[0041] Preliminary anomaly type determination: A preset anomaly scoring threshold θ (θ=0.75, which can be dynamically adjusted according to the warehousing scenario) is set. When s(x,n)≥θ, it is determined to be an anomaly. The anomaly is then preliminarily classified in combination with the modal information in the fusion features, such as "personnel entering the restricted area", "forklift overload", "temperature and humidity exceeding the standard", etc.

[0042] The above technical solutions are explained as follows: The anomaly detection module, based on an improved isolated forest model, achieves accurate identification of warehouse anomalies, solving the pain points of high false alarms and high false negatives in traditional detection methods. In terms of efficiency improvement, a split attribute selection mechanism based on feature density information is introduced, enhancing the model's processing efficiency for high-dimensional fused features. The anomaly detection response time is controlled within 1 second, significantly shorter than traditional isolated forest models, meeting the needs of real-time warehouse monitoring. For the actual work of warehouse management personnel, the automated anomaly scoring mechanism (0-1 quantitative indicators) is clear and intuitive, eliminating the need for manual data analysis and quickly screening high-risk events, reducing anomaly investigation time by more than 70%. In terms of risk control, the CIoU loss function improves the accuracy of bounding box regression. Combined with anomaly scoring threshold determination, this enhances anomaly detection accuracy and accurately captures early risks such as "abnormal shelf stress" and "minor cargo displacement."

[0043] Target Behavior Analysis Module: For anomalies (such as personnel violations or forklift operation abnormalities) initially identified by the anomaly detection module that are related to the target, further analyze the temporal correlation of the target behavior to improve the accuracy of anomaly judgment and avoid misjudgments caused by temporary occlusion of the target.

[0044] Behavioral feature extraction: Based on the output real-time coordinates of the target, calculate the target's motion features, including velocity V, acceleration A, and trajectory similarity. Velocity and acceleration are calculated as follows: ; ; in, for Target speed at all times; , , for Target coordinates at any given time; The time interval is 0.04s, corresponding to a video frame rate of 25fps. for Target acceleration at all times.

[0045] Behavior classification model: Construct a 3D-CNN+Transformer model, with the input being the motion features of the target in 16 consecutive frames and the video frame features, and the output being the behavior category (such as a person walking normally / climbing a shelf / a forklift driving normally / a forklift speeding, etc.).

[0046] The model uses cross-entropy loss for classification, as shown in the following formula: ; in, Cross-entropy loss; The number of samples; Number of behavior categories (default 8 categories); For the sample Category The true label (0 or 1); For the sample Category The predicted probability.

[0047] This module can be used to associate the behavior of personnel and equipment with the results of anomaly detection. For example, when it is detected that "personnel stay in the shelf area for more than 5 minutes and the acceleration is close to 0", the behavior analysis is combined to determine that it is an anomaly of "personnel staying in violation of regulations", thereby reducing the false alarm rate of single location detection.

[0048] The above technical solution is explained as follows: The target behavior analysis module conducts in-depth analysis of the dynamic behavior of personnel and equipment, achieving an upgrade in control from target existence to behavioral compliance. In terms of efficiency improvement, based on the motion feature calculation of target coordinates and the 3D-CNN+Transformer behavior classification model, it can automatically identify eight typical behaviors such as "personnel climbing shelves" and "forklift speeding," eliminating the need for 24-hour manual monitoring of video footage and significantly improving the efficiency of behavioral violation identification, especially suitable for unmanned warehouse scenarios at night. Regarding warehouse operation processes, this module can provide real-time feedback on forklift operation status, assisting managers in optimizing scheduling, reducing equipment idle time, and improving operational efficiency. In terms of risk control, temporal correlation analysis effectively avoids misjudgments caused by temporary obstruction. For example, when personnel are briefly obscured by goods, their status can be accurately determined through historical movement trajectories and acceleration characteristics, reducing the false alarm rate for behavioral violations. For dangerous operations such as rapid acceleration and sharp turns of forklifts, it can issue early warnings, reducing the incidence of collisions.

[0049] Environment and Equipment Status Assessment Module: Based on the environmental or equipment anomalies initially identified by the anomaly detection module, this module combines time-series information from multimodal fusion features to quantitatively assess environmental stability and equipment health status, providing a basis for anomaly tracing.

[0050] Environmental stability assessment: The fuzzy comprehensive evaluation method is used to construct an environmental factor evaluation matrix, as shown in the following formula: ; Where R is a 3×3 evaluation matrix (3 environmental factors: temperature and humidity T, dust concentration P, combustible gas concentration Q; 3 evaluation levels: stable 1, relatively stable 2, unstable 3); r ij The membership degree of the i-th environmental factor to the j-th evaluation level is calculated using a membership function (e.g., the membership function for temperature and humidity is Gaussian, with the central value being the suitable temperature and humidity for stored goods).

[0051] Combining environmental factor weight vector W env (Determined by the analytic hierarchy process), the environmental stability score E is obtained; E=W env ·R·[1,2,3] T T represents the real-time temperature and humidity data inside the warehouse; the larger the E value, the more unstable the environment.

[0052] Equipment health status assessment: For key equipment such as forklifts and racks, a health index (HealthIndex, HI) is constructed, as shown in the following formula: ; ∈[0,1], the closer HI is to 1, the better the device's health status. Wherein, , , For weights ( =0.5, =0.3, =0.2, highlighting the importance of rack stress and forklift load); for Constant shelf stress, , These are the maximum and minimum allowable stresses for the shelving, respectively. for Forklift load at all times This is the rated maximum load for the forklift; for Forklift speed at all times Speed ​​limits are set for forklifts within the warehouse.

[0053] When the environmental stability score E≥2.5 or the equipment health index HI≤0.3, it is determined to be an environmental or equipment anomaly, and a detailed assessment report is output.

[0054] The above technical solutions are explained as follows: The environmental and equipment status assessment module focuses on the health management of warehousing. In terms of efficiency improvement, the fuzzy comprehensive evaluation method and the quantitative assessment of the Equipment Health Index (HI) transform vague descriptions such as "unsuitable temperature and humidity" or "equipment has some problems" into precise indicators. Managers do not need professional instruments for testing; they can grasp the environmental stability and equipment health status through the system, thus improving assessment efficiency. For special scenarios such as cold chain warehousing, real-time environmental assessment can be linked to equipment control, avoiding delays caused by manual intervention and improving environmental control efficiency. In terms of risk prevention and control, automatic warnings are issued when the environmental stability score E≥2.5 or HI≤0.3, which can detect hidden risks such as "slowly exceeding temperature and humidity standards leading to mold growth" and "stress accumulation on shelves causing deformation" in advance. Compared to traditional manual inspections, the window for early detection of potential hazards is extended to several hours, significantly reducing the risk of asset loss. For hazardous chemical storage, real-time assessment of flammable gas concentration further strengthens the safety defense line.

[0055] Anomaly Cause Tracing Module: Existing technologies can only detect anomalies but cannot locate the root cause. This module is based on causal inference theory and combines the analysis results of the target behavior analysis module and the environment and equipment status assessment module to construct an anomaly cause-effect graph, thereby achieving accurate location of the anomaly root cause.

[0056] Cause-effect graph construction: Using abnormal events as result nodes and target behaviors, environmental states, and device states as cause nodes, a directed acyclic graph (DAG) is constructed, and the causal strength between nodes is calculated using Do-Calculus.

[0057] Causality strength calculation: Causality strength is calculated using the intervention probability, using the following formula: ; ∈[-1,1], the larger the absolute value, the stronger the causal relationship; C>0 indicates positive causality (I causes O to occur), and C<0 indicates negative causality. Among them, O is an abnormal event (O=1 indicates an abnormal event, O=0 indicates normal); I is a cause node (such as "forklift overload", "temperature and humidity exceeding the standard", etc.); do(·) is an intervention operation, which means forcibly setting I to 1 or 0.

[0058] Root cause identification: The causal strength of all cause nodes is ranked, and the two nodes with the largest absolute values ​​of C(I,O) are selected as the root causes of the anomalies. The causal order is then determined by combining this with temporal information (e.g., "forklift overload → increased rack stress → abnormal rack deformation," in which case "forklift overload" is the root cause). This module enables a breakthrough from "detecting anomalies" to "finding the root cause," addressing a core shortcoming of existing technologies.

[0059] The above technical solution is explained as follows: Based on Do-Calculus causal strength calculation and directed acyclic graph construction, the core root cause can be quickly located after an anomaly occurs, improving efficiency compared to the traditional method of tracing the source one by one. For example, in the case of abnormal shelf deformation, the system can quickly identify "forklift overloading" rather than "shelf aging," providing a clear direction for rectification and avoiding time-consuming ineffective rectification. For practical warehouse management, the root cause location results can automatically generate rectification suggestions, reducing the analysis and decision-making time for managers. In terms of risk prevention and control, accurate root cause location can solve problems at the source, preventing the recurrence of similar anomalies. For example, after optimizing air conditioning control for the root cause of "excessive temperature and humidity," the incidence of similar anomalies decreased. Simultaneously, causal sequence judgment can distinguish between direct and indirect causes, preventing rectification from remaining superficial and building a deep safety defense line.

[0060] Intelligent Decision and Strategy Generation Module: Based on the root causes of anomalies located by the anomaly causal tracing module and the analysis results of each module, a reinforcement learning (RL) decision model is constructed to generate targeted handling strategies, realizing the automated connection between "detection and decision".

[0061] RL Model Construction: Define the state space S as the fused feature F final(t) The system comprises: a set of anomaly types and root cause information; an action space A consisting of executable measures (such as "triggering audible and visual alarms," ​​"notifying inspection personnel," "controlling air conditioning to adjust temperature and humidity," "forcing forklifts to stop," etc., totaling 12 types of actions); and a reward function R, a quantitative indicator of decision-making effectiveness, with the following formula: ; in, A reward for resolving the anomaly (100 points are awarded after the anomaly is resolved). Penalties are imposed for handling costs (e.g., the cost of "forcing the forklift to stop" is higher than that of "notifying personnel," resulting in a penalty of 20 points vs. 5 points). The response is a delay penalty (delay time t(s), penalty score is t×2).

[0062] Policy generation: The Deep Deterministic Policy Gradient (DDPG) algorithm is used to train the RL model and output the optimal action sequence, as shown in the following formula: ; in, For state The optimal strategy under these circumstances; For state Next action The optimal action value function.

[0063] For example, when the root cause of the anomaly is "excessive temperature and humidity", the output strategy is "control the air conditioner to adjust the temperature and humidity, monitor environmental data in real time, and stop adjusting after the anomaly is eliminated"; when the root cause is "personnel illegally entering the restricted area", the output strategy is "trigger an audible and visual alarm, push a warning message to the mobile phone of the inspection personnel, and if they do not leave within 1 minute, link the access control to the restricted area exit".

[0064] The above technical solution is explained as follows: The intelligent decision-making and strategy generation module establishes a seamless connection between anomaly detection and automatic handling, solving the problem of traditional monitoring emphasizing detection but neglecting handling. In terms of efficiency improvement, the application of reinforcement learning (RL) models and the DDPG algorithm can dynamically generate optimal handling strategies based on anomaly type, root cause, and scenario, reducing response time to less than 3 seconds. Compared to the traditional model of manual alarm reception, analysis, and scheduling, handling efficiency is significantly improved. For example, when personnel illegally enter a restricted area, the system can automatically trigger an audible and visual alarm and push information, eliminating the need for manual intervention. To meet the needs of multi-scenario adaptation, the model's action space covers 12 types of handling measures, eliminating the need to rewrite rules for different warehouse types and improving system deployment efficiency.

[0065] Execution and Feedback Optimization Module: This module transforms the decision-making and strategy generation module's generated decision-making strategies into specific execution instructions, controls the actions of execution equipment (alarms, air conditioners, access control, equipment controllers, etc.) within the warehouse, and collects feedback data after strategy execution for model optimization, forming a closed-loop system.

[0066] Command execution: Through industrial Ethernet and Modbus protocol, the decision strategy is converted into standardized control commands (such as the air conditioning adjustment command format is "Device ID-Command Type-Target Temperature-Target Humidity"), and the execution delay is controlled within 1 second.

[0067] Feedback data collection: Collect multi-source data within 5 minutes after execution, including whether the anomaly has been eliminated, changes in environmental / equipment status, and handling costs, as feedback samples.

[0068] Model optimization: An online learning mechanism is used, adding feedback samples to the training set to update the parameters of the models in the multimodal attention fusion module, anomaly detection module, and intelligent decision-making and policy generation module, as shown in the following formula: ; in, These are the updated model parameters; The parameters were before the update; The learning rate is set to 0.001. This represents the gradient of the loss function with respect to the parameters. This is a feedback sample.

[0069] The specific steps of this embodiment are as follows: The application of industrial Ethernet and Modbus protocol ensures that the control command execution delay is controlled within 1 second, ensuring rapid implementation of response measures. Simultaneously, the online learning mechanism can update model parameters without system downtime, avoiding monitoring interruptions caused by system upgrades. For changes in warehousing scenarios, such as adjustments to cargo types or equipment updates, the system can automatically adapt through feedback data, eliminating the need for manual model retraining and improving efficiency. In terms of risk control, feedback data collection within 5 minutes allows for timely verification of response effectiveness. If the anomaly is not eliminated, the strategy is automatically adjusted to prevent the risk from escalating due to response failure. Continuous model optimization gradually improves the system's anomaly detection accuracy over time, enhancing stability in special scenarios such as cold chain and hazardous chemicals, and reducing missed risk detections due to system adaptation issues.

[0070] The specific usage method of this embodiment is as follows: Step 1: Deploy and start the multi-source data acquisition equipment; High-definition network cameras are installed in key locations such as warehouse aisles, shelving areas, and entrances and exits. Environmental sensors for temperature, humidity, dust concentration, and gas are distributed throughout the warehouse space. Data acquisition interfaces are connected to mobile devices such as forklifts and AGVs, and stress detection equipment is installed on key parts of the shelving.

[0071] Start all data acquisition devices and enable the time synchronization function based on the network time protocol to ensure that the video data captured by the camera, the environmental data collected by various sensors, and the equipment operation data are consistent in time. The data will be automatically stored in the system database in the format of "device identifier-time stamp-data content".

[0072] Step 2: Activate the target detection and localization function; The improved target detection model is activated in the system background. This model has preset parameters adapted to the warehouse scenario and does not require manual adjustment.

[0073] The system will automatically analyze the pre-processed video data, identify targets such as people, forklifts, AGVs, and goods in the scene in real time, and calculate the specific location of these targets in the physical space of the warehouse through the coordinate transformation relationship calibrated by the camera. Finally, it will output core information such as target type, real-time location, and size.

[0074] Step 3: Initiate multimodal data fusion analysis; The system automatically calls the multimodal attention fusion module, which can complete the feature extraction of various types of data without manual intervention, and extract key feature information from video target data, environmental sensing data and equipment operation data respectively.

[0075] The module will dynamically adjust the importance of different data through a dual-branch attention mechanism. It will automatically increase the weight of video data in scenarios such as cargo accumulation and increase the weight of environmental data in scenarios such as abnormal temperature and humidity. At the same time, it will highlight the data features of key time nodes before and after the anomaly occurs, and finally generate comprehensive feature data that integrates multi-dimensional information.

[0076] Step 4: Activate the anomaly detection function; Once the improved isolated forest anomaly detection model is activated, the system will automatically perform anomaly analysis using the fused features generated in step 3 as input.

[0077] The model quantifies the degree of anomaly by calculating the path length of data samples in an isolated tree. When the anomaly score reaches the preset standard, it is determined to be an abnormal event. Combined with the modal information in the fused features, the model initially identifies the anomaly type, such as personnel entering a restricted area, forklift overloading, and excessive temperature and humidity. At the same time, a prompt is issued on the system interface.

[0078] Step 5: Conduct in-depth analysis of target behavior; For the abnormal events related to personnel and equipment that were initially identified in step 4, the system automatically activates the target behavior analysis function.

[0079] By combining real-time location change data of the target, the target's motion state and behavior pattern are analyzed, and the behavioral characteristics are correlated with the anomaly detection results to eliminate misjudgments caused by temporary occlusion of the target, and to accurately confirm the attributes of abnormal behavior, such as personnel staying in violation of regulations or forklifts speeding.

[0080] Step 6: Conduct an environmental and equipment status assessment; For any environmental or equipment anomalies initially identified in step 4, the system automatically activates the status assessment module.

[0081] The fuzzy comprehensive evaluation method is used to assess the stability of environmental factors such as temperature, humidity and dust concentration. By constructing a health index, the health status of equipment such as forklift load and rack stress is assessed, and a detailed assessment report is generated to clarify the degree of environmental instability or the level of equipment health hazards.

[0082] Step 7: Activate the exception root cause tracing function; The system uses confirmed abnormal events as its core to automatically construct a causal relationship diagram that includes factors such as target behavior, environmental status, and equipment status.

[0083] By calculating the causal strength, the degree of correlation between each factor and the abnormal event is quantified. After ranking, the root cause and indirect cause of the abnormality are determined, and the causal logic is clarified, such as the correlation between "forklift overload, increased shelf stress, and shelf deformation", which provides a basis for subsequent handling.

[0084] Step 8: Generate and confirm the handling strategy; Based on the root cause of the anomaly and the evaluation results, the system invokes a reinforcement learning decision-making model to automatically generate targeted response strategies.

[0085] The strategy will cover specific actions to be taken, such as triggering audible and visual alarms, notifying inspection personnel, adjusting the temperature and humidity of the air conditioning, and forcing the forklift to stop. Staff can view the strategy and confirm its execution on the system interface. In emergency situations, the system supports the automatic execution of critical actions.

[0086] Step 9: Execute the strategy and complete feedback optimization; Based on the confirmed handling strategy, the system sends control commands to execution devices such as alarms, air conditioners, access control systems, and equipment controllers through the industrial network to ensure accurate execution of actions.

[0087] After the strategy is executed, the system automatically collects multi-source feedback data over a period of time, including whether the anomaly has been eliminated, changes in the environment and equipment status, and handling costs. This data is used as samples to update the model training set, automatically optimizing the parameters of data fusion, anomaly detection, and decision-making models, thereby improving the accuracy and adaptability of the system in subsequent operation.

[0088] In summary, in this embodiment, the multi-source data acquisition module acts as a sensing probe, constructing a comprehensive, synchronous, and clean data source to lay the foundation for subsequent analysis, efficiently acquiring multi-dimensional data without manual inspection. The target detection and positioning module overcomes the limitation of seeing objects but not their locations, upgrading video recognition from qualitative judgment to quantitative positioning, accurately tracking the dynamics of goods, equipment, and personnel, and significantly reducing the workload of monitoring personnel. The multimodal attention fusion module, as a core innovation, transforms simple superposition of multi-source data into intelligent collaboration through dynamic weight allocation, improving the efficiency and comprehensiveness of anomaly analysis. The anomaly detection module optimizes the model to balance detection speed and accuracy, reducing false alarms and missed detections, building a reliable early warning barrier for warehouse safety. The target behavior analysis module deepens the control dimension, extending from static positioning to dynamic prediction, accurately identifying violations and providing early warnings, improving the foresight of risk prevention and control. The environmental and equipment status assessment module transforms ambiguous states into quantitative indicators, enabling refined management of the environment and equipment, and capturing hidden risks in advance. The anomaly causal tracing module overcomes the limitations of existing technologies, quickly locating the root cause of anomalies and providing rectification directions, reducing the recurrence of similar problems from the source, and constructing a closed-loop risk management system. The intelligent decision-making and strategy generation module achieves seamless integration of anomaly detection and automatic handling, dynamically generating optimal solutions based on the scenario, significantly improving handling efficiency and reducing the need for manual intervention. The execution and feedback optimization module endows the system with self-evolution capabilities, adapting to changes in the scenario through online learning, ensuring long-term stable and efficient operation.

[0089] Although embodiments of the invention have been shown and described, the scope of the invention will be defined by the appended claims and their equivalents by those skilled in the art.

Claims

1. An intelligent warehouse monitoring system based on artificial intelligence, characterized in that, include: Multi-source data acquisition module: Synchronously collects video data, environmental sensor data and equipment operation data in the warehouse scenario, and uses a timestamp synchronization mechanism to ensure the time consistency of multi-source data. The acquisition frequency is dynamically adjusted according to the data type. Target detection and localization module: Based on preprocessed video data, it realizes real-time detection and location of personnel, forklifts, AGVs and goods in the warehouse. It adopts an improved YOLOv8 model and further improves the recognition accuracy in target occlusion scenarios by introducing an attention mechanism. Multimodal attention fusion module: Deeply fuses preprocessed environmental / device sensing data with output target information, dynamically assigns weights to different data through an attention mechanism, and highlights key information for anomaly detection; Anomaly detection module: Based on the fused features of the output, an improved isolated forest model is constructed to detect warehouse anomalies; Target Behavior Analysis Module: Further analyzes the temporal correlation of target behaviors based on the anomalies initially identified by the anomaly detection module that are related to the target. Environment and Equipment Status Assessment Module: Based on the environmental or equipment anomalies initially identified by the anomaly detection module, this module combines time-series information from multimodal fusion features to quantitatively assess environmental stability and equipment health status, providing a basis for anomaly tracing. Anomaly Cause Tracing Module: Based on causal inference theory, and combined with the analysis results of the target behavior analysis module and the environment and equipment status assessment module, an anomaly cause-effect graph is constructed to locate the root cause of the anomaly; Intelligent Decision and Strategy Generation Module: Based on the root causes of anomalies located by the anomaly causal tracing module and the analysis results of each module, a reinforcement learning decision model is constructed to generate targeted handling strategies; Execution and Feedback Optimization Module: This module transforms the decision-making strategies generated by the intelligent decision-making and strategy generation module into specific execution instructions, controls the actions of execution equipment within the warehouse, and collects feedback data after strategy execution for model optimization, forming a closed-loop system.

2. The intelligent warehouse monitoring system based on artificial intelligence according to claim 1, characterized in that, The multi-source data acquisition module specifically includes the following: Data types and acquisition devices: Video data is acquired through high-definition network cameras deployed in warehouse aisles, shelving areas, and entrances / exits; environmental data is acquired through distributed sensor nodes, including temperature and humidity sensors, dust concentration sensors, and gas sensors; equipment data is acquired through the OBD interfaces of forklifts and AGV mobile devices and shelving stress sensors, including equipment operating speed, load weight, battery level, and real-time shelving stress. Data preprocessing: Differentiated preprocessing methods are adopted based on the noise characteristics of different types of data; Video data denoising: Gaussian filtering is used to remove image noise, as shown in the following formula: ； in, The grayscale value of the pixel after filtering; The current pixel coordinates; , σ represents the pixel coordinates of the center of the convolution kernel; σ is the standard deviation of the Gaussian function. The original pixel grayscale value; * indicates a convolution operation; Sensor data standardization: Due to the large difference in dimensions between environmental and device data, Z-score standardization is used to eliminate the influence of dimensions. The formula is as follows: ； in, The data is standardized. This is the raw sensor data; This is the mean of this type of data; This represents the standard deviation of this type of data; Data time synchronization: The timestamp alignment method based on the network time protocol is adopted. All acquisition devices are synchronized to the clock of the warehouse monitoring center. The time error is controlled within 50ms. The synchronized data is stored in the format of "device ID, timestamp and data value".

3. The intelligent warehouse monitoring system based on artificial intelligence according to claim 2, characterized in that, The target detection and localization module specifically includes the following: Object detection model: An improved YOLOv8 neck network is added, incorporating a coordinate attention module to enhance the model's ability to extract spatial location information of objects. The model loss function adopts CIoU loss, as shown in the following formula: ； in, The intersection-union ratio (IUU) of the predicted bounding box and the ground truth bounding box reflects the degree of overlap between the two. Center point of the prediction box Center point of the real frame The Euclidean distance; It is the diagonal length of the smallest bounding rectangle that simultaneously contains both the predicted bounding box and the ground truth bounding box; This is the balance coefficient; A parameter used to measure aspect ratio consistency; Target location localization: Based on the detected target bounding box, calculate the real-time position of the target in the warehouse coordinate system; First, the conversion relationship between pixel coordinates and world coordinates is obtained through camera calibration, using the following formula: ； in, This is the camera intrinsic parameter matrix; It is a rotation matrix; (X,Y,Z) is the translation vector; (X,Y,Z) are the coordinates of the target in the world coordinate system. This module outputs information such as target type, real-time world coordinates (X,Y,Z), and target size, with positioning error controlled within ±10cm.

4. The intelligent warehouse monitoring system based on artificial intelligence according to claim 3, characterized in that, The multimodal attention fusion module specifically includes the following: Feature extraction: Feature extraction is performed on each modality of data separately; Video target features are obtained by improving the output feature map of YOLOv8, with a dimension of D1=512; Environmental sensor data, including temperature and humidity (T), dust concentration (P), and combustible gas concentration (Q), are extracted using a 3-layer fully connected network with a dimension of D2=64. Device data are extracted using a 1D-CNN to extract temporal features with a dimension of D3=64. Attention fusion model: Construct a two-branch attention mechanism, including a modal attention branch and a temporal attention branch; Modal attention weight calculation: Considering the differences in the contribution of different modalities to anomaly detection, the attention weight for each modality is calculated using the following formula: ； in, For the first Attention weights for each modality =1 corresponds to video target features. =2 corresponds to environmental characteristics. =3 corresponds to the device characteristics; For the first Feature vectors of each modality; S is the average vector of the three modal features; S(·) is the similarity calculation function; Temporal attention weight calculation: Considering the temporal correlation of warehouse data, an LSTM network is used to extract temporal features and calculate temporal weights, as shown in the following formula: ； ； in, for time The hidden state; for The modal features at each time step are weighted by modal weights; for Always hide your status; for Time-order weights at specific moments; , Here are the weight matrix and bias terms; T is the time window length; Final fusion feature calculation: Combining modality weights and temporal weights, the final fusion features are obtained, as shown in the following formula: ； in, for The final fused feature at each moment has a dimension of D = 512 + 64 + 64 = 640.

5. The intelligent warehouse monitoring system based on artificial intelligence according to claim 4, characterized in that, The anomaly detection module specifically includes the following: Anomaly detection model: Improve the isolated tree construction process of isolated forest by prioritizing the selection of dimensions with lower density in the fusion features when randomly selecting splitting attributes; The feature dimension density Density(d) is defined as the reciprocal of the standard deviation of the feature values ​​in that dimension, and the split attribute selection probability P(d) is positively correlated with Density(d). Anomaly score calculation: The anomaly score is obtained by calculating the path length of the sample across all isolated trees in the isolated forest, using the following formula: ； The closer the score is to 1, the more likely the sample is to be an anomaly. The closer the sample is to 0, the more normal it is. in, For the sample Abnormal scores; This represents the number of training samples; For the sample Path length in a single isolated tree; for The average value across all isolated trees; As a correction factor, when When the value is greater than 2, c(n) = 2H(n-1) - (2(n-1)) / n, where H(n) is the nth harmonic number (H(n) = ln(n) + γ, where γ ≈ 0.5772 is Euler's constant). Preliminary anomaly type determination: A preset anomaly scoring threshold θ is used. When s(x,n)≥θ, it is determined to be an anomaly, and the modal information in the fusion features is used for preliminary classification.

6. The intelligent warehouse monitoring system based on artificial intelligence according to claim 5, characterized in that, The target behavior analysis module specifically includes the following: Behavioral feature extraction: Based on the output real-time coordinates of the target, calculate the target's motion features, including velocity V, acceleration A, and motion trajectory similarity; Velocity and acceleration are calculated as follows: ； ； in, for Target speed at all times; , , for Target coordinates at any given time; For time intervals; for Target acceleration at any moment; Behavior classification model: Construct a 3D-CNN+Transformer model, with the input being the motion features of the target in 16 consecutive frames and the video frame features, and the output being the behavior category; The model uses cross-entropy loss for classification, as shown in the following formula: ； in, Cross-entropy loss; The number of samples; Number of behavior categories; For the sample Category The true label; For the sample Category The predicted probability.

7. The intelligent warehouse monitoring system based on artificial intelligence according to claim 6, characterized in that, The environmental and equipment condition assessment module specifically includes the following: Environmental stability assessment: The fuzzy comprehensive evaluation method is used to construct an environmental factor evaluation matrix, as shown in the following formula: ； Where R is a 3×3 evaluation matrix; r ij The membership degree of the i-th environmental factor belonging to the j-th evaluation level is calculated using the membership function. Combining environmental factor weight vector W env The environmental stability score E is obtained; E=W env ·R·[1,2,3] T T represents the real-time temperature and humidity data inside the warehouse; the larger the E value, the more unstable the environment. Equipment health status assessment: For key equipment such as forklifts and racks, an equipment health index is constructed using the following formula: ； ∈[0,1], the closer HI is to 1, the better the device health status; where, , , As weight; for Constant shelf stress, , These are the maximum and minimum allowable stresses for the shelving, respectively. for Forklift load at all times This is the rated maximum load for the forklift; for Forklift speed at all times Speed ​​limits for forklifts within the warehouse; When the environmental stability score E≥2.5 or the equipment health index HI≤0.3, it is determined to be an environmental or equipment anomaly, and a detailed assessment report is output.

8. The intelligent warehouse monitoring system based on artificial intelligence according to claim 7, characterized in that, The anomaly causation tracing module specifically includes the following: Cause-effect graph construction: Using abnormal events as result nodes and target behavior, environmental state, and device state as cause nodes, a directed acyclic graph is constructed, and the causal strength between nodes is calculated using Do-Calculus. Causality strength calculation: Causality strength is calculated using the intervention probability, using the following formula: ； in, This represents an abnormal event; I represents the cause node; do(·) represents the intervention operation, indicating that I should be forcibly set to 1 or 0; Root cause localization: Sort all cause nodes by causal strength, select the two nodes with the largest absolute value of C(I,O) as the root cause of the anomaly, and combine the time sequence information to determine the causal order.

9. The intelligent warehouse monitoring system based on artificial intelligence according to claim 8, characterized in that, The intelligent decision-making and strategy generation module specifically includes the following: RL Model Construction: Define the state space S as the fused feature F final(t) The set of anomaly types and root cause information; the action space A represents the executable measures; the reward function R is a quantitative indicator of the decision-making effect, as shown in the following formula: ； in, Rewards are given for resolving anomalies. Penalties for handling costs; In response to the delay penalty; Policy generation: The deep deterministic policy gradient algorithm is used to train the RL model and output the optimal action sequence, as shown in the following formula: ； in, For state The optimal strategy under these circumstances; For state Next action The optimal action value function; The execution and feedback optimization module specifically includes the following: Command execution: Through industrial Ethernet and Modbus protocol, decision-making strategies are converted into standardized control commands, with execution delay controlled within 1 second; Feedback data collection: Collect multi-source data within 5 minutes after execution, including whether the anomaly has been eliminated, changes in environmental / equipment status, and handling costs, as feedback samples; Model optimization: An online learning mechanism is adopted, and feedback samples are added to the training set to update the parameters of the models in the multimodal attention fusion module, anomaly detection module, and intelligent decision-making and policy generation module. The formula is as follows: ； in, These are the updated model parameters; The parameters were before the update; The learning rate; This represents the gradient of the loss function with respect to the parameters. This is a feedback sample.

10. A warehouse intelligent monitoring method based on artificial intelligence as described in claim 9, characterized in that, Specifically, the following steps are included: Step 1: Deploy and start the multi-source data acquisition equipment; High-definition network cameras are installed in key locations such as storage aisles, shelving areas, and entrances and exits. Environmental sensors for temperature, humidity, dust concentration, and gas are distributed throughout the storage space. Data acquisition interfaces are connected to mobile devices such as forklifts and AGVs. Stress detection equipment is also installed in key parts of the shelving. Start all data acquisition devices and enable the time synchronization function based on the network time protocol to ensure that the video data captured by the camera, the environmental data collected by various sensors, and the equipment operation data are consistent in time. The data will be automatically stored in the system database in the format of "device identifier-time stamp-data content". Step 2: Activate the target detection and localization function; Activate the improved target detection model in the system background. The model has preset parameters for warehouse scenarios and does not require manual adjustment. The system will automatically analyze the pre-processed video data, identify targets such as people, forklifts, AGVs, and goods in the scene in real time, and calculate the specific location of these targets in the physical space of the warehouse through the coordinate transformation relationship calibrated by the camera. Finally, it will output core information such as target type, real-time location, and size. Step 3: Initiate multimodal data fusion analysis; The system automatically calls the multimodal attention fusion module, which can complete the feature extraction of various types of data without manual intervention, and extract key feature information from video target data, environmental sensing data and device operation data respectively; The module will dynamically adjust the importance of different data through a dual-branch attention mechanism. In scenarios such as cargo accumulation, it will automatically increase the weight of video data and in scenarios such as abnormal temperature and humidity, it will automatically increase the weight of environmental data. At the same time, it will highlight the data features of key time nodes before and after the anomaly occurs, and finally generate comprehensive feature data that integrates multi-dimensional information. Step 4: Activate the anomaly detection function; Once the improved isolated forest anomaly detection model is activated, the system will automatically perform anomaly analysis using the fused features generated in step 3 as input. The model quantifies the degree of anomaly by calculating the path length of data samples in an isolated tree. When the anomaly score reaches the preset standard, it is determined to be an abnormal event. Combined with the modal information in the fused features, the model initially identifies the anomaly type and issues a prompt on the system interface. Step 5: Conduct in-depth analysis of target behavior; For the abnormal events related to personnel and equipment initially identified in step 4, the system automatically activates the target behavior analysis function; By combining the target's real-time location change data, the target's motion state and behavior pattern are analyzed, and the behavior characteristics are correlated with the anomaly detection results to eliminate misjudgments caused by temporary occlusion of the target and accurately confirm the abnormal behavior attributes. Step 6: Conduct an environmental and equipment status assessment; In response to the environmental or equipment anomalies initially identified in step 4, the system automatically activates the status assessment module; Fuzzy comprehensive evaluation method is used to assess the stability of environmental factors such as temperature, humidity and dust concentration. By constructing a health index, the health status of equipment such as forklift load and rack stress is assessed, and a detailed assessment report is generated to clarify the degree of environmental instability or the level of equipment health hazards. Step 7: Activate the exception root cause tracing function; The system uses confirmed abnormal events as its core to automatically construct a causal relationship diagram that includes factors such as target behavior, environmental state, and equipment state; By calculating the causal strength, the degree of correlation between each factor and the abnormal event is quantified, and after ranking, the root cause and indirect cause of the abnormality are determined, thus clarifying the causal logic. Step 8: Generate and confirm the handling strategy; Based on the root cause of the anomaly and the evaluation results, the system invokes a reinforcement learning decision-making model to automatically generate targeted response strategies. The strategy will cover specific actions to be taken. Staff can view the strategy and confirm its execution on the system interface. In case of emergency, the system supports the automatic execution of key actions. Step 9: Execute the strategy and complete feedback optimization; Based on the confirmed handling strategy, the system sends control commands to the execution equipment such as alarms, air conditioners, access control systems, and equipment controllers through the industrial network to ensure accurate execution of actions; After the strategy is executed, the system automatically collects multi-source feedback data over a period of time, including whether the anomaly has been eliminated, changes in the environment and equipment status, and handling costs. This data is used as samples to update the model training set, automatically optimizing the parameters of data fusion, anomaly detection, and decision-making models, thereby improving the accuracy and adaptability of the system in subsequent operation.

Images