A warehouse operation behavior compliance automatic identification and early warning method and device based on multi-sensor fusion

Abstract

The application discloses a kind of based on multi-sensor fusion's warehouse operation behavior compliance automatic identification and early warning method and equipment, it is related to warehouse intelligent monitoring technical field, through the data of multiple-source sensors such as magnetic attraction type master control terminal, layer plate gravity perception module, roadway personnel perception device collaborative collection, utilize edge computing gateway to complete multi-source data feature fusion extraction and operation behavior compliance multidimensional judgment in local, realize to the automatic identification and grading real-time early warning of three typical illegal behaviors such as wrong goods, overreach operation, goods stagnation;Meanwhile, cloud end time series causal analysis platform uses the comprehensive causal strength calculation algorithm innovatively proposed in the application, combined with granger causality test and mutual information method constructs time series causal reasoning model, quantifies the deep causal association between illegal events, generates leading event-sequela causal chain, so as to improve the standardization management level of warehouse operation.

Description

Technical Field

[0001] This invention relates to the field of intelligent warehouse monitoring technology, specifically to a method and device for automatic identification and early warning of compliance of warehouse operations based on multi-sensor fusion. Background Technology

[0002] With the rapid development of the warehousing and logistics industry, automated storage and retrieval systems and intelligent warehousing systems have been widely used. However, in actual operation, there are still many manual processes, such as picking, shelving, and inventory counting. Non-standard manual operations, such as misplacement of goods, unauthorized operation, and goods being left unattended, can not only lead to chaotic inventory management and low picking efficiency, but may also cause goods to be lost, damaged, or even cause safety accidents.

[0003] Currently, monitoring of warehousing operations mainly relies on two methods: manual inspection and video surveillance. Manual inspection has drawbacks such as low efficiency, limited coverage, and inability to detect problems in real time. Although traditional video surveillance systems can record the situation on site, they require manual real-time viewing or post-event playback and cannot automatically identify violations and provide real-time warnings, resulting in significant lag.

[0004] In recent years, although some computer vision-based behavior recognition systems have emerged, these systems generally suffer from the following problems: First, they rely solely on visual data, making them susceptible to environmental factors such as lighting and occlusion, resulting in high false alarm and false negative rates. Second, they require extensive wiring and installation, leading to high upgrade costs and making rapid deployment in existing warehousing systems difficult. Third, data processing primarily relies on the cloud, resulting in significant network latency and hindering real-time early warning and intervention. Fourth, their functionality is limited, only able to identify a few types of violations, failing to meet the compliance monitoring needs of the entire warehousing operation process. Fifth, they lack the ability to deeply analyze historical violation data, only able to discover isolated violations, unable to uncover causal relationships and systemic risk factors between different events, and can only conduct post-event accountability, unable to achieve pre-event prevention. Sixth, causal analysis relies heavily on human experience and lacks quantitative algorithmic models, resulting in insufficient accuracy and objectivity in risk identification. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of existing technologies and provide a method and device for automatic identification and early warning of compliance in warehousing operations based on multi-sensor fusion. This method utilizes a multi-source sensor system, including a magnetic main control terminal, a shelf gravity sensing module, and aisle personnel sensing devices, to collaboratively collect data. An edge computing gateway is used to locally perform multi-source data feature fusion extraction and multi-dimensional judgment of operational compliance, enabling automatic identification and graded real-time early warning of three typical violations: misplaced goods, unauthorized operations, and goods retention. Simultaneously, a cloud-based temporal causal analysis platform employs the innovative comprehensive causal strength calculation algorithm proposed in this invention, combined with Granger causality testing and mutual information methods to construct a temporal causal reasoning model. This model quantifies and mines deep causal relationships between violation events, generating a leading event-secondary event causal chain, thereby improving the standardized management level of warehousing operations.

[0006] To address the aforementioned technical problems, this invention provides the following technical solution: On one hand, a method for automatic identification and early warning of compliance in warehouse operations based on multi-sensor fusion, comprising the following specific steps: S1: Complete equipment deployment and parameter configuration, enter basic warehouse data, personnel permission data and storage location binding data, and establish the basic environment for system operation; S2: When the infrared human body sensing module detects that a person has entered the detection range, it triggers the wide-angle camera, the floor gravity sensing module and the roadway personnel sensing device to start data acquisition simultaneously, and preprocesses the acquired raw data. S3: The edge computing gateway performs feature fusion on the preprocessed multi-source data, extracts four core features: personnel dwell time, shelf weight change, cargo label information, and personnel operation location, and determines the current operation type based on the features; S4: Based on the extracted features and operation types, compliance judgments are made from three dimensions: goods matching, personnel permissions, and operation timeliness, and judgment results are generated. S5: Trigger the corresponding level of warning based on the judgment result, and at the same time, record the time, location, personnel, operation content and on-site images of the violation and report them to the cloud platform. S6: The cloud platform aggregates all historical data from the warehouse, uses a comprehensive causal strength calculation algorithm to build a time-series causal reasoning model to mine the causal relationships of violations, and iteratively optimizes the operation behavior recognition model and causal reasoning model based on incremental learning algorithms.

[0007] Furthermore, in step S3, the step of feature fusion of the preprocessed multi-source data by the edge computing gateway specifically includes: Extract the cumulative time spent by personnel in each shelf area, the average weight difference of the shelf before and after the operation, the barcode / QR code / text information of the cargo label, and the specific aisle and shelf number of the personnel currently operating; When the weight reduction of the shelf is greater than or equal to the preset picking threshold and the personnel stay time is greater than or equal to 2 seconds, it is determined as a picking action; When the increase in shelf weight is greater than or equal to the preset loading threshold and the personnel stay time is greater than or equal to 3 seconds, it is determined to be a loading action.

[0008] Furthermore, in S4, the multi-dimensional judgment of operational compliance specifically includes: Goods matching judgment: If it is determined to be a goods placement action and the category of the identified goods label is inconsistent with the preset category of the shelf, it is determined to be a misplacement. Personnel access control: If a person stays in an unauthorized shelf area for more than 5 seconds, it is considered an unauthorized operation. Operation timeliness judgment: If, after a pickup action is determined, no corresponding goods are detected being placed on the target shelf within a preset reasonable time of 30 minutes, it is judged as a goods retention behavior.

[0009] Furthermore, in S5, the tiered early warning and full-process recording specifically include: Tiered warning triggering: Level 1 warning corresponds to misplacement and unauthorized operation, triggering a rapid flashing red indicator light and a buzzer alarm, and pushing the warning to the operator's handheld terminal; Level 2 warning corresponds to goods being left unattended, triggering a slow flashing yellow indicator light and pushing the warning to the warehouse manager's terminal; Level 3 warning corresponds to the same person committing 3 or more violations within 24 hours, pushing the warning to the warehouse department head and generating a statistical report. Full-process recording: The time, location, personnel number, operation content, on-site images, and early warning handling status of violations are fully recorded to form a traceable violation record file.

[0010] Furthermore, in S6, the temporal causal reasoning and model self-optimization specifically include: Multi-dimensional time-series data aggregation: Store violation event data, personnel data, equipment data, environmental data, and operational data of the entire warehouse in alignment with timestamps to build a unified time-series data warehouse; Temporal feature extraction: Extracting statistical features of violations at different temporal, spatial, and personnel granularities; Causal structure learning and strength calculation: A constraint-based PC algorithm is used to construct a causal graph of violation events, and domain prior knowledge is introduced to eliminate false causal relationships. A comprehensive causal strength calculation algorithm is used to calculate the strength value of each causal relationship and screen strong causal associations. Causal chain generation and risk prediction: Generate a complete causal chain of leading event - intermediate event - secondary event, and predict high-risk periods, areas and personnel in the next 24-72 hours; Management optimization suggestion generation: Based on the causal chain analysis results, targeted management optimization suggestions are automatically generated and pushed to the management department; Model self-optimization: Based on newly labeled data, incremental learning algorithms are used to iteratively optimize the job behavior recognition model and the temporal causal reasoning model, and the updated model parameters are periodically sent to the edge computing gateway.

[0011] Furthermore, in S6, a constraint-based PC algorithm is used to construct a causal graph of violation events. Specifically, this involves: sorting and collecting observation variables across all dimensions of warehousing operations, including violation event types such as misplacement, unauthorized use, and lingering in different shelf areas, while also incorporating variables related to work periods, personnel scheduling, employee job qualifications, shelf sensing equipment operating conditions, and daily warehousing workload; using the PC algorithm to perform conditional independence tests on each pair of variables, and initially eliminating variable connections with no significant correlation based on partial correlation analysis results; formulating domain prior constraint rules based on actual warehousing operation processes, following the chronological logic of causal events, job operation authority specifications, and the inherent logic of location management business, and filtering out false causal association structures that do not conform to common sense in warehousing operations; gradually simplifying redundant variable association paths, retaining effective association links that simultaneously satisfy statistical independence tests and business constraints, and finally constructing a complete causal graph of violation events that maps the topological relationships between operators, work periods, work areas, equipment operating conditions, and various warehousing violation events.

[0012] Furthermore, in step S6, a comprehensive causal strength calculation algorithm is used to calculate the strength value of each causal relationship, the expression of which is: ,in, This is a comprehensive causal strength value; the larger the value, the stronger the causal relationship between the two variables. It is Granger causality test The value ranges from [0,1]. The smaller the value, the more significant the Granger causality. It is a variable With variables Mutual information values ​​between them Represents a pilot event, Representing a secondary event, it is used to measure the degree of non-linear dependence between two variables. It is the maximum mutual information value among all pairs of variables, used for normalizing mutual information values. These are the weighting coefficients of the Granger causality test results. These are the weight coefficients of the mutual information results, and satisfy... .

[0013] Furthermore, the aforementioned It is a comprehensive causal strength value, when When a strong causal relationship is identified, a complete causal chain of leading event-intermediate event-secondary event is generated based on the strong causal relationship, and risk prediction and management optimization suggestions are generated.

[0014] On the other hand, an automatic identification device for compliance of warehouse operations based on multi-sensor fusion, the device comprising: Magnetic main control terminal: integrates a wide-angle camera, infrared human body sensing module, gravity sensor, three-color indicator light, low-power main control chip, wireless communication module and rechargeable lithium battery, and is magnetically attached to a designated position on the shelf column. At least one shelf gravity sensing module: attached to the bottom of the shelf, used to detect changes in shelf weight in real time and generate gravity data; Lane personnel sensing device: Deployed at the entrance and key nodes of warehouse lanes to obtain coarse personnel positioning data, and is compatible with existing warehouse personnel positioning system patents; Edge computing gateway: It is wirelessly connected to the magnetic main control terminal, the shelf gravity sensing module and the roadway personnel sensing device, respectively, and locally deploys a pre-trained operation behavior recognition model for real-time processing of multi-source data and generating early warning instructions; Cloud-based temporal causal analysis platform: Communicatively connected to the edge computing gateway, including a time-series database, a causal inference engine, a risk prediction module, a management suggestion generation module, and a model optimization module, used to perform temporal causal inference and model self-optimization functions.

[0015] Compared with existing technologies, this method and equipment for automatic identification and early warning of compliance of warehouse operations based on multi-sensor fusion has the following beneficial effects: This invention constructs a causal graph of violation events using a constraint-based PC algorithm and combines it with a comprehensive causal strength calculation algorithm to quantify the correlation between events. This allows for the accurate discovery of the inherent causal relationships between leading and secondary events behind violations at different times, in different areas, and by different personnel. The entire system employs a magnetic and adhesive, wire-free installation structure, eliminating the need for structural modifications to existing shelving and adapting to various existing warehousing sites. Furthermore, it leverages an edge computing architecture to achieve real-time local data analysis and processing, effectively reducing network transmission latency and ensuring the immediacy of violation identification and early warning response. By utilizing historical warehousing time-series data to iteratively update the algorithm model, this invention can not only accurately identify typical violations such as misplaced goods, unauthorized operations, and goods retention, achieving tiered audible and visual early warnings and traceable archiving of violation records, but also analyze the root causes of violations from a business perspective, predicting high-risk operating periods, high-risk areas, and high-risk personnel in advance. This effectively standardizes manual warehousing operations and reduces problems such as inventory discrepancies, goods retention, and product damage.

[0016] Other advantages, objectives and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination or study, or may be learned from the practice of the invention. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0018] Figure 1 A flowchart of a method for automatic identification and early warning of compliance of warehouse operation behavior based on multi-sensor fusion; Figure 2 This is a structural block diagram of an automatic identification and early warning device for compliance of warehouse operations based on multi-sensor fusion; Figure 3 This is a structural block diagram of a cloud-based time-series causal analysis platform for automatic identification and early warning of compliance in warehouse operations based on multi-sensor fusion. Detailed Implementation

[0019] To further illustrate the technical means and effects of the present invention in achieving its intended purpose, the following detailed description of the specific implementation methods, structures, features, and effects of the present invention, in conjunction with the accompanying drawings and preferred embodiments, is provided below.

[0020] This invention proposes an automatic identification and early warning method and device for compliance of warehousing operations based on multi-sensor fusion. It adopts a data processing mechanism of edge feature fusion of multi-source heterogeneous sensors, temporal causal reasoning and incremental learning self-optimization. It can achieve accurate classification of warehousing operation behavior types and real-time judgment of violations without relying on manual inspection or increasing the operational burden of operators. It can also perform highly robust and high-confidence automatic identification and hierarchical early warning of warehousing operations throughout the entire process.

[0021] like Figure 1 As shown, the automatic identification and early warning method for compliance of warehousing operations based on multi-sensor fusion in this embodiment specifically includes: S1: Complete equipment deployment and parameter configuration, enter basic warehouse data, personnel permission data and storage location binding data, and establish the basic environment for system operation; S2: When the infrared human body sensing module detects that a person has entered the detection range, it triggers the wide-angle camera, the floor gravity sensing module and the roadway personnel sensing device to start data acquisition simultaneously, and preprocesses the acquired raw data. S3: The edge computing gateway performs feature fusion on the preprocessed multi-source data, extracts four core features: personnel dwell time, shelf weight change, cargo label information, and personnel operation location, and determines the current operation type based on the features; S4: Based on the extracted features and operation types, compliance judgments are made from three dimensions: goods matching, personnel permissions, and operation timeliness, and judgment results are generated. S5: Trigger the corresponding level of warning based on the judgment result, and at the same time, record the time, location, personnel, operation content and on-site images of the violation and report them to the cloud platform. S6: The cloud platform aggregates all historical data from the warehouse, uses a comprehensive causal strength calculation algorithm to build a time-series causal reasoning model to mine the causal relationships of violations, and iteratively optimizes the operation behavior recognition model and causal reasoning model based on incremental learning algorithms.

[0022] Specifically, the magnetic main control terminal is fixed at a height of 1.5m on each row of warehouse shelf uprights using strong magnetic adsorption. During installation, horizontal calibration is performed to ensure the optical axis of the wide-angle camera is parallel to the shelf panel, covering all shelf areas horizontally and the entire range from the ground to the highest shelf vertically. Shelf gravity sensing modules are evenly attached to the four corners of each shelf using adhesive, with four modules per shelf. Multi-point weighted average calculation of the total shelf weight eliminates detection errors caused by uneven cargo placement. Personnel sensing devices are deployed at the entrance of each warehouse aisle and at key nodes every 50m, with the detection range of adjacent devices overlapping by at least 1m for seamless coverage of coarse personnel positioning. Edge computing gateways are deployed in a well-ventilated and dry area near the warehouse control room, establishing communication connections with all front-end sensors via wireless LAN. The communication distance between the gateway and sensors is controlled within 100m to ensure stable data transmission.

[0023] During the system initialization phase, basic data such as warehouse rack number, shelf number, preset product categories for storage locations, and maximum shelf load capacity are entered; personnel permission data such as operator number, job qualification, authorized work area, and temporary permission validity period are entered; product category code, name, and corresponding storage location binding data are entered; sampling frequency, trigger threshold, communication parameters, and data reporting cycle of each sensor are configured; the shelf gravity sensing module is calibrated under no-load and full-load conditions to eliminate inherent drift and nonlinear errors of the sensors; distortion correction and focal length calibration are performed on the wide-angle camera to ensure the accuracy of image recognition; finally, the basic operating environment of the system is built.

[0024] For example, at 1000m 2 In the e-commerce small parcel warehouse, a total of 20 magnetic main control terminals, 800 shelf gravity sensing modules, 12 aisle personnel sensing devices, and 2 edge computing gateways were deployed. The permission data of 120 operators were entered, and 3,000 storage locations were bound to corresponding goods categories. The gravity sensor sampling frequency was configured to be 10Hz, the data reporting cycle to be 100ms, the wide-angle camera sampling frequency to be 30fps, and the infrared human body sensing module detection distance to be 5m. After no-load zero-point calibration and full-load calibration, the shelf weight detection error was controlled within ±0.5kg, and the camera image distortion rate was less than 1%, meeting the accuracy requirements for judging operation behavior.

[0025] Specifically, when the infrared human body sensing module detects that a person has entered its 5m detection range, it immediately generates a trigger signal and sends it to the edge computing gateway with a local timestamp. After receiving the trigger signal, the edge computing gateway determines the corresponding detection area based on the source of the trigger signal and simultaneously sends a collection instruction to the wide-angle camera, all floor gravity sensing modules, and adjacent lane personnel sensing devices in that area. The instruction contains a unified collection start timestamp to ensure the time synchronization of multi-sensor data.

[0026] The collected raw data is preprocessed as follows: For gravity data, a moving average filtering algorithm is used to remove high-frequency vibration noise generated by forklift movement and personnel movement. At the same time, outlier removal rules are set. When a single weight change exceeds 10% of the maximum load-bearing capacity of the shelf, it is judged as outlier data and removed. For camera image data, radial distortion correction is first performed to eliminate barrel distortion of wide-angle cameras, and Gaussian filtering is used to remove image noise. For personnel positioning data, a Kalman filtering algorithm is used to smooth the positioning trajectory, predict the movement direction and speed of personnel, and correct positioning jump errors. All preprocessed data are aligned with the unified timestamp of the edge computing gateway, and the time synchronization error is controlled within 10ms to ensure the spatiotemporal consistency of multi-source data.

[0027] For example, the moving average filter window size is set to 5, the process noise covariance of the Kalman filter is 0.01, and the measurement noise covariance is 0.1. When personnel enter the tunnel entrance, the infrared sensing module triggers all sensors to start data acquisition within 100ms. After preprocessing, the noise fluctuation of gravity data is reduced from ±2kg to ±0.3kg, the trajectory error of personnel positioning data is controlled within ±0.2m, and the time synchronization error of multi-source data is less than 10ms.

[0028] Specifically, the edge computing gateway performs spatiotemporal alignment and fusion on the preprocessed multi-source data to extract the following four core features: Personnel dwell time characteristics: Combining infrared human body sensing signals and roadway personnel positioning data, timing starts when a person enters the field of view of a camera corresponding to a certain floor and stops when the person leaves the field of view. The total dwell time of the person in that floor area is accumulated, while excluding short-term passage cases with a single dwell time of less than 0.5 seconds. Shelf weight change characteristics: The average weight collected by the shelf gravity sensing module within 3 seconds before the operation is taken as the baseline weight before the operation, and the average weight within 3 seconds after the operation is taken as the baseline weight after the operation. The difference between the two is calculated as the shelf weight change. When the absolute value of the weight change is less than 1kg, it is determined that there is no effective weight change. Cargo label information features: Extract the region of interest (ROI) from the operation area image captured by the wide-angle camera, locate the area where the cargo label is located, and then call the barcode recognition algorithm, QR code recognition algorithm and OCR text recognition algorithm respectively to recognize the label area. Combine the three recognition results and take the cargo category information with the highest matching degree as the final recognition result. Personnel operation location characteristics: The aisle number of the personnel is determined by the aisle personnel sensing device, the shelf row and shelf number of the personnel are determined by the image recognition of the wide-angle camera, and the trigger position of the shelf gravity sensing module is cross-verified to finally determine the specific warehouse location coordinates of the personnel operation, namely aisle number-shelf number-shelf number.

[0029] Operation type determination is based on the above four core characteristics: when the reduction in shelf weight is greater than or equal to the preset picking threshold and the personnel dwell time is greater than or equal to 2s, it is determined to be a picking action; when the increase in shelf weight is greater than or equal to the preset placing threshold and the personnel dwell time is greater than or equal to 3s, it is determined to be a placing action; if any of the above conditions are not met, it is determined to be a non-operational behavior.

[0030] For example, the preset picking threshold is 5kg, and the placing threshold is 5kg. When the system detects that the average weight of a shelf before operation is 120kg, the average weight after operation is 112kg, the weight reduction is 8kg ≥ 5kg, and the cumulative time the person stays in the shelf area is 3s ≥ 2s, the system determines that the behavior is a picking action. When the system detects that the average weight of a shelf before operation is 95kg, the average weight after operation is 103kg, the weight increase is 8kg ≥ 5kg, and the cumulative time the person stays in the shelf area is 4s ≥ 3s, the system determines that the behavior is a placing action.

[0031] Specifically, if a delivery action is determined, the preset list of goods categories for that shelf is retrieved, and the identified goods label category is compared with the preset category list. If the identified category is not in the preset list, it is determined to be a misplacement. If the highest matching degree of the label recognition result is less than 0.8, it is marked as an event to be confirmed and pushed to the warehouse administrator terminal for manual review. Retrieve the authorized area data and daily shift schedule of the operators. If the cumulative time a person spends in an unauthorized shelf area exceeds the preset threshold of 5 seconds, and the person has a shift record for that day, it is considered an unauthorized operation. If the person has no shift record for that day, entering any work area is considered an unauthorized operation. Temporarily authorized personnel entering authorized areas within the authorization period are not considered to be operating beyond their authority. If a pickup action is detected, a timer is started to keep track of the pickup. At the same time, the target storage location information of the pickup task is linked. If no delivery action of the corresponding goods to the target storage location is detected within a preset reasonable time of 30 minutes, and no record of the goods being transferred to the temporary storage area is detected, the goods are determined to be in a lingering situation. If the goods are transferred to the temporary storage area after pickup, the system will automatically reset the timer after recognizing the delivery action in the temporary storage area, and extend the reasonable time to 60 minutes.

[0032] For example, if a shelf is pre-selected as a daily necessities category, and the system detects a placement action, it identifies the goods label as food with a matching degree of 0.95, which is not in the pre-selected category list, thus determining it as misplacement; if an operator is only authorized to operate shelves 1-3, and their cumulative dwell time on a shelf area of ​​shelf 4 is 6s≥5s, and there is a scheduling record for that day, this is determined as unauthorized operation; if the system detects a pickup action at 10:00, with the target location being shelf 5, second floor, and no corresponding goods placement action is detected at the target location by 10:35, nor is there a placement record in the temporary storage area, this is determined as goods being left unattended.

[0033] Specifically, based on the compliance assessment results, corresponding warnings are triggered: Level 1 warnings correspond to misplacement and unauthorized operations, triggering the red indicator light on the magnetic main control terminal to flash rapidly at a frequency of 1Hz and the buzzer to emit an 80dB alarm sound. At the same time, the warning information (time, location, and type of violation) is pushed to the operator's handheld terminal, which displays a pop-up window and vibrates continuously. Level 2 warnings correspond to goods being left unattended, triggering the yellow indicator light on the magnetic main control terminal to flash slowly at a frequency of 0.2Hz and the warning information is pushed to the warehouse manager's terminal. Level 3 warnings correspond to the same person committing 3 or more violations within 24 hours, and a statistical report containing all violation records is pushed to the warehouse department head's terminal.

[0034] Set up an early warning escalation mechanism: If a Level 1 early warning is not handled within 5 minutes after being triggered, it will automatically escalate to a Level 2 early warning and be pushed to the warehouse administrator's terminal; if a Level 2 early warning is not handled within 30 minutes after being triggered, it will automatically escalate to a Level 3 early warning and be pushed to the warehouse department head's terminal.

[0035] The time, location, personnel number, operation content, on-site images (including three screenshots before, during, and after the operation), and early warning processing status of the violation are completely stored locally on the edge computing gateway for a period of no less than 90 days. At the same time, they are synchronously reported to the cloud platform through an encrypted channel to form an unalterable and traceable violation record archive.

[0036] For example, if a worker misplaces an item, the magnetic main control terminal will immediately trigger a flashing red indicator light and a buzzer alarm. If the violation is not corrected after 5 minutes, the system will automatically push a warning message to the warehouse administrator's terminal. If a worker misplaces an item 3 times in a day, the system will automatically generate a statistical report containing the time, location, content, and processing status of the violation, and push it to the warehouse department head's email address at 18:00 on the same day.

[0037] Specifically, the cloud platform stores all warehouse violation data, personnel data, equipment data, environmental data, and operational data in a timestamp-aligned manner to build a unified time-series data warehouse with a data storage period of no less than 3 years; it extracts statistical characteristics of violations at different time granularities (hours, days, weeks), spatial granularities (aisles, shelves, pallets), and personnel granularities (individuals, teams, positions), including violation occurrence rate, violation type distribution, and violation time period distribution.

[0038] A constraint-based PC algorithm is used to construct a causal graph of violation events. First, a comprehensive set of observational variables for warehousing operations is compiled, including types of violations such as misplacement, unauthorized access, and demurrage in different shelving areas. Related variables such as work hours, personnel scheduling, employee qualifications, shelving sensing equipment operating conditions, and daily warehouse workload are also included. Next, conditional independence tests are performed on each pair of variables, with a significance level set at 0.05. Based on partial correlation analysis results, edges connecting variables with no significant correlation are initially removed. Then, domain-specific prior constraints are formulated based on actual warehousing operations, adhering to the chronological logic of causal events, job operation authority specifications, and the inherent logic of location management, thus eliminating false causal relationships that do not conform to common sense in warehousing operations. Finally, redundant variable association paths are gradually simplified, retaining only valid association links that simultaneously satisfy statistical independence tests and business constraints. Ultimately, a complete causal graph of violations is constructed, mapping the topological relationships between personnel, work hours, work areas, equipment conditions, and various warehousing violations.

[0039] The algorithm calculates the strength of each causal relationship using a comprehensive causal strength calculation algorithm, and selects strong causal associations with a strength value ≥ 0.7. Based on the strong causal associations, a complete causal chain of leading event-intermediate event-secondary event is generated to predict high-risk periods, areas and personnel in the next 24-72 hours. Based on the causal chain analysis results, targeted management optimization suggestions are automatically generated, including personnel scheduling adjustments, training plan formulation, equipment maintenance arrangements, etc., and pushed to the management department.

[0040] In the model self-optimization phase, based on the newly added labeled violation data, an incremental learning algorithm is used to iteratively optimize the job behavior recognition model and the temporal causal reasoning model. Each optimization only updates some parameters of the model to avoid catastrophic amnesia. The model update cycle is 7 days. After the updated model parameters are tested and verified, they are automatically distributed to all edge computing gateways to achieve continuous model evolution.

[0041] The expression for the comprehensive causal strength calculation algorithm is as follows: In the formula, S is the comprehensive causal strength value, the larger the value, the stronger the causal relationship between the two variables; p is the p-value of the Granger causality test, the value range is [0,1], the smaller the p-value, the more significant the Granger causal relationship; The mutual information value between the leading event variable X and the secondary event variable Y is used to measure the degree of non-linear dependence between the two variables. This is the maximum mutual information value among all variable pairs, used for normalizing the mutual information value; These are the weighting coefficients of the Granger causality test results. These are the weight coefficients of the mutual information results, and satisfy... .

[0042] For example, setting =0.6, =0.4; PC algorithm analysis yielded a variable combination of early shift, new employees, high workload, and misplacement behavior, with a Granger causality test p-value of 0.02 and a mutual information value of... =0.8, the maximum mutual information value of all variable pairs. =1.0; Substituting into the formula, we get S=0.908>0.7, which is determined to be a strong causal relationship; Based on this causal chain, it is predicted that the No. 3 shelf area during the morning shift in the next 24 hours will be a high-risk area, and the system automatically generates management optimization suggestions and pushes them to the warehouse management department.

[0043] Optionally, the automatic identification and early warning method for compliance of warehouse operations based on multi-sensor fusion further includes a multi-sensor data confidence weighted fusion optimization step, used to improve the accuracy of operation type determination in complex environments. This step acquires the image recognition confidence of the wide-angle camera, the weight detection confidence of the shelf gravity sensing module, and the positioning confidence of the aisle personnel sensing device. Specifically, the image recognition confidence is calculated based on the matching degree of tag recognition, the weight detection confidence is calculated based on the sensor calibration status and ambient temperature, and the positioning confidence is calculated based on signal strength. The confidence of each sensor is used to weight and fuse the corresponding feature data to obtain weighted comprehensive feature data, which is used for subsequent operation type determination and compliance assessment. Specifically, during weighted fusion, sensor data with higher confidence has a larger weight, and sensor data with lower confidence has a smaller weight. When the confidence of a certain sensor is below 0.5, its weight is automatically reduced to 0.1, and the judgment mainly relies on data from other sensors.

[0044] For example, when the ambient light intensity is below 50 lux, the image recognition confidence level decreases from 0.9 to 0.6. At this time, the weights are adjusted to image feature weight 0.3, gravity feature weight 0.5, and positioning feature weight 0.2, increasing the weight ratio of gravity and positioning features. After weighted fusion optimization, the system's accuracy in determining the operation type in low-light scenarios increases from 88% to 95%, effectively reducing misjudgments caused by environmental factors.

[0045] This invention also provides an automatic identification device for compliance of warehouse operations based on multi-sensor fusion, which implements the above method, such as... Figure 2 As shown, it specifically includes the following components: a magnetic main control terminal, at least one layer gravity sensing module, a roadway personnel sensing device, an edge computing gateway, and a cloud-based time-series causal analysis platform; The magnetic main control terminal integrates a wide-angle camera, an infrared human body sensor module, a three-color indicator light, a low-power main control chip, a wireless communication module, and a rechargeable lithium battery. The entire unit features an integrated waterproof and dustproof design with an IP65 protection rating. It is fixed to a designated position on the shelf upright via strong magnetic adsorption, requiring no drilling for installation and allowing for easy disassembly. The wide-angle camera uses a 1080P resolution CMOS sensor with a 120° horizontal and 90° vertical viewing angle, fully covering four shelves in a single row of shelves. It supports automatic exposure and automatic white balance. The infrared human body sensor module uses pyroelectric infrared... The external sensor has a detection range of 0.5-5m, a detection angle of 110°, and a response time of ≤100ms, effectively distinguishing between the human body and other objects. The three-color indicator light supports red, yellow, and green, and can achieve different flashing frequencies to correspond to warning, standby, and normal operation states, respectively. A low-power main control chip handles local data preprocessing and warning command execution. The wireless communication module uses LoRa+WiFi dual-mode communication, with LoRa used for low-power control command transmission and WiFi for high-definition image data transmission. The rechargeable lithium battery is a 5000mAh high-capacity lithium battery.

[0046] The shelf gravity sensing module is evenly attached to the four corners of the shelf bottom using high-strength adhesive. It is used to detect changes in shelf weight in real time and generate gravity data. Each shelf is equipped with four gravity sensing modules. The total weight of the shelf is calculated by multi-point weighted average to eliminate detection errors caused by uneven placement of goods. The shelf gravity sensing module uses a high-precision resistance strain gauge sensor with a range of 0-200kg, a static measurement accuracy of ±0.5kg, a dynamic response time of ≤200ms, and a sampling frequency of 10Hz. It has a built-in low-power processing chip and wireless communication module, and is powered by a button battery. It supports sleep-wake function. When the shelf weight does not change, it enters sleep mode. When the weight change exceeds a preset threshold, it automatically wakes up and reports data, reducing the power consumption of wireless communication.

[0047] The personnel sensing device is deployed at the entrance of the warehouse aisle and at key nodes every 50m interval to obtain coarse personnel positioning data and determine the aisle and approximate area where the personnel are located. The device is compatible with existing warehouse personnel positioning system patents and can directly interface with mainstream personnel positioning devices such as UWB and RFID, without the need to repeatedly deploy positioning tags.

[0048] The edge computing gateway wirelessly connects to the magnetic main control terminal, the shelf gravity sensing module, and the roadway personnel sensing device. It deploys a pre-trained operation behavior recognition model locally to process multi-source data in real time, extract core features, determine operation types, make compliance judgments, and generate early warning instructions. The edge computing gateway uses an industrial-grade processor and supports simultaneous access to more than 200 front-end sensors. It also supports the function of resuming data transmission after network interruption. When the network is interrupted, it stores the violation data locally and automatically synchronizes the unuploaded data after the network is restored.

[0049] The cloud-based time-series causal analysis platform is connected to the edge computing gateway via 4G / 5G or fiber optic communication, employing a distributed cloud server architecture that supports simultaneous access from more than 10 warehouse nodes. Figure 3 As shown, the system includes a time-series database, a causal inference engine, a risk prediction module, a management suggestion generation module, and a model optimization module. These modules are used to perform time-series causal inference and model self-optimization. The time-series database stores multi-dimensional data from the entire warehouse, aligned to timestamps. The causal inference engine incorporates a constraint-based PC algorithm and a comprehensive causal strength calculation algorithm to construct a causal graph of violation events and calculate causal strength. The risk prediction module generates high-risk warnings for the next 24-72 hours based on causal chain analysis results. The management suggestion generation module automatically outputs targeted management optimization solutions based on causal analysis results. The model optimization module iteratively updates core model parameters based on incremental data, with a model update cycle of 7 days.

[0050] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A method for automatic identification and early warning of compliance of warehousing operations based on multi-sensor fusion, characterized in that, The method includes the following specific steps: S1: Complete equipment deployment and parameter configuration, enter basic warehouse data, personnel permission data and storage location binding data, and establish the basic environment for system operation; S2: When the infrared human body sensing module detects that a person has entered the detection range, it triggers the wide-angle camera, the floor gravity sensing module and the roadway personnel sensing device to start data acquisition simultaneously, and preprocesses the acquired raw data. S3: The edge computing gateway performs feature fusion on the preprocessed multi-source data, extracts four core features: personnel dwell time, shelf weight change, cargo label information, and personnel operation location, and determines the current operation type based on the features; S4: Based on the extracted features and operation types, compliance judgments are made from three dimensions: goods matching, personnel permissions, and operation timeliness, and judgment results are generated. S5: Trigger the corresponding level of warning based on the judgment result, and at the same time, record the time, location, personnel, operation content and on-site images of the violation and report them to the cloud platform. S6: The cloud platform aggregates all historical data from the entire warehouse, uses a comprehensive causal strength calculation algorithm to build a time-series causal reasoning model to mine the causal relationships of violation events, and iteratively optimizes the operation behavior recognition model and causal reasoning model based on incremental learning algorithms.

2. The method for automatic identification and early warning of compliance of warehouse operation behavior based on multi-sensor fusion according to claim 1, characterized in that, In step S3, the edge computing gateway performs feature fusion on the preprocessed multi-source data, specifically including: Extract the cumulative time spent by personnel in each shelf area, the average weight difference of the shelf before and after the operation, the barcode / QR code / text information of the cargo label, and the specific aisle and shelf number of the personnel currently operating; When the weight reduction of the shelf is greater than or equal to the preset picking threshold and the personnel stay time is greater than or equal to 2 seconds, it is determined as a picking action; When the increase in shelf weight is greater than or equal to the preset loading threshold and the personnel stay time is greater than or equal to 3 seconds, it is determined to be a loading action.

3. The method for automatic identification and early warning of compliance of warehouse operation behavior based on multi-sensor fusion according to claim 1, characterized in that, In S4, the multi-dimensional judgment of work behavior compliance specifically includes: Goods matching judgment: If it is determined to be a goods placement action and the category of the identified goods label is inconsistent with the preset category of the shelf, it is determined to be a misplacement. Personnel access control: If a person stays in an unauthorized shelf area for more than 5 seconds, it is considered an unauthorized operation. Operation timeliness judgment: If, after a pickup action is determined, no corresponding goods are detected being placed on the target shelf within a preset reasonable time of 30 minutes, it is judged as a goods retention behavior.

4. The method for automatic identification and early warning of compliance of warehouse operation behavior based on multi-sensor fusion according to claim 1, characterized in that, In S5, the tiered early warning and full-process recording specifically include: Tiered warning triggering: Level 1 warning corresponds to misplacement and unauthorized operation, triggering a rapid flashing red indicator light and a buzzer alarm, and pushing the warning to the operator's handheld terminal; Level 2 warning corresponds to goods being left unattended, triggering a slow flashing yellow indicator light and pushing the warning to the warehouse manager's terminal; Level 3 warning corresponds to the same person committing 3 or more violations within 24 hours, pushing the warning to the warehouse department head and generating a statistical report. Full-process recording: The time, location, personnel number, operation content, on-site images, and early warning processing status of violations are fully recorded to form a traceable violation record file.

5. The method for automatic identification and early warning of compliance of warehouse operation behavior based on multi-sensor fusion according to claim 1, characterized in that, In S6, temporal causal reasoning and model self-optimization specifically include: Multi-dimensional time-series data aggregation: Store violation event data, personnel data, equipment data, environmental data, and operational data of the entire warehouse in alignment with timestamps to build a unified time-series data warehouse; Temporal feature extraction: Extracting statistical features of violations at different temporal, spatial, and personnel granularities; Causal structure learning and strength calculation: A constraint-based PC algorithm is used to construct a causal graph of violation events, and domain prior knowledge is introduced to eliminate false causal relationships. A comprehensive causal strength calculation algorithm is used to calculate the strength value of each causal relationship and screen strong causal associations. Causal chain generation and risk prediction: Generate a complete causal chain of leading event - intermediate event - secondary event, and predict high-risk periods, areas and personnel in the next 24-72 hours; Management optimization suggestion generation: Based on the causal chain analysis results, targeted management optimization suggestions are automatically generated and pushed to the management department; Model self-optimization: Based on newly labeled data, incremental learning algorithms are used to iteratively optimize the job behavior recognition model and the temporal causal reasoning model, and the updated model parameters are periodically sent to the edge computing gateway.

6. The method for automatic identification and early warning of compliance of warehouse operation behavior based on multi-sensor fusion according to claim 5, characterized in that, In S6, a constraint-based PC algorithm is used to construct a causal graph of violation events. Specifically, this involves: collecting and organizing observation variables across all dimensions of warehousing operations, including violation event types such as misplacement, unauthorized use, and lingering in different shelf areas, while also incorporating variables related to work hours, personnel scheduling, employee qualifications, shelf sensing equipment operating conditions, and daily warehousing workload; using the PC algorithm to perform conditional independence tests on each pair of variables, and initially eliminating variable connections with no significant correlation based on partial correlation analysis results; formulating domain prior constraint rules based on actual warehousing operation processes, following the chronological logic of causal events, job operation authority specifications, and the inherent logic of location management, and filtering out false causal association structures that do not conform to common sense in warehousing operations; gradually simplifying redundant variable association paths, retaining effective association links that simultaneously satisfy statistical independence tests and business constraints, and finally constructing a complete causal graph of violation events that maps the topological relationships between operators, work hours, work areas, equipment operating conditions, and various warehousing violation events.

7. The method for automatic identification and early warning of compliance of warehouse operation behavior based on multi-sensor fusion according to claim 5, characterized in that, In step S6, the strength value of each causal relationship is calculated using a comprehensive causal strength calculation algorithm, the expression of which is: ,in, This is a comprehensive causal strength value; the larger the value, the stronger the causal relationship between the two variables. It is Granger causality test The value ranges from [0,1]. The smaller the value, the more significant the Granger causality. It is a variable With variables Mutual information values ​​between them Represents a pilot event, Representing a secondary event, it is used to measure the degree of non-linear dependence between two variables. It is the maximum mutual information value among all pairs of variables, used for normalizing mutual information values. These are the weighting coefficients of the Granger causality test results. These are the weight coefficients of the mutual information results, and satisfy... .

8. The method for automatic identification and early warning of compliance of warehouse operation behavior based on multi-sensor fusion according to claim 5, characterized in that, The It is a comprehensive causal strength value, when When a strong causal relationship is identified, a complete causal chain of leading event-intermediate event-secondary event is generated based on the strong causal relationship, and risk prediction and management optimization suggestions are generated.

9. An automatic identification device for compliance of warehouse operation behavior based on multi-sensor fusion, implementing the method of any one of claims 1-8, characterized in that, include: Magnetic main control terminal: integrates a wide-angle camera, infrared human body sensing module, gravity sensor, three-color indicator light, low-power main control chip, wireless communication module and rechargeable lithium battery, and is magnetically attached to a designated position on the shelf column. At least one shelf gravity sensing module: attached to the bottom of the shelf, used to detect changes in shelf weight in real time and generate gravity data; Lane personnel sensing device: Deployed at the entrance and key nodes of warehouse lanes to obtain coarse personnel positioning data, and is compatible with existing warehouse personnel positioning system patents; Edge computing gateway: It is wirelessly connected to the magnetic main control terminal, the shelf gravity sensing module and the roadway personnel sensing device, respectively, and locally deploys a pre-trained operation behavior recognition model for real-time processing of multi-source data and generating early warning instructions; Cloud-based time-series causal analysis platform: Communicatively connected to the edge computing gateway, including a time-series database, a causal inference engine, a risk prediction module, a management suggestion generation module, and a model optimization module, used to perform time-series causal inference and model self-optimization functions.

Images