Storage abnormity monitoring method and system based on federal learning and computer vision

Abstract

The invention discloses a federated learning and computer vision-based storage abnormity monitoring method and system, and the method comprises the steps: collecting a storage scene image in real time by each storage node, carrying out the standardization, data enhancement and denoising processing of the collected image data, only extracting a goods region image containing a goods region, and carrying out the recognition of the goods region image; carrying out rough labeling on the abnormity of the goods area image; inputting the marked image into a federated learning framework based on a star-shaped framework to train a position anomaly detection network model and a state anomaly detection model; each storage node trains a local position anomaly detection model and a local state anomaly detection model, and then uploads training parameters to the central server; the central server performs secure aggregation on the received encrypted local model parameters, generates encrypted global model parameters, issues the encrypted global model parameters to each storage node, and iteratively updates the local model until the model converges; and the converged model is used for anomaly detection and identification of the stored goods.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, machine learning and warehouse management, and particularly relates to a medicine and food warehouse abnormality monitoring method and system based on federated learning and computer vision. BACKGROUND

[0002] In the supply chain system of high safety requirements such as medicines and food, the warehouse link is the core node of guaranteeing product quality and safety and controlling circulation risk. Such products usually have strict storage condition requirements (such as specific temperature and humidity ranges), clear shelf life management specifications and strict packaging integrity standards. Once an abnormality occurs in the warehouse management process, for example, sorting errors caused by incorrect placement of goods, product contamination caused by packaging damage, and expired circulation caused by late warning of near-expiration products, not only will it cause huge direct economic losses, but also may cause serious public health and safety incidents such as medicine failure and food deterioration. Industry statistics show that under the traditional warehouse management mode, the sorting error rate of medicines due to incorrect placement is about 3%-5%, and the loss rate of food due to packaging damage and missed detection of near-expiration products can be as high as 8%-12%, and more than 70% of the errors are caused by manual intervention.

[0003] At present, the common warehouse abnormal management methods mainly include the following: first, the method based on manual scanning of bar code, which needs the operator to scan and record one by one, and the efficiency is low and the missed detection rate is high; second, the method relying on visual inspection, which is subjective and the inspection results depend on the experience and state of the workers, and it is difficult to achieve comprehensive and consistent abnormal coverage; third, the radio frequency identification (RFID) technology is used, but this technology is easily affected by metal packaging, liquid medium and cold chain environment, and has problems such as low recognition rate and high label cost; fourth, the automatic detection system based on preset logic rules, although it realizes a certain degree of automation, but the system flexibility is poor, and it is difficult to adapt to the complex and variable actual situation of the warehouse site, and the recognition ability for the abnormal mode not preset is weak. With the development of computer vision and artificial intelligence technology, the target detection method based on deep learning has been gradually applied to the field of warehouse management, which can realize the automatic recognition of the position, category and part of the state of the goods. However, when the existing deep learning technology is directly applied to the warehouse abnormal detection of high safety industry, there are still significant technical bottlenecks and challenges: first, the data island problem is prominent. Due to the consideration of data privacy and business secrets, the warehouse data of different manufacturers or different places are isolated from each other, and it is difficult to effectively share and converge, which leads to the insufficient generalization ability of the trained detection model, and it is difficult to adapt to various heterogeneous scenes. Second, the labeling cost is high. Deep learning models, especially those used for fine abnormal detection (such as minor damage and blurred bar code), rely heavily on a large amount of high-quality and accurate training data. The data labeling for warehouse abnormalities requires professional knowledge, and the cost is extremely high and the cycle is long, which restricts the iteration and optimization of the model. Third, the environmental robustness is insufficient. The actual warehouse environment is complex (such as light change, goods dense shielding, and various stacking methods), which leads to a significant decrease in the detection accuracy of the existing model in actual deployment. Finally, the performance and real-time performance are difficult to balance. High-precision models usually have high computational complexity, and it is difficult to meet the timeliness requirements of real-time monitoring and early warning on the edge computing devices in the warehouse site.

[0004] Therefore, an innovative abnormal detection scheme needs to be designed, which can effectively reduce the dependence on high-quality labeled data while fully protecting the data privacy of each participant, and at the same time improve the detection accuracy, robustness and real-time processing capability in complex and variable real warehouse scenes, to meet the extreme requirements of high safety industries such as medicine and food on warehouse management. SUMMARY

[0005] In order to overcome the defects of the prior art, the present application proposes a warehouse abnormal monitoring method and system based on federated learning and computer vision, which can realize efficient and accurate detection of the abnormal state of goods in the process of medicine and food warehouse, thereby improving the intelligent level and safety of warehouse management.

[0006] The first aspect of the present application is to provide a warehouse abnormality monitoring method based on federated learning and computer vision, comprising:

[0007] Step one: each warehouse node collects warehouse scene images in real time, the warehouse scene images include product barcodes, packaging areas and date information; after standardization, data enhancement and denoising processing of the collected image data, only the product area image containing the product area is extracted for subsequent processing;

[0008] Step two: determining whether it is abnormal according to the displacement of the product area image obtained in step one, the clarity of the product packaging boundary and the date determination rule of the product barcode, and performing batch coarse annotation on the identified abnormalities according to the determined rule;

[0009] Step three: input the image coarsely annotated in step two into the federated learning framework based on the star architecture to train the position abnormality detection network model and the state abnormality detection model;

[0010] Among them, the federated learning framework architecture adopts a star architecture including a central server and multiple warehouse nodes; the central server distributes an initial global model to each warehouse node; each warehouse node uses the locally preprocessed product area image to train the local position abnormality detection model and state abnormality detection model in parallel, generating local model parameters;

[0011] Each warehouse node uses a homomorphic encryption algorithm to encrypt the local model parameters and upload them to the central server;

[0012] The central server securely aggregates the received encrypted local model parameters to generate encrypted global model parameters and distributes them to each warehouse node; each warehouse node decrypts the encrypted global model parameters to update the local model;

[0013] Iteratively update the federated learning framework until the model converges;

[0014] Step four: at each warehouse node, input the real-time collected and preprocessed product area image into the position abnormality detection model and state abnormality detection model trained in step three; when the position abnormality detection model outputs an abnormal confidence exceeding a preset position abnormality confidence, generate a position abnormality warning and coordinate information; when the state abnormality detection model identifies at least one of packaging damage, near expiration or barcode abnormality, generate a state abnormality warning and category information; push the warning information to the warehouse management system for subsequent processing.

[0015] Further, each warehouse node in the federated learning framework uses homomorphic encryption to encrypt the model parameters when transmitting data to the central server.

[0016] Further, the working process of the federated learning framework comprises:

[0017] Initialization: the central server issues initial model parameters of the position anomaly detection network model and the state anomaly detection model to each warehouse node;

[0018] Local training: each warehouse node inputs the batch of rough-labeled data in step two into the position anomaly detection network model and the state anomaly detection model for training;

[0019] Samples with both model prediction confidence and rule confidence greater than a preset value are further labeled as high-confidence pseudo labels;

[0020] The high-confidence pseudo label samples and pre-labeled samples are mixed as a training set, and the position anomaly detection network model and the state anomaly detection model are retrained. After each round of training, samples with no labels among samples with both prediction confidence and rule confidence greater than a preset value are supplemented as high-confidence pseudo labels, and the training set is continuously expanded for iteration;

[0021] Encrypted upload and issuance of global model parameters: after the data is encrypted using a homomorphic encryption algorithm, it is uploaded to the central server, the data is globally parameterized, the aggregation weight is calculated according to the data volume and data quality of each warehouse node, then the encrypted parameters are weighted and averaged to obtain the global model parameters; the encrypted global model parameters are obtained and issued to each warehouse node;

[0022] Retraining: each warehouse node updates the issued global model parameters to retrain the position anomaly detection network model and the state anomaly detection model until the model self-optimization setting is met.

[0023] Further, the model self-optimization setting comprises at least one of the following: no new high-confidence pseudo labels, reaching a preset maximum number of iterations, running out of unlabeled samples, or showing signs of overfitting.

[0024] Further, the position anomaly detection network model is used to detect position anomalies of goods in preprocessed goods images, including cases where goods exceed the boundaries of the grid and multiple goods are stacked in the grid, and records the abnormal coordinate information; the state anomaly detection model is used to identify goods state anomalies, outputting the state anomaly category and confidence, and the state anomaly category includes packaging damage, near expiration, and barcode anomaly.

[0025] In an embodiment, the position anomaly detection network model is improved based on YOLOv5s, the backbone network adopts CSPDarknet-18, a grid anchor layer is added at the neck, pre-defined shelf grid coordinates are used to preferentially learn the position features of goods in the grid during model training, and the combination of classification loss and positioning loss is set.

[0026] In an embodiment, the state anomaly detection model is divided into three parts: a feature extraction layer, a deformable alignment module, and a multi-modal fusion layer; the feature extraction layer adopts ResNet-18 to extract visual features of package texture and edges, and adopts an OCR model to extract text features of dates and barcodes; the deformable alignment module dynamically adjusts the receptive field of the convolution kernel according to the size of the goods package to realize the positioning of local anomalies; the multi-modal fusion layer performs weighted fusion on visual features and text features through an attention mechanism to output a state anomaly category and a confidence; wherein the state anomaly category includes package damage, near-expiration, and barcode anomaly.

[0027] The second aspect of the application provides a warehouse anomaly monitoring system based on federated learning and computer vision, comprising a central server and a plurality of warehouse nodes.

[0028] The central server is configured to receive local model parameters processed by homomorphic encryption from the plurality of warehouse nodes, securely aggregate the received encrypted local model parameters to generate encrypted global model parameters, and distribute the encrypted global model parameters to each warehouse node to coordinate the completion of the federated learning training process.

[0029] Each of the warehouse nodes comprises an image acquisition and processing module and a local learning module.

[0030] The image acquisition and processing module is configured to acquire warehouse scene images in real time, and after standardization, data enhancement, and denoising processing of the acquired image data, only extract the goods region images containing the goods region.

[0031] The local learning module uses the goods region images processed by the local image acquisition and processing module to locally train the position anomaly detection model and the state anomaly detection model, and encrypts the local model parameters obtained by training using a homomorphic encryption algorithm and uploads them to the central server.

[0032] After the model training is completed, the real-time acquisition and pre-processing of the goods region images are input into the position anomaly detection model and the state anomaly detection model for anomaly monitoring, and the generated position anomaly warning and / or state anomaly warning information is pushed to the warehouse management system.

[0033] Furthermore, the central server is deployed on a third-party neutral platform; the location anomaly detection model is used to detect abnormal product location and generate location anomaly warnings and coordinate information when the anomaly confidence exceeds a preset threshold; the status anomaly detection model is used to identify at least one status anomaly among packaging damage, near-expiration, or barcode anomaly, and generate status anomaly warnings and category information.

[0034] Furthermore, 2-4 cameras are deployed in each shelf aisle, and the network layer controls the transmission latency to within 50 milliseconds.

[0035] Furthermore, the network layer adopts a hybrid networking mode of 5G and Wi-Fi 6, wherein the 5G network is used to cover large warehouses with an area of ​​more than 10,000 square meters, for communication of mobile cameras, and / or transmission of sensitive data.

[0036] Furthermore, Wi-Fi 6 networks operate in the 5GHz band and are allocated non-overlapping channels to avoid co-channel interference.

[0037] The beneficial effects of this invention are as follows:

[0038] The warehouse anomaly monitoring method and system based on federated learning and computer vision described in this invention introduces a federated learning framework. The local raw image data of each warehouse node does not need to be uploaded to the central server, but is only used locally for model training. Moreover, the uploaded model parameters are encrypted using a homomorphic encryption algorithm, which fundamentally eliminates the risk of leakage of sensitive data during transmission and centralized processing.

[0039] The federated learning framework follows a process of initialization → local training → encrypted upload → aggregation → distribution → retraining. Furthermore, it constructs a "local training-encrypted aggregation-global update" process for training, which can efficiently utilize distributed computing resources while ensuring data isolation, enabling collaborative modeling of multiple nodes and accelerating model convergence.

[0040] The system significantly improves detection accuracy, with a false negative rate of less than 5%, effectively reducing warehouse operation risks. It also significantly reduces data labeling and hardware deployment costs, and has good adaptability to complex scenarios and cross-industry scalability. Attached Figure Description

[0041] Figure 1 This is a flowchart of the warehouse anomaly monitoring method based on federated learning and computer vision described in this invention.

[0042] Figure 2 This is a schematic diagram illustrating the working principle of the state anomaly detection model constructed in step three of the warehouse anomaly monitoring method based on federated learning and computer vision.

[0043] Figure 3A flowchart illustrating the learning and training process for the federated learning framework constructed in step four. Detailed Implementation

[0044] To make the objectives, technical solutions, beneficial effects, and significant advancements of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings provided in the examples of the present invention. Obviously, all the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0045] In the description of this application, unless otherwise expressly specified and limited, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance; the term "multiple" refers to two or more; unless otherwise specified or explained, the terms "connected," "fixed," etc., should be interpreted broadly. For example, "connected" can be a fixed connection, a detachable connection, an integral connection, or an electrical connection; "connected" can be a direct connection or an indirect connection through an intermediate medium. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances.

[0046] This embodiment uses a regional warehousing center of a large chain pharmacy as an application scenario. The warehousing center stores more than 1,200 types of medicines, covering various packaging types such as tablets, capsules, and oral liquids. The storage area is 5,000 square meters, with 100 sets of shelves. The average daily inbound and outbound medicines are 30,000 pieces. The original abnormality monitoring method used was manual barcode scanning and visual inspection, with a missed detection rate of 18% and an average processing volume of 220 pieces per person per hour. The expiration loss rate caused by missed detection of near-expiry products was about 5%.

[0047] Hardware and software deployment:

[0048] Edge layer: within 9 aisles of 10 shelving units,

[0049] Two 1080P cameras (25 FPS, 90°-120° field of view) are deployed in each aisle. The cameras are 2.5m high and facing the shelf openings to ensure coverage of all product placement areas.

[0050] Each channel is equipped with one AI edge computing device for target detection (21 TOPS computing power), which is connected to the camera via a PoE switch, and the embedded device is connected to the cloud server via a 5G module. The AI ​​edge computing device deploys and runs a location anomaly detection network model, which is based on an improved YOLOv5s, with an inference speed of 32 FPS and a location anomaly detection confidence threshold of 80%.

[0051] Cloud layer: Deploy 1 server (128GB memory, 2TB SSD), install Ubuntu 20.04 operating system, Docker containerized environment, TensorFlow 2.8 deep learning framework; deploy an anomaly detection model (ResNet-18+CRNN-Lite, 12.5M parameters), inference speed 28 FPS, packaging damage confidence threshold set to 85%, near-expiry judgment threshold set to 30 days (less than 30 days from expiry date is considered near-expiry), barcode recognition accuracy threshold set to 80%.

[0052] Network Layer: Employing 5G / Wi-Fi 6 wireless communication technology, high-speed data transmission between edge devices and cloud servers is achieved, with transmission latency controlled within 50ms. In large warehouse multi-camera scanning scenarios, Wi-Fi 6 is generally applicable but requires optimization: Leveraging OFDMA technology and a single AP speed of 3-4Gbps, it can support dozens or even hundreds of concurrent 1080P cameras (2-4Mbps per channel), with an actual latency of 10-30ms meeting the ≤50ms requirement. It is also cost-effective and flexible in deployment. However, a single AP only covers 500-1000㎡, requiring deployment at a density of 800-1000㎡ per AP (e.g., 6-8 APs for a 5000㎡ warehouse). Furthermore, the 5GHz frequency band must be selected, non-overlapping channels allocated, and bandwidth priority assigned to cameras to avoid interference from metal shelves and signal blind spots. For warehouses exceeding 10000㎡, with mobile cameras, or transmitting sensitive data, a hybrid deployment with 5G can be implemented.

[0053] like Figure 1 As shown, a computer vision and federated learning method for pharmaceutical and food warehousing includes:

[0054] Step 1: Multi-source image data acquisition and preprocessing:

[0055] S11: Data Acquisition: Using cameras deployed on the top of the aisles between warehouse shelves, real-time image data of goods was collected for 7 consecutive days, 12 hours per day (covering different lighting periods in the morning, noon, and evening), totaling 120,000 images. These images covered various packaging types, including tablets (30%), capsules (25%), oral liquids (20%), ointments (15%), and others (10%). Among these, 96,000 images were normal, and 24,000 images were abnormal (8,000 images with abnormal locations, 6,000 images with damaged packaging, 6,000 images nearing their expiration date, and 4,000 images with abnormal barcodes). The collected image data covered the barcode area, the intact packaging area, and the production date / expiration date printing area.

[0056] S12: Standardization: The acquired images are uniformly resized to 1920×1080 resolution to eliminate the impact of different camera resolutions on the model input;

[0057] S13: Data Augmentation: The images standardized in S12 were augmented by random cropping, rotation, lighting simulation, and Gaussian noise addition to obtain 360,000 augmented images, improving the model's robustness to changes in warehouse lighting and deviations in the angle of goods placement. Specifically, the cropping ratio was set to 0.8-1.0, the brightness was set to ±30% and the contrast to ±20% during lighting simulation, the variance was set to 0.01-0.03 during Gaussian noise addition, and the image rotation was set to -15° to 15°.

[0058] S14: Noise Removal: Gaussian filtering (kernel size=3×3) and adaptive threshold binarization are applied to the image processed in S13 to highlight key text areas such as barcodes and dates and suppress background noise interference.

[0059] S15: Region Segmentation: A pre-trained visual model is used to divide the image into "product regions" and "background regions." Subsequent detection is performed only on the product regions, reducing background interference and improving inference speed; thus obtaining pre-processed image and environmental data. The visual model is SCRTNet, a visual feature extraction backbone network that fuses local and global information. SCRTNet is a combination of ResNet and Transformer.

[0060] Step 2: Weakly Supervised Pseudo-Label Generation and Model Optimization

[0061] S21: Construct a rule base: Location rules include: if the offset between the barcode center and the compartment center is >5mm, the location is considered abnormal; if the number of goods in the same compartment is >1, the stacking is considered abnormal; Status rules include: if the grayscale gradient of the packaging edge is >30, the damage is suspected; if the clarity of the date character is <0.7, the code is considered to be awaiting verification (0-1 score, 1 being the clearest); if the barcode character length is ≠13 digits, the barcode is considered abnormal.

[0062] S22: Perform batch coarse annotation based on the determined rule engine: Input the preprocessed image from step one into the rule engine determined in S21 to automatically generate basic pseudo-labels. The annotation format is: image path - anomaly type - anomaly coordinates - confidence level. The initial pseudo-label coverage reaches over 80%, and the accuracy is approximately 85%.

[0063] In this embodiment, pseudo-labels are automatically generated for 360,000 images based on a rule engine. After removing about 60,000 images due to issues such as dim lighting and camera movement speed, the final coverage is 300,000 images (coverage rate of 83.3%). The accuracy rate of pseudo-labels for abnormal positions is 86%, and the accuracy rate of pseudo-labels for abnormal states is 84%. The remaining 60,000 images are manually labeled and used as a validation set and a small amount of training set.

[0064] Step 3: Input the image coarsely annotated in Step 2 into the pre-trained federated learning framework to simultaneously identify location anomalies and state anomalies; the federated learning framework includes a location anomaly detection network model and a state anomaly detection model.

[0065] The location anomaly detection network model is used to detect location anomalies in the preprocessed product images. Location anomalies include products exceeding the grid boundaries and multiple products stacked within the grid. The model also records the anomaly coordinates. The state anomaly detection model is used to identify state anomalies and output the state anomaly category and confidence level. State anomaly categories include damaged packaging, near-expiry date, and barcode anomalies.

[0066] The location anomaly detection network model is constructed as follows:

[0067] Model architecture: Based on YOLOv5s improvements, the backbone network adopts CSPDarknet-18 (replacing the original CSPDarknet-53), reducing the number of parameters (from 8.7M to 7.2M). At the same time, a grid anchoring layer is added to the neck, and the grid coordinates are predefined (e.g., each grid is 300×200 pixels). During model training, the positional features of the goods in the grid are learned first.

[0068] Loss function: A combination of "classification loss (Focal Loss) + localization loss (CIoU Loss)" is used. Focal Loss solves the class imbalance problem between outlier samples (accounting for about 5%-10%) and normal samples, while CIoU Loss improves the localization accuracy of bounding boxes.

[0069] like Figure 2 As shown, the state anomaly detection model is constructed as follows:

[0070] Model Architecture: The model consists of three parts: a feature extraction layer, a deformable alignment module (DAM), and a multimodal fusion layer. The feature extraction layer uses ResNet-18 to extract visual features such as packaging texture and edges, while using a lightweight OCR model (such as CRNN-Lite) to extract text features such as dates and barcodes. The DAM module dynamically adjusts the receptive field of the convolutional kernels according to the size of the product packaging (e.g., using 3×3 kernels for small-sized pharmaceutical packaging and 7×7 kernels for large-sized food boxes) to achieve accurate localization of anomalies (such as minor damage). The multimodal fusion layer uses an attention mechanism to weight and fuse visual features (weight 0.6) and text features (weight 0.4), outputting the state anomaly category (packaging damage, near expiration, barcode anomaly) and confidence level.

[0071] Step 4: Construct a federated learning framework, embed the training of the location anomaly detection network model and state anomaly detection model constructed in Step 3 into the federated learning framework, and train the model based on the image data labeled with pseudo-labels in Step 2.

[0072] like Figure 3 As shown, the federated learning framework architecture is designed as a star topology with a central server and multiple storage nodes. The central server is deployed on a neutral third-party platform within the industry association's cloud platform, while each storage node (including different pharmacies and food warehouses) deploys a local training module. Furthermore, homomorphic encryption (Paillier algorithm) is used to encrypt model parameters when transmitting data between each storage node and the central server.

[0073] In application scenarios involving multiple warehouse locations and data isolation from multiple vendors, parameter aggregation is employed to achieve collaborative model training, ensuring data privacy and security while improving model generalization ability. Homomorphic encryption and differential privacy processing are combined during parameter transmission to ensure data security throughout the entire process. Based on improvements to the FedAvg algorithm, a distributed training process of "local training - encrypted aggregation - global update" is constructed to solve the data isolation problem across multiple warehouse locations.

[0074] The specific work process is as follows:

[0075] S41: Initialization: The central server distributes the initial model parameters (random initialization or pre-trained parameters based on public datasets) of the location anomaly detection network model and the state anomaly detection model to each warehouse node.

[0076] S42: Local Training: Each warehouse node inputs the data labeled in step two (including 300,000 pseudo-label samples labeled by the rule engine and 60,000 manually labeled samples) into the location anomaly detection network model and state anomaly detection model created in step three for training. The training rounds are 5-10, and the batch size is adjusted according to the node hardware configuration (edge ​​device batch size=8-16, server batch size=32-64). Only the model parameters are saved during the training process (excluding the original data).

[0077] S43: Further annotation: Samples with both model prediction confidence and rule confidence > 90% are selected as high-confidence pseudo-labels, improving accuracy to 92%-95%;

[0078] S44: Model Iterative Training: Mix high-confidence pseudo-label samples (70%) with a small number of manually labeled samples (30%) as the training set, retrain the dual model, and after 3-5 iterations, the model's accuracy in recognizing pseudo-label samples will be improved by 10%-15%, while reducing the dependence on manually labeled samples.

[0079] S45: Pseudo-label update: After each training round, unlabeled samples with a new model prediction confidence and rule confidence greater than 90% are supplemented as high-confidence pseudo-labels. The training set is continuously expanded in S44 to achieve model self-optimization. The model self-optimization process of pseudo-label update can be stopped when the model performance converges and stabilizes, no new high-confidence pseudo-labels are added, the preset maximum number of iterations (set to 500) is reached, the unlabeled samples are exhausted, or overfitting symptoms appear.

[0080] The specific security and aggregation mechanisms of the federated learning framework are as follows:

[0081] In the federated learning framework, each repository node uses the Paillier homomorphic encryption algorithm to encrypt the local model parameters and upload them to the central server. The encryption key is kept independently by each node. The central server can only receive and manipulate the encrypted parameters and cannot decrypt them to obtain the original model parameters, thus ensuring the privacy and security of the data of each node.

[0082] In the federated learning framework, a dynamic weighted global parameter aggregation strategy is adopted: After receiving the encrypted parameters uploaded by each node, the central server uses an improved FedAvg algorithm to aggregate the global parameters. The aggregation weight is not a simple average, but is calculated sequentially based on the data volume (proportional weight) and data quality (labeling accuracy weight) of each node, and then the encrypted parameters are weighted and averaged to obtain the global model parameters. Differential privacy noise (Laplace noise, scale parameter 0.01) is added during the aggregation process to prevent the inference of local data features from the aggregated parameters.

[0083] The weight of each node's data volume is as follows: a basic weight is allocated based on the data volume of each storage point. The weight of the top 30% of nodes is set between 0.15 and 0.2; the weight of the bottom 70% of nodes is set between 0.05 and 0.1.

[0084] Data quality weight: The weight is adjusted based on the labeling accuracy of the local data of each node. The aggregate weight of nodes with a labeling accuracy of >95% is increased by 20% on the basis of the base weight.

[0085] Model update distribution: The central server in the federated learning framework decrypts the aggregated global model parameters (only verifies integrity, does not obtain the original parameters) and distributes them to each repository node. Each node loads the global parameters as the initial parameters for a new round of local training. Iterative training is conducted for 10-20 rounds until the evaluation accuracy of the global model fluctuates by less than 1%, which is considered as model convergence.

[0086] Building upon the federated learning framework, multi-layered security measures are employed to enhance data privacy and system security, including:

[0087] Local data retention: The original image data and annotation data of each node are stored on the local server and are not uploaded to the central server or other nodes, thus protecting data privacy from the source;

[0088] Parameter transmission encryption: HTTPS + homomorphic encryption dual encryption protocol is used to prevent parameters from being stolen or tampered with during transmission;

[0089] Node authentication: The central server records the training logs of each node (such as parameter upload time and data volume) through blockchain technology. Nodes must be authenticated (such as digital certificates) when joining or leaving to prevent malicious node attacks.

[0090] The abnormal response monitoring and processing system, constructed and trained using computer vision and federated learning methods in pharmaceutical and food warehousing, is as follows:

[0091] Cameras located at each warehouse node collect real-time image data of pharmaceuticals in the warehouse environment; after standardization, data enhancement and noise reduction of the collected image data, only the product area of ​​the image is extracted for subsequent processing;

[0092] The processed images are input into the pre-trained location anomaly detection network model and state anomaly detection model respectively, to identify both location anomalies and state anomalies simultaneously;

[0093] An anomaly detection network model detects whether there are any anomalies in the product area of ​​an image. If the anomaly confidence level is >80%, an anomaly warning is immediately generated, and the images of the normally located product areas and the anomaly-identified product areas, along with their coordinate information, are uploaded to the cloud. The inference process of the anomaly detection network model is as follows: The AI ​​edge computing device loads the anomaly detection network model, performs inference on the preprocessed product images, and outputs the product location coordinates and anomaly confidence level. When the confidence level is >80%, an anomaly is identified. Anomalies include products exceeding the grid boundary and multiple products stacked within the grid. The anomaly coordinate information is recorded, and the images of the normally located product areas and the anomaly-identified product areas, along with their coordinate information, are sent to the cloud layer.

[0094] A pre-trained state anomaly detection model in the cloud performs state anomaly detection on uploaded images. If at least one of the following anomalies is detected: damaged packaging, near-expiry date, or abnormal barcode, a state anomaly warning is generated, abnormal area features are extracted, and the anomaly category is reported in the form of a label. The reasoning process of the state anomaly detection model is as follows: The AI ​​edge computing device uploads images that are normal (or the goods area in the image with abnormal location) detected by the location anomaly detection network model to the cloud. The location anomaly detection network model in the cloud performs reasoning. When the confidence level of damaged packaging is >85%, the confidence level of near-expiry date (date <30 days from expiry) is >90%, and the barcode recognition accuracy is <80%, it is determined to be a state anomaly of the corresponding type, and abnormal area features (such as the coordinates of the damaged location, the date text content, and unclear barcodes) are extracted.

[0095] Subsequently, alerts regarding abnormal locations and / or statuses are merged and pushed to the mobile devices of warehouse management personnel and the existing warehouse management system (WMS).

[0096] After receiving the alert, the management personnel will go to the location of the anomaly to verify it and confirm the anomaly type through the mobile APP (such as "confirmed minor damage to packaging" or "false alarm cleared"). The system records the verification results for subsequent model optimization (such as adding false alarm samples to the training set and adjusting the model threshold).

[0097] After the exception is handled, the system automatically updates the exception status (such as "handled" or "pending return / exchange") and generates an exception handling report (including exception statistics, handling time, and cause analysis), which can be exported in PDF / Excel format.

[0098] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style of the specification is merely for clarity. Those skilled in the art should regard the specification as a whole, and the technical solutions in the embodiments can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A warehouse abnormality monitoring method based on federated learning and computer vision, characterized in that, The application relates to a warehouse scene image real-time collection method based on a federated learning framework. Step 1: each warehouse node collects a warehouse scene image in real time, wherein the warehouse scene image comprises a product barcode, a packaging area and date information; After standardization, data enhancement and denoising processing are performed on the collected image data, only a product area image containing a product area is extracted for subsequent processing; Step 2: whether an exception exists is determined according to the displacement of the product position of the product area image obtained in step 1, the clarity of the product packaging boundary and the date determination rule of the product barcode, and batch coarse labeling is performed on the identified exception according to the determined rule; Step 3: the image coarsely labeled in step 2 is input into a position anomaly detection network model and a state anomaly detection model based on a star-shaped architecture federated learning framework; The federated learning framework adopts a star-shaped architecture comprising a central server and multiple warehouse nodes; the central server distributes an initial global model to each warehouse node; each warehouse node utilizes a locally preprocessed product area image to train the local position anomaly detection model and state anomaly detection model in parallel, and generates local model parameters; Each warehouse node encrypts the local model parameters by using a homomorphic encryption algorithm and uploads the local model parameters to the central server; The central server securely aggregates the received encrypted local model parameters, generates encrypted global model parameters and distributes the global model parameters to each warehouse node; each warehouse node decrypts the encrypted global model parameters to update the local model; The federated learning framework is iteratively updated until the model converges; Step 4: at each warehouse node, a real-time collected and preprocessed product area image is input into the position anomaly detection model and the state anomaly detection model trained in step 3; when the position anomaly detection model outputs an abnormal confidence exceeding a preset position anomaly confidence, a position anomaly early warning and coordinate information are generated; when the state anomaly detection model identifies at least one of a packaging damage, a near-expiration date or a barcode anomaly, a state anomaly early warning and category information are generated; and the early warning information is pushed to a warehouse management system for subsequent processing.

2. The warehouse anomaly monitoring method based on federated learning and computer vision according to claim 1, characterized in that, The working process of the federated learning framework comprises: Initialization: the central server distributes initial model parameters of a position anomaly detection network model and a state anomaly detection model to each warehouse node; Local training: each warehouse node inputs the batch coarse labeled data in step 2 into the position anomaly detection network model and the state anomaly detection model for training; Samples with a model prediction confidence and a rule confidence both greater than a preset value are further labeled as high-confidence pseudo labels; The high-confidence pseudo label samples and pre-labeled samples are mixed as a training set, and the position anomaly detection network model and the state anomaly detection model are retrained; after each round of training, samples without labels in the samples with a prediction confidence and a rule confidence both greater than a preset value are supplemented as high-confidence pseudo labels, and the training set is continuously expanded for iteration. Encrypting uploading and issuing global model parameters: after the data is encrypted by a homomorphic encryption algorithm, the data is uploaded to a central server, the data is aggregated for global parameters, the aggregation weight is calculated according to the data volume and data quality of each storage node, then the encrypted parameters are weighted and averaged to obtain the global model parameters; the encrypted global model parameters are obtained and issued to each storage node; Retraining: each storage node updates and re-trains the position anomaly detection network model and the state anomaly detection model according to the global model parameters issued, and trains until the model self-optimization setting is met. 3.The warehouse abnormality monitoring method based on federated learning and computer vision of claim 2, wherein, The model self-optimization setting includes at least one of the following: no new high-trust pseudo-label, reaching a preset maximum iteration number, no labeled sample depletion or overfitting sign. 4.The warehouse abnormality monitoring method based on federated learning and computer vision of claim 1, wherein, The position anomaly detection network model is used to detect the position anomaly of the preprocessed goods image, and the position anomaly includes the case that the goods exceed the boundary of the grid or the goods are stacked in the grid, and the abnormal coordinate information is recorded. The state anomaly detection model is used to identify the state anomaly of the goods, and outputs the state anomaly category and confidence, and the state anomaly category includes package damage, near expiration, and barcode anomaly. 5.A warehouse abnormality monitoring system based on federated learning and computer vision, characterized in that The central server and a plurality of storage nodes are included. The central server is used to receive the local model parameters processed by homomorphic encryption from the plurality of storage nodes; securely aggregates the received encrypted local model parameters to generate encrypted global model parameters; and issues the encrypted global model parameters to each storage node to coordinate the completion of the federated learning training process. Each storage node includes an image acquisition and processing module and a local learning module. The image acquisition and processing module is used to acquire real-time warehouse scene images, and after standardization, data enhancement and denoising processing of the acquired image data, only the goods region image containing the goods region is extracted. The local learning module uses the goods region image processed by the local image acquisition and processing module to locally train the position anomaly detection model and the state anomaly detection model, and encrypts the local model parameters obtained by training by using a homomorphic encryption algorithm and uploads them to the central server. After the model training is completed, the preprocessed goods region image is input into the position anomaly detection model and the state anomaly detection model for anomaly monitoring, and the generated position anomaly warning and / or state anomaly warning information is pushed to the warehouse management system.

6. The federated learning and computer vision based warehouse anomaly monitoring system of claim 5, wherein The central server is deployed on a third-party neutral platform; the position anomaly detection model is used to detect the position anomaly of the goods, and generates a position anomaly warning and coordinate information when the abnormal confidence exceeds a preset threshold; and the state anomaly detection model is used to identify at least one of the package damage, near expiration or barcode anomaly, and generates a state anomaly warning and category information.

Images