Warehouse material flow anomaly processing method, device and equipment and storage medium

Abstract

This invention relates to the field of warehouse material flow handling technology, and discloses a method, apparatus, equipment, and storage medium for handling warehouse material flow anomalies. The method includes acquiring warehouse monitoring video streams, IoT sensor data, and text data of the target material; determining a structured spatiotemporal point sequence characterizing the handling trajectory of the target material based on the warehouse monitoring video streams and IoT sensor data; determining the probability of anomaly risk in the current handling state of the target material based on the structured spatiotemporal point sequence and combined with the text data; and sending an intervention command to the material handling equipment corresponding to the target material when the probability of anomaly risk exceeds a preset threshold, so that the material handling equipment responds to the intervention command and executes an intervention action, the intervention action being determined based on the intervention command. This solution improves the efficiency of handling logistics flow anomalies.

Description

Technical Field

[0001] This invention belongs to the field of warehouse material monitoring technology, specifically relating to methods, devices, equipment, and storage media for handling abnormal warehouse material flow. Background Technology

[0002] In the current warehouse logistics system, material flow involves complex interactions between forklifts, AGVs, personnel and the environment. Traditional monitoring methods have three technical bottlenecks: (1) Single perception dimension, such as relying only on temperature or vibration sensors, which cannot capture spatial behavior anomalies such as speeding and path deviation, resulting in low anomaly detection accuracy; (2) Fragmented multi-source data, with video, IoT signals and WMS instructions lacking semantic association, making it difficult to build causal reasoning chains and causing data silos; (3) Lagging response mechanism, with a serious delay between anomaly detection and intervention, making it difficult to meet the safety requirements of highly dynamic warehouses.

[0003] While existing video understanding frameworks incorporate spatiotemporal annotation (such as timestamps and bounding boxes), they face the problem of a disconnect between coordinate representation and temporal inference in warehouse scenarios. This leads to significant detection latency, high false positive rates, and severe token length redundancy. For example, existing technologies (such as CN116229272A) employ fixed grids and single PPO optimization, which cannot handle spatiotemporal collapse and excessively long inference chains in dynamic scenes. Therefore, there is an urgent need for a technical solution that can uniformly encode spatiotemporal information into structured point sequences and support real-time inference and closed-loop intervention. Summary of the Invention

[0004] In view of the shortcomings of the prior art, the purpose of this invention is to provide a method, apparatus, equipment and storage medium for handling abnormal material flow in warehouses.

[0005] According to one aspect of this application, a method for handling abnormal warehouse material flow is disclosed, the method comprising: Acquire warehouse monitoring video streams, IoT sensor data, and text data of the target material from the warehouse management system; Based on the warehouse monitoring video stream and the IoT sensor data, a structured spatiotemporal point sequence characterizing the target material handling trajectory is determined, wherein the structured spatiotemporal point sequence is in the format of plain text containing an object identifier, a set of integer quantized coordinate points within a predetermined range, and a timestamp. Based on the structured spatiotemporal point sequence and combined with the text data, the probability of abnormal risk in the current handling state of the target material is determined. When the probability of the abnormal risk exceeds a preset threshold, an intervention command is sent to the material handling equipment corresponding to the target material, so that the material handling equipment responds to the intervention command and performs an intervention action, the intervention action being determined based on the intervention command.

[0006] In some embodiments, determining the structured spatiotemporal point sequence characterizing the target material handling trajectory based on the warehouse monitoring video stream and the IoT sensor data includes: Each frame of the warehouse monitoring video stream is divided into multiple target grids; wherein the density parameter of the target grid is adaptively adjusted according to the proportion of the pixels of the target logistics in the corresponding image frame; Determine the video spatial probability distribution for each target grid, wherein the video spatial probability distribution is a heatmap, and the probability value at each location represents the visual confidence that a material key point exists at that location; Based on the IoT sensor data, the material key points whose visual confidence meets the preset confidence threshold are subjected to weighted fusion processing to obtain the fused key points. The pixel coordinates of the fusion key points are quantized into integer quantized coordinates within a predetermined range based on the quantization formula, and the integer quantized coordinates are sorted in chronological order to obtain the structured spatiotemporal point sequence. The quantification formula is as follows: ; In the formula, The coordinates are quantized integers. The original pixel coordinates of the key points are used for fusion. The width of the frame image, in pixels.

[0007] In some embodiments, determining the structured spatiotemporal point sequence characterizing the target material handling trajectory based on the warehouse monitoring video stream and the IoT sensor data includes: Determine the initial model; Obtain a warehouse trajectory training set, which includes real trajectory labeled samples and trajectory augmentation samples; A batch of samples are extracted from the warehouse trajectory training set and input into the initial model for forward propagation to obtain a spatiotemporal point prediction sequence. Determine the spatiotemporal distance loss value between the predicted spatiotemporal point sequence and the actual spatiotemporal point sequence labeled in the sample; Based on the spatiotemporal distance loss value, the parameter gradient of the initial model is determined by the backpropagation algorithm, and the parameters of the initial model are updated and optimized by the gradient descent optimization algorithm to minimize the spatiotemporal distance loss. Repeat the backpropagation and update optimization steps until the preset training rounds are reached to obtain the supervised fine-tuned model.

[0008] A reinforcement learning algorithm that integrates GRPO and GSPO strategies is adopted, and the supervised fine-tuning model is optimized based on a composite reward function to obtain an optimized spatiotemporal point sequence prediction model. The composite reward function includes at least an accuracy reward term, a sequence length penalty term, and a temporal consistency reward term. The warehouse monitoring video stream is input into the spatiotemporal point sequence prediction model to obtain the structured initial spatiotemporal point sequence output by the spatiotemporal point sequence prediction model. Based on the IoT sensor data, the structured initial spatiotemporal point sequence is corrected to obtain the structured spatiotemporal point sequence.

[0009] In some embodiments, the composite reward function is: ; ; In the formula, The accuracy bonus is calculated based on the ROUGE score, visual crossover ratio, or temporal crossover ratio between the predicted and true sequences. This is a sequence length penalty term, which penalizes excessively long predicted trajectory sequences based on the cosine function; For adaptive temporal reward items, the evaluation time window σ gradually decreases as the number of training rounds increases; The time-gated intersection-union reward term only applies when the predicted timestamp t is compared to the actual timestamp. The calculation is activated when the absolute value of the difference is less than the time window σ, i.e., when the condition is met. Effective immediately; λ1, λ2, and λ3 are the weight coefficients corresponding to each reward item.

[0010] In some embodiments, determining the probability of abnormal risk in the current handling state of the target material based on the structured spatiotemporal point sequence and the text data includes: At least one kinematic feature parameter is parsed from the structured spatiotemporal point sequence. The kinematic feature parameter includes instantaneous velocity, average velocity, motion acceleration, motion direction angle, or path curvature radius. From the text data, extract at least one business rule related to the current task. The business rule includes area speed limit value, preset path coordinates, restricted area boundary or material attribute identifier. The parsed kinematic feature parameters are matched and compared with the corresponding business rules to calculate at least one compliance deviation value; Based on the material attributes and task priorities in the text data, dynamic weights are assigned to the compliance deviation values; The weighted compliance deviation values ​​are input into a preset risk assessment model to obtain the abnormal risk probability value output by the risk assessment model, wherein the risk assessment model is a Bayesian network model or a machine learning model trained based on historical abnormal data.

[0011] In some embodiments, the material key points whose visual confidence meets a preset confidence threshold are weighted and fused based on the IoT sensor data to obtain fused key points; The gate threshold is determined based on the vibration frequency in the IoT sensor data; The fusion probability distribution is determined by combining the fusion formula with the video modal probability distribution obtained from the key points of the material and the IoT modal probability distribution obtained from the IoT sensor data. The key points of fusion are determined based on the fusion probability distribution; The fusion formula is as follows: ; In the formula, For the fusion probability distribution, The video modal probability distribution obtained for the key points of the material. This refers to the IoT modal probability distribution obtained from IoT sensor data. Preset gate weights.

[0012] In some embodiments, the video modal probability distribution obtained based on the material key points includes: A video frame containing the key points of the material is input into a pre-trained target detection neural network to obtain a video spatial heatmap output by the target detection neural network. The video spatial heatmap represents the visual confidence that the target material exists at each spatial location in the video frame, and the visual confidence is defined as the video modal probability distribution. Determining the fusion probability distribution based on the IoT modal probability distribution obtained from the IoT sensor data includes: IoT sensor data is input into a pre-trained spatiotemporal mapping model to obtain an IoT spatial probability map output by the model. The IoT spatial probability map represents the spatial position probability of the target material in the video frame coordinate system inferred based on the physical state of the device, and the spatial position probability is defined as the IoT modal probability distribution.

[0013] According to another aspect of this application, a warehouse material flow anomaly handling device is also disclosed, the device comprising: The data acquisition module is used to acquire warehouse monitoring video streams, IoT sensor data, and text data of the target material from the warehouse management system. The structured spatiotemporal point sequence determination module determines a structured spatiotemporal point sequence characterizing the target material handling trajectory based on the warehouse monitoring video stream and the IoT sensor data. The structured spatiotemporal point sequence is in the format of plain text containing an object identifier, a set of integer quantized coordinate points within a predetermined range, and a timestamp. An abnormal risk probability determination module is used to determine the abnormal risk probability of the current handling state of the target material based on the structured spatiotemporal point sequence and the text data. An intervention instruction sending module is used to send an intervention instruction to the material handling equipment corresponding to the target material when the probability of the abnormal risk exceeds a preset threshold, so that the material handling equipment responds to the intervention instruction and performs an intervention action, the intervention action being determined based on the intervention instruction.

[0014] According to another aspect of this application, an electronic device is also disclosed, the electronic device including a memory and at least one processor, the memory storing instructions; the at least one processor invokes the instructions in the memory to cause the electronic device to perform various steps of the warehouse material flow anomaly handling method as described in any of the preceding claims.

[0015] According to another aspect of this application, a computer-readable storage medium is also disclosed, on which instructions are stored, which, when executed by a processor, implement the various steps of the warehouse material flow anomaly handling method as described in any of the preceding claims.

[0016] The present invention includes, but is not limited to, the following beneficial effects: (1) This scheme encodes heterogeneous data into a concise structured spatiotemporal point sequence through grid partitioning, key point sampling, and especially cross-modal fusion based on gating mechanism. This sequence is a highly compressed data format, achieving low-latency processing; (2) Through dynamic grid partitioning and video spatial heat map generation, the system can adaptively adjust the analysis granularity according to the target size and quantify the visual confidence, thereby achieving more refined and robust key point positioning in warehouse environments with varying target scales and complex lighting conditions, improving the accuracy of trajectory perception from the source; (3) This scheme does not simply rely on vision, but uses IoT sensor data as an independent information source to cross-validate and correct high-confidence visual key points, so that the system can still output data even when vision is obstructed, blurred, or interfered with. (4) This scheme integrates GRPO and GSPO strategies, balancing rapid exploration and stable optimization in the reinforcement learning process, avoiding the model from getting stuck in local optima or experiencing violent fluctuations when pursuing high rewards, ensuring the robustness of the training process and the reliability of the final model performance; (5) This scheme innovatively introduces a mechanism for allocating dynamic weights based on material attributes and task priorities, which enables the system to understand complex business logics such as the same degree of speeding, the higher risk of transporting hazardous chemicals than transporting empty pallets, or the higher acceptability of slight path deviations in emergency orders. Risk calculation is no longer a simple rule matching, but an intelligent decision with business context awareness, which greatly improves the rationality and operability of alarms and reduces invalid alarms caused by rule rigidity. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below.

[0018] Figure 1 This is a flowchart of warehouse material flow anomaly handling according to an embodiment of this application; Figure 2 This is a flowchart illustrating a structured spatiotemporal point sequence representing the trajectory of a target material transport, according to an embodiment of this application. Figure 3 This is a flowchart illustrating the key fusion points obtained after fusion in an embodiment of this application; Figure 4 This is another flowchart illustrating how to determine a structured spatiotemporal point sequence representing the trajectory of a target material transport according to an embodiment of this application; Figure 5 This is a flowchart illustrating the determination of the probability of abnormal risks in the current handling state of the target material, according to an embodiment of this application. Figure 6This is a structural block diagram of the warehouse material flow anomaly handling device according to an embodiment of this application; Figure 7 It is a diagram of an electronic device. Detailed Implementation

[0019] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” or “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0020] For ease of understanding, the specific process of the embodiments of the present invention will be described below. Figure 1 This is a flowchart of a warehouse material flow anomaly handling method according to an embodiment of this application. (See attached document.) Figure 1 It includes the following steps: S100: Acquire warehouse monitoring video streams, IoT sensor data, and text data of the target material from the warehouse management system.

[0021] Specifically, real-time video streams obtained from high-definition cameras deployed at key warehouse nodes (such as aisles, shelving areas, and entrances / exits) can be used as monitoring video streams. Further, computer vision algorithms can be used to identify the identity, location, direction of movement, posture, and relative positional relationships of material handling equipment (such as forklifts and AGVs) with personnel, shelving, and other equipment from the monitoring video streams. Furthermore, IoT sensor data can be based on readings from various sensors installed on the material handling equipment, typically including accelerometers and gyroscopes from inertial measurement units (IMUs) to detect rapid acceleration, deceleration, and sharp turns; vibration sensors to monitor equipment stability; speed or odometers to provide speed and displacement data; and weight sensors to monitor load data. Text data consists of structured task instructions and rules from the Warehouse Management System (WMS). For example, task instructions could include, "Forklift A, retrieve pallet C from location B01 and transport it to loading area D," and business rules such as "Speed ​​limit 5 km / h in aisle E," "Area F is a restricted area," and "Material G is hazardous and must be handled with care."

[0022] S102. Based on warehouse monitoring video streams and IoT sensor data, determine a structured spatiotemporal point sequence that characterizes the trajectory of the target material handling.

[0023] Specifically, in one example, Figure 2 This is a flowchart illustrating a structured spatiotemporal point sequence characterizing the trajectory of a target material transport, as described in this application. The flowchart is an exemplary illustration of step S102. (See attached document.) Figure 2 It includes the following steps: S200: Divide each frame of the warehouse monitoring video stream into multiple target grids.

[0024] Specifically, after acquiring the warehouse monitoring video stream, each frame can be dynamically divided into an N×N uniform grid, where N is a preset positive integer. The density parameter of the target grid is adaptively adjusted based on the proportion of pixels of the target material in the corresponding image frame. That is, when the pixel proportion of the target object in the image exceeds a preset proportion threshold of the total pixels, a grid encryption mechanism is triggered, increasing the value of N for finer division. For example, in the target grid division step, N can be 32 or 64, with a preset proportion threshold of 2%; when the pixel proportion of the target object exceeds 2%, N is adjusted from 32 to 64.

[0025] S202. Determine the video spatial probability distribution for each target grid. The video spatial probability distribution is a heatmap.

[0026] The probability value for each location represents the visual confidence that a material key point exists at that location.

[0027] Specifically, the system utilizes a pre-trained object detection neural network, such as an improved CenterNet, to process each frame of video. The core task of this model is to locate and identify targets of interest in the image. This model is optimized based on the standard CenterNet by introducing a channel attention module after the output of the Feature Pyramid Network (FPN) to enhance the response to specific features in the warehouse environment, such as metal shelves and forklifts. The model is initialized using the publicly available MS COCO dataset and fine-tuned using a warehouse-specific dataset containing over 5000 labeled samples collected by our system. For each target class, such as forklifts, the object detection neural network outputs a heatmap corresponding to the size of the input image. The value of each pixel on the heatmap (usually between 0 and 1) represents the probability that the location is a keypoint; the closer the value is to 1, the more confident the model is that a keypoint exists at that location. After the target grid division step, the system examines each N×N grid. The probability set formed by the heat values ​​corresponding to all pixels within the grid is the spatial probability distribution. The system samples the points with the highest probabilities (e.g., up to 5) from this distribution as representative key points of the grid. Further, video frames containing material key points are input into a pre-trained target detection neural network to obtain a video spatial heatmap output by the network. The video spatial heatmap represents the visual confidence that target material exists at each spatial location in the video frame, and the visual confidence is defined as the video modal probability distribution.

[0028] S204. Based on IoT sensor data, perform weighted fusion processing on material key points whose visual confidence meets the preset confidence threshold to obtain fused key points.

[0029] Specifically, Figure 3 This is a flowchart illustrating the fusion key points obtained in an embodiment of this application. This step is an exemplary description of step S204. See reference... Figure 3 It includes the following steps: S300: Determine the gate threshold based on the vibration frequency in the IoT sensor data.

[0030] For example, when the vibration frequency is >40Hz (abnormal or severe equipment condition): set the gate threshold to 0.7. At this time, the system trusts the video more because visual information is more critical and reliable for judging whether an abnormality such as a collision or sudden turn has occurred during severe vibration.

[0031] When the vibration frequency is ≤40Hz (the device is in a stable state): set the gate threshold to 0.3. This means that the system trusts sensor inference more, because in a stable state, sensor data is stable, and the position inferred by the sensor may better supplement and correct small errors or missed detections in video detection.

[0032] S302. Combining the fusion formula, based on the video modal probability distribution obtained from the key points of the material and the IoT modal probability distribution obtained from the IoT sensor data, determine the fusion probability distribution; The fusion formula is as follows: ; In the formula, For the fusion probability distribution, The video modal probability distribution obtained from key points of the material. This refers to the IoT modal probability distribution obtained from IoT sensor data. Preset gate weights.

[0033] Specifically, the IoT modal probability distribution obtained based on IoT sensor data involves inputting the IoT sensor data into a pre-trained spatiotemporal mapping model to obtain an IoT spatial probability map output by the model. This IoT spatial probability map represents the spatial position probability of a target material in the video frame coordinate system, inferred from the physical state of the device, and the spatial position probability is defined as the IoT modal probability distribution. For example, IoT sensor data (such as vibration signals within a window period) is fed into the spatiotemporal mapping model and transformed into a feature vector through the model's feature extraction network (such as 1D CNN or LSTM). This vector encodes the device's state and behavioral pattern at that moment; for example, severe high-frequency vibration might correspond to high-speed turning or bumping. During the training phase, the system learns a mapping relationship by pairing sensor data with a large number of synchronized video frames. This mapping relationship associates the sensor feature vector with a spatial region distribution in the image. For example, a spatiotemporal mapping model may learn that severe vibrations of a certain pattern often occur simultaneously with a forklift appearing in the lower left region of the image and tilting its posture. Then, during the inference phase, for the real-time sensor data, the system uses the learned mapping relationship to generate a probability distribution map overlaid on the video frame. This map indicates the probability of the target appearing in which regions of the image based on the current device state. This spatial position probability of the target material in the video frame coordinate system inferred from the physical state of the device is defined as the IoT modal probability distribution.

[0034] The spatiotemporal mapping model is a neural network whose input layer receives multi-dimensional IoT sensor data (dimension [T, D], where D is the number of sensor channels) within a time window T, and whose output layer generates a two-dimensional probability map proportional to the video frame resolution (e.g., 1 / 4). The model's structure includes: a three-layer one-dimensional CNN encoder for extracting spatiotemporal features of sensor signals; a feature flattening layer; and a decoder consisting of two fully connected layers and a reshaping layer. Training is conducted under supervised learning using a large amount of synchronous (sensor sequences, target real-world locations) data. The real-world location is converted into a two-dimensional Gaussian heatmap centered at that location as the training label, and the loss function is the mean squared error (MSE).

[0035] S304. Determine the key points of fusion based on the fusion probability distribution.

[0036] Specifically, the fusion probability distribution determined in step S302 is a two-dimensional array (matrix), where the value of each element represents the comprehensive confidence that the corresponding image location is a target keypoint. In one example, the Argmax peak search algorithm can be used to search this fusion probability distribution map to find one or more peak points with the highest probability values. The coordinates (x_max, y_max) of the peak point are considered to be the most likely location of the target keypoint in the current frame. If multiple keypoints are allowed in a frame, such as the two front corners of a forklift, the system will set a distance threshold. After finding the global maximum value, other peaks in a certain area around it are suppressed, and then the next global maximum value is searched, and so on, to ensure that the extracted keypoints have spatial distinguishability. The coordinates (x_max, y_max) of the aforementioned peak point are used as the final output of this step. These coordinates are the fusion keypoint that integrates visual information and IoT inference information.

[0037] S206. Based on the quantization formula, the pixel coordinates of the fused key points are quantized into integer quantized coordinates within a predetermined range, and the integer quantized coordinates are sorted in chronological order to obtain a structured spatiotemporal point sequence.

[0038] The quantification formula is as follows: ; In the formula, The coordinates are quantized integers. The original pixel coordinates of the key points are used for fusion. The width of the frame image, in pixels.

[0039] Specifically, W is the width (in pixels) of the frame image. It is first normalized from the original pixel range of [0, W-1], then linearly scaled to a relative scale range of [0.0, 1.0]. This eliminates the absolute coordinate differences caused by different camera resolutions. For example, for a 1920x1080 video, W=1920. After normalization, it is approximately 0.3387. Further scaling is performed, enlarging the normalized scale of 0.3387 to the integer range of [0, 999]. At this point, 0.3387 × 999 ≈ 338.3. Further, integer quantization is performed, such as rounding 338.3 to 338. Thus, the original floating-point pixel coordinate 650.3 is converted to the integer 338 between 0 and 999. Further, the video stream is processed continuously. Each frame (or every few frames) generates a set (or more) of quantized coordinates for the target. Each coordinate point is bound to a unique target identifier (ID), and each coordinate point is stamped with a high-precision timestamp of its generation time. All coordinate points of the same target ID are arranged into a list according to the order of their timestamps, ultimately generating a structured spatiotemporal point sequence, for example: forklift_001[ <338> <194> <340> <192> <345> <189> ]@[10:30:01.500,10:30:01.533,10:30:01.566], where forklift_001 is the target object, [ <338> <194> [] represents the trajectory path in the quantization space of 0-999, and @[10:30:01.500,...] represents the time corresponding to each location point.

[0040] S104. Based on structured spatiotemporal point sequences and combined with text data, determine the probability of abnormal risks in the current handling status of the target material.

[0041] Specifically, Figure 4 This is another flowchart illustrating the determination of a structured spatiotemporal point sequence characterizing the trajectory of a target material in an embodiment of this application. This flowchart is another exemplary description of step S102. (See attached document.) Figure 4 It includes the following steps: S400, Determine the initial model.

[0042] Specifically, a suitable neural network architecture can be chosen as the starting point for learning, such as an encoder-decoder architecture. The encoder (e.g., ResNet, ViT) is responsible for understanding the visual content of the input video frames and extracting high-level features. The decoder (e.g., Transformer Decoder, LSTM) is responsible for autoregressively generating sequential outputs based on the encoded features.

[0043] S402. Obtain the warehouse trajectory training set.

[0044] The warehouse trajectory training set includes real trajectory labeled samples and trajectory augmentation samples.

[0045] Specifically, in real warehouse monitoring videos, the movement trajectories of equipment such as forklifts are annotated frame-by-frame or keyframe by manual or semi-automatic tools. Each sample contains a video segment and corresponding ground truth labels [coordinate sequence, timestamp sequence]. Furthermore, to increase data diversity and quantity, existing real trajectories are augmented by transformations such as rotation, scaling, and adding noise to obtain augmented trajectory samples.

[0046] S404. Extract a batch of samples from the warehouse trajectory training set, input them into the initial model for forward propagation, and obtain the spatiotemporal point prediction sequence.

[0047] S406. Determine the spatiotemporal distance loss value between the predicted spatiotemporal point sequence and the actual spatiotemporal point sequence labeled in the sample.

[0048] Specifically, a loss function needs to be defined to quantify the difference between the predicted sequence and the true sequence. Spatiotemporal distance loss may include spatial location loss and temporal alignment loss. Spatial location loss is based on calculating the square of the Euclidean distance or Manhattan distance between each predicted point and the corresponding true point. Temporal alignment loss can use dynamic time warping (DTW) loss to align the two sequences before calculating the distance, or use loss based on timestamp differences.

[0049] S408. Based on the spatiotemporal distance loss value, the parameter gradient of the initial model is determined by the backpropagation algorithm, and the parameters of the initial model are updated and optimized by the gradient descent optimization algorithm to minimize the spatiotemporal distance loss.

[0050] For example, the chain rule can be used to calculate backwards from the model output layer to the input layer to obtain the contribution (gradient) of each trainable parameter (weight) in the model to the total loss. Then, an optimizer (such as Adam) can be used to make small updates to all the model parameters along the gradient in the opposite direction (i.e. the direction of reducing loss) to minimize the spatiotemporal distance loss.

[0051] S410, repeat the backpropagation steps and update optimization steps until the preset training rounds are reached, and obtain the supervised fine-tuned model after supervised fine-tuning.

[0052] Specifically, repeat steps S404-S408 until the preset number of training rounds is reached to obtain the supervised fine-tuned model.

[0053] S412. A reinforcement learning algorithm that integrates GRPO and GSPO strategies is adopted, and the supervised fine-tuning model is optimized based on a composite reward function to obtain an optimized spatiotemporal point sequence prediction model.

[0054] The composite reward function includes at least an accuracy reward term, a sequence length penalty term, and a temporal consistency reward term.

[0055] Specifically, the composite reward function is as follows: ; ; In the formula, The accuracy bonus is calculated based on the ROUGE score, visual crossover ratio, or temporal crossover ratio between the predicted and true sequences. This is a sequence length penalty term, which penalizes excessively long predicted trajectory sequences based on the cosine function; For adaptive temporal reward items, the evaluation time window σ gradually decreases as the number of training rounds increases; The time-gated intersection-union reward term only applies when the predicted timestamp t is compared to the actual timestamp. The calculation is activated when the absolute value of the difference is less than the time window σ, i.e., when the condition is met. Effective immediately; λ1, λ2, and λ3 are the weight coefficients corresponding to each reward item.

[0056] S414. Input the warehouse monitoring video stream into the spatiotemporal point sequence prediction model to obtain the structured initial spatiotemporal point sequence output by the spatiotemporal point sequence prediction model.

[0057] Specifically, the real-time warehouse monitoring video stream is input into the optimized spatiotemporal point sequence prediction model obtained by S412. The model performs one forward propagation and directly outputs the structured initial spatiotemporal point sequence.

[0058] S416. Based on IoT sensor data, the structured initial spatiotemporal point sequence is corrected to obtain a structured spatiotemporal point sequence.

[0059] Specifically, to ensure strict temporal synchronization between IoT data (such as IMU and velocity data) and the video stream, the initial structured spatiotemporal point sequence needs to be corrected. For example, if the IoT data shows smooth device movement (low vibration, uniform speed), but the visual sequence exhibits severe jitter, the visual sequence is subjected to Kalman filtering or smoothing. If the IoT data detects severe vibration or abrupt stops, while the visual sequence shows uniform motion, the key points of the visual sequence may be overlaid or adjusted using displacements estimated by the IoT. Confidence levels are assigned to both the visual sequence and the IoT-based estimated sequence (e.g., low confidence for high vibration), and a weighted fusion is performed to obtain the final corrected structured spatiotemporal point sequence.

[0060] S106. When the probability of abnormal risk exceeds the preset threshold, an intervention command is sent to the material handling equipment corresponding to the target material so that the material handling equipment responds to the intervention command and performs an intervention action.

[0061] Specifically, Figure 5 This is a flowchart illustrating the determination of the probability of abnormal risks in the current handling state of the target material, as described in this application embodiment. Figure 5 It includes the following steps: S500. Extract at least one kinematic feature parameter from the structured spatiotemporal point sequence.

[0062] Kinematic characteristic parameters include instantaneous velocity, average velocity, acceleration, direction angle, or radius of curvature of the path. For example, instantaneous velocity can be calculated using the quantized coordinate difference and timestamp difference between two adjacent points. For instance, from point one to point two: coordinates (5, -5), Δ time 0.033 seconds, then the instantaneous velocity... The average velocity can be determined by calculating the ratio of the total displacement to the total time of the entire sequence. The acceleration can be calculated from the rate of change of velocity of three consecutive points. The direction angle can be determined by calculating the direction of the line connecting two consecutive points, such as the angle with due north. The radius of curvature of the path can be calculated by analyzing the arc formed by three consecutive points. The smaller the radius of curvature, the sharper the turn.

[0063] S502. Extract at least one business rule related to the current task from the text data.

[0064] Specifically, the business rules include area speed limits, preset path coordinates, restricted area boundaries, or material attribute identifiers. The area speed limit can be extracted based on the current area; for example, in the packaging area, the speed limit is 5 km / h. Planned path points such as [P1, P2, P3] are extracted as preset path coordinates. Restricted areas, such as the polygonal coordinate range of shelf area A, are extracted as restricted area boundaries. Material attribute identifiers, such as fragile glass or priority, can be extracted.

[0065] S504. Match and compare the parsed kinematic feature parameters with the corresponding business rules, and calculate at least one compliance deviation value.

[0066] The deviation values ​​can include: overspeed deviation: (current instantaneous speed - area speed limit) / area speed limit, with a positive result indicating the percentage of overspeed; path deviation: calculates the shortest Euclidean distance between the current point and the preset path [P1, P2, P3], or calculates the average distance between the entire trajectory and the preset path; restricted area intrusion deviation: determines whether the current point coordinates are within the restricted area polygon, if so, it is 1 (serious violation), otherwise it is 0; sharp turn / sharp acceleration / deceleration deviation: compares the calculated acceleration or radius of curvature with the safety threshold.

[0067] S506. Based on the material attributes and task priorities in the text data, assign dynamic weights to compliance deviation values.

[0068] Specifically, dynamic weights include material attribute weights: for the same speeding deviation, the weight might be 1.0 for transporting ordinary cardboard boxes, but it might increase to 2.0 or 3.0 for transporting hazardous chemicals or fragile glass. It can also include task priority weights: a slight deviation in the path of an urgent, high-priority task might have a lower weight than the same deviation in a regular, low-priority task, because the former has a higher timeliness requirement.

[0069] S508. Input the weighted compliance deviation values ​​into the preset risk assessment model to obtain the abnormal risk probability value output by the risk assessment model.

[0070] The risk assessment model is either a Bayesian network model or a machine learning model trained on historical anomaly data.

[0071] Specifically, the pre-defined risk assessment model is a trained risk assessment function. It can be a Bayesian network model: a probabilistic graphical model. Each deviation value is input as an evidence node, and the posterior probability of an anomaly is calculated through predefined conditional probability relationships within the network (which can be determined based on historical data). Alternatively, it can be a machine learning model such as a gradient boosting tree or a neural network: trained using a large amount of historical data (containing various deviation values ​​and labels indicating whether an anomaly occurred). The model learns the complex non-linear relationship between deviation values ​​and anomaly outcomes. Inputting a weighted deviation of new data, the model directly outputs a risk probability ranging from 0% to 100%.

[0072] Furthermore, Figure 6 This is a structural block diagram of the warehouse material flow anomaly handling device according to an embodiment of the application, such as... Figure 6 As shown, the device includes: The data acquisition module is used to acquire warehouse monitoring video streams, IoT sensor data, and text data of the target materials from the warehouse management system. The structured spatiotemporal point sequence determination module, based on warehouse monitoring video streams and IoT sensor data, determines a structured spatiotemporal point sequence that characterizes the trajectory of target material handling. The structured spatiotemporal point sequence is in the format of plain text containing an object identifier, a set of integer quantized coordinate points within a predetermined range, and a timestamp. The abnormal risk probability determination module is used to determine the abnormal risk probability of the current handling status of the target material based on structured spatiotemporal point sequences and combined with text data. The intervention command sending module is used to send an intervention command to the material handling equipment corresponding to the target material when the probability of abnormal risk exceeds a preset threshold, so that the material handling equipment can respond to the intervention command and perform an intervention action. The intervention action is determined based on the intervention command.

[0073] The application of the relevant modules of the device in this example can be referred to the relevant introduction of the method principle above, and will not be repeated here.

[0074] above Figure 6 The warehouse material flow anomaly handling device in this embodiment of the invention will be described in detail from the perspective of modular functional entities. The electronic equipment in this embodiment of the invention will be described in detail from the perspective of hardware processing.

[0075] Figure 7 This is a schematic diagram of the structure of an electronic device 700 provided in an embodiment of the present invention. The electronic device 700 can vary significantly due to differences in configuration or performance, and may include one or more central processing units (CPUs) 710 (e.g., one or more processors) and a memory 720, and one or more storage media 730 (e.g., one or more mass storage devices) for storing application programs 733 or data 732. The memory 720 and storage media 730 can be temporary or persistent storage. The program stored in the storage media 730 may include one or more modules (not shown in the diagram), each module including a series of instruction operations on the electronic device 700. Furthermore, the processor 710 may be configured to communicate with the storage media 730 and execute the series of instruction operations in the storage media 730 on the electronic device 700.

[0076] Electronic device 700 may also include one or more power supplies 740, one or more wired or wireless network interfaces 750, one or more input / output interfaces 760, and / or one or more operating systems 731, such as Windows Server, MacOSX, Unix, Linux, FreeBSD, etc. Those skilled in the art will understand that... Figure 7 The illustrated electronic device structure does not constitute a limitation on electronic devices and may include more or fewer components than illustrated, or combine certain components, or have different component arrangements.

[0077] The present invention also provides a computer-readable storage medium, which can be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium, wherein the computer-readable storage medium stores instructions that, when executed on a computer, cause the computer to perform the steps of any of the above-described warehouse material flow anomaly handling methods.

[0078] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the system, device, or unit described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0079] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0080] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for handling abnormal material flow in a warehouse, characterized in that, The method includes: Acquire warehouse monitoring video streams, IoT sensor data, and text data of the target material from the warehouse management system; Based on the warehouse monitoring video stream and the IoT sensor data, a structured spatiotemporal point sequence characterizing the target material handling trajectory is determined, wherein the structured spatiotemporal point sequence is in the format of plain text containing an object identifier, a set of integer quantized coordinate points within a predetermined range, and a timestamp. Based on the structured spatiotemporal point sequence and combined with the text data, the probability of abnormal risk in the current handling state of the target material is determined. When the probability of the abnormal risk exceeds a preset threshold, an intervention command is sent to the material handling equipment corresponding to the target material, so that the material handling equipment responds to the intervention command and performs an intervention action, the intervention action being determined based on the intervention command.

2. The warehouse material flow anomaly handling method according to claim 1, characterized in that, The determination of the structured spatiotemporal point sequence characterizing the target material handling trajectory based on the warehouse monitoring video stream and the IoT sensor data includes: Each frame of the warehouse monitoring video stream is divided into multiple target grids; wherein the density parameter of the target grid is adaptively adjusted according to the proportion of the pixels of the target logistics in the corresponding image frame; Determine the video spatial probability distribution for each target grid, wherein the video spatial probability distribution is a heatmap, and the probability value at each location represents the visual confidence that a material key point exists at that location; Based on the IoT sensor data, the material key points whose visual confidence meets the preset confidence threshold are subjected to weighted fusion processing to obtain the fused key points. The pixel coordinates of the fusion key points are quantized into integer quantized coordinates within a predetermined range based on the quantization formula, and the integer quantized coordinates are sorted in chronological order to obtain the structured spatiotemporal point sequence. The quantification formula is as follows: ； In the formula, The coordinates are quantized integers. The original pixel coordinates of the key points are used for fusion. The width of the frame image, in pixels.

3. The warehouse material flow anomaly handling method according to claim 1, characterized in that, The determination of the structured spatiotemporal point sequence characterizing the target material handling trajectory based on the warehouse monitoring video stream and the IoT sensor data includes: Determine the initial model; Obtain a warehouse trajectory training set, which includes real trajectory labeled samples and trajectory augmentation samples; A batch of samples are extracted from the warehouse trajectory training set and input into the initial model for forward propagation to obtain a spatiotemporal point prediction sequence. Determine the spatiotemporal distance loss value between the predicted spatiotemporal point sequence and the actual spatiotemporal point sequence labeled in the sample; Based on the spatiotemporal distance loss value, the parameter gradient of the initial model is determined by the backpropagation algorithm, and the parameters of the initial model are updated and optimized by the gradient descent optimization algorithm to minimize the spatiotemporal distance loss. Repeat the backpropagation and update optimization steps until the preset number of training rounds is reached to obtain the supervised fine-tuned model. A reinforcement learning algorithm that integrates GRPO and GSPO strategies is adopted, and the supervised fine-tuning model is optimized based on a composite reward function to obtain an optimized spatiotemporal point sequence prediction model. The composite reward function includes at least an accuracy reward term, a sequence length penalty term, and a temporal consistency reward term. The warehouse monitoring video stream is input into the spatiotemporal point sequence prediction model to obtain the structured initial spatiotemporal point sequence output by the spatiotemporal point sequence prediction model. Based on the IoT sensor data, the structured initial spatiotemporal point sequence is corrected to obtain the structured spatiotemporal point sequence.

4. The warehouse material flow anomaly handling method according to claim 3, characterized in that, The composite reward function is: ； ； In the formula, The accuracy bonus is calculated based on the ROUGE score, visual crossover ratio, or temporal crossover ratio between the predicted and true sequences. This is a sequence length penalty term, which penalizes excessively long predicted trajectory sequences based on the cosine function; For adaptive temporal reward items, the evaluation time window σ gradually decreases as the number of training rounds increases; The time-gated intersection-union reward term only applies when the predicted timestamp t is compared to the actual timestamp. The calculation is activated when the absolute value of the difference is less than the time window σ, i.e., when the condition is met. Effective immediately; λ1, λ2, and λ3 are the weight coefficients corresponding to each reward item.

5. The warehouse material flow anomaly handling method according to claim 1, characterized in that, The determination of the probability of abnormal risk in the current handling state of the target material based on the structured spatiotemporal point sequence and the text data includes: At least one kinematic feature parameter is parsed from the structured spatiotemporal point sequence. The kinematic feature parameter includes instantaneous velocity, average velocity, motion acceleration, motion direction angle, or path curvature radius. From the text data, extract at least one business rule related to the current task. The business rule includes area speed limit value, preset path coordinates, restricted area boundary or material attribute identifier. The parsed kinematic feature parameters are matched and compared with the corresponding business rules to calculate at least one compliance deviation value; Based on the material attributes and task priorities in the text data, dynamic weights are assigned to the compliance deviation values; The weighted compliance deviation values ​​are input into a preset risk assessment model to obtain the abnormal risk probability value output by the risk assessment model, wherein the risk assessment model is a Bayesian network model or a machine learning model trained based on historical abnormal data.

6. The warehouse material flow anomaly handling method according to claim 2, characterized in that, Based on the IoT sensor data, the material key points with visual confidence levels that meet the preset confidence threshold are subjected to weighted fusion processing to obtain the fused key points; The gate threshold is determined based on the vibration frequency in the IoT sensor data; The fusion probability distribution is determined by combining the fusion formula with the video modal probability distribution obtained from the key points of the material and the IoT modal probability distribution obtained from the IoT sensor data. The key points of fusion are determined based on the fusion probability distribution; The fusion formula is as follows: ； In the formula, For the fusion probability distribution, The video modal probability distribution obtained for the key points of the material. This refers to the IoT modal probability distribution obtained from IoT sensor data. Preset gate weights.

7. The warehouse material flow anomaly handling method according to claim 4, characterized in that, The video modal probability distribution obtained based on the material key points includes: A video frame containing the key points of the material is input into a pre-trained target detection neural network to obtain a video spatial heatmap output by the target detection neural network. The video spatial heatmap represents the visual confidence that the target material exists at each spatial location in the video frame, and the visual confidence is defined as the video modal probability distribution. The IoT modal probability distribution obtained based on the IoT sensor data includes: IoT sensor data is input into a pre-trained spatiotemporal mapping model to obtain an IoT spatial probability map output by the model. The IoT spatial probability map represents the spatial position probability of the target material in the video frame coordinate system inferred based on the physical state of the device, and the spatial position probability is defined as the IoT modal probability distribution.

8. A warehouse material flow anomaly handling device, characterized in that, The device includes: The data acquisition module is used to acquire warehouse monitoring video streams, IoT sensor data, and text data of the target material from the warehouse management system. The structured spatiotemporal point sequence determination module determines a structured spatiotemporal point sequence characterizing the target material handling trajectory based on the warehouse monitoring video stream and the IoT sensor data. The structured spatiotemporal point sequence is in the format of plain text containing an object identifier, a set of integer quantized coordinate points within a predetermined range, and a timestamp. An abnormal risk probability determination module is used to determine the abnormal risk probability of the current handling state of the target material based on the structured spatiotemporal point sequence and the text data. An intervention instruction sending module is used to send an intervention instruction to the material handling equipment corresponding to the target material when the probability of the abnormal risk exceeds a preset threshold, so that the material handling equipment responds to the intervention instruction and performs an intervention action, the intervention action being determined based on the intervention instruction.

9. An electronic device, characterized in that, The electronic device includes a memory and at least one processor, the memory storing instructions; the at least one processor invokes the instructions in the memory to cause the electronic device to perform the steps of the warehouse material flow anomaly handling method as described in any one of claims 1-7.

10. A computer-readable storage medium storing instructions thereon, characterized in that, When the instructions are executed by the processor, they implement the various steps of the warehouse material flow anomaly handling method as described in any one of claims 1-7.

Images