Video AI intelligent point package method

Abstract

The application relates to the field of industrial detection, and discloses a video AI intelligent point package method, which comprises the following steps: collecting a cement conveyor belt video stream through an RTSP protocol and preprocessing; an end-to-end joint optimization network is constructed, which comprises a target detection module, a multi-target tracking module and a counting module; a bidirectional counting-detection coupling loss function is constructed in a training stage to jointly update parameters of the three modules; a scene self-adaptive dynamic hypergraph correlation method is used to perform track-detection correlation in a counting process; the structural parameters of the dynamic hypergraph are dynamically determined according to the scene density of a current frame; and a hyperedge is used to connect more than three nodes and evaluate the group motion rationality among multiple cement packages. The application solves the technical problems that in the existing AI vision scheme, detection adaptability is insufficient, ID frequently switches and tracking is lost in a dense stacking scene, and counting errors cannot be jointly optimized through backward propagation. The counting accuracy rate of the conventional scene is higher than 99.9%, and the counting accuracy rate of the stacking shielding scene is higher than 99.2%.

Description

Technical Field

[0001] This invention relates to the field of industrial inspection, and more particularly to a video AI-powered intelligent package detection method. Background Technology

[0002] In process industries such as cement, chemicals, and grain, automatic counting of bagged products is one of the core processes in production and shipping. Traditionally, this process mainly relies on manual counting or automatic counting devices based on photoelectric sensors. Manual counting requires workers to keep their eyes on the conveyor belt for long periods of time, which is labor-intensive and prone to omissions and errors due to fatigue. Photoelectric sensor counting uses infrared transmitters and receivers installed on both sides of the conveyor belt to accumulate electrical pulse signals generated by the blockage of the light beam by the bagged products passing by. In recent years, with the development of deep learning technology, computer vision-based target detection algorithms (such as the YOLO series) and multi-target tracking algorithms (such as SORT, DeepSORT, ByteTrack, BoTSORT, etc.) have been gradually introduced into the field of industrial vision, providing a new technical path for achieving automated visual counting. Based on this, some solutions connect detection and tracking to form a "detection-tracking-counting" pipeline. The detector identifies the target, the tracker maintains the consistency of the target's identity, and the counting module triggers counting based on the trajectory crossing the virtual line, aiming to replace the traditional photoelectric counting method.

[0003] However, the aforementioned existing technical solutions all have significant limitations in actual industrial scenarios. The efficiency and accuracy of manual counting methods are no longer sufficient to meet the needs of modern large-scale production. In scenarios such as continuous packaging, sticking packaging, and stacking, photoelectric switch counting may result in multiple closely connected products being counted as one, leading to serious undercounting. When cement dust adheres to the conveyor belt, it may also mistakenly block the infrared beam, causing overcounting. The error rate in normal scenarios is about 5% to 10%, and it can be as high as 30% to 50% in stacking scenarios. While existing AI vision solutions have alleviated the aforementioned problems to some extent, they still suffer from three major shortcomings: First, general object detection models experience a significant drop in accuracy under complex scenarios such as uneven lighting, dust occlusion, and product stacking, leading to missed detections and false detections. Second, existing tracking algorithms such as DeepSORT and BoTSORT struggle to model the collective motion patterns of multiple targets in scenarios with densely stacked and high-speed movement, as they only consider pairwise relationships between trajectories and detection boxes, making it difficult to model these patterns. This can easily result in ID switching and tracking loss. Third, existing solutions generally employ a serial pipeline architecture of "detection-tracking-counting," where the detection, tracking, and counting modules operate independently and are optimized separately. Counting errors cannot be backpropagated to guide parameter updates in preceding modules, resulting in a structural bottleneck in improving the overall system accuracy. Summary of the Invention

[0004] The purpose of this invention is to provide a video AI intelligent packet counting method, which solves the technical problems in existing AI vision solutions such as insufficient scene adaptability of the detection module, frequent ID switching and tracking loss caused by the lack of group motion modeling ability in densely stacked scenes, and the inability of counting errors in the "detection-tracking-counting" serial architecture to be backpropagated to the preceding modules for joint optimization.

[0005] This invention provides a video AI intelligent packet-spotting method, which includes the following steps: S1. Acquire real-time video streams in the industrial cement bag delivery scenario using the RTSP protocol, and preprocess the video frames to obtain standardized image data. S2. Construct an end-to-end joint optimization network, which includes a target detection module, a multi-target tracking module, and a counting module. Input the standardized image data into the end-to-end joint optimization network. The target detection module outputs the bounding box, confidence score, and category information of the cement bags. The multi-target tracking module assigns a unique tracking ID to each cement bag based on the detection results and continuously tracks its motion trajectory. The counting module executes counting logic based on the tracking ID crossing a preset virtual counting line and outputs the counting result of the cement bags. S3. During the training phase of the end-to-end joint optimization network, a bidirectional counting-detection coupling loss function is constructed to perform end-to-end joint parameter updates for the target detection module, multi-target tracking module, and counting module. S4. During the execution of the counting logic, the multi-target tracking module adopts the scene adaptive dynamic hypergraph association method to perform cement bag trajectory-detection association. The structural parameters of the dynamic hypergraph are dynamically determined according to the cement bag scene density of the current frame. The hyperedges in the dynamic hypergraph are used to connect more than three nodes. The hyperedge cost term corresponding to each hyperedge is used to evaluate the rationality of the group movement among multiple cement bags in a set of association combinations.

[0006] A video AI-powered smart package-taking device includes: a processor and a storage medium; the processor loads and executes instructions and data in the storage medium to implement a video AI-powered smart package-taking method.

[0007] The beneficial effects provided by this invention are as follows: By constructing an end-to-end joint optimization network comprising a target detection module, a multi-target tracking module, and a counting module, a bidirectional counting-detection coupling loss function is introduced during the training phase. This allows the counting error to synchronously optimize the feature representation of the detection and tracking modules through backpropagation, overcoming the structural bottleneck of traditional serial architectures where counting errors cannot be fed back to preceding modules. During counting, the multi-target tracking module employs a scene-adaptive dynamic hypergraph association method. It dynamically determines the hypergraph structure parameters based on the cement bag scene density of the current frame, connecting three or more nodes via hyperedges to model the group behavior patterns of multiple cement bags. The hyperedge cost term is used to evaluate the rationality of the group movement among multiple cement bags in a set of association combinations, effectively solving the problem of frequent ID switching and tracking loss caused by mutual occlusion between targets in densely stacked scenes. Actual production line testing has verified that this invention achieves a counting accuracy of over 99.9% in conventional scenarios and over 99.2% in stacked occlusion scenarios. It can completely replace traditional photoelectric counting equipment, effectively reducing missed and duplicate counting problems, while also possessing significant advantages such as data traceability and low maintenance costs. Attached Figure Description

[0008] Figure 1 This is a simplified flowchart of the method of the present invention; Figure 2 This is a schematic diagram of the hardware device operation according to an embodiment of the present invention. Detailed Implementation

[0009] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0010] Before formally describing the present invention, a general description of the solution of the present invention will be given first to facilitate understanding.

[0011] Example 1: Please refer to Figure 1 The present invention provides a video AI intelligent packet-spotting method, comprising: S1. Acquire real-time video streams in the industrial cement bag delivery scenario using the RTSP protocol, and preprocess the video frames to obtain standardized image data. As one embodiment, step S1 of the present invention is implemented as follows: The video stream acquisition unit obtains real-time video streams from industrial cameras via the RTSP protocol. These industrial cameras are high-definition network cameras with resolutions of 2 megapixels or higher, supporting H.264 / H.265 encoding. They feature wide dynamic range, low-light performance, and dust and water resistance, enabling them to withstand the harsh environmental conditions of cement workshops, characterized by high dust levels, strong vibrations, and large temperature fluctuations. One camera is installed in each packaging lane, positioned 2 to 3 meters above the conveyor belt with the lens pointing vertically downwards to ensure complete coverage of the entire conveyor belt width. The acquired video streams are decoded using FFmpeg to obtain YUV420P video frames, which are then converted to RGB format for subsequent processing. The video resolution is 1920×1080, and the frame rate is 25 FPS.

[0012] Before further processing, the captured video frames undergo preprocessing, including: (1) Median filtering for noise reduction: A 3×3 median filter kernel is used to filter the image to remove salt-and-pepper noise and impulse noise. Median filtering can effectively protect the edge information of the image while removing noise, making it very suitable for processing images from industrial sites.

[0013] (2) Adaptive Histogram Equalization: The Limiting Contrast Adaptive Histogram Equalization (CLAHE) algorithm is used to enhance image contrast and improve the problem of uneven illumination. CLAHE divides the image into multiple 8×8 blocks, performs histogram equalization on each block separately, and then uses bilinear interpolation to stitch the blocks together, avoiding the problem of local over-enhancement caused by global histogram equalization.

[0014] (3) Perspective Transformation Geometric Correction: Since the camera installation angle cannot be completely vertical downward, the image will have a certain perspective distortion. During system initialization, a standard rectangular calibration plate is placed on the conveyor belt to calculate the perspective transformation matrix. Then, perspective transformation is performed on each frame of the image to correct the perspective distortion and ensure that the shape of the cement bag in the image is accurate.

[0015] (4) Image scaling: The original 1920×1080 image is scaled to 640×640, which is the standard input size of YOLO11, and can maximize the processing speed while ensuring detection accuracy.

[0016] The preprocessed video frames are sent to the YOLO11-SCE detection unit for target detection.

[0017] As a concrete example, in the packaging workshop of a cement plant, a camera is installed 2.5 meters directly above the conveyor belt, with the lens pointing vertically downwards. After the system starts up, the camera captures a 1920×1080 resolution video stream at a frame rate of 25 FPS. After decoding with FFmpeg, the video frames are obtained in YUV420P format. After being converted to RGB format, they undergo 3×3 median filtering for noise reduction, CLAHE histogram equalization, perspective transformation and geometric correction, and finally scaled down to 640×640 standardized image data, which is then input into the subsequent YOLO11-SCE detection unit for processing.

[0018] S2. Construct an end-to-end joint optimization network, which includes a target detection module, a multi-target tracking module, and a counting module. Input the standardized image data into the end-to-end joint optimization network. The target detection module outputs the bounding box, confidence score, and category information of the cement bags. The multi-target tracking module assigns a unique tracking ID to each cement bag based on the detection results and continuously tracks its motion trajectory. The counting module executes counting logic based on the tracking ID crossing a preset virtual counting line and outputs the counting result of the cement bags. As one embodiment, step S2 of the present invention is implemented as follows: The end-to-end joint optimization network constructed in this invention is a large-scale differentiable neural network architecture, comprising three sub-modules: an object detection module, a multi-object tracking module, and a counting module. These three sub-modules are connected in series to form a complete processing pipeline, and end-to-end joint parameter optimization is performed through a joint loss function throughout the entire network training process.

[0019] (a) Target Detection Module The target detection module employs the improved YOLO11-SCE (Scene Condition Expert) detection unit of this invention. YOLO11 is the latest generation of YOLO series target detection algorithms released in 2025, featuring advantages such as high speed, high accuracy, and small parameter count. The standard YOLO11 consists of three parts: a backbone network, a Neck feature fusion network, and a Head detection head. The Neck part adopts an FPN+PAN structure, outputting feature maps at three scales: P3 (high resolution), P4 (medium resolution), and P5 (low resolution).

[0020] This invention innovatively improves upon the standard YOLO11 by replacing the C3k2 convolutional block in the P4 medium-resolution branch with the Scene Condition Expert (SCE) module. The reason for choosing the P4 branch for replacement is that cement bags are medium-sized targets in video images, and the feature extraction capability of the P4 branch (downsampled by 16 times) is crucial to detection accuracy.

[0021] The SCE module includes four types of expert networks specifically designed for industrial scenarios and one dual-scale gated routing network. The four types of expert networks are as follows: Standard Single Bag Specialist (E1): Responsible for handling single cement bags that are well-lit, unobstructed, and free from adhesion; Stacking and Adhesion Expert (E2): Specializes in handling scenarios where 2 to 10 cement bags are stacked and stuck together; Dust containment expert (E3): Handling scenarios with high levels of dust in the workshop and where cement bags are partially covered; Uneven Illumination Expert (E4): Handles scenes with abnormal lighting conditions such as backlighting, reflections, and partial exposure.

[0022] Each expert network is an independent lightweight convolutional subnetwork, consisting of 3×3 spatial convolutions, BatchNorm batch normalization, SiLU activation function, and 1×1 pointwise projection. The four expert classes share the same network structure but differ in their training data: the normal single-packet expert is trained using 50,000 normal single-packet images, the stacked and adhered expert using 20,000 stacked and adhered images, the dust occlusion expert using 20,000 dust occlusion images, and the uneven lighting expert using 10,000 uneven lighting images.

[0023] The dual-scale gated routing network consists of a global routing branch and a local routing branch. The global routing branch extracts global scene features from the entire frame to determine the overall environment type of the current frame; the local routing branch extracts local features for each candidate target to determine the specific state of a single cement bag. The outputs of the two branches are weighted and fused (the global-local fusion weight β is set to 0.3, i.e., it focuses more on the local features of a single target) to obtain the final expert selection criteria.

[0024] During actual inference, the SCE module receives the P4 mid-resolution feature map output by the YOLO11 backbone network. (Where B is the batch size, C is the number of channels, H is the height, and W is the width) and the initial set of candidate boxes output by the detection head. (in Let J be the bounding box coordinates of the j-th candidate box. (N is the number of candidate boxes, corresponding to the confidence level). Global routing branch pairs. After global average pooling, a global routing probability vector is output through a two-layer perceptron. The local routing branch performs ROI pooling on each candidate box and outputs a local routing probability vector through a two-layer perceptron. The final route probability is .

[0025] The number of activated experts is dynamically determined based on the real-time load of the inference device and the detection confidence of the candidate boxes. : If GPU utilization is greater than 80% or detection confidence is high ,but ; If GPU utilization is less than 50% and detection confidence is... ,but ; Other situations .

[0026] Select the most suitable route based on the probability of route fusion. Each expert network has its weights normalized. The input feature map is then fed into all activated expert networks for parallel computation. Finally, the refined feature map is obtained by weighted summation according to the normalized weights.

[0027] in The normalized routing weight for the k-th activated expert. A dedicated expert network for the k-th activated scenario.

[0028] The refined feature maps are fed into the YOLO11 Neck network and detection head to obtain the final detection results. It includes the bounding box coordinates, confidence score, and category information for each cement bag.

[0029] (II) Multi-target tracking module The multi-target tracking module employs the improved AdaBoTSORT adaptive target tracking algorithm of this invention, and embeds a scene-adaptive dynamic hypergraph association method within it (see step S4 for details). AdaBoTSORT makes three core improvements on the BoTSORT algorithm: velocity-aware full-state camera motion compensation, two-dimensional reliability dynamic weight association, and confidence-weighted sliding window feature update.

[0030] The complete workflow of the multi-target tracking module is as follows: (1) Receive the detection results output by the YOLO11-SCE detection unit, including the bounding box, confidence score and ReID feature of each cement bag (the ReID feature is a 128-dimensional normalized feature vector, output by an independent branch in the detection head). (2) Perform Kalman filtering prediction on all active trajectories to obtain the predicted state of the current frame. The Kalman filter uses a constant velocity motion model, and the state vector of each trajectory is 8-dimensional: ,in The coordinates of the bounding box center are For width, For height, For the velocity components of the corresponding variable; (3) Perform speed-sensing full-state camera motion compensation to correct the predicted state of all trajectories; (4) Calculate the dynamic fusion weights for each pair of trajectory-detection boxes and construct the fusion cost matrix; (5) Use the Hungarian algorithm to perform the first-stage matching of high-confidence detection boxes and trajectories; (6) Perform a second-stage IoU matching on unmatched trajectories and low-confidence detection boxes; (7) Update the ReID feature library of successfully matched trajectories using a confidence-weighted sliding window feature update mechanism; (8) Update the Kalman filter state of the trajectory using the matched detection boxes; (9) Perform trajectory lifecycle management, delete 30 consecutive unmatched death trajectories, and initialize new trajectories; (10) Output the set of active trajectories in the current frame, including the tracking ID, bounding box and motion state of each trajectory.

[0031] (III) Counting Module The counting module accurately counts the cement bags based on the tracking trajectory output by the multi-target tracking module. Users can draw a horizontal virtual counting line and counting area on the video screen using a mouse through a web management interface. The virtual counting line is typically positioned at the conveyor belt exit, approximately 1 meter from the packaging machine exit.

[0032] The counting logic unit monitors the movement status of each tracking ID in real time. When the bottom of the cement bag's bounding box touches the virtual counting line, it is marked as "to be counted". When more than 50% of the cement bag's bounding box crosses the virtual counting line, a single valid count is triggered, and the tracking ID is marked as "counted". The same tracking ID is counted only once during its entire lifecycle.

[0033] Each time a valid count is triggered, the system increments the total number of packages in the current lane by 1 and sends the count result to the backend service unit in real time. Simultaneously, the system captures the current frame image and 5-second video clips before and after it, storing them in association with the count record for later verification.

[0034] As a concrete example, in the actual operation of a cement plant's packaging workshop, cameras capture video streams at a frame rate of 25 FPS, which are then preprocessed to obtain a standardized 640×640 image. This image is first input into the YOLO11-SCE detection unit. The dual-scale gated routing network determines that the current frame is a stacked and adhered scene, activating two expert networks (K=2): a stacked and adhered expert network and a normal single-bag expert network. After weighted fusion, the bounding boxes of 12 cement bags are output (each box contains center coordinates x, y, width w, height h, confidence score s, and 128-dimensional ReID features). The multi-object tracking module receives these 12 detection results and performs association matching with the existing 8 active trajectories. Ten detection boxes are successfully associated with existing trajectories and their trajectories are updated, while two detection boxes are initialized as new trajectories (assigned new tracking IDs: T009 and T010). The counting module monitors the position of all tracks in real time. When the bottom of the bounding box of the cement bag with tracking ID T005 touches the virtual counting line and more than 50% of its area crosses the counting line, counting is triggered, and the total number of bags changes from 10,847 to 10,848.

[0035] S3. During the training phase of the end-to-end joint optimization network, a bidirectional counting-detection coupling loss function is constructed to perform end-to-end joint parameter updates for the target detection module, multi-target tracking module, and counting module. As one embodiment, step S3 of the present invention is implemented as follows: Traditional detection, tracking, and counting systems employ a pipelined approach of "detection → tracking → counting," with each module trained independently, and counting errors unable to be fed back to the detection and tracking modules. This invention breaks this paradigm by treating the three modules as a whole for end-to-end joint training.

[0036] (a) Construction of the training dataset This invention uses images of cement bags actually collected in a factory as the training dataset. The dataset covers various scenarios with different lighting conditions (sunny days, cloudy days, and artificial lighting at night), different angles, different stacking states (normal single bag, stacks of 2 to 5 bags, and stacks of 5 to 10 bags), and different dust concentrations (clean, light dust, and heavy dust). A total of 100,000 images were collected and divided into training, validation, and test sets in an 8:1:1 ratio.

[0037] For each training image, the annotation information includes: The bounding box coordinates (x_min, y_min, x_max, y_max) for each cement bag; Each cement bag has a category label (normal single bag / stacked bag / dust-covered bag / abnormal light exposure bag); The total number of cement bags in the current frame; Density distribution map (Generated by convolving the center point of each labeled bounding box with a Gaussian kernel).

[0038] For video sequence training data, additional annotation is required: The association relationship of the same target across frames (i.e., trajectory ID); the true count value in the entire video segment. .

[0039] (ii) Loss detection

[0040] Detection loss The standard YOLO11 loss function consists of three parts: Bounding box regression loss: using CIoU loss, the calculation formula is as follows: ,in For Euclidean distance, Center of the prediction box Centered at the true bounding box This is the diagonal distance of the smallest closure box that simultaneously contains both the predicted and ground truth boxes. These are the weighting coefficients. A measure of aspect ratio consistency; Confidence loss: Use binary cross-entropy loss; Classification loss: Use cross-entropy loss.

[0041] The weighted sum of the three losses yields the following result: The weighting coefficients are respectively set as , , .

[0042] (iii) Tracking Losses

[0043] Tracking loss This represents the identity association loss during trajectory-detection association in the multi-object tracking module. Specifically, for each frame, the negative log-likelihood of all correctly associated pairs (i.e., the true match between the trajectory and the detection box) is calculated:

[0044] in The set of all true matching trajectory-detection pairs in the current frame. To match the number of pairs, The correlation probability between trajectory i and detection j is given by the output of the differentiable hypergraph neural network, see step S44 for details.

[0045] (iv) Forward counting consistency loss

[0046] While outputting the count value, the counting module generates a density distribution map of the current frame through a lightweight decoder. Each pixel value represents the probability density of the presence of the cement bag target center at that location. This lightweight decoder consists of two deconvolutional layers and one 1×1 convolutional layer, progressively upsampling the feature map of the tracking module to the original image size (640×640).

[0047] Forward counting consistency loss By using density distribution maps Integrating over the entire image plane yields the predicted count value:

[0048] Then calculate the predicted count value and the actual count value. Smoothing L1 loss between:

[0049] The smoothed L1 loss is defined as follows:

[0050] (v) Reverse density sensing loss

[0051] The reverse density sensing loss is one of the core innovations of this invention, which realizes the "reverse" information flow from the counting module to the detection module.

[0052] First, according to the density distribution map Generate a spatial attention mask :

[0053] in Density distribution map Mean across the entire image plane ( ), The preset scaling factor (taken in this embodiment) ), For the Sigmoid function, .

[0054] The physical meaning of this attention mask is: in high-density areas (where multiple cement bags are stacked), the mask value is close to 1, making the detection module pay more attention to this area; in low-density areas, the mask value is close to 0, suppressing the detection module's response to the background area. The Sigmoid function smoothly maps the mask value to the (0,1) interval.

[0055] Then, the attention mask The feature map size is downsampled to the size of the target detection module, and then weighted:

[0056] in This is the intermediate layer feature map of the target detection module (specifically, the feature map output by the third C3k2 module of the YOLO11 backbone network, with a size of 80×80×512). For element-wise multiplication, For bilinear interpolation downsampling, the attention mask is downsampled from 640×640 to 80×80. Weighted feature map. The cement bag target detection continues through subsequent network layers of the target detection module.

[0057] Inverse density-sensing loss Mean squared error loss for the density distribution plot:

[0058] in This is the true density distribution map generated by convolving the center points of the true bounding boxes with a Gaussian kernel. Specifically, for each labeled cement bag center point... A two-dimensional Gaussian kernel is generated with it as the center. After superimposing all Gaussian kernels, we get ,in Take 15 pixels.

[0059] (vi) Joint Loss Function and End-to-End Optimization The bidirectional counting-detection coupling loss function is expressed as:

[0060] The values ​​for each weight coefficient are: , , , These weighting coefficients were determined through grid search on the validation set.

[0061] During training, backpropagation is used to... The gradients are simultaneously passed to the object detection module, multi-object tracking module, and counting module. The AdamW optimizer is used to jointly update all learnable parameters of the end-to-end joint optimization network. The initial learning rate is 0.001, the weight decay coefficient is 0.0005, the batch size is 32, and the number of training epochs is 200.

[0062] The effect of this end-to-end joint optimization is that the counting error will propagate back to the detection module and the tracking module, forcing the detection module to extract higher quality features in areas that are prone to counting errors (such as stacked areas and boundary areas); at the same time, the prediction error of the density distribution map will adjust the detection module's attention to dense and sparse areas through the attention masking mechanism, forming a reverse information flow of "counting → detection".

[0063] As a concrete example, during training, a batch of input contained 32 images, one of which showed 8 cement bags stacked together. The object detection module initially predicted only 6 cement bags (missing 2), and the density distribution map integral predicted the count value. Actual count value Forward counting consistency loss The value is approximately 1.7. Simultaneously, the density distribution map predicts lower values ​​in the stacked region, resulting in a lower weight (approximately 0.3) for the generated attention mask in the stacked region, leading to a reverse density-aware loss. Significant. Combined losses. Through backpropagation, gradients flow simultaneously to both the detection and counting modules: the detection module learns to extract more features from the stacked regions, while the counting module learns to predict the density distribution more accurately. After 200 rounds of training, the detection accuracy in this scenario improved from 81.3% to 99.2%.

[0064] S4. During the execution of the counting logic, the multi-target tracking module adopts the scene adaptive dynamic hypergraph association method to perform cement bag trajectory-detection association. The structural parameters of the dynamic hypergraph are dynamically determined according to the cement bag scene density of the current frame. The hyperedges in the dynamic hypergraph are used to connect more than three nodes. The hyperedge cost term corresponding to each hyperedge is used to evaluate the rationality of the group movement among multiple cement bags in a set of association combinations.

[0065] As one embodiment, step S4 of the present invention is implemented as follows: Traditional multi-object tracking algorithms typically employ bipartite graph matching (such as the Hungarian algorithm) when performing trajectory-detection association, considering only the one-to-one relationship (univariate cost) between the trajectory and the detection box. This method performs well in scenarios with sparse targets and simple motion, but in industrial scenarios with densely stacked cement bags and high-speed movement, it is prone to ID switching and tracking loss. The scene-adaptive dynamic hypergraph association method proposed in this invention fundamentally solves this problem by constructing hyperedges connecting three or more nodes to model the group behavior patterns among cement bags.

[0066] (I) Scene density estimation and hypergraph structure parameter determination First, count the number of cement bag targets detected in the current frame. and scene area Scene area Pre-calibrated by the user, representing the area (in square meters) of the effective detection zone on the conveyor belt. Calculate the scene density factor:

[0067] Scene density factor The unit is "units / square meter", which reflects the density of cement bags in the current frame.

[0068] Based on scene density factor Dynamically determine the structural parameters of the hypergraph: like If so, it is determined to be a densely stacked scene, and the super-edge order is [not specified]. Furthermore, hyperedges are constructed using a fully connected approach. In this embodiment, the preset high-density threshold is taken as... Units per square meter (i.e., more than 3 cement bags per square meter are considered dense scenes).

[0069] like If so, it is determined to be a medium-density scene with a hyperedge order of [number missing]. Furthermore, the hyperedge is constructed using the k-nearest neighbor method, where k takes the value of .in In this embodiment, the preset low-density threshold is taken as... Units per square meter.

[0070] like If the order is 0, it is determined to be a sparse scene, and the super-edge order is 0. It degenerates into a standard second-order edge (i.e., a normal bipartite graph matching).

[0071] The above density threshold and The value of was obtained through experimental optimization in real-world scenarios based on the size of the cement bags (the standard cement bag size is approximately 50cm × 30cm × 10cm) and the width of the conveyor belt (approximately 80cm to 120cm). In dense scenarios (more than 3 cement bags per square meter), the spacing between cement bags is less than 10cm, and stacking and occlusion occur frequently, requiring high-order hyperedges (4th order) to model complex group behavior; in medium-density scenarios, 3rd order hyperedges are sufficient to capture the mutual influence between targets; in sparse scenarios, the mutual influence between targets is small, and degenerating to second-order edges can reduce computational complexity.

[0072] (II) Construction of dynamic hypergraphs Constructing dynamic hypergraphs ,in: Node set It consists of all candidate association pairs in the current frame. Each node This represents a candidate association hypothesis between an existing trajectory and a detection box. For example, if the current frame has 8 active trajectories and 10 detection boxes, there are a total of 80 candidate association pairs, or 80 nodes.

[0073] Hyperedge set Connect according to the structural parameters determined in step S41 There are 10 nodes. Each hyperedge corresponds to a group behavior pattern of cement bags on the conveyor belt, such as "three cement bags moving forward side by side" or "four cement bags following closely in front and behind".

[0074] The hyperedge is constructed as follows: (1) For each detection box Find the one that is spatially closest to it. Other bounding boxes (all other bounding boxes in fully connected mode); (2) For each trajectory Find the location that is spatially closest to its predicted location. Another trajectory; (3) Perform a Cartesian product between the above set of detection boxes and the set of trajectories to generate a set of candidate hyperedges; (4) Filter the candidate superedges and retain the superedges that are spatially adjacent to the candidate association pairs corresponding to the nodes inside the superedge.

[0075] (III) Definition of Global Energy Function Define a global energy function on a dynamic hypergraph:

[0076] in: Selection variables for all candidate associations; This indicates whether the i-th candidate association is selected (1 indicates selected, 0 indicates not selected); It is a univariate potential function that measures the reasonableness of a single candidate association; It is the hyperedge potential function, which measures the group rationality when all candidate associations within the hyperedge are selected simultaneously.

[0077] (iv) Solving Differentiable Hypergraphs via Neural Networks Since the exact minimization of the global energy function is an NP-hard problem, this invention uses a differentiable hypergraph neural network for approximate solution. The differentiable hypergraph neural network performs bidirectional message passing between nodes and hyperedges through hyperedge convolution operations, iteratively updates the node embedding vectors, and finally outputs the value of each node through a classification layer. Association probability .

[0078] The specific computational process of differentiable hypergraph neural networks will be detailed in the following description.

[0079] Ultimately, based on the association probability Determine the globally optimal association result: when Greater than the preset association threshold (In this embodiment, we take) When the i-th candidate association is established, the Hungarian algorithm is used to ensure that each detection box is associated with at most one trajectory and each trajectory is associated with at most one detection box.

[0080] As a concrete example, in one frame of an image, eight cement bags are densely packed on a conveyor belt (approximately four cement bags per square meter), constituting a densely stacked scene. The system detected this. , Calculated Therefore, a 4th-order fully connected hyperedge approach is adopted. The constructed hypergraph contains 80 nodes (8 trajectories × 10 detection boxes) and approximately 560 hyperedges (each hyperedge connects 4 nodes). After three layers of message passing, the differentiable hypergraph neural network outputs the association probability of each node. Specifically, the association probability between the 3rd trajectory and the 5th detection box is 0.92, and the association probability between the 3rd trajectory and the 7th detection box is 0.08, ultimately successfully associating the 3rd trajectory with the 5th detection box. In contrast, traditional bipartite graph matching only considers univariate cost, resulting in two ID switching operations in this dense scenario, while the method of this invention achieves zero ID switching operations.

[0081] Example 2: It should be noted that the bidirectional counting-detection coupling loss function in step S3 specifically includes the forward counting consistency loss and the reverse density sensing loss: S31: While outputting the count value, the counting module generates a density distribution map of the current frame through a lightweight decoder. Each pixel value represents the probability density of the presence of the cement bag target center at that location.

[0082] As one embodiment, step S31 of the present invention is implemented as follows: The lightweight decoder in the counting module consists of two deconvolutional layers and one 1×1 convolutional layer. The deconvolutional layers are configured as follows: First deconvolutional layer: 256 input channels, 128 output channels, 4×4 kernel, stride 2, padding 1, output size is twice the input size; Second deconvolutional layer: 128 input channels, 64 output channels, 4×4 kernel, stride 2, padding 1, output size is twice the input size. The 1×1 convolutional layer maps 64 channels to 1 channel, outputting a density distribution map.

[0083] The decoder takes as input a feature map (160×160×256 pixels) output from the last layer of the multi-object tracking module. After two deconvolution upsampling operations, it outputs a density distribution map with a size of 640×640×1. The size is consistent with the input image. Each pixel value in the density distribution map ranges from [0,1], representing the probability density of the presence of the cement bag target center at that location. The integral value of the entire density distribution map equals the predicted number of cement bags in the current frame.

[0084] As a concrete example, in a certain frame of the image, the density distribution map output by the decoder forms eight Gaussian-shaped peak regions at the center of the eight cement bales, with peak values ​​ranging from 0.7 to 0.9, while the background region values ​​are close to 0. The integral value of the entire density distribution map is 7.8, indicating that the current frame is predicted to have approximately 7.8 cement bales (actually 8).

[0085] S32: The forward counting consistency loss By the density distribution map Integrating over the entire image plane to obtain the predicted count value And calculate its difference from the actual count value. Smoothing L1 loss between: .

[0086] As one embodiment, step S32 of the present invention is implemented as follows: Density distribution map The size is Where H=640, W=640. The predicted count values ​​are obtained by discrete integration over the entire image plane:

[0087] The physical meaning of this summation operation is to accumulate the probability density of each pixel to obtain the expected value of the number of targets. Since each cement bag target forms a Gaussian kernel with an integral value of 1 in the density distribution map, therefore... This directly corresponds to the predicted number of targets.

[0088] True count value This represents the total number of labeled cement bags in the current frame, obtained through manual annotation. For example, if an image in a training sample contains 8 cement bags, then... .

[0089] The smoothed L1 loss is defined as:

[0090] in Smoothing L1 loss combines the advantages of L1 loss (robust to large errors) and L2 loss (smooth and differentiable to small errors).

[0091] For example, if , ,but , , .like , ,but , , .

[0092] S33: The reverse density sensing loss Specifically: S331: According to the density distribution map Generate a spatial attention mask : ,in The density distribution map The mean value across the entire image plane. The preset scaling factor, This is the Sigmoid function.

[0093] As one embodiment, step S331 of the present invention is implemented as follows: First, calculate the global mean of the density distribution map:

[0094] global mean This reflects the average density level of the cement bales in the current frame. For example, if ,but .

[0095] Then, the attention mask value is calculated for each pixel location:

[0096] Among them, the proportionality factor The sensitivity of the attention mask to density differences is controlled. In this embodiment, we take... The basis for this value is: the typical range of the peak value in the density distribution map is... to The mean is approximately multiplied by Afterwards, the input value for the peak region is approximately 0.3 to 0.8, and the Sigmoid output is approximately 0.57 to 0.69. The input value for the background region is approximately -0.2 to -0.1, and the Sigmoid output is approximately 0.45 to 0.48, which can effectively distinguish between high-density and low-density regions.

[0097] Sigmoid function The input values ​​are smoothly mapped to the (0,1) interval, making the attention mask a smooth weight map.

[0098] The physical meaning of this attention mask is: in areas with density higher than the mean (areas containing cement particles), the mask value is greater than 0.5, making the detection module pay more attention to these areas; in areas with density lower than the mean (background areas), the mask value is less than 0.5, suppressing the detection module's response to the background.

[0099] S332: Apply the attention mask The feature map is downsampled to the size of the target detection module, and then weighted: ,in This is the intermediate layer feature map of the target detection module. For element-wise multiplication; the weighted feature map is... The cement bag target detection continues through the subsequent network layers of the target detection module.

[0100] As one embodiment, step S332 of the present invention is implemented as follows: Intermediate layer feature map of the object detection module Take the feature map output from the 3rd C3k2 module of the YOLO11 backbone network, with a size of [size missing]. (i.e., H_F=80, W_F=80, C_F=512). This feature map contains rich semantic information and is suitable for attention weighting.

[0101] Attention mask The size is It needs to be downsampled to using bilinear interpolation. :

[0102] After downsampling, each channel of the feature map is weighted element-wise: For all

[0103] The physical meaning of this weighting operation is that in high-density regions (where multiple cement bags are stacked), the characteristic responses of all channels are enhanced; in low-density regions, the characteristic responses of all channels are suppressed. This "channel-independent" weighting method preserves the relative relationships between channels of the feature map while simultaneously guiding spatial attention.

[0104] Weighted feature map Target detection continues through subsequent network layers of YOLO11 (including the FPN+PAN structure of the Neck network and the Head detection head).

[0105] S333: Calculate the reverse density sensing loss ,in This is the true density distribution map generated by convolving the center points of the true bounding boxes with a Gaussian kernel.

[0106] As one embodiment, the specific implementation of step S333 of the present invention is as follows: True density distribution map The generation process is as follows: For each labeled cement bag bounding box, extract the coordinates of its center point. Then, a two-dimensional Gaussian kernel is generated centered on this point:

[0107] The standard deviation of the Gaussian kernel We selected 15 pixels. This value is based on the fact that a standard cement bag is approximately 40×60 pixels in a 640×640 image, and the Gaussian kernel coverage is approximately 3 times the standard deviation (45 pixels), which can cover the entire cement bag area well.

[0108] By superimposing all Gaussian kernels, the true density distribution map is obtained:

[0109] The true density distribution map satisfies .

[0110] The mean squared error loss is:

[0111] S34: Define the detection loss The bounding box regression loss, confidence loss, and classification loss of the target detection module are weighted sums; the tracking loss is defined. The identity association loss during the cement bag trajectory-detection association in the multi-target tracking module; the bidirectional counting-detection coupling loss function is expressed as: in , , , The preset weighting coefficients are used; backpropagation is then used to... The gradient is simultaneously transmitted to the target detection module, the multi-target tracking module, and the counting module, and the AdamW optimizer is used to jointly update all learnable parameters of the end-to-end joint optimization network.

[0112] As one embodiment, step S34 of the present invention is implemented as follows: The weight coefficients of each loss term were determined through grid search on the validation set. Specifically, for each weight coefficient, a search was performed on the validation set within the range [0.1, 5.0] with a step size of 0.1 to select the weight combination that maximized the counting accuracy on the validation set. The final determined weight coefficients are as follows: , , , .in The value is relatively large (2.0) because the counting task is the ultimate goal of this invention and needs to be given a higher optimization priority.

[0113] The AdamW optimizer is configured with the following initial learning rate: The exponential decay rate estimated by the first moment The exponential decay rate estimated by the second moment Weight decay coefficient Batch size The learning rate employs a cosine annealing scheduling strategy, gradually decreasing from 0.001 to 0.00001 over 200 training rounds.

[0114] During the reverse propagation process, The gradients of all learnable parameters in the network are calculated using the chain rule. It is particularly noteworthy that... The gradient will pass through the attention mask. The information is passed to the feature extraction layer of the detection module to realize the reverse information flow of "counting module → detection module"; The gradient will be simultaneously transmitted to the counting module (which optimizes the prediction of the density distribution map) and the detection module (which optimizes the prediction of the number of targets).

[0115] As a concrete example, in a training iteration, a certain sample's , , , ,but After backpropagation, the AdamW optimizer updates the network weights based on the gradients of each parameter, with the current learning rate being 0.0008.

[0116] It should be noted that the scene adaptive dynamic hypergraph association method in step S4 specifically includes: S41: Count the number of cement bag targets detected in the current frame. and scene area Calculate scene density factor And according to the scene density factor Dynamically determine the structural parameters of the hypergraph: like If so, it is determined to be a densely stacked scene, and the super-edge order is [not specified]. Furthermore, hyperedges are constructed using a fully connected approach. like If so, it is determined to be a medium-density scene with a hyperedge order of [number missing]. Furthermore, the hyperedge is constructed using the k-nearest neighbor method, where k takes the value of ; like If the order is 0, it is determined to be a sparse scene, and the super-edge order is 0. It degenerates into a standard second-order edge; in and This is the preset density threshold.

[0117] As one embodiment, step S41 of the present invention is implemented as follows: Scene area The area is obtained by the user through calibration during system initialization. The specific calibration method is as follows: a rectangular calibration plate of known dimensions (e.g., a 1m × 0.5m checkerboard calibration plate) is placed on the conveyor belt plane. The image coordinates are mapped to world coordinates through perspective transformation, thereby calculating the area of ​​the effective detection region of the conveyor belt. (Unit: square meters). In this embodiment, the effective detection area of ​​the conveyor belt in a cement plant is 2 square meters.

[0118] Number of cement bag targets detected in the current frame Output by the object detection module. For example, if 8 cement bags are detected in a frame, then... .

[0119] Scene density factor Units per square meter.

[0120] The density threshold value is: Units / square meter The thresholds were determined based on the following: a standard cement bag is approximately 50cm × 30cm in size, with a projected area of ​​approximately 0.15 square meters. In dense scenes (more than 3 cement bags per square meter), the spacing between cement bags is less than 10cm, and stacking and occlusion occur frequently, requiring a 4th-order hyperedge to model group behavior; in medium-density scenes (1 to 3 cement bags per square meter), the spacing between cement bags is 10cm to 30cm, and a 3rd-order hyperedge is sufficient to capture the mutual influence between targets; in sparse scenes (less than 1 cement bag per square meter), the mutual influence between targets is small, and degenerating to a 2nd-order hyperedge can reduce computational cost.

[0121] In the k-nearest neighbor approach, k takes the value of That is, each node can connect to a maximum of 10 nearest neighbor nodes, and connects to all other nodes when the number of targets is small.

[0122] S42: Constructing a Dynamic Hypergraph , where the set of nodes It consists of all candidate association pairs in the current frame, each node Represents a candidate association hypothesis between a cement bag trajectory and a cement bag detection box; hyperedge set Connect according to the structural parameters determined in step S41 Each node represents a hyperedge that corresponds to a group behavior pattern of cement bags on the conveyor belt.

[0123] As one embodiment, step S42 of the present invention is implemented as follows: Assuming the current frame has Active trajectory and If there are 10 detection boxes, then the number of candidate association pairs is 10. Each candidate association pair constitutes a node in the hypergraph.

[0124] The hyperedge is constructed as follows: (1) For each detection box Calculate the spatial distance (Euclidean distance, in pixels) between it and all other detection boxes, and select the closest one. Each detection box (select all other detection boxes in fully connected mode); (2) For each trajectory Calculate the spatial distance between its predicted location and all other predicted trajectory locations, and select the closest one. A trajectory; (3) Combine the set of detection boxes selected in step (1) with the set of trajectories selected in step (2) to generate candidate hyperedges. Specifically, for the selected hyperedges... A detection box and For each trajectory, select the one whose spatial position is closest to the others in their Cartesian product. Each candidate association pair is a node contained within a superedge.

[0125] Each hyperedge corresponds to a group behavior pattern. For example, in a densely stacked scenario, a 4th-order hyperedge may correspond to a group pattern of "four cement bags arranged in pairs, closely following each other"; in a medium-density scenario, a 3rd-order hyperedge may correspond to a group pattern of "three cement bags arranged in a triangle".

[0126] S43: Define a global energy function on the dynamic hypergraph. ,in For the selection variable of all candidate associations, Indicates the first Whether a candidate association is selected. It is a univariate potential function. It is the hyperboundary potential function.

[0127] As one embodiment, step S43 of the present invention is implemented as follows: The global energy function transforms the association problem into a combinatorial optimization problem: selecting a set of candidate associations ( This minimizes the global energy. The selected candidate associations are the final trajectory-detection matching results.

[0128] Univariate potential function Evaluating the reasonableness of a single candidate association. Hyperedge potential function. The group rationality is measured when all candidate associations within a superedge are selected simultaneously.

[0129] S44: The dynamic hypergraph is encoded using a differentiable hypergraph neural network. This differentiable hypergraph neural network performs bidirectional message passing between nodes and hyperedges through hyperedge convolution operations, iteratively updates node embedding vectors, and finally outputs each node through a classification layer. Association probability According to the association probability Determine the globally optimal association result.

[0130] As one embodiment, step S44 of the present invention is implemented as follows: Since the exact minimization of the global energy function is an NP-hard problem (the search space grows exponentially with the number of nodes), this invention uses a differentiable hypergraph neural network for approximate solution. The differentiable hypergraph neural network uses a message-passing mechanism to iteratively propagate information between nodes and hyperedges, ultimately outputting the association probability of each node, thus transforming the combinatorial optimization problem into a node classification problem.

[0131] The specific computational process of differentiable hypergraph neural networks will be described in detail later.

[0132] It should be noted that the univariate potential function in step S43 Specifically: in This represents the intersection-union ratio (IoU) of the predicted trajectory bounding box and the detected bounding box of the cement bag. For feature cosine similarity, The appearance characteristics of the cement bag trajectory. The appearance features of the cement bag inspection frame. and The preset weighting coefficients; the univariate potential function For a preset constant .

[0133] As one embodiment, the specific implementation of the univariate potential function in step S43 of the present invention is as follows: Univariate potential function Measure the cost of matching the i-th candidate association pair (trajectory i with detection box j). This indicates that the match has been selected. This indicates that you do not choose this option.

[0134] When matching is selected ( The cost consists of two parts: (1) IoU cost: Where IoU (Intersection over Union) is the area of ​​the intersection between the predicted trajectory bounding box and the detected bounding box divided by the area of ​​their union:

[0135] The IoU value ranges from [0,1]. A larger IoU indicates greater overlap between the two bounding boxes, resulting in a more reliable match. Therefore... The smaller the value, the lower the cost.

[0136] (2) Cost of appearance features: .in The appearance feature vector maintained for the trajectory (128-dimensional, maintained by a confidence-weighted sliding window feature update mechanism, see the background technology section for details). Here is the ReID feature vector of the detection box (128-dimensional, output by the detection head). Cosine similarity is defined as:

[0137] The cosine similarity score ranges from -1 to 1, with a larger value indicating greater similarity between the two features. Therefore... The smaller the value, the lower the cost.

[0138] Weighting coefficient and satisfy In this embodiment, we take , This means it places greater emphasis on motion information (IoU). This value was determined through grid search optimization on the validation set.

[0139] When no matching is selected ( The cost is a preset constant. In this embodiment, we take... The physical meaning of this value is: the cost of a mismatch is equivalent to a moderate matching cost (when IoU=0 and cosine similarity is 0). This causes the model to tend to mismatch when the matching cost is higher than 1.0.

[0140] As a concrete example, in a candidate association pair, the IoU between the predicted trajectory box and the detected box is 0.85, and the cosine similarity is 0.92. Then:

[0141] Since 0.122 < 1.0 ( The model tends to select this match.

[0142] It should be noted that the hyper-edge potential function in step S43 Specifically: in Let be the order of the superedge. , , These are preset weighting coefficients; This is a spatial topological consistency sub-item used to constrain the association combination of any two cement bags within a hyperedge. and Consistency of relative distance between adjacent frames: ,in The normalized spatial distance between the two cement bag targets in the current frame is defined as follows: the normalization is performed using the diagonal length of the current frame image as a reference. This is the normalized spatial distance from the previous frame; This is a motion vector consistency sub-item used to constrain the association combination of any two cement bags within the hyperedge. and The consistency of the motion vector in both direction and magnitude: ,in , These are the motion vectors of the two cement bag targets, calculated by the difference in center positions between adjacent frames; This is a group velocity distribution consistency sub-item used to constrain the association combination of any two cement bags within the hyperedge. and The velocity is consistent with the overall velocity distribution of all cement bag targets in the scene: ,in , Let the instantaneous velocities of the two cement bags be denoted. , This represents the mean and standard deviation of the velocity of all cement bag targets in the current frame.

[0143] As one embodiment, the specific implementation of the hyperboundary potential function in step S43 of the present invention is as follows: Hyperboundary potential function Only when all nodes inside the hyperedge are selected (i.e. For all A non-zero cost is only generated when a set of candidate associations are selected simultaneously. This means that group consistency is only considered when a set of candidate associations are selected at the same time. If no node inside the hyperedge is selected, the hyperedge potential function is 0.

[0144] Weighting coefficient , , satisfy In this embodiment, we take , , This means that spatial topological consistency has the highest weight. This value is determined through grid search optimization on the validation set.

[0145] (1) Spatial topology consistency sub-item

[0146] Let this be the normalized spatial distance between the two cement bag targets in the current frame. Let the coordinates of the center points of the two targets be... and The length of the image diagonal is pixels, then:

[0147] The normalized spatial distance ranges from [0, 1]. Similarly, the normalized spatial distance of the previous frame is calculated. .

[0148] This reflects the degree of change in the relative distance between two targets between adjacent frames. On a conveyor belt, cement bales typically maintain a relatively stable spacing, therefore... The smaller the value, the more consistent the spatial topology of the group.

[0149] For example, if the distance between two cement bags is 200 pixels in the current frame and 190 pixels in the previous frame, then... , , .

[0150] (2) Motion Vector Consistency Sub-item

[0151] The motion vectors of the two cement bag targets are respectively and It is calculated from the center position difference between adjacent frames:

[0152] Cosine similarity:

[0153] This reflects the similarity in the direction and magnitude of the motion of the two targets. On a conveyor belt, cement bags typically move at the same speed and in the same direction, therefore... The smaller the value, the better the consistency of group movement.

[0154] For example, if the motion vectors of two cement bags are (12, -3) and (11, -2) respectively, their cosine similarity is approximately 0.995. .

[0155] (3) Consistency of group velocity distribution

[0156] The instantaneous velocities of the two cement bag targets are respectively and (Unit: pixels per frame).

[0157] The average velocity of all cement bag targets in the current frame: The standard deviation of the velocity of all cement bag targets in the current frame:

[0158] This reflects the degree to which the average velocity of the two targets deviates from the overall velocity distribution of the scene. The smaller the value, the closer the speed of the two targets is to the average level of the scenario, and the better the group consistency.

[0159] For example, if the velocities of two targets are 14 pixels / frame and 16 pixels / frame respectively, the scene mean is 15 pixels / frame, and the standard deviation is 2 pixels / frame, then... .

[0160] As a concrete example, in densely stacked scenarios ( Given a 4th-order hyperedge containing 4 candidate associations, all nodes within the hyperedge are selected. Calculate all 6 pairs of nodes ( The hyperedge potential function is the sum of the three sub-items of the equation. If all sub-items are small (spatial distance variation < 0.02, motion vector cosine similarity > 0.98, velocity close to the mean), then the hyperedge potential function value is small, indicating that the group motion of the four cement bags is very consistent, and the association result is reasonable. Conversely, if the motion direction of a cement bag is significantly different from the other three, then the hyperedge potential function value is large, and the model will tend not to select this association combination.

[0161] It should be noted that the differentiable hypergraph neural network in step S44 specifically includes: S441: Node Initialization: Initialize each node initial embedding vector Let it be the unary feature vector of its corresponding candidate association pair. ,in The normalized Euclidean distance between the predicted position of the cement bag trajectory and the detected position of the cement bag is defined as follows: the normalization refers to normalization based on the diagonal length of the current frame image. S442: Hyperedge Convolution Operation: In the... In a layered network, message passing from a node to a superedge is... ,in To find the aggregation function, The weight matrix is ​​learnable; message passing from hyperedge to node is... ,in It is a multilayer perceptron. The weight matrix is ​​a learnable matrix; S443: Iterative execution steps S442 total This process is repeated to obtain the final embedding vector for each node. ,in The preset number of layers in the hypergraph neural network; S444: The final embedding vector Input a fully connected classification layer, activate it with Softmax, and output each node. Association probability ,in and These are the learnable weight matrix and bias vector; S445: Based on the aforementioned association probability and preset association threshold ,when Time to determine the first The candidate associations are established; then the Hungarian algorithm is used to solve the problem so that each cement bag detection box is associated with at most one trajectory and each cement bag trajectory is associated with at most one detection box, thus obtaining the global optimal association result; the detection boxes that are successfully matched in the global optimal association result are used to update the Kalman filter state of the corresponding cement bag trajectory, the unmatched detection boxes are initialized as new trajectories, and the unmatched trajectories are marked as temporarily unassociated.

[0162] As one embodiment, step S44 of the present invention is implemented as follows: S441: Node initialization: Each node initial embedding vector It is a 3-dimensional vector composed of three unary features: (1) : Intersection over Union (IoU) ratio between the predicted trajectory bounding box and the detected bounding box, with a value range of [0,1]; (2) : Cosine similarity between trajectory appearance features and detected appearance features, with a value range of [-1, 1]. In this embodiment, it is truncated to [0, 1] (negative values ​​are 0). (3) The normalized Euclidean distance between the predicted trajectory location and the detected location. Let the predicted trajectory center be... The center of the detection frame is The length of the image diagonal is pixels, then:

[0163] The normalized distance ranges from [0, 1].

[0164] Therefore, the initial embedding vector All three components are in the range [0,1].

[0165] S442: Superedge convolution operation: Differentiable hypergraph neural networks have a total of Layers. Each layer contains two steps: (1) Message passing from node to hyperedge: For each hyperedge It aggregates information from all the nodes it contains:

[0166] in As a learnable weight matrix, the 3D node embeddings are mapped to a 32-dimensional edge message space. The SUM aggregation function sums the information of all nodes, so that the hyperedge message contains comprehensive information about all nodes within the hyperedge.

[0167] (2) Message passing from the hyperedge to the node: For each node Aggregate all messages that contain the hyperedge of that node:

[0168] in The weight matrix is ​​learnable, and the MLP is a two-layer perceptron (32-dimensional → 64-dimensional → 32-dimensional), with ReLU activation function. The node embedding dimension remains 32-dimensional throughout.

[0169] The physical meaning of this convolution operation is that each node receives information (group information) from other nodes within the same hyperedge, enabling iterative updates of node embeddings. After multiple iterations, the node embedding not only includes its own unary features but also incorporates group context information.

[0170] S443: Iterative execution: Iterative execution steps S442 total Second-rate. The value of was determined experimentally: Insufficient message delivery Smoothing occurred at times. This achieves optimal performance. The final embedding vector for each node is then obtained. .

[0171] S444: Classification layer: The final embedding vector Input a fully connected classification layer:

[0172] in , Output after Softmax activation:

[0173] This represents the probability that the i-th candidate association is selected. This indicates the probability of not being selected.

[0174] S445: Globally optimal association: Based on association probability and preset association threshold ,when The i-th candidate association is determined to be valid. Then, the Hungarian algorithm is used to solve the problem, ensuring that each detection box is associated with at most one trajectory, and each trajectory is associated with at most one detection box.

[0175] The specific implementation of the Hungarian algorithm is as follows: Constructing the cost matrix ,in ( Let be the correlation probability between trajectory i and detection j. The Hungarian algorithm finds the matching scheme that minimizes the total cost while satisfying the one-to-one constraint.

[0176] After matching: Successfully matched detection boxes are used to update the Kalman filter state of the corresponding trajectory (performing the Kalman filter update step); Unmatched bounding boxes (i.e., bounding boxes that do not match any trajectory) are initialized as new trajectories and assigned new tracking IDs; Unmatched trajectories (i.e. trajectories that do not match any detection boxes for multiple consecutive frames) are marked as temporarily unassociated. If a trajectory remains unmatched for 30 consecutive frames, it is deleted.

[0177] As a concrete example, a frame contains 8 trajectories and 10 bounding boxes. The micro-hypergraph neural network outputs 80 association probabilities. After thresholding, 12 candidate association probabilities are greater than 0.5. The Hungarian algorithm selects 8 matches from these 12 candidates (each trajectory and bounding box is matched at most once), ultimately resulting in all 8 trajectories being successfully matched, and 2 bounding boxes being initialized as new trajectories.

[0178] It should be noted that the target detection module in step S2 adopts the YOLO11-SCE scene condition expert detection unit. The YOLO11-SCE detection unit replaces the C3k2 convolutional block of the P4 resolution branch of the standard YOLO11 with the scene condition expert SCE module. The SCE module includes four industry-specific expert networks: normal single bag expert, stacking and adhesion expert, dust occlusion expert, and uneven lighting expert, as well as a dual-scale gated routing network. The four expert networks are all built for the cement bag detection scenario. The dual-scale gated routing network dynamically selects and activates at least one expert network to extract cement bag features based on global scene features and local target features. Finally, the output feature maps of all activated experts are weighted and summed according to normalized routing weights to obtain the refined P4 feature map.

[0179] As one embodiment, the specific implementation of the target detection module in step S2 of the present invention has been described in detail in part S2. Here, we supplement the training details of the expert network and the specific parameters of the routing mechanism.

[0180] Training of expert networks: The four expert networks have the same structure, but different training data: Normal single-bag expert: trained using 50,000 single-bag images with normal lighting, no occlusion, and no adhesion; Stacking and Adhesion Expert: Trained using 20,000 images containing 2 to 10 stacked and adhered cement bags; Dust masking expert: trained using 20,000 images of large amounts of dust in the workshop and partially masked cement bags; Illumination Unevenness Expert: Trained using 10,000 images with lighting anomalies such as backlighting, reflection, and partial exposure.

[0181] Each expert network was trained independently for 50 epochs with an initial learning rate of 0.001 and using the AdamW optimizer.

[0182] Parameters of a two-scale gated routing network: Global routing branch: for P4 feature map Perform global average pooling to obtain A 4-dimensional global feature vector is processed through a two-layer perceptron (256→128→4) to output a 4-dimensional global routing probability. .

[0183] Local routing branch: For each candidate box, perform ROI pooling (output 7×7×256), flatten it, and then pass it through a two-layer perceptron (12544→128→4) to output a 4-dimensional local routing probability. .

[0184] Fusion weights ,Right now .

[0185] The rules for dynamically determining the number of experts K have been detailed in section S2.

[0186] As a concrete example, in a certain frame of an image, the global route probability output by the dual-scale gated routing network is: (By a stacking and sticking expert), the local routing probability of a candidate box is After fusion Select the Stacking and Adhesion Expert (highest probability). If GPU utilization is <50% and detection confidence is <0.7, then K=3, simultaneously activating the Stacking and Adhesion Expert, Normal Single Packet Expert, and Dust Masking Expert.

[0187] It should be noted that the stacked adhesion expert network embeds a differentiable physical optimization layer based on the Poisson equation: the differentiable physical optimization layer transforms the depth estimation problem of stacked adhesion cement bales into the Poisson equation. The problem to be solved, in which For the depth field to be estimated, The source term is predicted from the input feature map; the differentiable physical optimization layer uses the differentiable conjugate gradient method to iteratively solve the Poisson equation during the forward propagation of the neural network to obtain the depth field. According to the depth field Local gradient magnitude generation of uncohesion weight map The disaggregation weight map The input feature map of the stacked adhesion expert network is multiplied element-wise before being fed into subsequent convolutional layers to suppress the feature response of the adhesion region and separate the stacked cement bags. Each iteration of the differentiable conjugate gradient method is a differentiable operation, and the gradient in the solution process can be propagated to the feature extraction layer of the stacked adhesion expert network through backpropagation, so that the feature extraction layer adaptively learns the cement bag feature representation that is conducive to solving the Poisson equation.

[0188] As one embodiment, the specific implementation of the stacked adhesion expert network in step S2 of the present invention is as follows: (I) Physical Background In computer vision, the depth estimation problem can be modeled as solving the Poisson equation. For stacked cement bales, different depth layers create different grayscale patterns in the image, with the gradient magnitude of the depth field being larger at the target boundary and smaller inside the target. After obtaining the depth field by solving the Poisson equation, different target instances can be distinguished based on the gradient magnitude of the depth field.

[0189] (ii) Discretization of the Poisson equation Poisson equation The form on a two-dimensional discrete grid is:

[0190] in For position Depth value at that location, For source terms.

[0191] Rewrite the above equation in matrix form: ,in The sparse matrix corresponding to the Laplace operator (size: (where N is the total number of pixels). and The depth field and source terms are vectorized.

[0192] (III) Source Term Prediction The input feature map of the stacked adhesion expert network is (C=256, H=40, W=40). Predict source terms using a 3×3 convolutional layer (256 input channels, 1 output channel). .

[0193] (iv) Solving by differentiable conjugate gradient method Iterative solution using the conjugate gradient method The iterative formula for the conjugate gradient method is: initialization: , ,

[0194] For the k-th iteration (k=0,1,...,K-1,K=10):

[0195] Each iteration of the conjugate gradient method involves only matrix-vector multiplication. The gradients of the entire solution process, including vector dot products and scalar operations, are all differentiable. Therefore, the gradients of the entire solution process can be backpropagated to the source term using the chain rule. The prediction network.

[0196] In this embodiment, the number of iterations K=10, which is sufficient to bring the residual to less than 1% of the initial residual.

[0197] (v) Generation of the unadhesion weight map The depth field is obtained by solving. Then, calculate its gradient magnitude:

[0198] Generate a de-adhesion weight map:

[0199] The physical meaning of this weight map is: at the boundaries of the depth field (where the gradient magnitude is large), the weight values ​​are small, suppressing the feature response; in flat regions of the depth field (where the gradient magnitude is small), the weight values ​​are large, preserving the feature response. This helps to separate stacked cement bales—the boundaries of different cement bales correspond to the boundaries of the depth field, and the feature responses in these regions are suppressed, thereby reducing the adhesion between targets.

[0200] (vi) Feature weighting and subsequent processing De-adhesion weight map Upsample to the input feature map size (40×40), then multiply element-wise with the input feature map:

[0201] in This is for bilinear interpolation upsampling.

[0202] Weighted feature map The data is fed into the subsequent convolutional layer of the stacked and adherent expert network (a 3×3 convolutional layer with 256 input channels and 256 output channels, activated by BatchNorm+SiLU), and finally outputs the refined feature map.

[0203] As a concrete example, in an image containing five stacked cement bags, the depth field obtained by solving the Poisson equation has a large gradient magnitude (>0.5) at the boundaries of the five cement bags, while the disjoint weight map has a small value (<0.6) at these locations, suppressing the feature response at the boundaries. Inside the cement bags, the gradient magnitude is small (<0.1), while the weight value is large (>0.9), preserving the internal features. The weighted feature map makes it easier for subsequent convolutional layers to distinguish different cement bag instances.

[0204] It should be noted that the dust masking expert network embeds a differentiable transmittance estimation layer based on an atmospheric scattering model: the differentiable transmittance estimation layer models transmittance estimation as an atmospheric scattering model. Solve the inverse problem, where To observe the image, For fog-free images, Transmittance, The atmospheric light value is used as the atmospheric light value; the differentiable transmittance estimation layer first estimates the atmospheric light value through a lightweight subnetwork. and scene depth Then, based on the scene depth Calculate transmittance ,in The atmospheric scattering coefficient is preset; based on the transmittance and atmospheric light value Perform occlusion removal enhancement on the input feature map: ,in To be a small constant to prevent division by zero; The output feature map of the dust-masking expert network is used to enhance the feature identifiability of cement bags under dust-masking conditions; all parameters of the lightweight sub-network are optimized end-to-end through backpropagation of the bidirectional counting-detection coupling loss function, so that the transmittance estimation layer adaptively learns the optimal transmittance estimate under the dust-masking scenario of cement bags.

[0205] As one embodiment, the specific implementation of the dust masking expert network in step S2 of the present invention is as follows: (I) Physical Background Atmospheric scattering models describe the attenuation process of light as it propagates through media such as haze and dust.

[0206] in: Images showing fog / dust. Images of realistic scenes with no fog / dust; Transmittance represents the degree of attenuation of light as it travels from a scene point to the camera, and its value ranges from (0,1). is the atmospheric light value, representing the brightness of the atmosphere at infinity.

[0207] In a cement plant workshop, dust obscures images of cement bags, causing blurring and reduced contrast. This is addressed by estimating transmittance. and atmospheric light value It can be seen from the observed images Restore the unobstructed, realistic scene. .

[0208] (ii) Lightweight subnetwork The input feature map of the dust masking expert network is (C=256, H=40, W=40). The lightweight subnetwork consists of two parallel branches: (1) Atmospheric light value estimation branch: global average pooling is performed on the input feature map (outputting a 256-dimensional vector), and the atmospheric light value is output through a two-layer perceptron (256→128→1). (Scalar).

[0209] (2) Scene depth estimation branch: Use two 3×3 convolutional layers (256→128→1) to estimate the scene depth map. The depth value ranges from [0, 10] meters and is obtained by multiplying the Sigmoid activation function by 10.

[0210] (III) Transmittance Calculation According to atmospheric scattering models, transmittance has an exponential relationship with scene depth:

[0211] in The atmospheric scattering coefficient is taken as in this embodiment. This value corresponds to a scenario with moderate dust concentration (visibility of approximately 10 meters). A higher value indicates a denser dust layer, and a faster decrease in transmittance with depth.

[0212] transmittance The value range is (0,1]. When the depth is 0, the transmittance is 1 (no attenuation). As the depth increases, the transmittance approaches 0 (complete attenuation).

[0213] (iv) Enhancement by removing occlusion Based on the inverse transform of the atmospheric scattering model, unobstructed features are recovered from the observed feature map:

[0214] in It is a small constant (taken in this embodiment) This is to prevent division by zero when the transmittance is 0.

[0215] The physical meaning of this transformation is: subtracting the atmospheric light component from the observed features ( ), then divide by transmittance Compensation attenuation restores the unobstructed features.

[0216] (v) End-to-end optimization All parameters of the lightweight subnetwork (including the convolutional and fully connected layer parameters of the atmospheric light value estimation branch and the scene depth estimation branch) are optimized end-to-end through backpropagation of the bidirectional counting-detection coupled loss function. This means that the network not only learns to estimate transmittance and atmospheric light value, but also learns to make the de-occluded feature map most favorable for subsequent cement bag detection and counting tasks.

[0217] As a concrete example, in a workshop setting with high dust concentrations, the observed feature map of a cement bag is blurred due to dust obscuring the image. A lightweight subnetwork estimates atmospheric light values. Scene depth Meters, calculate transmittance After de-occlusion enhancement: The eigenvalues ​​were amplified and the atmospheric light component was removed, resulting in a significant enhancement of the edge and texture features of the cement pack.

[0218] Finally, the counting logic in step S4 specifically includes: presetting a horizontal virtual counting line and a counting area; when the bottom of the cement bag boundary frame corresponding to any tracking ID touches the virtual counting line, and more than 50% of the boundary frame area enters the counting area, a single valid count is triggered, and the tracking ID is marked as "counted"; the system maintains a global set of counted IDs, and any tracking ID triggers a count only once in its complete life cycle; when the conveyor belt stops moving, the system locks the counting status of all counted IDs until the conveyor belt restarts and unlocks.

[0219] Example 3: Please see Figure 2 , Figure 2 This is a schematic diagram of the hardware device in operation according to an embodiment of the present invention. The hardware device specifically includes: a video AI intelligent point-and-click device 401, a processor 402, and a storage medium 403.

[0220] A video AI intelligent package counting device 401: The video AI intelligent package counting device 401 implements the video AI intelligent package counting method.

[0221] Processor 402: The processor 402 loads and executes the instructions and data in the storage medium 403 to implement the video AI intelligent point-and-click method.

[0222] Storage medium 403: The storage medium 403 stores instructions and data; the storage medium 403 is used to implement the video AI intelligent point-and-click method.

[0223] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A video AI intelligent packet counting method, characterized in that: Includes the following steps, S1. Acquire real-time video streams in the industrial cement bag delivery scenario using the RTSP protocol, and preprocess the video frames to obtain standardized image data. S2. Construct an end-to-end joint optimization network, which includes a target detection module, a multi-target tracking module, and a counting module. Input the standardized image data into the end-to-end joint optimization network. The target detection module outputs the bounding box, confidence score, and category information of the cement bags. The multi-target tracking module assigns a unique tracking ID to each cement bag based on the detection results and continuously tracks its motion trajectory. The counting module executes counting logic based on the tracking ID crossing a preset virtual counting line and outputs the counting result of the cement bags. S3. During the training phase of the end-to-end joint optimization network, a bidirectional counting-detection coupling loss function is constructed to perform end-to-end joint parameter updates for the target detection module, multi-target tracking module, and counting module. S4. During the execution of the counting logic, the multi-target tracking module adopts the scene adaptive dynamic hypergraph association method to perform cement bag trajectory-detection association. The structural parameters of the dynamic hypergraph are dynamically determined according to the cement bag scene density of the current frame. The hyperedges in the dynamic hypergraph are used to connect more than three nodes. The hyperedge cost term corresponding to each hyperedge is used to evaluate the rationality of the group movement among multiple cement bags in a set of association combinations.

2. The video AI intelligent packet-spotting method as described in claim 1, characterized in that, The bidirectional counting-detection coupling loss function in step S3 specifically includes forward counting consistency loss and reverse density sensing loss: S31. While outputting the count value, the counting module generates a density distribution map of the current frame through a lightweight decoder. Each pixel value represents the probability density of the presence of the cement bag target center at that location; S32, the forward counting consistency loss By using the density distribution map Integrating over the entire image plane to obtain the predicted count value And calculate its difference from the actual count value. Smoothing L1 loss between: ; S33, the reverse density sensing loss Specifically: S331, According to the density distribution map Generate a spatial attention mask : ,in The density distribution map The mean value across the entire image plane. The preset scaling factor, For the Sigmoid function; S332, Apply the attention mask The feature map is downsampled to the size of the target detection module, and then weighted: ,in This is the intermediate layer feature map of the target detection module. For element-wise multiplication; the weighted feature map The cement bag target detection continues through the subsequent network layers of the target detection module; S333, Calculate the reverse density sensing loss. ,in This is the true density distribution map generated by convolving the center points of the true bounding boxes with a Gaussian kernel; S34. Define detection loss The bounding box regression loss, confidence loss, and classification loss of the target detection module are weighted sums; the tracking loss is defined. The identity association loss during the cement bag trajectory-detection association in the multi-target tracking module; the bidirectional counting-detection coupling loss function is expressed as: ， in , , , The preset weighting coefficients are used; the weights are then passed through backpropagation. The gradient is simultaneously transmitted to the target detection module, the multi-target tracking module, and the counting module, and the AdamW optimizer is used to jointly update all learnable parameters of the end-to-end joint optimization network.

3. The video AI intelligent packet-spotting method as described in claim 1, characterized in that: The scene-adaptive dynamic hypergraph association method in step S4 specifically includes: S41. Count the number of cement bag targets detected in the current frame. and scene area Calculate scene density factor And according to the scene density factor Dynamically determine the structural parameters of the hypergraph: like If so, it is determined to be a densely stacked scene, and the super-edge order is [not specified]. Furthermore, hyperedges are constructed using a fully connected approach. like If so, it is determined to be a medium-density scene with a hyperedge order of [number missing]. Furthermore, the hyperedge is constructed using the k-nearest neighbor method, where k takes the value of ; like If the order is 0, it is determined to be a sparse scene, and the super-edge order is 0. It degenerates into a standard second-order edge; in and This is a preset density threshold; S42. Constructing a dynamic hypergraph , where the set of nodes It consists of all candidate association pairs in the current frame, each node Represents a candidate association hypothesis between a cement bag trajectory and a cement bag detection box; hyperedge set Connect according to the structural parameters determined in step S41 Each node, and each hyperedge, corresponds to a group behavior pattern of cement bags on the conveyor belt; S43. Define a global energy function on the dynamic hypergraph. ,in For the selection variable of all candidate associations, Indicates the first Whether a candidate association is selected. It is a univariate potential function. It is the hyperboundary potential function; S44. The dynamic hypergraph is encoded using a differentiable hypergraph neural network. The differentiable hypergraph neural network performs bidirectional message passing between nodes and hyperedges through hyperedge convolution operations, iteratively updates the node embedding vectors, and finally outputs each node through a classification layer. Association probability According to the association probability Determine the globally optimal association result.

4. The video AI intelligent packet counting method as described in claim 3, characterized in that, The univariate potential function in step S43 Specifically: ， in This represents the intersection-union ratio (IoU) of the predicted trajectory bounding box and the detected bounding box of the cement bag. For feature cosine similarity, The appearance characteristics of the cement bag trajectory. The appearance features of the cement bag inspection frame. and The preset weighting coefficients; the univariate potential function For a preset constant .

5. The video AI intelligent packet-spotting method as described in claim 3, characterized in that, Hyperboundary potential function in step S43 Specifically: in Let be the order of the superedge. , , These are preset weighting coefficients; This is a spatial topological consistency sub-item used to constrain the association combination of any two cement bags within a hyperedge. and Consistency of relative distance between adjacent frames: ,in The normalized spatial distance between the two cement bag targets in the current frame is defined as follows: the normalization is performed using the diagonal length of the current frame image as a reference. This is the normalized spatial distance from the previous frame; This is a motion vector consistency sub-item used to constrain the association combination of any two cement bags within the hyperedge. and The consistency of the motion vector in both direction and magnitude: ,in , These are the motion vectors of the two cement bag targets, calculated by the difference in center positions between adjacent frames; This is a group velocity distribution consistency sub-item used to constrain the association combination of any two cement bags within the hyperedge. and The velocity is consistent with the overall velocity distribution of all cement bag targets in the scene: ,in , Let the instantaneous velocities of the two cement bags be denoted. , This represents the mean and standard deviation of the velocity of all cement bag targets in the current frame.

6. The video AI intelligent packet counting method as described in claim 3, characterized in that, The differentiable hypergraph neural network in step S44 specifically includes: S441, Node Initialization: Initialize each node initial embedding vector Let it be the unary feature vector of its corresponding candidate association pair. ,in The normalized Euclidean distance between the predicted position of the cement bag trajectory and the detected position of the cement bag is defined as follows: the normalization refers to normalization based on the diagonal length of the current frame image. S442, Superedge Convolution Operation: In the... In a layered network, message passing from a node to a superedge is... ,in To find the aggregation function, The weight matrix is ​​learnable; message passing from hyperedge to node is... ,in It is a multilayer perceptron. The weight matrix is ​​a learnable matrix; S443, Iterative execution steps S442 total This process is repeated to obtain the final embedding vector for each node. ,in The preset number of layers in the hypergraph neural network; S444, The final embedded vector Input a fully connected classification layer, activate it with Softmax, and output each node. Association probability ,in and These are the learnable weight matrix and bias vector; S445, Based on the aforementioned association probability and preset association threshold ,when Time to determine the first The candidate associations are established; then the Hungarian algorithm is used to solve the problem so that each cement bag detection box is associated with at most one trajectory and each cement bag trajectory is associated with at most one detection box, thus obtaining the global optimal association result; the detection boxes that are successfully matched in the global optimal association result are used to update the Kalman filter state of the corresponding cement bag trajectory, the unmatched detection boxes are initialized as new trajectories, and the unmatched trajectories are marked as temporarily unassociated.

7. The video AI intelligent packet-spotting method as described in claim 2, characterized in that, In step S2, the target detection module uses the YOLO11-SCE scene conditional expert detection unit. The YOLO11-SCE detection unit replaces the C3k2 convolutional block of the P4 resolution branch of the standard YOLO11 with the scene conditional expert SCE module. The SCE module includes four industry-specific expert networks: normal single-pack expert, stacking and adhesion expert, dust occlusion expert, and uneven lighting expert, as well as a dual-scale gated routing network. The four expert networks are all built for the cement bag detection scenario. The dual-scale gated routing network dynamically selects and activates at least one expert network to extract cement bag features based on global scene features and local target features. Finally, the output feature maps of all activated experts are weighted and summed according to normalized routing weights to obtain the refined P4 feature map.

8. The video AI intelligent packet-spotting method as described in claim 7, characterized in that, The stacked adhesion expert network embeds a differentiable physical optimization layer based on the Poisson equation: this differentiable physical optimization layer transforms the depth estimation problem of stacked adhesion cement bales into the Poisson equation. The problem to be solved, in which For the depth field to be estimated, The source term is predicted from the input feature map; the differentiable physical optimization layer uses the differentiable conjugate gradient method to iteratively solve the Poisson equation during the forward propagation of the neural network to obtain the depth field. According to the depth field Local gradient magnitude generation of uncohesion weight map The deadhesion weight map The input feature map of the stacked adhesion expert network is multiplied element-wise before being fed into subsequent convolutional layers to suppress the feature response of the adhesion region and separate the stacked cement bags. Each iteration of the differentiable conjugate gradient method is a differentiable operation, and the gradient in the solution process can be propagated to the feature extraction layer of the stacked adhesion expert network through backpropagation, so that the feature extraction layer adaptively learns the cement bag feature representation that is conducive to solving the Poisson equation.

9. The video AI intelligent packet counting method as described in claim 7, characterized in that, The dust masking expert network embeds a differentiable transmittance estimation layer based on an atmospheric scattering model: this differentiable transmittance estimation layer models transmittance estimation as an atmospheric scattering model. Solve the inverse problem, where To observe the image, For fog-free images, Transmittance, The atmospheric light value is used as the atmospheric light value; the differentiable transmittance estimation layer first estimates the atmospheric light value through a lightweight subnetwork. and scene depth Then, based on the scene depth Calculate transmittance ,in The atmospheric scattering coefficient is preset; based on the transmittance and atmospheric light value Perform occlusion removal enhancement on the input feature map: ,in To be a small constant to prevent division by zero; The output feature map of the dust masking expert network is used to enhance the feature identifiability of cement bags under dust masking conditions. All parameters of the lightweight subnetwork are optimized end-to-end through backpropagation of the bidirectional counting-detection coupling loss function, enabling the transmittance estimation layer to adaptively learn the optimal transmittance estimate under the cement bag dust occlusion scenario.

10. A video AI intelligent ordering device, characterized in that: include: A processor and a storage medium; the processor loads and executes instructions and data in the storage medium to implement the video AI intelligent point-and-click method according to any one of claims 1 to 8.

Images