Warehouse material monitoring method and system based on multi-view spatial attention guidance

Abstract

This invention discloses a method and system for monitoring warehouse materials based on multi-view spatially guided attention. Addressing the problems of complex stacking and viewpoint occlusion, it constructs a processing framework consisting of four stages: feature extraction, structural modeling, text generation, and rule judgment. First, a cross-view Transformer mechanism is designed to spatially register and fuse images from multiple cameras, generating voxel feature representations in a unified coordinate system. Second, a structure perception module is constructed, fusing viewpoint consistency and category-guided attention to extract spatial arrangement relationships and semantic structures. Then, a structure-guided text generation network is proposed, based on an improved Transformer autoregressive decoder, to generate structured semantic text describing the location and stacking logic of materials. Finally, automatic compliance judgment and risk alerts are achieved through rule base comparison. This invention improves spatial understanding and semantic expression capabilities in multi-target occlusion scenarios, providing reliable support for intelligent monitoring in complex warehouse environments.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and intelligent monitoring technology, and relates to a method and system for monitoring warehouse materials. Specifically, it relates to a structured monitoring method and system for warehouse materials based on a multi-view spatial guidance attention mechanism, which can be applied to intelligent perception and anomaly alarm in scenarios such as power storage and hazardous materials supervision. Background Technology

[0002] With the rapid development of smart warehousing and the Industrial Internet of Things, traditional static image-based target detection and recognition methods have gradually revealed problems such as low accuracy and weak structural modeling capabilities in scenarios with severe spatial occlusion and multi-camera collaborative processing. Currently, most mainstream methods focus on single-frame image processing or single-view video analysis, making it difficult to achieve modeling of spatial structural relationships and globally consistent representation.

[0003] Significant progress has been made in object detection and semantic understanding, especially with the widespread application of general-purpose visual models such as YOLOv8 (Reference 1), DETR (Reference 2), and the Segment Anything Model (Reference 3) in various scenarios. However, these models primarily use bounding boxes as the modeling unit, lacking the ability to model the spatial layout, stacking relationships, and structural semantics between objects. For warehousing scenarios with clear structural constraints and placement specifications, such as those involving electrical materials or hazardous materials, relying solely on single-frame or single-view images is insufficient to accurately capture stacking logic and violation risks. On the other hand, multi-view collaborative perception, as an important trend in recent years, has gained attention in areas such as 3D reconstruction (Reference 4), behavior understanding (Reference 5), and multi-camera object tracking (Reference 6), but its application to structured semantic generation and rule-matching risk alerts remains a research gap.

[0004] Existing structure-aware methods such as MonoScene (Reference 7) and VoxelNeXt (Reference 8) attempt to reconstruct semantic voxels from single-view images, but they suffer from poor stability in occluded regions and blurred boundaries. Cross-View Attention mechanisms have been introduced into multi-view scene understanding tasks in recent years. Works such as BEVFormer (Reference 9) and PETR (Reference 10) improve spatial perception accuracy through cross-view feature interaction. Meanwhile, image-text generation models (such as BLIP-2 (Reference 11)) perform well in cross-modal representation learning, but their structure generation logic still relies on explicit annotations and lacks integration with 3D structures.

[0005] References:

[0006] Reference 1: Jocher G, et al. YOLOv8: Ultralytics.https: / / github.com / ultralytics / ultralytics, 2023.

[0007] Reference 2: Carion N, et al. "End-to-End Object Detection withTransformers." ECCV, 2020.

[0008] Reference 3: Kirillov A, et al. "Segment Anything." arXiv preprint arXiv:2304.02643, 2023.

[0009] Reference 4: Peng S, et al. "Neural Body: Implicit Neural Representationswith Structured Latent Codes for Novel View Synthesis of Dynamic Humans."CVPR, 2021.

[0010] Reference 5: Zhang Y, et al. "STAR: Spatio-Temporal Action Reasoning forMulti-Camera Action Recognition." ICCV, 2021.

[0011] Reference 6: Chen J, et al. "TrackFormer: Multi-Object Tracking withTransformers." CVPR, 2022.

[0012] Reference 7: Cao J, et al. "MonoScene: Monocular 3D Semantic SceneCompletion." CVPR, 2022.

[0013] Reference 8: Li J, et al. "VoxelNeXt: Next Generation VoxelNet for 3D ObjectDetection." CVPR, 2023.

[0014] Document 9: Li Y, et al. "BEVFormer: Learning Bird's-Eye-ViewRepresentation from Multi-Camera Images via Spatiotemporal Transformers." ECCV, 2022.

[0015] Document 10: Liu Y, et al. "PETR: Position Enhanced Transformer for 3DObject Detection." ECCV, 2022.

[0016] Document 11: Li J, et al. "BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models." ICML, 2023. Summary of the Invention

[0017] To address the spatial occlusion and complex stacking issues encountered in existing warehouse material monitoring technologies and to improve the automation and intelligence of material monitoring, this invention proposes a structured monitoring method and system for warehouse materials based on a multi-view spatially guided attention mechanism. Through multi-camera collaborative modeling, voxel-level structural fusion, and structure-guided text generation, it achieves the extraction of structurally consistent and semantically rich multi-view fusion representations from video streams, and combines a rule knowledge base for anomaly detection and alarm output.

[0018] The technical solution adopted by the present invention is: a warehouse material monitoring method based on multi-view spatially guided attention, characterized by comprising the following steps:

[0019] Step 1: Acquire multi-view images from different cameras, perform spatial registration and information fusion on the multi-view images, generate voxel feature representations in a unified coordinate system, and output multi-view feature reprojection vectors after mapping processing.

[0020] Step 2: Based on the spatial structure perception feature generation network, the multi-view feature reprojection vectors are fused with view consistency and category-guided attention to obtain a voxel joint vector with structural alignment and semantic focus. ;

[0021] Step 3: Based on the structure-guided text generation network, generate structured semantic text describing the location and stacking logic of materials;

[0022] Step 4: Perform fuzzy matching between the structured semantic text and the preset rule base to achieve automatic compliance judgment and intelligent risk warning output.

[0023] Preferably, in step 1, a Swing Transformer network is used to process the input image from each viewpoint. Perform independent encoding to obtain its two-dimensional encoded feature map. ,in N represents the number of cameras; H and W represent the image width and height respectively; R represents the real number field; and C represents the number of network channels. These represent the length and width of the encoded image, respectively.

[0024] Preferably, in step 1, the spatial registration and information fusion of multi-view images is performed based on the intrinsic parameters of each camera. With extrinsic matrix Construct perspective prompts;

[0025] ;

[0026] in, , The focal lengths of the camera in the horizontal and vertical directions. , Principal point coordinates; extrinsic parameter matrix This represents the transformation from the world coordinate system to the camera coordinate system, including the rotation matrix. Translation matrix For each spatial point Its projection position in the image coordinate system under the world coordinate system , ;in, This indicates that they are proportionally equal in homogeneous coordinates. These represent the horizontal and vertical pixel coordinates of the spatial point in the image coordinate system, respectively.

[0027] Encoded feature maps based on each viewpoint image and its corresponding perspective cues Based on the set of anchor voxels Location of each voxel Project the image onto the image planes of each viewpoint to obtain its image coordinates under the i-th camera:

[0028] ;

[0029] Where M is the preset number of anchor voxels, and j represents the index number of the voxel in the set, from the 1st to the Mth voxel;

[0030] Using bilinear interpolation from encoded feature maps Extract corresponding features voxel positions were obtained. A collection of observations from all perspectives; then embeddings using a voxel query. Using the query value and features from each perspective as the key and value, an attention mechanism is constructed, outputting a set of voxel features. ;

[0031] ;

[0032] in, Let represent the fused cross-view feature set of the i-th camera, and d represent the scaling factor dimension in the attention mechanism.

[0033] Preferably, in step 1, the voxel feature set is... Map back to image space to generate structure-aligned multi-view feature reprojection vectors;

[0034] Define interpolation kernel Represents the image coordinates u and the projection point The spatial correlation weighting function maps voxel features to a two-dimensional image plane:

[0035] ;

[0036] Wherein, the weight function satisfies This ensures that the features at each pixel location in the image are consistent with the local structure of the fusion result.

[0037] Preferably, in step 2, the spatial structure perception feature generation network includes a spatial voxel construction unit, a perspective consistency fusion unit, and a category-guided attention unit.

[0038] The spatial voxel construction unit is used to reproject multi-view feature vectors Projecting onto a unified 3D space and constructing a multi-view voxel-level joint representation; firstly, through a voxel feature stacking layer, the features from N views under each anchor voxel are concatenated into a feature block of the form [N,d], where d represents the scaling factor dimension in the attention mechanism; then, preliminary structural modeling is performed through three 3D convolutional extraction layers; after convolution, a BatchNorm3D normalization layer and a ReLU activation function are applied to improve feature robustness and non-linear expressive ability; next, a 1×1×1 convolutional layer is used to compress the number of channels and reduce redundancy; finally, multi-scale contextual information is extracted through an SPPF layer to form a fused encoded voxel feature set f1;

[0039] The viewpoint consistency fusion unit includes a 3×3×3 convolutional layer, a feature normalization layer LayerNorm, a ReLU activation function layer, a softmax layer, a full max layer, and a Fused Voxel unit connected in sequence. The output of the ReLU activation function layer is pixel-wise added to the output of the full max layer, and then pixel-wise added to the output of the full max layer to output the viewpoint consistency-enhanced voxel representation f2. The Fused Voxel unit consists of a ReLU activation function layer and a BatchNorm3D normalization layer, used to further enhance nonlinear feature representation and training stability.

[0040] The category-guided attention unit includes a category embedding extension layer (Classembedding), an upsampling layer (Upsample), a key-value mapping convolutional layer, a query embedding extension layer (Queryembedding), a matrix multiplication layer (MatMul), a scaling layer (Scale), a mask layer (Mask), and a softmax layer. The output of the 1×1×1 convolutional layer of the spatial voxel construction unit is processed by the Class embedding extension layer (Classembedding), the upsampling layer (Upsample), and the key-value mapping convolutional layer to output a value vector and a key vector. The f2 vector is processed by the query embedding extension layer (Queryembedding) to output a query vector, which is then processed by the matrix multiplication layer (MatMul) and sequentially input into the scaling layer (Scale), the mask layer (Mask), and the softmax layer. Finally, it is added pixel-by-pixel to the value vector to output a structurally aligned and semantically focused voxel joint representation. .

[0041] Preferably, the SPPF layer includes a sequentially connected initial convolutional layer, three parallel max pooling layers (first MaxPool2d layer, second MaxPool2d layer, and third MaxPool2d layer), a feature concatenation layer (Concat layer), and a fusion convolutional layer. The initial convolutional layer is used to compress the channel dimension of the input features. The three max pooling layers use the same kernel size (k=5) and stride (s=1) configuration, and extract multi-scale receptive fields from the initial convolutional output. The outputs of the three layers are concatenated with the initial convolutional output along the channel dimension, fused by the Concat layer, and then input into the fusion convolutional layer to further extract multi-scale contextual information.

[0042] Preferably, the full max layer is a per-view feature maximum response selection module, used to perform maximum value operation on the channel dimension of the feature response of each voxel position under multiple views; let the feature vector of the j-th anchor voxel under the i-th view be: Then the output of the full max layer is:

[0043] ;

[0044] The maximum activation feature vector is used for subsequent fusion with the ReLU layer output to enhance the responsiveness of the consistency region.

[0045] Preferably, in step 3, a structure-guided text generation network is used to further process the voxel joint vectors. [M,d], where M represents the length of the structure voxel sequence and d is the feature dimension of each voxel; first, it is input to the linear mapping layer Linear(d,d) for feature regularization and to generate key vectors, value vectors, and query vectors as cross-attention modules; then, it is added pixel by pixel to the position-aware input sequence and input to the multi-layer decoder stack layer. Each layer consists of a masked multi-head attention mechanism layer, a first addition and normalization processing layer, a multi-head attention mechanism layer, a second addition and normalization processing layer, and a feedforward network layer; the key vector and value vector are processed by the masked multi-head attention mechanism layer and the first addition and normalization processing layer, and then input together with the query vector into the multi-head attention mechanism layer, the second addition and normalization processing layer, and the feedforward network layer; the output of the multi-layer decoder stack layer is processed by two linear layers and a linear projection layer Softmax in sequence to output a probability distribution, and the word with the highest probability is selected as the currently generated token, thus completing the autoregressive generation of structured semantic text in sequence;

[0046] The position-aware input sequence is a sequence of tokens generated during text generation that is mapped to a sequence of the form [T, e] through a word embedding layer (dictionary size V, embedding dimension e), where T is the current number of tokens; the embedded sequence is further augmented with a position encoding vector to form the position-aware input sequence.

[0047] Preferably, in step 4, the preset rule base is a structured knowledge base built for warehouse safety specifications, which includes various rule items such as stacking specifications, material attribute rules, safety distance requirements, and risk identification rules.

[0048] The technical solution adopted by the system of this invention is: a warehouse material monitoring method system based on multi-view spatially guided attention, comprising:

[0049] One or more processors;

[0050] A storage device for storing one or more programs, which, when executed by one or more processors, enable the one or more processors to implement the warehouse material monitoring method based on multi-view spatial guided attention.

[0051] Compared with the prior art, the present invention has the following beneficial effects:

[0052] (1) This invention proposes for the first time a warehouse material monitoring method based on a multi-view spatial guided attention mechanism. It integrates video information from multiple cameras and effectively improves the material identification and structural understanding capabilities in occluded scenarios by combining spatial structure perception and semantic modeling. This method is significantly superior to traditional single-view or static image analysis methods.

[0053] (2) The present invention designs a cross-view Transformer fusion mechanism and a spatial voxel consistency modeling method, which can construct a multi-view voxel-level feature representation with consistent structure and complete semantics, effectively alleviate the redundancy conflict problem of multi-source images in spatial alignment and semantic fusion, and has good generalization and scalability.

[0054] (3) This invention proposes a structure-guided text generation decoder to realize the automatic transformation from fused features to structured semantic description. Combined with industry knowledge base, it can realize the automatic comparison and alarm of warehouse stacking specifications, material attributes and risk factors, and has high interpretability and engineering implementation value.

[0055] (4) The present invention can flexibly adapt to different numbers and locations of camera deployment schemes, and is suitable for intelligent monitoring and compliance auditing needs in various warehousing scenarios such as power, hazardous chemicals, and military industry, and has broad application prospects. Attached Figure Description

[0056] The technical solutions of the present invention will be further illustrated below using embodiments and specific implementation methods. In addition, some accompanying drawings are used in the description of the technical solutions. Those skilled in the art can obtain other drawings and the intent of the present invention from these drawings without any creative effort.

[0057] Figure 1 This is a flowchart of a method according to an embodiment of the present invention;

[0058] Figure 2 This is a network structure diagram for generating spatial structure perception features according to an embodiment of the present invention.

[0059] Figure 3 This is a structural diagram of the text generation network in an embodiment of the present invention. Detailed Implementation

[0060] To facilitate understanding and implementation of the present invention by those skilled in the art, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0061] While existing warehouse management systems have partially adopted video analytics and object detection technologies to enhance the digital acquisition of material status, they still face various technical challenges in practical applications. Especially in complex environments with multiple materials, categories, and levels of stacking, traditional methods struggle to accurately identify the structural relationships between materials and lack intelligent monitoring capabilities that are deeply integrated with management rules.

[0062] This embodiment mainly addresses the following two types of technical problems existing in the prior art;

[0063] Technical Issue 1: Limited single-view analysis capabilities make it difficult to handle complex scenes with stacked occlusions;

[0064] Most existing intelligent monitoring methods for warehouse materials rely on a single camera input, only acquiring partial information from a fixed angle. When materials are diverse, stacked in complex layers, partially obscured, or placed in corners, a single perspective easily leads to incomplete object detection, incorrect category recognition, and failed posture assessment. Furthermore, camera blind spots and obstructed areas may remain unmonitored for extended periods, posing a threat to warehouse safety management. Therefore, the lack of a multi-view joint reasoning mechanism is a significant bottleneck preventing existing systems from adapting to real-world, complex scenarios.

[0065] Technical Issue 2: Insufficient ability to model the spatiotemporal relationships of materials, making it difficult to generate structured descriptions;

[0066] Warehouse management focuses not only on the types and quantities of materials, but more importantly on their spatial relationships (such as stacking vertically, side-by-side, and sequential order) and temporal evolution (such as handling changes and stack updates). However, existing technologies mainly focus on static target detection or simple motion trajectory analysis, lacking fine-grained modeling of the spatiotemporal arrangement relationships between materials. This makes it difficult to express structured semantic information such as "material A is stacked on top of B" or "materials C and D maintain a 0.5-meter safe distance." Furthermore, it lacks the ability to perform semantic-level comparisons with warehouse management rules, hindering accurate intelligent alerts and automatic recording.

[0067] Please see Figure 1 This embodiment provides a method for monitoring warehouse materials based on multi-view spatially guided attention, which includes the following steps:

[0068] Step 1: Acquire multiple RGB video images covering the warehouse area, and extract features from each image using an image encoder to generate a two-dimensional visual feature map for each viewpoint; at the same time, generate viewpoint cue information aligned with the feature map based on the geometric parameters of each camera, fuse the feature maps from different viewpoints through a cross-viewpoint Transformer mechanism to generate fused structural features, and then map them uniformly to the spatial coordinate system through a feature reprojection module.

[0069] This embodiment employs fixed multi-view cameras at multiple key locations within the warehouse to continuously monitor the warehouse environment, capturing RGB video streams covering the entire warehouse area. A time synchronization module performs frame-level time alignment of video frames from different perspectives to ensure consistent time reference for subsequent multi-view fusion operations.

[0070] In one implementation, a Swing Transformer network is used to process the input image at each viewpoint. Perform independent encoding to obtain its two-dimensional encoded feature map. ,in N represents the number of cameras; H and W represent the image width and height respectively; R represents the real number field; and C represents the number of network channels. These represent the length and width of the encoded image, respectively.

[0071] In one implementation, the spatial registration and information fusion of multi-view images is based on the intrinsic parameters of each camera. With extrinsic matrix Construct perspective prompts;

[0072] ;

[0073] in, , The focal lengths of the camera in the horizontal and vertical directions. , Principal point coordinates; extrinsic parameter matrix This represents the transformation from the world coordinate system to the camera coordinate system, including the rotation matrix. Translation matrix For each spatial point Its projection position in the image coordinate system under the world coordinate system , ;in, This indicates that they are proportionally equal in homogeneous coordinates. These represent the horizontal and vertical pixel coordinates of the spatial point in the image coordinate system, respectively.

[0074] Encoded feature maps based on each viewpoint image and its corresponding perspective cues Based on the set of anchor voxels Location of each voxel Project the image onto the image planes of each viewpoint to obtain its image coordinates under the i-th camera:

[0075] ;

[0076] Where M is the preset number of anchor voxels, and j represents the index number of the voxel in the set, from the 1st to the Mth voxel;

[0077] Using bilinear interpolation from encoded feature maps Extract corresponding features voxel positions were obtained. A collection of observations from all perspectives; then embeddings using a voxel query. Using the query value and features from each perspective as the key and value, an attention mechanism is constructed, outputting a set of voxel features. ;

[0078] ;

[0079] in, Let represent the fused cross-view feature set of the i-th camera, and d represent the scaling factor dimension in the attention mechanism.

[0080] Finally, the voxel feature set Map back to image space to generate structure-aligned multi-view feature reprojection vectors;

[0081] Define interpolation kernel Represents the image coordinates u and the projection point The spatial correlation weighting function maps voxel features to a two-dimensional image plane:

[0082] ;

[0083] Wherein, the weight function satisfies This ensures that the features at each pixel location in the image are consistent with the local structure of the fusion result.

[0084] Step 2: Based on the fused spatial features, a spatial structure-aware feature generation network is used to project the surface features of materials from different perspectives into a unified 3D voxel space; the projection consistency of voxel points under different perspectives is calculated, and voxel features are fused based on consistency weights; the feature representation of structurally related regions is strengthened through a category-guided attention mechanism; finally, a structure-preserving encoder is used to output a spatial structure-aware feature vector. ;

[0085] Please see Figure 2 In one embodiment, the spatial structure perception feature generation network includes a spatial voxel construction unit, a perspective consistency fusion unit, and a category-guided attention unit.

[0086] The spatial voxel construction unit is used to reproject multi-view feature vectors Projecting onto a unified 3D space and constructing a multi-view voxel-level joint representation; firstly, through a voxel feature stacking layer, the features from N views under each anchor voxel are concatenated into a feature block of the form [N,d], where d represents the scaling factor dimension in the attention mechanism; then, preliminary structural modeling is performed through three 3D convolutional extraction layers; after convolution, a BatchNorm3D normalization layer and a ReLU activation function are applied to improve feature robustness and non-linear expressive ability; next, a 1×1×1 channel compression convolutional layer is used to compress the number of channels and reduce redundancy; finally, multi-scale contextual information is extracted through an SPPF layer to form a fused encoded voxel feature set f1;

[0087] The viewpoint consistency fusion unit includes a 3×3×3 convolutional layer, a feature normalization layer LayerNorm, a ReLU activation function layer, a softmax layer, a full max layer, and a Fused Voxel unit connected in sequence. The output of the ReLU activation function layer is pixel-wise added to the output of the full max layer, and then pixel-wise added to the output of the full max layer to output the viewpoint consistency-enhanced voxel representation f2. The Fused Voxel unit consists of a ReLU activation function layer and a BatchNorm3D normalization layer, used to further enhance nonlinear feature representation and training stability.

[0088] The category-guided attention unit includes a category embedding extension layer (Classembedding), an upsampling layer (Upsample), a key-value mapping convolutional layer, a query embedding extension layer (Queryembedding), a matrix multiplication layer (MatMul), a scaling layer (Scale), a mask layer (Mask), and a softmax layer. The output of the 1×1×1 channel compressed convolutional layer of the spatial voxel construction unit is processed by the category embedding extension layer (Classembedding), the upsampling layer (Upsample), and the key-value mapping convolutional layer to output a value vector and a key vector. The f2 vector is processed by the query embedding extension layer (Queryembedding) to output a query vector, which is then processed with the key vector by the matrix multiplication layer (MatMul) and sequentially input into the scaling layer (Scale), the mask layer (Mask), and the softmax layer. Finally, it is added pixel-by-pixel to the value vector to output a structurally aligned and semantically focused voxel joint representation. .

[0089] In one implementation, the SPPF layer includes a sequentially connected initial convolutional layer, three parallel max pooling layers (a first MaxPool2d layer, a second MaxPool2d layer, and a third MaxPool2d layer), a feature concatenation layer (Concat), and a fusion convolutional layer. The initial convolutional layer is used to compress the channel dimension of the input features. The three max pooling layers use the same kernel size (k=5) and stride (s=1) configuration, and extract multi-scale receptive fields from the initial convolutional output. The outputs of the three layers are concatenated with the initial convolutional output along the channel dimension, fused by the Concat layer, and then input into the fusion convolutional layer to further extract multi-scale contextual information.

[0090] In one implementation, the full max layer is a per-view feature maximum response selection module, used to perform a maximum value operation on the channel dimension of the feature response of each voxel position under multiple views; let the feature vector of the j-th anchorvoxel under the i-th view be... Then the output of the full max layer is:

[0091] ;

[0092] The maximum activation feature vector is used for subsequent fusion with the ReLU layer output to enhance the responsiveness of the consistency region.

[0093] Step 3: Input the structure-aware vector into the structure-guided text generation network to aggregate spatial semantics from different perspectives and form a unified structural semantic representation; then input this representation into the structure-guided text decoder to generate a structured semantic text description reflecting the material category, stacking order and spatial layout through the Transformer autoregressive decoding mechanism.

[0094] Please see Figure 3 In one implementation, the structure guides the text generation network for further processing of voxel joint vectors. [M,d], where M represents the length of the structure voxel sequence and d is the feature dimension of each voxel; first, it is input to the linear mapping layer Linear(d,d) for feature regularization and to generate key vectors, value vectors, and query vectors as cross-attention modules; then, it is added pixel by pixel to the position-aware input sequence and input to the multi-layer decoder stack layer. Each layer consists of a masked multi-head attention mechanism layer, a first addition and normalization processing layer, a multi-head attention mechanism layer, a second addition and normalization processing layer, and a feedforward network layer; the key vector and value vector are processed by the masked multi-head attention mechanism layer and the first addition and normalization processing layer, and then input together with the query vector into the multi-head attention mechanism layer, the second addition and normalization processing layer, and the feedforward network layer; the output of the multi-layer decoder stack layer is processed by two linear layers and a linear projection layer Softmax in sequence to output a probability distribution, and the word with the highest probability is selected as the currently generated token, thus completing the autoregressive generation of structured semantic text in sequence;

[0095] The position-aware input sequence is a sequence of tokens generated during text generation that is mapped to a sequence of the form [T, e] through a word embedding layer (dictionary size V, embedding dimension e), where T is the current number of tokens; the embedded sequence is further augmented with a position encoding vector to form the position-aware input sequence.

[0096] Step 4: Input the structured semantic text into the rule matching alarm module, load and parse the preset storage specifications, determine whether the current structure is compliant through fuzzy matching, and output alarm information when abnormal situations such as illegal stacking or occlusion risk are detected.

[0097] In one implementation, the preset rule base is a structured knowledge base built around warehouse safety specifications, containing various types of rule entries such as stacking specifications, material attribute rules, safety distance requirements, and risk identification rules; for example:

[0098]

[0099] The following comparative experiments further illustrate the method of this invention. During the testing phase, each scene in the test set is treated as an independent task, with the corresponding structured description as the target semantics. Other scenes constitute the comparison set. The model with the best performance during training is used for inference, outputting structured semantic text and performing rule comparison tasks to obtain the final violation detection results. Evaluation metrics include BLEU-4 text matching, mAP of the structured description, violation detection recall, and F1-score, used to measure the system's performance in terms of structural expression accuracy and risk judgment ability.

[0100] This experiment was conducted on the Warehouse-MVS dataset, a multi-view dataset for warehouse materials. This dataset is based on real-world power storage scenarios and includes common stacked materials such as cable reels, switch cabinets, electrical boxes, and fire extinguishers. It covers five different locations with fixed camera angles, collecting a total of 12,000 images. Each scenario corresponds to a set of structured spatial descriptions and compliance judgment labels.

[0101] Since this dataset was built in-house for this project and does not include publicly available standard partitioning, it was evenly divided into training and testing sets in the experiment. Specifically, 70% of the subframes in each scene were used as training data, and the remaining 30% were used as testing data. This ensured that the scene structure in the testing set did not appear in the training set, avoiding data leakage. During the testing phase, each scene in the testing set was treated as a query example, and the structured text was output and matched against a rule base to determine whether an alarm event was triggered.

[0102] Images in the Warehouse-MVS dataset were uniformly resized to 512×512. In the training data, input images were randomly rotated ±10 degrees for enhancement, with a 50% probability of adjusting brightness and contrast while preserving color consistency. The visual encoder was initialized using ImageNet-1K pre-trained parameters, with a patch size of 4 during the multi-view encoding stage. In the structured text generation process, a standard Transformer autoregressive decoding structure was employed, with a maximum sequence length of 64. In the cross-attention module, structure voxel features served as the key and value, and the generated token sequence served as the query, decoded using a masked multi-head attention mechanism. The loss function was cross-entropy loss.

[0103] During training, the AdamW optimizer was used with an initial learning rate of 0.0001, using cosine learning rate decay, and 100 training iterations. The training batch size was set to 16, divided by scene, with each batch containing 4 scenes and 4 frames per scene. During the testing phase, only the final structure vectors were used to generate semantic text for compliance judgment.

[0104] To verify the effectiveness of this invention, this experiment compares the method of this invention with a variety of multi-view scene perception methods, mainly including: (1) YOLOv8 + manual rule template (single-frame recognition + string matching). (2) MonoScene: constructing voxel semantic representation based on single-view images. (3) BEVFormer: constructing scene understanding features based on BEV space representation. (4) This invention was tested on the Warehouse-MVS dataset, and the results are shown in Table 1.

[0105] Table 1. Comparison of the performance of different methods in structured representation and violation detection on the Warehouse-MVS dataset.

[0106]

[0107] As shown in Table 1, compared with typical multi-view structure perception methods in recent years, the method proposed in this invention achieves significant performance improvements in both structured semantic representation and violation identification. The YOLOv8 method can only perform local object detection and generate text through preset rules, making it difficult to reconstruct the real spatial structure; MonoScene cannot handle occluded regions and has insufficient multi-view alignment capabilities; BEVFormer constructs a BEV space but lacks semantic representation capabilities. In BLEU-4 structured text evaluation, this invention improves by 7 percentage points and mAP by 8 percentage points compared to BEVFormer; in the violation alarm task, Recall is improved by 13% and F1-score by 12%. These results demonstrate that this invention has stronger structural reasoning capabilities and alarm accuracy in real-world power storage scenarios.

[0108] The experimental results fully verify the spatial modeling capability, structural semantic expression capability, and effectiveness of the present invention in compliance monitoring in multi-view occlusion and complex stacking scenarios.

[0109] The multi-view spatial perception and structured semantic modeling method proposed in this invention is not only applicable to violation detection and risk identification in complex warehousing environments, but can also be widely embedded in the next-generation intelligent material supply chain system as a core perception and structural understanding module of the intelligent warehousing system. By accurately modeling the location, stacking logic, and spatial relationships of materials in multi-view images, the system can automatically generate structured semantic text with category, layout, and compliance labels, achieving a deep expression from perception to understanding. This standardized semantic information can be seamlessly connected to the upper-level warehousing management platform and scheduling execution system to construct an intelligent collaborative closed loop of "multi-view perception—structure recognition—rule matching—process linkage". Based on this technical framework, the system can support dynamic perception of material status, automatic auditing of compliance, intelligent planning of sorting paths, and early warning of abnormal risks, significantly improving the automation level of warehousing operations and the response efficiency of material flow, providing key support capabilities for building a flexible, efficient, and self-driven intelligent supply chain.

[0110] It should be understood that the above experiments are only a part of the performance verification examples of the technical solution provided by this invention, intended to illustrate the advantages of this invention in multi-view spatial structure modeling and violation risk identification, and not to limit the implementation methods or applicable scenarios of this invention. The method and system of this invention can also be adapted to other multi-camera heterogeneous monitoring environments, other structured semantic generation tasks, or industry knowledge rule base construction applications.

[0111] Furthermore, the various functional modules and technical units provided by this invention (including a multi-view spatial fusion module, a structure-guided text generation network, rule matching and alarm mechanisms, etc.) can be flexibly combined or replaced according to actual needs to form various modified implementation methods. The connection order, network structure parameters, input and output formats, etc., between the modules can also be adjusted to adapt to deployment requirements or resource constraints in different industrial scenarios. As long as these adjustments and modifications still satisfy the core technical ideas defined in the claims of this invention, they should be considered to fall within the protection scope of this invention. If any modifications, equivalent substitutions, or functional enhancements are made by those skilled in the art without departing from the substantial innovative content of this invention, they should also be considered to fall within the protection scope of the claims of this invention.

[0112] It should be understood that the embodiments described above are only some, not all, of the embodiments of the present invention. Furthermore, the technical features of the various embodiments or individual embodiments provided by the present invention can be arbitrarily combined to form feasible technical solutions. Such combinations are not constrained by the order of steps and / or structural composition patterns, but must be based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by the present invention.

[0113] It should be understood that the above description of the preferred embodiments is quite detailed, but it should not be considered as a limitation on the scope of protection of this invention. Those skilled in the art, under the guidance of this invention, can make substitutions or modifications without departing from the scope of protection of the claims of this invention, and all such substitutions or modifications fall within the scope of protection of this invention. The scope of protection of this invention should be determined by the appended claims.

Claims

1. A warehouse material monitoring method based on multi-view spatial attention guidance, characterized in that, Includes the following steps: Step 1: Acquire multi-view images from different cameras, perform spatial registration and information fusion on the multi-view images, generate voxel feature representations in a unified coordinate system, and output multi-view feature reprojection vectors after mapping processing. Step 2: Based on the spatial structure perception feature generation network, the multi-view feature re-projection vector is fused with view consistency and class guided attention to obtain a structure aligned and semantic focused voxel joint vector ; The spatial structure perception feature generation network includes a spatial voxel construction unit, a perspective consistency fusion unit, and a category-guided attention unit. The spatial voxel construction unit is used to reproject multi-view feature vectors Projecting onto a unified 3D space and constructing a multi-view voxel-level joint representation; firstly, through a voxel feature stacking layer, the features from N views under each anchor voxel are concatenated into a feature block of the form [N,d], where d represents the scaling factor dimension in the attention mechanism; then, preliminary structural modeling is performed through three 3D convolutional extraction layers; after convolution, a BatchNorm3D normalization layer and a ReLU activation function are applied to improve feature robustness and non-linear expressive ability; next, a 1×1×1 convolutional layer is used to compress the number of channels and reduce redundancy; finally, multi-scale contextual information is extracted through an SPPF layer to form a fused encoded voxel feature set f1; The viewpoint consistency fusion unit includes a 3×3×3 convolutional layer, a feature normalization layer LayerNorm, a ReLU activation function layer, a softmax layer, a full max layer, and a Fused Voxel unit connected in sequence. The outputs of the ReLU activation function layer and the full max layer are added pixel by pixel to output a voxel representation f2 with enhanced viewpoint consistency. The Fused Voxel unit consists of a ReLU activation function layer and a BatchNorm3D normalization layer, which is used to further enhance nonlinear feature representation and training stability. The category-guided attention unit includes a category embedding extension layer (Classembedding), an upsampling layer (Upsample), a key-value mapping convolutional layer, a query embedding extension layer (Queryembedding), a matrix multiplication layer (MatMul), a scaling layer (Scale), a mask layer (Mask), and a softmax layer. The output of the 1×1×1 convolutional layer of the spatial voxel construction unit is processed by the Classembedding, Upsample, and key-value mapping convolutional layers to output a Value vector and a Key vector. The f2 vector is processed by the Queryembedding layer to output a Query vector, which is then processed by the MatMul layer and sequentially input into the Scale, Mask, and softmax layers. Finally, it is added element-wise with the Value vector to output a structurally aligned and semantically focused voxel joint representation. ; Step 3: Based on the structure-guided text generation network, generate structured semantic text describing the location and stacking logic of materials; The structure guides the text generation network for further processing of voxel joint vectors. [M,d], where M represents the length of the structure voxel sequence and d is the feature dimension of each voxel; first, it is input to the linear mapping layer Linear(d,d) for feature regularization and to generate key vectors, value vectors, and query vectors as cross-attention modules; then, it is added pixel by pixel to the position-aware input sequence and input to the multi-layer decoder stack layer. Each layer consists of a masked multi-head attention mechanism layer, a first addition and normalization processing layer, a multi-head attention mechanism layer, a second addition and normalization processing layer, and a feedforward network layer; the key vector and value vector are processed by the masked multi-head attention mechanism layer and the first addition and normalization processing layer, and then input together with the query vector into the multi-head attention mechanism layer, the second addition and normalization processing layer, and the feedforward network layer; the output of the multi-layer decoder stack layer is processed by two linear layers and a linear projection layer Softmax in sequence to output a probability distribution, and the word with the highest probability is selected as the currently generated token, thus completing the autoregressive generation of structured semantic text in sequence; The position-aware input sequence is a sequence of tokens generated during text generation that is mapped to a sequence of the form [T, e] through a word embedding layer (dictionary size V, embedding dimension e), where T is the current number of tokens; the embedded sequence is further augmented with a position encoding vector to form the position-aware input sequence. Step 4: Perform fuzzy matching between the structured semantic text and the preset rule base to achieve automatic compliance judgment and intelligent risk warning output.

2. The warehouse material monitoring method based on multi-view spatially guided attention as described in claim 1, characterized in that: In step 1, the Swing Transformer network is used to process the input image from each viewpoint. Perform independent encoding to obtain its two-dimensional encoded feature map. ,in N represents the number of cameras; H and W represent the original image's width and height, respectively; R represents the real number field; and C represents the number of network channels. These represent the length and width of the encoded image, respectively.

3. The warehouse material monitoring method based on multi-view spatially guided attention as described in claim 2, characterized in that: In step 1, the spatial registration and information fusion of multi-view images are based on the intrinsic parameters of each camera. With extrinsic matrix Construct perspective prompts; ； in, , The focal lengths of the camera in the horizontal and vertical directions. , Principal point coordinates; extrinsic parameter matrix This represents the transformation from the world coordinate system to the camera coordinate system, including the rotation matrix. Translation matrix For each spatial point Its projection position in the image coordinate system under the world coordinate system , ;in, This indicates that they are proportionally equal in homogeneous coordinates. These represent the horizontal and vertical pixel coordinates of the spatial point in the image coordinate system, respectively. Encoded feature maps based on each viewpoint image and its corresponding perspective cues Based on the set of anchor voxels Location of each voxel Project the image onto the image planes of each viewpoint to obtain its image coordinates under the i-th camera: ； Where M is the preset number of anchor voxels, and j represents the index number of the voxel in the set, from the 1st to the Mth voxel; Using bilinear interpolation from encoded feature maps Extract corresponding features voxel positions were obtained. A collection of observations from all perspectives; then embeddings using a voxel query. Using the query value and features from each perspective as the key and value, an attention mechanism is constructed, outputting a set of voxel features. ; ； in, Let represent the fused cross-view feature set of the i-th camera, and d represent the scaling factor dimension in the attention mechanism.

4. The warehouse material monitoring method based on multi-view spatially guided attention as described in claim 3, characterized in that: In step 1, the voxel feature set is... Map back to image space to generate structure-aligned multi-view feature reprojection vectors; Define interpolation kernel Represents the image coordinates u and the projection point The spatial correlation weighting function maps voxel features to a two-dimensional image plane: ； Wherein, the weight function satisfies This ensures that the features at each pixel location in the image are consistent with the local structure of the fusion result.

5. The warehouse material monitoring method based on multi-view spatially guided attention as described in claim 1, characterized in that: The SPPF layer includes a sequentially connected initial convolutional layer, three parallel max pooling layers, a feature concatenation layer (Concat), and a fusion convolutional layer. The initial convolutional layer is used to compress the channel dimension of the input features. The three max pooling layers use the same kernel size and stride configuration and extract multi-scale receptive fields from the output of the initial convolution. The outputs of the three layers are concatenated with the output of the initial convolution along the channel dimension, fused by the Concat layer, and then input into the fusion convolutional layer to further extract multi-scale contextual information.

6. The warehouse material monitoring method based on multi-view spatially guided attention according to claim 1, characterized in that: The full max layer is a per-view feature maximum response selection module, used to perform maximum value operation on the channel dimension of the feature response of each voxel position under multiple views; let the feature vector of the j-th anchor voxel under the i-th view be: Then the output of the full max layer is: ； Maximum Activation Feature Vector This is used for subsequent fusion with the ReLU layer output to enhance the responsiveness of the consistent region.

7. The warehouse material monitoring method based on multi-view spatially guided attention according to any one of claims 1-6, characterized in that: In step 4, the preset rule base is a structured knowledge base built for warehouse safety specifications, which includes stacking specifications, material attribute rules, safety distance requirements, and risk identification rules.

8. A warehouse material monitoring method system based on multi-view spatially guided attention, characterized in that, include: One or more processors; A storage device for storing one or more programs, which, when executed by one or more processors, cause the one or more processors to implement the warehouse material monitoring method based on multi-view spatially guided attention as described in any one of claims 1 to 7.

Images