A warehouse inventory method based on multi-modal perception technology, a robot and a warehouse system

Abstract

The application discloses a warehouse inventory method based on multi-modal perception technology, a robot and a warehouse system, belongs to the technical field of warehouse management and intelligent perception fusion, collects multi-modal data such as visual images, spatial depth, radio frequency perception and infrared temperature, constructs image feature vectors, three-dimensional point cloud models, radio frequency response matrices and temperature maps; a multi-channel fusion network based on an attention mechanism is used to generate a fusion recognition vector, and the modal weight is dynamically adjusted; an article space distribution map is constructed, and a missing area of perception is marked; low-confidence area data is automatically completed based on a priority scheduling algorithm; the original fusion result and the completion result are jointly verified to form a final inventory list; if the confidence of an article is lower than a warning threshold for a plurality of times in succession, an abnormal alarm is triggered, and a traceable perception sequence is generated; the method is suitable for high-precision inventory tasks in complex warehouse scenes, has the advantages of strong recognition robustness, intelligent completion mechanism and traceable abnormality.

Description

Technical Field

[0001] This invention relates to the field of warehouse management and intelligent sensing fusion technology, specifically to a warehouse inventory method, robot, and warehouse system based on multimodal sensing technology. Background Technology

[0002] Warehouse inventory counting is a core component of modern logistics management, and its results directly impact a company's assessment of inventory status, material turnover, and supply chain execution. Most current inventory counting methods rely on single sensing methods, such as RFID tag scanning or 2D image recognition systems. However, these systems face challenges in real-world scenarios, such as obstruction, lost tags, changes in lighting, or stacked items, leading to decreased recognition accuracy and increased false recognition rates.

[0003] While some existing technologies attempt to improve accuracy by deploying multiple sensing methods in parallel, their fusion methods are mostly simple weighting at the result layer or logical judgment based on confidence levels. They lack dynamic modeling mechanisms for potential contradictions between multimodal data, leading to frequent omissions or false positives at key inventory points. Furthermore, traditional systems often rely on repeated sensing or manual intervention for sensing areas with insufficient confidence, resulting in low response efficiency.

[0004] Therefore, there is an urgent need for an intelligent inventory method that integrates multimodal sensing data and has the capabilities of adaptive conflict handling, anomaly identification and completion, and dynamic feedback adjustment. This method can automatically identify the status of items in environments with complex stacking and obstruction interference, and proactively initiate a reconstruction sensing task in the event of conflicts or missing items, thereby achieving stable, accurate, and highly robust inventory management of the storage environment. Summary of the Invention

[0005] The purpose of this invention is to provide a warehouse inventory method, robot, and warehouse system based on multimodal perception technology to address the shortcomings of the prior art.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a warehouse inventory counting method based on multimodal sensing technology, comprising:

[0007] Collect multimodal sensing data within the storage area, including visual image data, spatial depth information, radio frequency sensing data, and infrared temperature feature data;

[0008] Feature extraction was performed on the multimodal sensing data to construct a multimodal feature set including image feature vectors, three-dimensional spatial point cloud models, radio frequency response matrices, and temperature distribution maps;

[0009] By utilizing the constructed multimodal feature set, a multi-channel fusion network based on an attention mechanism is used to generate a fusion recognition vector for the target item, and the weight coefficients of each modality are dynamically adjusted based on a confidence inference mechanism during the fusion process.

[0010] A spatial distribution map of items is constructed based on the fused recognition vectors, and a perception missing label is generated for target items in the recognition results with a confidence level lower than the first threshold.

[0011] For regions with missing perception markers, a dynamic priority scheduling algorithm is used to determine the modal channels and angle regions that need to be filled in first, thereby achieving automatic completion of low-confidence regions.

[0012] The fusion recognition results and the completed area are jointly verified to form the final inventory list, including the target item ID, spatial location, quantity, fusion confidence value and completion mark status;

[0013] If an item's fusion confidence level falls below the second warning threshold for S consecutive times, the anomaly alarm module is triggered, and a traceable perception data sequence is generated for subsequent manual verification.

[0014] Preferably, constructing an object spatial distribution map and generating perceptual missing markers includes:

[0015] The fused recognition vector is input into a heterogeneous feature tensor reconstruction network to construct a fused tensor set. The network is used to perform semantic layer weighted fusion of visual features, depth features and radio frequency response data.

[0016] The fused tensor set is mapped to a three-dimensional voxel space to construct a spatial distribution map of items with time labels, and a graph attention clustering algorithm is introduced to classify confidence boundaries.

[0017] For voxel units with confidence levels below the first threshold, the difference between their fusion feature residuals and the environmental baseline is calculated. When the residual magnitude exceeds a set perturbation threshold, a corresponding perception missing label is generated and marked in the atlas.

[0018] Preferably, calculating the fusion feature residual includes: comparing the original feature vector of the voxel unit in the multimodal fusion tensor with its historical fusion vector in the previous period in the temporal feature buffer using cosine similarity, calculating the difference in the residual vector, and denoting it as the fusion feature residual;

[0019] The calculation of environmental baseline difference includes: extracting the set of environmental baseline features in the local area to which the voxel unit belongs; constructing an environmental response function through the regional temperature and humidity map and the electromagnetic interference raster matrix; projecting the current fused vector into the environmental response function space and measuring the Euclidean distance with the baseline vector to obtain the magnitude of environmental difference, which is denoted as environmental baseline difference.

[0020] Preferably, the automatic completion of regions with perceptual missing markers includes:

[0021] Multimodal feature loss maps of the perception-deficient regions are extracted, and a modal angle cross-mapping table (MAM) is constructed. The MAM is used to record the response integrity and historical misjudgment rate of different modal channels under various observation angles.

[0022] The MAM is embedded as input into a graph-structured priority scheduling network, and a dynamic priority scoring matrix P is generated by combining the current environmental perturbation matrix and the boundary ambiguity map. The scoring matrix is ​​used to determine the optimal combination sequence of modal channels and angular regions in the completion task.

[0023] Based on the scoring matrix P, a completion perception task is triggered, and corresponding perception nodes are scheduled to reconstruct the multimodal input of the low-confidence region. The feature enhancement module performs fusion stabilization processing to generate a completed perception vector to replace the original missing region content.

[0024] Preferably, the task of triggering the completion perception based on the scoring matrix P includes:

[0025] Based on the priority combination sequence in the scoring matrix P, the movable sensing node is assigned to the missing region and instructed to collect multimodal raw data within a specified angular trajectory range, including structured light depth map, millimeter wave reflectance spectrum and multispectral image information.

[0026] Based on the collected multimodal input data, the channel response is automatically activated or suppressed based on the modal complementarity score, and the residual information between different modes is fused and iteratively optimized.

[0027] The enhanced and fused perception vector is output, and it is replaced to the missing region position through a spatial index mapping mechanism aligned with the original perception tensor to complete the completion and update operation. At the same time, the replacement mark is synchronously written into the global spatial distribution map.

[0028] Preferably, the joint verification of the fused recognition result and the completed region to form the final inventory list includes:

[0029] Extract the spatial index region covered by the original fusion recognition result and the completed perception vector, construct the index consistency map, perform vector alignment and confidence intersection calculation on the overlapping region, and generate the consistency score matrix Q;

[0030] For regions with low consistency in the scoring matrix Q, the original fusion result or the completed vector is dynamically determined as the final data benchmark by combining the residual information of the fusion source tensor and the trajectory of modal weight changes during the completion process.

[0031] Write the jointly verified inventory vectors into the final inventory list.

[0032] Preferably, the dynamic determination of using the original fusion result or the completed vector as the final data benchmark includes:

[0033] For the original fusion tensor and the completed sensing vector that cover the same spatial index position, the corresponding multimodal channel residual features are extracted respectively, and a fusion residual mapping map R is constructed to represent the residual signal-to-noise ratio of each mode in the conflict region;

[0034] Synchronously call the modal weight adjustment record during the completion process to construct a set of weight change trajectory curves W(t). The curves record the activation frequency, weight amplitude and adaptive suppression state of each modal channel during the completion process in the form of a time series.

[0035] R and W(t) are input into the joint consistency scoring function F(R, W) to calculate the degree of stability coupling between the two. When the function value exceeds the set stability deviation threshold, the completed vector is retained; otherwise, the original fusion result is kept as the final data benchmark.

[0036] The present invention also provides a robot, comprising:

[0037] The multimodal sensing and acquisition module collects multimodal sensing data within the storage area, including visual image data, spatial depth information, radio frequency sensing data, and infrared temperature feature data.

[0038] The modeling module extracts features from the multimodal sensing data and constructs a multimodal feature set including image feature vectors, a three-dimensional spatial point cloud model, a radio frequency response matrix, and a temperature distribution map.

[0039] The multi-channel fusion recognition module utilizes the constructed multi-modal feature set to generate the fusion recognition vector of the target item through a multi-channel fusion network based on an attention mechanism, and dynamically adjusts the weight coefficients of each modality based on a confidence inference mechanism during the fusion process.

[0040] The missing labeling module constructs a spatial distribution map of items based on fused recognition vectors, and generates perceptual missing labels for target items with confidence levels below the first threshold in the recognition results;

[0041] The reconstruction module determines the modal channels and angle regions that need to be filled in first based on a dynamic priority scheduling algorithm for regions with missing perception markers, thereby achieving automatic completion of low-confidence regions.

[0042] The inventory list generation module performs joint verification of the fusion recognition results and the completed area to form the final inventory list, including the target item ID, spatial location, quantity, fusion confidence value, and completion mark status;

[0043] The anomaly detection module triggers the anomaly alarm module if an item's fusion confidence level falls below the second warning threshold for S consecutive times, and generates a traceable perception data sequence for subsequent manual verification.

[0044] The present invention also provides a warehousing system, including the aforementioned robot.

[0045] The technical effects and advantages provided by the present invention in the above technical solution are as follows:

[0046] 1. This invention achieves robust and accurate identification of item information in a warehouse environment by constructing a multimodal perception and fusion mechanism, significantly improving inventory accuracy in complex scenarios such as occlusion, stacking, uneven lighting, and thermal interference. The fusion network introduces a channel-weighted structure based on an attention mechanism and dynamically adjusts modal contributions with a confidence inference strategy, ensuring stable output even when modal conflicts or missing modalities, thus solving the problems of low accuracy and easy failure in traditional single-sensor methods.

[0047] 2. This invention introduces a priority completion scheduling and multi-round joint verification mechanism for sensing missing regions. It establishes a consistency scoring system by combining fusion residuals and modal weight change trajectories, achieving automatic reconstruction and data identification of low-confidence regions, effectively avoiding false completion and overfitting. Furthermore, by combining an anomaly alarm module and a traceable sensing sequence, it supports the tracking management and manual review of long-term unstable targets, possessing good system self-optimization capabilities and engineering deployability. Attached Figure Description

[0048] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0049] Figure 1 This is a flowchart of the method of the present invention.

[0050] Figure 2 This is a flowchart of the system modules of the present invention. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0052] Example 1, please refer to Figure 1 As shown in this embodiment, a warehouse inventory counting method based on multimodal sensing technology includes:

[0053] Collect multimodal sensing data within the storage area, including visual image data, spatial depth information, radio frequency sensing data, and infrared temperature feature data;

[0054] Feature extraction was performed on the multimodal sensing data to construct a multimodal feature set including image feature vectors, three-dimensional spatial point cloud models, radio frequency response matrices, and temperature distribution maps;

[0055] By utilizing the constructed multimodal feature set, a multi-channel fusion network based on an attention mechanism is used to generate a fusion recognition vector for the target item, and the weight coefficients of each modality are dynamically adjusted based on a confidence inference mechanism during the fusion process.

[0056] A spatial distribution map of items is constructed based on the fused recognition vectors, and a perception missing label is generated for target items in the recognition results with a confidence level lower than the first threshold.

[0057] For regions with missing perception markers, a dynamic priority scheduling algorithm is used to determine the modal channels and angle regions that need to be filled in first, thereby achieving automatic completion of low-confidence regions.

[0058] The fusion recognition results and the completed area are jointly verified to form the final inventory list, including the target item ID, spatial location, quantity, fusion confidence value and completion mark status;

[0059] If an item's fusion confidence level falls below the second warning threshold for S consecutive times, the anomaly alarm module is triggered, and a traceable perception data sequence is generated for subsequent manual verification.

[0060] To make the technical solution of the present invention clearer, the present invention will be described in detail below with reference to specific embodiments. This embodiment is a preferred solution of the present invention, and is not limited to this implementation. All equivalent improvements and substitutions made within the scope of the present invention are within the protection scope of the present invention.

[0061] This embodiment provides a warehouse inventory method based on multimodal perception technology. This method combines visual image data, spatial depth information, radio frequency sensing data, and infrared temperature feature data. Through fusion modeling and dynamic weighting mechanisms, it improves the accuracy and stability of item identification in complex warehouse environments.

[0062] In this embodiment, multiple heterogeneous sensing nodes are deployed in the warehouse space, including:

[0063] RGB industrial camera (used to acquire visual image data);

[0064] TOF depth camera (used to acquire spatial depth information);

[0065] UWB or RFID antenna arrays (used to collect radio frequency sensing data);

[0066] Thermal imaging infrared sensor (used to capture infrared temperature characteristic data).

[0067] The aforementioned sensing nodes are connected to the data processing center through edge gateways, forming a distributed multimodal sensing network.

[0068] In actual operation, the system initiates the sensing process on a per-scanning-task basis. Each scan task is synchronously collected on a spatial partition basis to ensure temporal and spatial consistency of data from different modalities. The sampling frequency is uniformly set at 30Hz, and the spatial sampling interval is controlled within 0.5 meters.

[0069] After data collection is complete, the system performs feature extraction on various types of raw sensor data. The specific steps are as follows:

[0070] Visual image data: A ResNet50 backbone network is used to extract image feature vectors from RGB images, with an output size of [C_img, H / 32, W / 32]. C_img represents the number of channels in the image feature (e.g., 64, 128, 256, etc.), determined by the network structure, and is used to describe the depth of the image's semantic representation. H / 32 and W / 32 represent the image's height and width after downsampling via convolution and pooling, respectively. Color histograms and Local Pattern Recognition (LBP) are also extracted as auxiliary features.

[0071] Spatial depth information: The TOF depth map is converted into dense 3D point cloud data. The FPFH (Fast Point Feature Histograms) method is used to extract spatial geometric features and construct a voxelized 3D spatial model to form a spatial point cloud tensor with a size of [N_pts, 3 + F_d]. N_pts represents the total number of 3D point clouds extracted from the depth map or LiDAR. F_d represents additional depth feature dimensions other than x, y, and z, such as normal vector, intensity, reflectivity, curvature, etc.

[0072] Radio frequency sensing data: The radio frequency echo signal is converted into a two-dimensional response matrix through the CSI channel state information extraction module, and stable radio frequency features are extracted by performing multi-scale wavelet transform and encoded into response feature vectors with size [C_rf].

[0073] Infrared temperature feature data: The infrared thermal image is converted into a two-dimensional temperature distribution map. Thermal distribution features are extracted by temperature difference gradient and hot spot clustering. Finally, a temperature feature vector [C_ir] is formed by the centroid of the high-temperature region, temperature deviation, and thermal texture gradient.

[0074] Ultimately, the system combines the four types of features into a unified multimodal feature set, represented as: It is a three-dimensional point cloud spatial model.

[0075] To fully utilize multi-source heterogeneous sensing data, this embodiment employs a multi-channel fusion network based on an attention mechanism, the structure of which is as follows:

[0076] The input layer receives the aforementioned multimodal feature set;

[0077] A separate encoding subnetwork is set up for each modality feature channel to extract high-level semantic embeddings;

[0078] All modal channels undergo cross-attention modeling in the fusion layer to generate a fused recognition vector F_fused;

[0079] During the fusion process, the system introduces a confidence inference mechanism to dynamically adjust the weights of each modality, ensuring that the fusion result is robust to anomalous modalities.

[0080] Each modal encoder contains three sub-modules:

[0081] Feature normalization layer;

[0082] Residual reinforcement block;

[0083] Location embedding layer.

[0084] The fusion layer contains the following structure:

[0085] Modal weight adjustment coefficient Calculated using the following function: ;in: Let k be the characteristic variance of mode k; is the modal disturbance estimation factor; Z is the normalization constant. This represents the natural exponential function.

[0086] This mechanism enables the network to dynamically adjust the fusion bias in scenarios with physical occlusion, changes in lighting, or local thermal distortion, thereby improving recognition stability.

[0087] After fusion, the system outputs the fused recognition vector F_fused of the target item. This vector represents the multimodal fusion representation of each item and serves as the basic data structure for recognition, localization, completion and verification in subsequent steps.

[0088] In this embodiment of the invention, in order to achieve spatial localization and missing labeling of target items after multimodal fusion recognition, a spatial distribution modeling mechanism based on heterogeneous feature tensor reconstruction network and graph attention clustering algorithm is proposed. This mechanism can significantly improve the spatial accuracy and confidence verification capability of item recognition.

[0089] This step first inputs the extracted and fused multimodal recognition vector F_fused into a fusion processing structure called a heterogeneous feature tensor reconstruction network. This network is designed to accept and further model the semantic heterogeneity between visual features, depth features, and radio frequency response data, thereby improving the expressive power and contextual consistency of spatial mapping.

[0090] The heterogeneous feature tensor reconstruction network HFTR-Net includes the following key modules:

[0091] Multimodal embedding layer: Unifies the embedding of fusion vectors from different modalities into an equal-dimensional semantic space, and performs alignment processing through batch normalization and position embedding;

[0092] Semantic layer weighting module: Employs the cross-attention mechanism in the Transformer architecture to perform semantic-level fusion of the importance of each modal channel;

[0093] Residual connection mechanism: Preserves original modal features and reduces information loss caused by over-fusion.

[0094] The network outputs constitute a fusion tensor set T_fused, with dimensions [N_voxel, C_fused], where N_voxel is the number of voxel units obtained from the spatial partitioning, and C_fused is the fusion channel dimension.

[0095] The aforementioned fused tensor set T_fused is mapped to a three-dimensional voxel space to achieve a three-dimensional spatial distribution mapping of the target object. The mapping process includes:

[0096] A fixed spatial rasterization strategy is adopted to divide the warehouse area into voxel units according to the resolution Δx×Δy×Δz;

[0097] Each fused feature is bound to voxel coordinates (i, j, k) using the spatial location information contained in the fusion tensor (projected from the depth modality);

[0098] Time labels t_id are superimposed on the voxel tensor structure to form a distribution map of time labels, which facilitates subsequent comparison of multiple inventory sequences and detection of dynamic anomalies.

[0099] To identify voxel regions that may have blurred boundaries, occlusion, or identification conflicts, this embodiment introduces a graph attention clustering algorithm. A graph structure G = (V, E) is constructed, where: node V represents a voxel unit; edge E represents the spatial adjacency relationship between voxels; and the feature vector is derived from the corresponding vector in the fusion tensor T_fused.

[0100] A boundary confidence score β(i,j,k) for each voxel is calculated using an inter-node attention allocation mechanism. If this score is lower than a set first threshold θβ, the region is marked as a potential perceptual uncertainty area and added to the missing label candidate set. In this invention, the first threshold θβ is set to 0.4, which serves as a criterion for identifying regions with blurred boundaries or perceptual instability in spatial voxels. This value is derived from experimental results under numerous occlusion and interference scenarios, achieving a good balance between accuracy and recall. Regions with scores below this threshold are identified as perceptual uncertainty areas and added to the missing label candidate set to trigger subsequent completion processing.

[0101] For the above candidate voxel units, the following two types of metrics are performed to determine whether a perceptual missing label should be generated:

[0102] The calculation of the fusion feature residual includes: for a voxel unit (i,j,k), the system retrieves its current fusion feature vector F_curr and its historical fusion vector F_prev stored in the temporal feature buffer in the previous round of inventory, and calculates their cosine similarity cossim; further, the dissimilarity is taken as the difference in the residual vector. ΔF is the fusion feature residual of the voxel unit, which is used to reflect the magnitude of the fusion feature change and to determine whether there is abnormal drift in combination with the historical trajectory.

[0103] The calculation of environmental baseline difference includes: extracting the environmental baseline feature set E_base of the local space where the current voxel unit is located, including: regional temperature and humidity map (collected by an environmental sensor network); and electromagnetic interference grid matrix (generated by an RF sensor array). The current fused feature F_curr is projected into the environmental response function E(x, y, z) space, and the Euclidean distance ΔE between it and the baseline vector is measured. ΔE is the magnitude of the environmental baseline difference of the voxel unit, used to determine whether there is external environmental disturbance in the area.

[0104] The system will jointly determine the fusion feature residual ΔF and the environmental difference magnitude ΔE with preset thresholds T_ΔF and T_ΔE:

[0105] If ΔF ≥ T_ΔF and ΔE ≥ T_ΔE, the area is identified as a sensing failure unit. The thresholds T_ΔF and T_ΔE are used to jointly determine whether a voxel exhibits significant sensing bias. Based on extensive experimental statistics, a recommended value for T_ΔF is 0.25, and for T_ΔE is 0.3, corresponding to the sensitive boundaries of feature change magnitude and environmental disturbance response, respectively. The system only marks the region as a potential sensing failure area when both thresholds are simultaneously exceeded.

[0106] A sensory missing marker is added to the voxel unit in the spatial distribution map, and a time label is recorded. This missing marker will trigger a sensory supplementation strategy during subsequent priority completion and fusion repair processes.

[0107] This implementation provides an automatic completion mechanism for sensing gaps. By constructing a modal angle cross-mapping table, a graph structure scheduling network, and a feature enhancement module, it achieves adaptive reconstruction and repair of low-confidence sensing areas in warehouse inventory checks. This mechanism is particularly suitable for environments with severe occlusion, unstable modal signals, or frequent sensing blind spots.

[0108] When the system detects a voxel unit with a confidence level lower than a set first threshold θβ during the construction of a spatial distribution map, and the difference between the fusion feature residual and the environmental baseline is higher than the tolerance range, a perception missing marker is generated. When the system detects that the fusion confidence level of a voxel unit is lower than the preset first threshold θβ, and the fusion feature residual ΔF of that unit exceeds the threshold T_ΔF=0.25, and the difference ΔE between the fusion feature and the environmental baseline exceeds the threshold T_ΔE=0.3, it is considered an abnormal perception information. When all three conditions are met, the system marks the voxel as a perception missing region to trigger the completion scheduling mechanism. Subsequently, the system defines the set of all voxel regions with missing markers as Vmiss; for each missing region, the system extracts its corresponding fusion tensor channel feature missing map, statistically analyzes the response missing types of different modalities in that voxel region (such as blurred boundaries, thermal mismatch, etc.), and constructs a multimodal feature loss map.

[0109] For each missing region, the system constructs a Modal Angle Cross Mapping Table (MAM), which records the following information:

[0110] The historical perception integrity value of different modal channels under different observation angles is denoted as C_ij;

[0111] The misclassification rate ε_ij for each combination (modal i, angle j) in the historical inventory task;

[0112] For the currently missing region, the missing status flag of each combination in the most recent perception task.

[0113] To rationally allocate sensory resources and schedule paths for the completion task, this embodiment designs a graph-structured priority scheduling network, which receives the following inputs:

[0114] MAM modal angle mapping table;

[0115] Current local environmental disturbance matrix (such as radio frequency blocking index, infrared interference source density, etc.);

[0116] Boundary ambiguity map of the missing region (generated by spatial gradient statistics).

[0117] The scheduling network operates as a graph neural network, where nodes represent candidate modality angle combinations and edges represent resource conflicts or redundancy relationships between different combinations. Through iterative message passing and attention-weighted calculation, a dynamic priority scoring matrix P is output, with dimensions [M, N], where M is the number of modalities and N is the number of angle combinations. A larger value indicates that the combination is given higher priority for the completion task.

[0118] Based on the combination priority sorted by the scoring matrix P, the system calls mobile sensing nodes from the deployable sensing node pool and assigns them to the best angle acquisition positions around the missing region v_i.

[0119] Each node generates a view trajectory planning path based on the priority trajectory combination in the scoring matrix;

[0120] The types of data collected include: structured light depth maps, millimeter-wave reflectance spectra, and multispectral image information;

[0121] All collected data is uniformly synchronized to the fusion server cache pool.

[0122] The collected raw sensor data is input into the feature enhancement module, which has the following structure:

[0123] Modal channel gating unit: Automatically activates / suppresses each modal channel based on modal complementarity score;

[0124] Residual Iterative Fusion Layer: Uses a multi-stage residual synthesis strategy to perform inter-modal feature fusion and iteratively corrects conflict responses generated by differing modes;

[0125] Output layer: Generates an enhanced fusion vector F to complete i, with the same size as the original tensor vector.

[0126] The complementarity scoring function is defined as follows: ;in, Let Vk represent the complementarity score of mode k, Var represent the internal variance of the mode, and Corr represent the correlation coefficient with other modes. This represents the other modules. `f` is a combination function used to integrate the two indicators mentioned above and output a complementarity score. The combination function may be a weighted, nonlinear combination, or normalized scoring function, used to adjust the modality weights or select retention strategies during the fusion process.

[0127] Finally, the enhanced fusion vector F completes i by precisely aligning it with the missing position in the original fusion tensor through a spatial index mapping mechanism, and its spatial coordinates are controlled by the voxel index (i,j,k).

[0128] The replacement operation is synchronously written to the global spatial distribution map and marked with the status field "Reconstructed=TRUE". At the same time, the completion flag Rflag=1 is stored in the history table for subsequent model learning and failure detection backtracking.

[0129] To improve the reliability of the fusion recognition results and ensure data consistency and rationality of judgment during the perception completion process, this implementation provides a joint verification mechanism to comprehensively evaluate the data deviation between the original fusion vector and the completed perception vector in the same spatial index area, automatically determine the source of the final inventory vector, and generate a final inventory list with verification labels.

[0130] After completing the fusion sensing and completion operations, the system compares the original fused tensor Tfused with the completed sensing vector set F, and retrieves the overlapping region Voverlap in the spatial index tensor structure, i.e.: In the formula, This indicates an existence judgment function used to determine whether a voxel has valid data in a tensor / vector; This indicates the value of the completed sensing vector at position vi, suggesting that the voxel position has an updated value in the completion result.

[0131] Based on this, the system constructs an Index Consistency Graph (ICM), the structure of which is as follows:

[0132] Each node represents an overlapping voxel unit;

[0133] Node attributes include the semantic difference between the fused vector and the completed vector;

[0134] The edge weights are the differences in the feature space gradients between adjacent voxels.

[0135] The system performs semantic alignment and confidence intersection calculations on all vector pairs in Voverlap, including:

[0136] Vector normalization and alignment;

[0137] The intersection of confidence values ​​is Conf_intersect(v_i) = min(conf_T, conf_F completion);

[0138] The cosine distance between the difference vectors is Δ_cos(v_i).

[0139] Finally, a consistency score matrix Q is generated, which is used to indicate the alignment degree and confidence intersection of each overlapping region.

[0140] For voxel units with low consistency in Q (such as those with scores below the set threshold θ_consistency), the system enters a dynamic judgment process to determine whether the final inventory vector should be based on the original fusion result or the completed result.

[0141] For example, when the score of a voxel unit in the consistency scoring matrix Q is lower than the set threshold θ_consistency=0.6, the system determines that there is a fusion conflict. At this time, a dynamic judgment process is triggered, which combines the modal residual signal-to-noise ratio and the weight change trajectory to perform a joint score. If the joint score exceeds the stability threshold, the completed result is used as the benchmark; otherwise, the original fusion result is retained.

[0142] This dynamic judgment process is based on two key information sources:

[0143] Integrate source tensor residual information;

[0144] The trajectory of modal weight changes during the completion process.

[0145] For a conflicting voxel position vi, the system extracts the feature responses of the voxel in different modal channels from the original fused tensor T_fused and the completed sensing vector F, respectively, and calculates the channel-level differences: Where σk is the historical variance of channel k, and Rk reflects the residual signal-to-noise ratio. The system constructs a fusion residual mapping map R = {R1, R2, ..., Rk} to evaluate the stability deviation of multimodal systems in the conflict region.

[0146] The system simultaneously calls the weight adjustment records of each mode during the completion process to construct a set of modal weight trajectory curves W(t), the contents of which include:

[0147] : The weight magnitude of mode k;

[0148] : The adaptive suppression state flag (0 or 1) for the modal channel. If mode k is suppressed at time t, then =1, unsuppressed is 0.

[0149] Input R and W(t) into the joint consistency scoring function F(R, W) to calculate the degree of stability coupling at the current spatial index position, as defined below: ; This indicates that the weights are normalized; the larger the function result, the more stable and dominant the completed data is in that region. The system sets a stability deviation threshold Ts, and the judgment is as follows:

[0150] If F(R, W) ≥ Ts, then the completed vector Fcomplete(vi) is retained; otherwise, the original fusion result Tfused(vi) is retained as the final data benchmark. The vectors corresponding to each spatial index position after joint validation are used as the final inventory result. For example, in this invention, the threshold Ts is set to 0.5, which is derived from the median value of the statistical distribution of the stability scoring function F(R,W), ensuring that high-confidence completed regions are given priority during the evaluation process. A score higher than Ts indicates that the completed data has stronger stable coupling at the current voxel position, and is therefore more suitable as the final inventory result.

[0151] To further enhance the system's ability to warn of potential identification errors and sensing failure risks, this implementation design proposes an anomaly alarm mechanism based on fusion confidence cumulative detection. It combines historical sensing data to construct a traceable sequence, supporting subsequent manual verification and model optimization.

[0152] In this invention, the system assigns a unique identifier `item_id` to each identified item in each inventory round and records its corresponding fused identification vector `F_fused` and confidence level `conf_i`. If an item's fused confidence level `conf_i` remains below a set second warning threshold `θ_warn` for S consecutive inventory rounds, the system automatically marks the item as a "warning target" and triggers the anomaly alarm module `AAM`. The second warning threshold `θ_warn` is used to determine whether an item exhibits a persistent perceptual anomaly across multiple inventory rounds. In this invention, this threshold is set to 0.3, a value based on the critical point of the confidence level distribution of low-confidence misidentified samples in the training set, exhibiting high sensitivity and recognition accuracy. If an item's fused confidence level remains below the second warning threshold `θ_warn` for S consecutive inventory rounds, the system automatically marks it as a warning target and triggers the `AAM` anomaly alarm module to prompt manual review.

[0153] The abnormal alarm module performs the following processing steps:

[0154] Triggered alarm records: An alarm record item is generated in the alarm log, which includes the alarm timestamp, triggering item ID, continuous low confidence statistics, spatial location index and corresponding confidence sequence;

[0155] Generate a retrospective sensing data sequence: The system calls the multimodal raw sensing data and fusion process cache of the item in the first S rounds, including:

[0156] Raw image data, depth map, radio frequency signal and temperature spectrum;

[0157] Fusion of recognition vector and modal weight distribution;

[0158] Integrate residuals and consistency scores;

[0159] Construct a retrospective data structure: organize it into a structured data sequence in chronological order, with each item containing modal data, fusion confidence, spatial index, and timestamp;

[0160] Interface output and manual review: The system will input the structured data sequence into the manual verification module or store it in the model training database for subsequent manual review or adaptive learning correction of the fusion model.

[0161] Through this mechanism, the system can identify items with abnormal confidence levels in the early stages, issue timely warnings, and support highly traceable verification paths, ensuring the reliability and self-evolution capability of the inventory system in long-term operation.

[0162] Example 2, please refer to Figure 2 As shown, the robot described in this embodiment includes:

[0163] The multimodal sensing and acquisition module collects multimodal sensing data within the storage area, including visual image data, spatial depth information, radio frequency sensing data, and infrared temperature feature data.

[0164] The modeling module extracts features from the multimodal sensing data and constructs a multimodal feature set including image feature vectors, a three-dimensional spatial point cloud model, a radio frequency response matrix, and a temperature distribution map.

[0165] The multi-channel fusion recognition module utilizes the constructed multi-modal feature set to generate the fusion recognition vector of the target item through a multi-channel fusion network based on an attention mechanism, and dynamically adjusts the weight coefficients of each modality based on a confidence inference mechanism during the fusion process.

[0166] The missing labeling module constructs a spatial distribution map of items based on fused recognition vectors, and generates perceptual missing labels for target items with confidence levels below the first threshold in the recognition results;

[0167] The reconstruction module determines the modal channels and angle regions that need to be filled in first based on a dynamic priority scheduling algorithm for regions with missing perception markers, thereby achieving automatic completion of low-confidence regions.

[0168] The inventory list generation module performs joint verification of the fusion recognition results and the completed area to form the final inventory list, including the target item ID, spatial location, quantity, fusion confidence value, and completion mark status;

[0169] The anomaly detection module triggers the anomaly alarm module if an item's fusion confidence level falls below the second warning threshold for S consecutive times, and generates a traceable perception data sequence for subsequent manual verification.

[0170] Example 3: The warehousing system described in this example includes the robot mentioned above.

[0171] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A warehouse inventory counting method based on multimodal perception technology, applied to robots, characterized in that: include: Collect multimodal sensing data within the storage area, including visual image data, spatial depth information, radio frequency sensing data, and infrared temperature feature data; Feature extraction was performed on the multimodal sensing data to construct a multimodal feature set including image feature vectors, three-dimensional spatial point cloud models, radio frequency response matrices, and temperature distribution maps; By utilizing the constructed multimodal feature set, a multi-channel fusion network based on an attention mechanism is used to generate a fusion recognition vector for the target item, and the weight coefficients of each modality are dynamically adjusted based on a confidence inference mechanism during the fusion process. A spatial distribution map of items is constructed based on the fused recognition vectors, and a perception missing label is generated for target items in the recognition results with a confidence level lower than the first threshold. For regions with missing perception markers, a dynamic priority scheduling algorithm is used to determine the modal channels and angle regions that need to be filled in first, thereby achieving automatic completion of low-confidence regions. The fusion recognition results and the completed area are jointly verified to form the final inventory list, including the target item ID, spatial location, quantity, fusion confidence value and completion mark status; If an item's fusion confidence level falls below the second warning threshold for S consecutive times, the anomaly alarm module is triggered, and a traceable perception data sequence is generated for subsequent manual verification.

2. The warehouse inventory counting method based on multimodal sensing technology according to claim 1, characterized in that: in, Constructing a spatial distribution map of objects and generating perceptual missing markers includes: The fused recognition vector is input into a heterogeneous feature tensor reconstruction network to construct a fused tensor set. The network is used to perform semantic layer weighted fusion of visual features, depth features and radio frequency response data. The fused tensor set is mapped to a three-dimensional voxel space to construct a spatial distribution map of items with time labels, and a graph attention clustering algorithm is introduced to classify confidence boundaries. For voxel units with confidence levels below the first threshold, the difference between their fusion feature residuals and the environmental baseline is calculated. When the residual magnitude exceeds a set perturbation threshold, a corresponding perception missing label is generated and marked in the atlas.

3. The warehouse inventory counting method based on multimodal sensing technology according to claim 2, characterized in that: Calculating the fusion feature residual includes: comparing the original feature vector of the voxel unit in the multimodal fusion tensor with its historical fusion vector in the previous period in the temporal feature buffer using cosine similarity, calculating the difference in the residual vector, and denoting it as the fusion feature residual; The calculation of environmental baseline difference includes: extracting the set of environmental baseline features in the local area to which the voxel unit belongs; constructing an environmental response function through the regional temperature and humidity map and the electromagnetic interference raster matrix; projecting the current fused vector into the environmental response function space and measuring the Euclidean distance with the baseline vector to obtain the magnitude of environmental difference, which is denoted as environmental baseline difference.

4. The warehouse inventory method based on multimodal sensing technology according to claim 1, characterized in that: in, Automatic completion of regions with perceptual missing markers includes: Extract the multimodal feature loss map of the perception missing region and construct a modal angle cross-mapping table (MAM). The MAM is used to record the response integrity and historical misjudgment rate of different modal channels under various observation angles. The MAM is embedded as input into a graph-structured priority scheduling network, and a dynamic priority scoring matrix P is generated by combining the current environmental perturbation matrix and the boundary ambiguity map. The scoring matrix is ​​used to determine the optimal combination sequence of modal channels and angular regions in the completion task. Based on the scoring matrix P, a completion perception task is triggered, and corresponding perception nodes are scheduled to reconstruct the multimodal input of the low-confidence region. The feature enhancement module performs fusion stabilization processing to generate a completed perception vector to replace the original missing region content.

5. A warehouse inventory counting method based on multimodal sensing technology according to claim 4, characterized in that: in, The completion perception task triggered based on the scoring matrix P includes: Based on the priority combination sequence in the scoring matrix P, the mobile sensing node is assigned to the missing region and instructed to collect multimodal raw data within a specified angular trajectory range, including structured light depth map, millimeter wave reflectance spectrum and multispectral image information. Based on the collected multimodal input data, the channel response is automatically activated or suppressed based on the modal complementarity score, and the residual information between different modes is fused and iteratively optimized. The enhanced and fused perception vector is output and replaced in the missing region through a spatial index mapping mechanism aligned with the original perception tensor, thus completing the completion and update operation. At the same time, the replacement marker is synchronously written into the global spatial distribution map.

6. The warehouse inventory method based on multimodal sensing technology according to claim 1, characterized in that: in, The fusion recognition results and the completed regions are jointly verified to form the final inventory list, which includes: Extract the spatial index region covered by the original fusion recognition result and the completed perception vector, construct the index consistency map, perform vector alignment and confidence intersection calculation on the overlapping region, and generate the consistency score matrix Q; For regions with low consistency in the scoring matrix Q, the original fusion result or the completed vector is dynamically determined as the final data benchmark by combining the residual information of the fusion source tensor and the trajectory of modal weight changes during the completion process. Write the jointly verified inventory vectors into the final inventory list.

7. A warehouse inventory counting method based on multimodal sensing technology according to claim 6, characterized in that: in, Dynamic judgment based on the original fusion result or the completed vector as the final data benchmark includes: For the original fusion tensor and the completed sensing vector that cover the same spatial index position, the corresponding multimodal channel residual features are extracted respectively, and a fusion residual mapping map R is constructed to represent the residual signal-to-noise ratio of each mode in the conflict region; Synchronously call the modal weight adjustment record during the completion process to construct a set of weight change trajectory curves W(t). The curves record the activation frequency, weight amplitude and adaptive suppression state of each modal channel during the completion process in the form of a time series. R and W(t) are input into the joint consistency scoring function F(R, W) to calculate the degree of stability coupling between the two. When the function value exceeds the set stability deviation threshold, the completed vector is retained; otherwise, the original fusion result is kept as the final data benchmark.

8. A robot, characterized in that: The robot performs a warehouse inventory counting method based on multimodal perception technology as described in any one of claims 1-7, comprising: The multimodal sensing and acquisition module collects multimodal sensing data within the storage area, including visual image data, spatial depth information, radio frequency sensing data, and infrared temperature feature data. The modeling module extracts features from the multimodal sensing data and constructs a multimodal feature set including image feature vectors, a three-dimensional spatial point cloud model, a radio frequency response matrix, and a temperature distribution map. The multi-channel fusion recognition module utilizes the constructed multi-modal feature set and generates the fusion recognition vector of the target item through a multi-channel fusion network based on an attention mechanism. During the fusion process, it dynamically adjusts the weight coefficients of each modality based on a confidence inference mechanism. The missing labeling module constructs a spatial distribution map of items based on fused recognition vectors, and generates perceptual missing labels for target items with confidence levels below the first threshold in the recognition results; The reconstruction module determines the modal channels and angle regions that need to be filled in first based on a dynamic priority scheduling algorithm for regions with missing perception markers, thereby achieving automatic completion of low-confidence regions. The inventory list generation module performs joint verification of the fusion recognition results and the completed area to form the final inventory list, including the target item ID, spatial location, quantity, fusion confidence value, and completion mark status; The anomaly detection module triggers the anomaly alarm module if an item's fusion confidence level falls below the second warning threshold for S consecutive times, and generates a traceable perception data sequence for subsequent manual verification.

9. A warehousing system, characterized in that: Including the robot as described in claim 8.

Images