A Multimodal Warehouse Inventory Method and System
By employing a multimodal warehouse inventory method that combines image data, RFID reading data, and spatial perception data, and utilizing location window gating and cross-modal binding solutions, the problem of inconsistent inventory results in existing technologies has been solved. This approach achieves highly accurate and traceable warehouse inventory, reduces the risk of misjudgment, and improves inventory efficiency.
Patent Information
- Application Number
- CN202611143156.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-07-30
- Publication Date
- 2026-11-17
AI Technical Summary
In scenarios with dense shelving, small spacing between adjacent storage locations, and frequent obstruction, existing warehouse inventory technologies are prone to issues. RFID reading is susceptible to cross-location cross-reading due to multipath reflection and antenna coverage. Visual recognition also suffers from reading fluctuations due to changes in viewing angle, lighting interference, and packaging obstruction. This leads to inconsistencies in inventory results, increased probability of misattribution and duplicate counting, and a lack of process evidence for verification, affecting the reliability and traceability of inventory results.
A multimodal warehouse inventory method is adopted, which combines a warehouse location benchmark model and warehouse location window gating with image data, RFID reading data and spatial perception data for synchronous collection to generate visual candidate sets and RFID candidate sets. The set of warehouse location objects is output by solving cross-modal binding. Consistency scoring drives active review closed loop and generates inventory evidence package to ensure data consistency.
It improves the interpretability and reproducibility of inventory data, reduces the probability of misjudgment under dense shelving and obstruction interference, enhances the accuracy of material identification and storage location determination, improves the stability and anti-interference ability of inventory results, and reduces the workload of manual review and the risk of omission through a consistency scoring-driven review mechanism, thereby enhancing the efficiency of closed-loop inventory processing and audit traceability.
Smart Images

Figure CN123117979A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of warehousing and logistics informatization and intelligent inventory management, specifically to a multimodal warehouse inventory management method and system. Background Technology
[0002] As e-commerce and manufacturing warehousing evolve towards multi-variety, small-batch, and high-turnover operations, Warehouse Management Systems (WMS) need to continuously maintain consistency between inventory records and physical stock to support operations such as receiving and shipping verification, picking and replenishment, batch traceability, and inventory turnover. On-site inventory checks typically require the rapid completion of full or zone-based inventory counts without significantly impacting daily operational pace, and the output of a discrepancy list and location information for verification.
[0003] Current inventory methods mainly include manual barcode scanning using handheld terminals, batch identification using handheld or fixed RFID readers, and inspection and data collection using mobile robots equipped with cameras and RFID antenna arrays. The backend system typically matches the read labels or visually identified material identifiers with the inventory ledger to generate results such as missing, extra, misaligned, or quantity discrepancies. Some solutions also incorporate depth cameras or LiDAR for environmental mapping and positioning to plan inspection routes or help determine data collection points.
[0004] In scenarios with dense shelving, small spacing between adjacent storage locations, and frequent obstruction, RFID reading is susceptible to cross-location cross-reading due to multipath reflection and antenna coverage. Visual recognition may also experience reading fluctuations due to changes in viewing angle, lighting interference, and packaging obstruction. The collection results of different modalities are difficult to establish a stable correspondence with specific storage location boundaries, leading to inconsistencies in inventory discrepancies, increased probability of misattribution and duplicate counting, and a lack of process evidence for verification, affecting the reliability and traceability of inventory results. Summary of the Invention
[0005] To address the shortcomings of existing technologies, this invention provides a multimodal warehouse inventory method and system to solve the technical problems existing in the prior art.
[0006] The above-mentioned technical objective of the present invention is achieved through the following technical solution: A multimodal warehouse inventory method includes the following steps: S1: Generate inventory count tasks based on the inventory ledger of the warehouse management system (WMS) and establish a warehouse location benchmark model. The warehouse location benchmark model includes warehouse location identifiers, warehouse location space boundaries, and the material identifiers and quantity information corresponding to the warehouse location. S2: Control the inventory terminal to inspect the target warehouse area according to the inventory task, and obtain the pose sequence of the inventory terminal during the inspection process. The pose sequence includes position, attitude and timestamp. S3: Divide the target storage area into storage location windows based on the storage location benchmark model to obtain multiple storage location windows; S4: For each storage location window, perform multimodal synchronous acquisition based on the pose sequence to obtain the synchronous acquisition frame corresponding to the storage location window. The synchronous acquisition frame includes image data, RFID reading data and spatial perception data. Among them, the RFID reading data includes tag identification, received signal strength, phase information, antenna number, read / write power level and timestamp. The spatial perception data is depth image or point cloud data. S5: Generate a visual candidate set and an RFID candidate set in the storage location window. The visual candidate set is obtained by performing target detection and tag parsing on the image data, and the RFID candidate set is obtained by tag aggregation on the RFID read data. S6: Based on the spatial constraints of the storage location window, perform cross-modal binding solution on the visual candidate set and the RFID candidate set, and output the storage location object set. Each storage location object in the storage location object set includes a tag identifier, storage location identifier, material identifier, quantity result and consistency score. S7: Perform consistency verification between the collection of storage location objects and the storage location baseline model, obtain the inventory discrepancy results, and determine whether to trigger an active review based on the consistency score; S8: When the consistency score is lower than the preset threshold, the control inventory terminal performs active verification collection on the corresponding storage location window, and updates the storage location object set and its consistency score based on the active verification collection results. S9: Generate an inventory evidence package for the updated set of storage location objects, and write back the inventory discrepancy results and the inventory evidence package to WMS to complete the inventory closure loop; The inventory evidence package includes RFID reading data associated with the storage location objects, image data, spatial perception data, pose sequence fragments, consistency scores, and proactively reviewed collection records.
[0007] Preferably, the multimodal synchronous acquisition includes: Image data, RFID reading data, spatial perception data, and pose information are timestamped and aligned, and the aligned multimodal data are uniformly mapped to the same coordinate system. The coordinate system mapping includes performing coordinate transformations on the camera coordinate system, RFID antenna coordinate system, and spatial sensing coordinate system based on extrinsic calibration parameters.
[0008] Preferably, the partitioning of the storage location window includes: Based on spatial perception data, identify the structure of shelf uprights, beams or shelves to determine the boundaries of storage space. Align the spatial boundary of the storage location with the storage location identifier in the storage location benchmark model to obtain a storage location window that corresponds one-to-one with the storage location identifier; The data acquisition gating of the storage location window is activated when the inventory terminal's position enters the storage location space boundary, and deactivated when the inventory terminal's position leaves the storage location space boundary.
[0009] Preferably, the generation of the visual candidate set includes: Cargo instance detection is performed on the image data to obtain candidate bounding boxes; Perform barcode reading or character recognition on the area corresponding to the candidate target box to obtain the material identification; When barcode reading or character recognition fails, the material identifier is determined by similarity matching between the appearance feature vector of the candidate target box and the material master data. The generation of the RFID candidate set includes: The RFID reading data is segmented and aggregated according to the storage location window to generate a reading sequence, and reading records with fewer than a preset number of reads or a received signal strength lower than a preset strength threshold are removed.
[0010] Preferably, the cross-modal binding solution includes: For each visual candidate object and each RFID candidate tag, a matching score is calculated, and a matching score matrix is constructed. The matching score includes spatial consistency score, trajectory consistency score, and semantic consistency score; The binding relationship between the label identifier and the visual candidate object is solved based on the matching score matrix, and the binding relationship is mapped to the corresponding storage location identifier in the storage location window to output the storage location object set. The trajectory consistency score is determined by the correlation between the phase sequence or received signal strength sequence of the RFID candidate tag and the pose change sequence of the inventory terminal.
[0011] Preferably, the consistency check includes outputting one or more of the following difference types: missing difference, extra difference, misalignment difference, and quantity difference; A difference score is generated for the difference type, and the difference score is determined by the consistency score, the difference type weight, and the difference quantity magnitude. The proactive review and collection includes one or more of the following review actions: Control the inventory terminal to change the observation angle or observation distance to collect additional image data; Control the RFID reader / writer to adjust the read / write power level and switch the antenna number to perform directional repeat reading; The control inventory terminal performs multiple samplings at preset stop points and performs consistency fusion of the sampling results.
[0012] Preferably, the inventory evidence package further includes a digest verification value, which is determined by a hash digest generated from RFID reading data, image data, and spatial perception data, and is bound and stored with the inventory task batch number, storage location identifier, and tag identifier; After writing back to WMS, the following is also included: For storage location windows with difference scores exceeding the preset difference threshold, an abnormal work order is generated and associated with the corresponding inventory evidence package, which is then pushed to the review terminal. After verifying the inventory discrepancies at the verification terminal, update the warehouse location baseline model and record the version number of the warehouse location baseline model before and after the update, as well as the corresponding inventory task batch number.
[0013] A multimodal warehouse inventory system includes: The task generation module is used to generate inventory counting tasks based on the inventory ledger of the Warehouse Management System (WMS) and establish a warehouse location benchmark model. The pose acquisition module is used to control the inventory terminal to perform inspections and acquire pose sequences. The storage location window division module is used to divide the target storage area into multiple storage location windows based on the storage location benchmark model and perform data acquisition gating. The multimodal acquisition module is used to generate synchronous acquisition frames for each storage location window. The synchronous acquisition frames include image data, RFID reading data, spatial perception data, and pose information. The candidate generation module is used to generate visual candidate sets and RFID candidate sets; The cross-modal binding module is used to perform cross-modal binding solutions on the visual candidate set and the RFID candidate set and output the storage location object set, and calculate the consistency score for the storage location objects in the storage location object set; The consistency verification module is used to verify the consistency between the collection of storage location objects and the storage location baseline model and obtain the inventory difference results. The proactive review control module is used to control the multimodal acquisition module to perform proactive review acquisition on the corresponding storage location window and update the storage location object set and its consistency score when the consistency score is lower than the preset threshold. The evidence package generation module is used to generate and archive inventory evidence packages associated with the storage location object. The write-back module is used to write back the inventory discrepancy results and inventory evidence package to the WMS to complete the inventory loop.
[0014] Preferably, the system further includes a time synchronization and calibration module, which is used to timestamp align image data, RFID reading data, spatial perception data and pose information, and to realize coordinate transformation of the camera coordinate system, RFID antenna coordinate system and spatial perception coordinate system based on external parameter calibration parameters; The multimodal acquisition module includes an RFID reader and a multi-antenna array; The RFID reading data includes tag identification, received signal strength, phase information, antenna number, read / write power level, and timestamp.
[0015] Preferably, the cross-modal binding module is used to construct a matching score matrix and solve the binding relationship between the label identifier and the visual candidate object. The matching score matrix includes a spatial consistency score matrix, a trajectory consistency score matrix, and a semantic consistency score matrix. The active verification control module is used to control the RFID reader to adjust the reading and writing power level and switch the antenna number to perform directional repeat reading during the active verification and acquisition phase. The evidence package generation module is used to generate hash digests of the inventory evidence package and bind them to the inventory task batch number and storage location identifier for consistency verification and audit traceability. The system also includes an exception work order module, which is used to generate exception work orders for storage location windows with difference scores higher than preset difference thresholds and push them to the review terminal along with the inventory evidence package.
[0016] In summary, the present invention has the following main beneficial effects: This invention, through the collaborative constraints of the storage location benchmark model and the storage location window gating, confines the image data, RFID reading data, and spatial perception data collected by the inventory terminal during the inspection process to the same storage location window. It also achieves unified alignment of multimodal data through timestamp alignment and coordinate system mapping, thereby achieving a one-to-one correspondence of synchronous collection frames with the storage location as the smallest unit. This avoids inventory result drift caused by cross-storage reading, misplacement, and unclear collection boundaries in continuous reading mode, improves the interpretability and reproducibility of inventory data, and reduces the probability of misjudgment in scenarios such as dense shelving and obstruction interference.
[0017] This invention generates a visual candidate set and an RFID candidate set within the storage location window, and constructs a matching score matrix that includes spatial consistency, trajectory consistency, and semantic consistency. It then performs cross-modal binding solutions on the visual candidate objects and RFID candidate tags, thereby achieving the effect of outputting an object-level storage location object set and calculating a consistency score. This improves the inventory results from simple reading and counting to a constrained and verifiable binding relationship, effectively suppressing deviations caused by occasional readings and low-confidence identification, improving the accuracy of material identification and storage location attribution, and enhancing the stability and anti-interference ability of inventory results in complex environments.
[0018] This invention utilizes a proactive review closed loop driven by consistency scoring to control the inventory terminal to perform actions such as changing the observation angle to acquire additional images, adjusting read / write power, switching antenna directional rereading, and multiple sampling consistency fusion under low consistency conditions. After review, it updates the set of storage location objects and the discrepancy results, and simultaneously generates an inventory evidence package containing multimodal raw data, pose sequence fragments, review records, and summary verification values, which is then written back to the WMS. This achieves the effects of discrepancy convergence and full-process traceability, reducing the workload of manual review and the risk of missed judgments, while also providing verifiable evidence for abnormal work orders, thus improving the efficiency of inventory closed-loop processing and audit traceability capabilities. Attached Figure Description
[0019] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation
[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0021] Example 1 refer to Figure 1 A multimodal warehouse inventory method includes the following steps: S1: Generate inventory count tasks based on the inventory ledger of the warehouse management system (WMS) and establish a warehouse location benchmark model. The warehouse location benchmark model includes warehouse location identifiers, warehouse location space boundaries, and the material identifiers and quantity information corresponding to the warehouse location. S2: Control the inventory terminal to inspect the target warehouse area according to the inventory task, and obtain the pose sequence of the inventory terminal during the inspection process. The pose sequence includes position, attitude and timestamp. S3: Divide the target storage area into storage location windows based on the storage location benchmark model to obtain multiple storage location windows; S4: For each storage location window, perform multimodal synchronous acquisition based on the pose sequence to obtain the synchronous acquisition frame corresponding to the storage location window. The synchronous acquisition frame includes image data, RFID reading data and spatial perception data. Among them, the RFID reading data includes tag identification, received signal strength, phase information, antenna number, read / write power level and timestamp. The spatial perception data is depth image or point cloud data. S5: Generate a visual candidate set and an RFID candidate set in the storage location window. The visual candidate set is obtained by performing target detection and tag parsing on the image data, and the RFID candidate set is obtained by tag aggregation on the RFID read data. S6: Based on the spatial constraints of the storage location window, perform cross-modal binding solution on the visual candidate set and the RFID candidate set, and output the storage location object set. Each storage location object in the storage location object set includes a tag identifier, storage location identifier, material identifier, quantity result and consistency score. S7: Perform consistency verification between the collection of storage location objects and the storage location baseline model, obtain the inventory discrepancy results, and determine whether to trigger an active review based on the consistency score; S8: When the consistency score is lower than the preset threshold, the control inventory terminal performs active verification collection on the corresponding storage location window, and updates the storage location object set and its consistency score based on the active verification collection results. S9: Generate an inventory evidence package for the updated set of storage location objects, and write back the inventory discrepancy results and the inventory evidence package to WMS to complete the inventory closure loop; The inventory evidence package includes RFID reading data associated with the storage location objects, image data, spatial perception data, pose sequence fragments, consistency scores, and proactively reviewed collection records.
[0022] A multimodal warehouse inventory system includes an inventory terminal and an inventory platform, which are connected via wired or wireless networks.
[0023] The inventory terminal is a mobile inventory carrier, which can be an inventory robot, AMR, or AGV, and includes at least the following: 1) Image acquisition unit, used to acquire image data within the storage location window; the image acquisition unit can be an RGB camera or an RGB-D camera; 2) An RFID acquisition unit for acquiring RFID read data; the RFID acquisition unit includes an RFID reader and a multi-antenna array, wherein the RFID read data includes at least tag identification, received signal strength, phase information, antenna number, read / write power level, and timestamp; 3) Spatial sensing unit, used to collect spatial sensing data; the spatial sensing data is depth image or point cloud data, and the spatial sensing unit can be a lidar or a depth camera; 4) Pose acquisition unit, used to acquire the pose sequence of the inventory terminal during the inspection process; the pose sequence includes at least position, attitude and timestamp; 5) Edge computing unit, used to perform candidate generation, cross-modal binding solution, and consistency score calculation for storage location window, and to trigger active review collection when the consistency score is lower than the preset threshold.
[0024] The inventory platform can be deployed on a local warehouse server or a cloud server, and it includes at least: The task generation module is used to generate inventory counting tasks based on the inventory ledger of the warehouse management system (WMS) and to establish a warehouse location benchmark model. The pose acquisition module is used to receive the pose sequence from the inventory terminal, or to control the inventory terminal to perform inspection and acquire the pose sequence. The storage location window division module is used to divide the target storage area into multiple storage location windows based on the storage location benchmark model, and to perform data acquisition gating on the storage location windows; The multimodal acquisition module is used to generate synchronous acquisition frames for each storage location window; the synchronous acquisition frames include image data, RFID reading data, spatial perception data, and pose information; The candidate generation module is used to generate visual candidate sets and RFID candidate sets; The cross-modal binding module is used to construct the matching score matrix and solve the binding relationship between the label identifier and the visual candidate object, output the set of storage location objects, and calculate the consistency score for the storage location objects in the set of storage location objects; The consistency verification module is used to verify the consistency between the collection of storage location objects and the storage location baseline model and obtain the inventory difference results. The active verification control module is used to control the multimodal acquisition module to perform active verification acquisition on the corresponding storage location window when the consistency score is lower than the preset threshold, and to update the storage location object set and its consistency score. The evidence package generation module is used to generate and archive inventory evidence packages associated with the storage location object. The write-back module is used to write back the inventory discrepancy results and inventory evidence package to the WMS to complete the inventory loop. The abnormal work order module is used to generate abnormal work orders for storage location windows whose difference scores exceed the preset difference threshold and push them to the review terminal along with the inventory evidence package.
[0025] To ensure consistency and traceability in implementation, this embodiment uses the following data objects throughout the inventory process: An inventory task must include at least the inventory batch number, the target storage location set, the task time window, and route planning information; The warehouse location baseline model should at least include the warehouse location identifier, warehouse location spatial boundary, material identifier and quantity information corresponding to the warehouse location, and record the baseline model version number; The synchronous acquisition frame is a collection of acquisition data generated in units of storage location windows, which includes at least image data, RFID reading data, spatial perception data and pose information; The visual candidate set is a set of candidate objects obtained by image data detection and identifier parsing; the RFID candidate set is a set of candidate tags obtained by RFID reading data based on tag aggregation and stability filtering. The storage location object set is a set of object-level results output from the cross-modal binding solution. Each storage location object contains at least a tag identifier, storage location identifier, material identifier, quantity result, and consistency score.
[0026] A multimodal warehouse inventory method is proposed. The method takes the warehouse location as the smallest unit of inventory and achieves warehouse location-level alignment of multimodal data through warehouse location window gating. It outputs object-level inventory results by solving cross-modal binding. When the consistency score is insufficient, it triggers active review to converge the differences. Finally, it generates an evidence package and writes it back to WMS to form a closed loop.
[0027] The inventory platform obtains the inventory ledger from the WMS (Work Management System). The inventory ledger includes at least the location identifier, material identifier, and book quantity. The platform generates inventory tasks based on the inventory ledger and assigns an inventory batch number to each task. Each inventory task includes at least the target storage area, the target set of storage locations, and a task time window to limit the execution scope and time of the inventory terminal.
[0028] The inventory platform establishes a storage location benchmark model based on the inventory ledger, and the storage location benchmark model includes at least: Storage location identifier, used to uniquely identify a specific storage location; Storage location space boundary, used to describe the spatial extent of the storage location window in the warehouse coordinate system; The material identifier and quantity information corresponding to the storage location are used as a benchmark for consistency verification.
[0029] To ensure traceability, the baseline model at the warehouse location further records the baseline model version number. The baseline model version number used in the inventory task is fixed so that the inventory results can be linked to a specific baseline source.
[0030] The inventory platform controls the inventory terminal to inspect the target storage area according to the inventory task. During the inspection, the inventory terminal outputs a pose sequence. The pose sequence includes position, attitude, and timestamp. The pose sequence can be obtained by the positioning algorithm of the inventory terminal, such as using LiDAR point cloud and odometry for positioning, or using visual mapping and inertial measurement for positioning. The pose sequence is used for subsequent temporal alignment and spatial mapping of multimodal data to avoid ambiguity caused by the inability of data from different sensors to be aligned within the same storage location window.
[0031] The inventory platform divides the target warehouse area into multiple warehouse location windows based on the warehouse location benchmark model. To improve the spatial accuracy of the warehouse location windows, this embodiment further utilizes spatial perception data to identify the structure of shelf uprights, beams, or shelves to determine the warehouse location space boundaries. Specifically, planar and linear structures can be extracted from point cloud data to obtain the key geometric structure of the shelf. Then, the geometric structure is aligned with the warehouse location space boundaries in the warehouse location benchmark model to obtain warehouse location windows that correspond one-to-one with the warehouse location identifiers.
[0032] To achieve data alignment at the storage location level, this embodiment employs a data acquisition gating mechanism using storage location windows: When the inventory terminal's position enters the boundary of the storage space, the data acquisition gating of the storage space window is activated; when the inventory terminal's position leaves the boundary of the storage space, the data acquisition gating of the storage space window is deactivated.
[0033] This gating mechanism ensures that subsequent synchronous acquisition frames correspond one-to-one with storage location identifiers, reducing the risk of cross-location misreading caused by data mixing at the data acquisition entry point.
[0034] For each storage location window, the inventory terminal performs multimodal synchronous acquisition based on the pose sequence to obtain a synchronous acquisition frame corresponding to the storage location window. The synchronous acquisition frame includes at least: Image data; RFID data reading; RFID data reading includes tag identification, received signal strength, phase information, antenna number, read / write power level, and timestamp; Spatial perception data; spatial perception data is depth images or point cloud data.
[0035] To avoid ambiguity caused by different sampling frequencies of images, RFID, and spatial sensing, this embodiment performs timestamp alignment on synchronously acquired frames and maps the aligned multimodal data to the same coordinate system. Specifically: 1) Timestamp alignment: Based on the unified clock inside the inventory terminal, the timestamps of image data acquisition, RFID reading data acquisition, spatial perception data acquisition, and pose timestamps are aligned so that the same synchronous acquisition frame can correspond to a unified reference time. 2) Coordinate system mapping: Based on the external parameter calibration parameters, coordinate transformation is performed on the camera coordinate system, RFID antenna coordinate system and spatial perception coordinate system so that each modal data can be mapped to the inventory terminal base coordinate system; and combined with the pose sequence, it is further mapped to the warehouse coordinate system so as to perform spatial constraint calculation within the storage location window.
[0036] To enable those skilled in the art to directly implement coordinate mapping, this embodiment provides a standard homogeneous transformation expression. Let the warehouse coordinate system be... The coordinate system of the inventory terminal base is The camera coordinate system is The spatial perception coordinate system is , No. The coordinate system of each RFID antenna is Then we have: ; ; ; in, Let be the homogeneous transformation matrix of the pose sequence at time t. , , The fixed transformation matrix obtained by calibrating the extrinsic parameters.
[0037] Visual candidate sets and RFID candidate sets are generated separately within the storage location window.
[0038] 1) Visual candidate set generation: The inventory terminal performs cargo instance detection on the image data to obtain candidate target boxes. Barcode reading or character recognition is then performed on the corresponding areas of the candidate target boxes to obtain material identification.
[0039] When barcode reading or character recognition fails, the appearance feature vector of the candidate target box is extracted and similarity matching is performed with the appearance feature vector in the material master data to determine the material identifier.
[0040] Visual candidate objects should at least record the candidate target bounding box, material identifier, and visual confidence level for subsequent semantic consistency constraints.
[0041] 2) Generation of RFID candidate sets: The RFID reading data is segmented and aggregated according to the storage location window to generate a reading sequence. That is, using the tag identifier as the key, multiple reading records during the gate opening period are aggregated into a reading sequence of the same tag.
[0042] To suppress misjudgments caused by sporadic readings and long-distance cross-reading, this embodiment removes reading records with fewer than a preset threshold number of reads and reading records with received signal strength lower than a preset threshold strength. After stability screening, an RFID candidate set is obtained, ensuring that candidate tags have an effective sequence length that can be used for trajectory consistency calculation.
[0043] This embodiment performs cross-modal binding solution on the visual candidate set and RFID candidate set under the spatial constraints of the storage location window, and outputs a storage location object set. Each storage location object in the storage location object set includes a tag identifier, storage location identifier, material identifier, quantity result, and consistency score.
[0044] To ensure the reproducibility of cross-modal binding solutions, this embodiment constructs a matching score matrix and solves for the optimal binding relationship. A matching score is calculated for each visual candidate object and each RFID candidate tag, and a matching score matrix is constructed. The matching score includes spatial consistency score, trajectory consistency score, and semantic consistency score.
[0045] The overall matching score is determined by the following formula: ; in, Spatial consistency score; Scoring is given based on trajectory consistency. Score the semantic consistency. For the weighting coefficients, satisfying .
[0046] 1) Spatial Consistency Score: Spatial consistency is used to constrain the binding relationship to conform to the spatial distribution within the storage location window. Spatial perception data is used to determine the occupancy outline or occupancy area within the storage location window, and the position of the visual candidate object is obtained by mapping the candidate target box. The spatial constraint of the RFID candidate tag comes from the relative orientation features formed by the storage location window gating and multi-antenna pointing reading. The spatial consistency score can be determined as follows: ; in, This represents the position vector of the visual candidate object in the coordinate system of the storage location window. To estimate the spatial location of RFID candidate tags within the storage location window. This is the spatial attenuation parameter.
[0047] 2) Trajectory Consistency Score: The trajectory consistency score is determined by the correlation between the phase sequence or received signal strength sequence of the RFID candidate tag and the pose change sequence of the inventory terminal. This embodiment uses the phase sequence as an example to calculate the trajectory consistency score: ; in, This represents the phase change sequence of RFID candidate tags during gating. To inventory the sequence of relative distance changes between the terminal and visual candidate objects, This is the Pearson correlation coefficient. The Pearson correlation coefficient is calculated as follows: ; in, correspond , The mean; corresponding , The mean is denoted as N; N is the number of gated sampling points. When using the received signal strength sequence, Simply replace it with an RSSI variation sequence.
[0048] 3) Semantic Consistency Score: Semantic consistency is used to constrain the consistency between the material identifier bound to the label and the visually interpreted material identifier. If they are consistent, the semantic consistency score is higher; if they are inconsistent, the semantic consistency score is lower. This embodiment uses a strict matching method: ; After obtaining the matching score matrix, the binding relationship between the label identifiers and the visual candidate objects is solved, and the binding relationship is mapped to the corresponding storage location identifiers in the storage location window to output the storage location object set. In this embodiment, the binding solution is expressed as a maximum matching score problem:
[0049]
[0050] Where M represents the number of visual candidate objects and K represents the number of RFID candidate tags. This indicates that a visual object is bound to a label.
[0051] After binding is complete, a set of storage location objects is output. Each storage location object includes at least: label identifier, storage location identifier, material identifier, quantity result, and consistency score. The quantity result can be obtained by counting the objects in the binding relationship, and anomaly counts can be verified by combining the occupancy contours of spatial awareness data. The consistency score is used to characterize the reliability of the storage location object binding. In this embodiment, the consistency score is defined as the normalized result of the comprehensive matching score: ; in, Assess the consistency of the storage location objects. and Match the minimum and maximum scores for the current storage location window.
[0052] The consistency of the collection of storage location objects with the storage location baseline model is verified to obtain the inventory discrepancy results. The inventory discrepancy results include one or more of the following: missing discrepancies, extra discrepancies, misalignment discrepancies, and quantity discrepancies.
[0053] Missing discrepancies refer to situations where the baseline exists but the inventory count was not output. The extra discrepancy is due to a situation where the inventory output does not exist; Misalignment discrepancies occur when storage location identifiers do not match. Quantity discrepancy refers to a situation where the inventory quantity does not match the baseline quantity.
[0054] To facilitate anomaly priority control, this embodiment generates a difference score for each difference type. The difference score is determined by the consistency score, the difference type weight, and the magnitude of the difference quantity. The difference score can be calculated using the following formula: ; in, Scoring the consistency of the storage location objects. Weights for difference types, For the magnitude of the difference in quantity, ,, This is the rating coefficient.
[0055] After the consistency verification is completed, the consistency score determines whether to trigger an active review.
[0056] When the consistency score is lower than a preset threshold, the control inventory terminal performs active verification and collection on the corresponding storage location window, and updates the storage location object set and its consistency score based on the active verification and collection results. The active verification and collection includes one or more of the following verification actions: 1) Change the observation angle or observation distance to acquire additional image data: The inventory terminal performs a short-distance attitude adjustment in front of the storage location window to cover the obstructed area with the camera's field of view. After re-capturing the image, it regenerates the visual candidate set and rebinds it.
[0057] 2) Adjust the read / write power level and switch the antenna number to perform directional repeat reading: During the verification phase, the inventory terminal adjusts the RFID read / write power to a lower level and switches the antenna number for directional repeat reading to narrow the reading range and reduce the probability of cross-location reads. Directional repeat reading works in conjunction with location window gating; RFID reading is only activated when the inventory terminal's position is within the current location window gating range, and reading is deactivated when it exceeds the gating range.
[0058] 3) Perform multiple samplings at preset stop points and consistently fuse the data: The inventory terminal performs multiple samplings at a preset stop point in front of the storage location window, forming multiple sets of synchronous collection frames. The binding relationships obtained from multiple samplings are fused for consistency. If the binding results are stable, the consistency score is improved; otherwise, the storage location window is marked as a candidate for manual review.
[0059] After the review is completed, update the set of storage location objects and the consistency score, and re-execute the consistency check to converge the differences.
[0060] An inventory evidence package is generated for the updated set of storage location objects, and the inventory discrepancy results and the inventory evidence package are written back to the WMS to complete the inventory closure loop. The inventory evidence package includes RFID read data associated with the storage location objects, image data, spatial perception data, pose sequence fragments, consistency scores, and proactive verification collection records.
[0061] To enhance the verifiability of the evidence package, the inventory evidence package further includes a digest checksum. The digest checksum is determined by a hash digest generated from the RFID read data, image data, and spatial sensing data, and is stored in conjunction with the inventory task batch number, storage location identifier, and tag identifier. The digest checksum can be calculated as follows: ; in, The evidence package is a serialized byte stream that includes at least RFID reading data, image data index or content, spatial perception data index or content, pose sequence fragments, consistency scores and review records.
[0062] After writing back to the WMS, an exception work order is generated for storage location windows with difference scores exceeding the preset difference threshold, and the corresponding inventory evidence package is pushed to the review terminal. After confirming the inventory difference results, the review terminal updates the storage location baseline model and records the version number before and after the update, as well as the corresponding inventory task batch number, for subsequent traceability.
[0063] In real-world warehousing environments, common problems include RFID cross-reading due to multi-path metal shelving, visual reading failures caused by outer packaging obstruction, and misidentification due to similar-looking materials. This embodiment achieves result convergence through location window gating, cross-modal binding solution, and active verification closed-loop. The specific process is as follows: When the inventory terminal enters the gated area of a specific storage location window, multimodal synchronous acquisition is initiated. The image acquisition unit acquires an image of the storage location window, the spatial perception unit acquires point cloud or depth data of that window, and the RFID acquisition unit performs multiple reads using a multi-antenna array, recording the phase and received signal strength sequences. The candidate generation module first eliminates tags with low read counts and low strength to reduce cross-reading interference over long distances, and then constructs a matching score matrix based on image candidate objects and RFID candidate tags.
[0064] If visual recognition fails, the visual candidate set is supplemented with material identifiers by matching the appearance feature vectors with the material master data to maintain semantic constraints. If RFID cross-reading leads to unstable binding relationships, the trajectory consistency score will decrease due to insufficient correlation between the phase sequence and pose change sequence, causing the consistency score to fall below the threshold and triggering active review. During the review phase, low-power directional rereading and attitude adjustment are used to converge the RFID reading coverage to the current storage location window and ensure the visual viewpoint covers obstructed areas. After review, the binding relationship stabilizes, the difference score decreases, and closed-loop write-back is completed.
[0065] Through the above mechanism, the inventory results are supported by object-level binding relationships and have the traceability of evidence packages and hash digests, which is different from the implementation method of simply reading RFID or directly outputting quantity differences after visual recognition, thus forming a stable landing point in terms of novelty and inventiveness.
[0066] The working principle of this embodiment is to establish a warehouse location baseline model with the warehouse location as the smallest inventory unit, and to limit the multimodal data collected by the inventory terminal during the inspection process to a single warehouse location window through warehouse location window gating, thereby providing a consistent data boundary for subsequent object-level cross-modal binding. Specifically, the inventory platform generates inventory tasks based on the WMS inventory ledger and establishes a warehouse location baseline model. The warehouse location baseline model includes at least the warehouse location identifier, the warehouse location spatial boundary, and the material identifier and quantity information corresponding to the warehouse location. The inventory terminal inspects the target warehouse area according to the inventory task and continuously outputs the pose sequence. When the inventory terminal pose enters a warehouse location spatial boundary, the data acquisition gating of that warehouse location window is opened, and the gating is closed when it leaves, so that each synchronous acquisition frame corresponds one-to-one with the corresponding warehouse location identifier. During the gating period, the inventory terminal synchronously collects image data, RFID reading data, and spatial perception data, and performs timestamp alignment and coordinate system mapping on each modal data, so that spatial constraint calculations can be performed in the same coordinate system. Subsequently, visual candidate sets and RFID candidate sets are generated in the storage location window. The visual candidate set is obtained by cargo instance detection and identification parsing of image data, while the RFID candidate set is obtained by aggregating RFID read data by tag and filtering it through read count thresholds and signal strength thresholds to suppress the interference of occasional readings and long-distance cross-reading on the results.
[0067] After the candidate set is generated, the system performs cross-modal binding solutions under the spatial constraints of the storage location window. It constructs a matching score matrix for visual candidate objects and RFID candidate tags, and calculates a comprehensive matching score based on spatial consistency, trajectory consistency, and semantic consistency. Spatial consistency is used to constrain the tag reading source to match the occupied area and visual object position within the storage location window. Trajectory consistency utilizes the correlation between the RFID phase sequence or received signal strength sequence and the inventory terminal pose change sequence to determine whether the tag response changes with the inspection movement conform to the observation pattern of the storage location window. Semantic consistency is used to constrain the consistency relationship between RFID-bound material identifiers and visually resolved material identifiers. The system solves for the binding relationship between tag identifiers and visual candidate objects under one-to-one binding constraints, and maps the binding relationship to the storage location identifiers in the current storage location window, forming a storage location object set. Each storage location object in the set contains at least a tag identifier, a storage location identifier, a material identifier, a quantity result, and a consistency score. The consistency score is used to characterize the reliability of the binding relationship. When the consistency score reaches the preset threshold, the system will perform consistency verification between the collection of storage location objects and the storage location benchmark model, and output one or more of the following: missing difference, extra difference, misalignment difference, and quantity difference. The system will also combine the consistency score, the weight of the difference type, and the magnitude of the difference quantity to generate a difference score, which will serve as the basis for prioritizing anomaly handling.
[0068] When the consistency score falls below a preset threshold, the system triggers an active review loop, performing review data collection and updating the set of storage location objects and their consistency scores for the corresponding storage location window. The review actions include at least changing the observation angle or distance to collect supplementary image data, adjusting the RFID read / write power level and switching antenna numbers for directional rereading, and performing multiple samplings at preset stopping points and performing consistency fusion on the sampling results. The directional rereading is coordinated with the storage location window gating; reading is initiated when the inventory terminal's pose meets the storage location window constraints and deactivated when it leaves the gating range, further reducing the probability of cross-storage location cross-reading. After the review is completed, the system re-executes cross-modal binding solution and consistency verification, causing the discrepancies to converge to a stable state under the constraints of the review actions. Finally, the system generates an inventory evidence package for the updated set of storage location objects and writes the inventory discrepancy results and the inventory evidence package back to the WMS to complete the inventory closure loop. The inventory evidence package includes at least RFID reading data, image data, spatial perception data, pose sequence fragments, consistency scores, and proactive review collection records associated with the storage location objects. It further generates a summary verification value and binds it to the inventory batch number, storage location identifier, and tag identifier for storage, for post-audit traceability and consistency verification. For storage location windows with discrepancy scores higher than a preset threshold, the system generates an exception work order and pushes the corresponding evidence package to the review terminal. After review confirmation, the system updates the storage location benchmark model and records the version number and inventory batch number, so that the inventory results and benchmark data form a traceable closed-loop link, thereby improving the accuracy and process verifiability of storage location inventory in complex warehousing environments.
[0069] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.
Claims
1. A warehouse inventory method based on multimodal analysis, characterized in that, Includes the following steps: S1: Generate inventory count tasks based on the inventory ledger of the warehouse management system (WMS) and establish a warehouse location benchmark model. The warehouse location benchmark model includes warehouse location identifiers, warehouse location space boundaries, and the material identifiers and quantity information corresponding to the warehouse location. S2: Control the inventory terminal to inspect the target warehouse area according to the inventory task, and obtain the pose sequence of the inventory terminal during the inspection process. The pose sequence includes position, attitude and timestamp. S3: Divide the target storage area into storage location windows based on the storage location benchmark model to obtain multiple storage location windows; S4: For each storage location window, perform multimodal synchronous acquisition based on the pose sequence to obtain the synchronous acquisition frame corresponding to the storage location window. The synchronous acquisition frame includes image data, RFID reading data and spatial perception data. Among them, the RFID reading data includes tag identification, received signal strength, phase information, antenna number, read / write power level and timestamp. The spatial perception data is depth image or point cloud data. S5: Generate a visual candidate set and an RFID candidate set in the storage location window. The visual candidate set is obtained by performing target detection and tag parsing on the image data, and the RFID candidate set is obtained by tag aggregation on the RFID read data. S6: Based on the spatial constraints of the storage location window, perform cross-modal binding solution on the visual candidate set and the RFID candidate set, and output the storage location object set. Each storage location object in the storage location object set includes a tag identifier, storage location identifier, material identifier, quantity result and consistency score. S7: Perform consistency verification between the collection of storage location objects and the storage location baseline model, obtain the inventory discrepancy results, and determine whether to trigger an active review based on the consistency score; S8: When the consistency score is lower than the preset threshold, the control inventory terminal performs active verification collection on the corresponding storage location window, and updates the storage location object set and its consistency score based on the active verification collection results. S9: Generate an inventory evidence package for the updated set of storage location objects, and write back the inventory discrepancy results and the inventory evidence package to WMS to complete the inventory closure loop; The inventory evidence package includes RFID reading data associated with the storage location objects, image data, spatial perception data, pose sequence fragments, consistency scores, and proactively reviewed collection records.
2. The warehouse inventory method based on multimodal analysis according to claim 1, characterized in that, The multimodal synchronous acquisition includes: Image data, RFID reading data, spatial perception data, and pose information are timestamped and aligned, and the aligned multimodal data are uniformly mapped to the same coordinate system. The coordinate system mapping includes performing coordinate transformations on the camera coordinate system, RFID antenna coordinate system, and spatial sensing coordinate system based on extrinsic calibration parameters.
3. The warehouse inventory method based on multimodal analysis according to claim 2, characterized in that, The storage location window division includes: Based on spatial perception data, identify the structure of shelf uprights, beams or shelves to determine the boundaries of storage space. Align the spatial boundary of the storage location with the storage location identifier in the storage location benchmark model to obtain a storage location window that corresponds one-to-one with the storage location identifier; The data acquisition gating of the storage location window is activated when the inventory terminal's position enters the storage location space boundary, and deactivated when the inventory terminal's position leaves the storage location space boundary.
4. The warehouse inventory method based on multimodal operation according to claim 3, characterized in that, The generation of the visual candidate set includes: Cargo instance detection is performed on the image data to obtain candidate bounding boxes; Perform barcode reading or character recognition on the area corresponding to the candidate target box to obtain the material identification; When barcode reading or character recognition fails, the material identifier is determined by similarity matching between the appearance feature vector of the candidate target box and the material master data. The generation of the RFID candidate set includes: The RFID reading data is segmented and aggregated according to the storage location window to generate a reading sequence, and reading records with fewer than a preset number of reads or a received signal strength lower than a preset strength threshold are removed.
5. A warehouse inventory method based on multimodal analysis according to claim 4, characterized in that, The cross-modal binding solution includes: For each visual candidate object and each RFID candidate tag, a matching score is calculated, and a matching score matrix is constructed. The matching score includes spatial consistency score, trajectory consistency score, and semantic consistency score; The binding relationship between the label identifier and the visual candidate object is solved based on the matching score matrix, and the binding relationship is mapped to the corresponding storage location identifier in the storage location window to output the storage location object set. The trajectory consistency score is determined by the correlation between the phase sequence or received signal strength sequence of the RFID candidate tag and the pose change sequence of the inventory terminal.
6. The warehouse inventory method based on multimodal operation according to claim 5, characterized in that, The consistency check includes outputting one or more of the following difference types: missing difference, extra difference, misaligned difference, and quantity difference; A difference score is generated for the difference type, and the difference score is determined by the consistency score, the difference type weight, and the difference quantity magnitude. The proactive review and collection includes one or more of the following review actions: Control the inventory terminal to change the observation angle or observation distance to collect additional image data; Control the RFID reader / writer to adjust the read / write power level and switch the antenna number to perform directional repeat reading; The control inventory terminal performs multiple samplings at preset stop points and performs consistency fusion of the sampling results.
7. A warehouse inventory method based on multimodal analysis according to claim 6, characterized in that, The inventory evidence package further includes a digest verification value, which is determined by a hash digest generated from RFID reading data, image data, and spatial perception data, and is bound and stored with the inventory task batch number, storage location identifier, and tag identifier. After writing back to WMS, the following is also included: For storage location windows with difference scores exceeding the preset difference threshold, an abnormal work order is generated and associated with the corresponding inventory evidence package, which is then pushed to the review terminal. After verifying the inventory discrepancies at the verification terminal, update the warehouse location baseline model and record the version number of the warehouse location baseline model before and after the update, as well as the corresponding inventory task batch number.
8. A multimodal warehouse inventory system applicable to the multimodal warehouse inventory method described in any one of claims 1-7, characterized in that, include: The task generation module is used to generate inventory counting tasks based on the inventory ledger of the Warehouse Management System (WMS) and establish a warehouse location benchmark model. The pose acquisition module is used to control the inventory terminal to perform inspections and acquire pose sequences. The storage location window division module is used to divide the target storage area into multiple storage location windows based on the storage location benchmark model and perform data acquisition gating. The multimodal acquisition module is used to generate synchronous acquisition frames for each storage location window. The synchronous acquisition frames include image data, RFID reading data, spatial perception data, and pose information. The candidate generation module is used to generate visual candidate sets and RFID candidate sets; The cross-modal binding module is used to perform cross-modal binding solutions on the visual candidate set and the RFID candidate set and output the storage location object set, and calculate the consistency score for the storage location objects in the storage location object set; The consistency verification module is used to verify the consistency between the collection of storage location objects and the storage location baseline model and obtain the inventory difference results. The proactive review control module is used to control the multimodal acquisition module to perform proactive review acquisition on the corresponding storage location window and update the storage location object set and its consistency score when the consistency score is lower than the preset threshold. The evidence package generation module is used to generate and archive inventory evidence packages associated with the storage location object. The write-back module is used to write back the inventory discrepancy results and inventory evidence package to the WMS to complete the inventory loop.
9. A dynamic target detection system based on a three-dimensional low-altitude surveillance radar according to claim 8, characterized in that, The system also includes a time synchronization and calibration module, which is used to align image data, RFID reading data, spatial perception data and pose information with timestamps, and to realize coordinate transformation between the camera coordinate system, RFID antenna coordinate system and spatial perception coordinate system based on external parameter calibration parameters. The multimodal acquisition module includes an RFID reader and a multi-antenna array; The RFID reading data includes tag identification, received signal strength, phase information, antenna number, read / write power level, and timestamp.
10. A dynamic target detection system based on a three-dimensional low-altitude surveillance radar according to claim 9, characterized in that, The cross-modal binding module is used to construct a matching score matrix and solve the binding relationship between the label identifier and the visual candidate object. The matching score matrix includes a spatial consistency score matrix, a trajectory consistency score matrix, and a semantic consistency score matrix. The active verification control module is used to control the RFID reader to adjust the reading and writing power level and switch the antenna number to perform directional repeat reading during the active verification and acquisition phase. The evidence package generation module is used to generate hash digests of the inventory evidence package and bind them to the inventory task batch number and storage location identifier for consistency verification and audit traceability. The system also includes an exception work order module, which is used to generate exception work orders for storage location windows with difference scores higher than preset difference thresholds and push them to the review terminal along with the inventory evidence package.
