Master-slave robot asset hierarchical inventory system based on multi-modal scanning

Abstract

This invention discloses a multimodal scanning-based asset hierarchical inventory system using main and auxiliary robots, comprising a main and auxiliary robot unit, a multimodal AI scanning unit, a visual matching processing unit, an asset hierarchical control unit, a data interaction unit, and a cloud management platform. The main and auxiliary robot unit consists of a main robot module and an auxiliary robot module. The multimodal AI scanning unit is integrated into the main and auxiliary robot unit and includes a visual image scanning module, an RFID scanning module, and a laser contour scanning module. The asset hierarchical control unit divides assets into a core layer, an important layer, and a general layer, configuring differentiated scanning strategies, inventory frequencies, and matching similarity thresholds for different layers. The visual matching processing unit selects corresponding scan data based on the asset layer and compares it with the asset's baseline features. When the matching similarity is lower than the corresponding layer's threshold, an abnormal signal is output. This invention achieves comprehensive inventory coverage and refined asset identification control in complex warehousing environments.

Description

Technical Field

[0001] This invention relates to the field of intelligent warehouse management technology, and in particular to a multimodal scanning-based primary and secondary robot asset hierarchical inventory system. Background Technology

[0002] Asset inventory is a crucial part of corporate asset management. By regularly verifying the quantity, condition, and location of assets, it ensures consistency between accounts and actual assets, preventing asset loss and management risks. In industries such as warehousing, manufacturing, healthcare, and finance, the number of managed assets is vast, diverse, and geographically dispersed. Traditional manual inventory methods are inefficient and prone to errors, failing to meet the precision and frequency requirements of modern enterprises for asset control.

[0003] With the development of mobile robot technology and automatic identification technology, robot-based automated inventory solutions are gradually being applied. Existing inventory robot solutions typically use a single robot platform equipped with an RFID reader or visual recognition module, which autonomously travels along a preset path and completes asset scanning. However, the above solutions have the following shortcomings:

[0004] Firstly, due to their size and mobility limitations, single robots struggle to enter narrow aisles between shelves or effectively scan high-level assets, resulting in blind spots in inventory coverage and making it impossible to achieve full-area inventory in complex warehousing environments.

[0005] Secondly, existing solutions mostly use a single sensing method (RFID or visual recognition only) for asset verification. In scenarios such as obstruction, uneven lighting, or damaged tags, the risk of identification failure is high, and the accuracy and reliability of the inventory results are difficult to guarantee.

[0006] Third, the existing inventory plan adopts a uniform inventory strategy for all assets, failing to configure differentiated inventory frequency, scanning accuracy and verification rules according to the differences in asset value and importance. This results in insufficient control accuracy for high-value assets and waste of scanning resources for low-value assets, and both inventory efficiency and control effectiveness need to be improved. Summary of the Invention

[0007] This invention provides a hierarchical inventory system for master and auxiliary robot assets based on multimodal scanning, which can achieve full coverage, multimodal fusion recognition, and differentiate inventory strategies according to the importance of assets, thus partially solving or alleviating the above-mentioned shortcomings in the prior art.

[0008] The technical solution of this invention is:

[0009] The main and auxiliary robot asset hierarchical inventory system based on multimodal scanning is characterized by including a main and auxiliary robot unit, a multimodal AI scanning unit, a visual matching processing unit, an asset hierarchical management and control unit, a data interaction unit, and a cloud management platform.

[0010] The main and auxiliary robot unit includes a main robot module and at least one auxiliary robot module;

[0011] The multimodal AI scanning unit is integrated into the main and auxiliary robot units and includes a visual image scanning module, an RFID scanning module, and a laser contour scanning module.

[0012] The asset hierarchical management unit divides assets into a core layer, an important layer, and a normal layer. It configures tri-modal, bi-modal, and uni-modal scanning strategies for the three layers respectively, and sets differentiated inventory frequency and matching similarity thresholds.

[0013] The visual matching processing unit selects corresponding scan data based on the asset level and compares it with the asset benchmark features stored in the cloud management platform. When the matching similarity is lower than the corresponding level threshold, an abnormal signal is output.

[0014] The data interaction unit connects the aforementioned units to the cloud management platform; the cloud management platform stores the baseline characteristic information of each asset and is responsible for issuing inventory tasks.

[0015] In some embodiments, the main and auxiliary robot units further include an autonomous navigation module, an obstacle avoidance module, and a positioning module; the autonomous navigation module plans the inventory path based on SLAM (Simultaneous Localization and Mapping) technology; the obstacle avoidance module identifies obstacles and dynamically adjusts the path by linking lidar and visual sensors; and the positioning module uses UWB (Ultra-Wideband) positioning technology to achieve precise positioning of the robot.

[0016] In some embodiments, the auxiliary robot module is a small tracked structure, and at least one of the auxiliary robot modules is equipped with a lifting mechanism. The lifting height of the lifting mechanism is in the range of 0.5m to 3m, and it is used to complete the inventory operation of high-rise assets.

[0017] In some embodiments, the multimodal AI scanning unit further includes an environment adaptive adjustment module, which collects real-time light intensity, temperature and humidity parameters of the scene, and automatically adjusts the exposure time of the visual image scanning module, the laser power of the laser contour scanning module and the reading distance of the RFID scanning module according to the collected parameters.

[0018] In some embodiments, the visual matching processing unit includes a data preprocessing module, a multimodal feature extraction module, a visual matching module, and an anomaly recognition module. The multimodal feature extraction module uses a CNN convolutional neural network to extract deep visual features from visual images and morphological features from laser contour data, and encodes RFID data to form an identity feature vector, generating a multi-dimensional feature vector. The visual matching module performs fast coarse matching of visual features using the ORB algorithm, then performs fine matching optimization of the matching results using the SIFT algorithm, and finally combines deep learning embedding matching technology to calculate the similarity between the multi-dimensional feature vector and the asset baseline feature vector. The anomaly recognition module compares the calculated matching similarity with the corresponding level threshold. If the similarity is lower than the threshold, it is determined to be an inventory anomaly. Anomaly types include missing inventory, incorrect inventory, asset damage, and position deviation.

[0019] In some embodiments, the visual matching module further employs the RANSAC method to perform geometric verification on the matching results of visual features, and outputs the final matching result after eliminating mismatched feature point pairs.

[0020] In some embodiments, the asset hierarchical management unit also has a hierarchical dynamic adjustment function, which updates the asset hierarchy according to changes in asset value and importance, and synchronizes the update results to the cloud management platform and the main and auxiliary robot units, so that the scanning strategy, inventory frequency and matching similarity threshold of the corresponding assets are switched synchronously with the hierarchy update.

[0021] In some embodiments, the system further includes an anomaly handling unit, which is bidirectionally connected to the asset hierarchical management unit and the cloud management platform. The anomaly handling unit receives anomaly signals output by the visual matching processing unit. For missing inventory anomalies, it schedules the main and auxiliary robot units to return to the corresponding area for rescanning. For misplaced inventory anomalies, it compares the asset baseline information with the scan data and sends the misplaced inventory details to the cloud management platform. For asset damage or positional deviation anomalies, it controls the main and auxiliary robot units to collect images of the abnormal scene and update the status information of the corresponding asset.

[0022] In some embodiments, the data interaction unit adopts a dual-mode communication method of 5G and Wi-Fi and encrypts the transmitted data; the data interaction unit also has an offline data caching function, which automatically stores inventory data and operation records when the network is interrupted, and synchronizes the offline inventory data to the cloud management platform after the network is restored.

[0023] In some embodiments, the cloud management platform further includes a task management module, a real-time monitoring module, a data management module, a statistical analysis module, and an interface docking module; the task management module supports the splitting and collaborative scheduling of inventory tasks; the real-time monitoring module monitors the inventory site in real time through the visual sensors of the main and auxiliary robot units; the data management module stores asset baseline information, inventory data, and abnormal data; the statistical analysis module performs statistical analysis on the inventory data and generates an inventory report; the interface docking module docks with external management systems through standardized interfaces to achieve bidirectional synchronization of asset information.

[0024] The beneficial effects of this invention are as follows:

[0025] This invention utilizes a master-slave collaborative architecture with master and auxiliary robot modules. The master robot module is responsible for global task scheduling and data aggregation, while the auxiliary robot module is responsible for inventory operations in confined spaces and high-rise assets. Through collaborative operation, the two achieve full coverage of different spatial forms in complex warehouse environments, solving the problem that a single robot cannot cover all inventory scenarios due to its size and mobility limitations.

[0026] The multimodal AI scanning unit integrates a visual image scanning module, an RFID scanning module, and a laser contour scanning module. The three scanning modes are independent of each other and complement each other, overcoming the risk of recognition failure in scenarios such as occlusion, uneven lighting, or damaged tags when using a single sensing method. This improves the accuracy of asset identification and environmental adaptability.

[0027] The asset tiered management unit divides assets into core, important, and ordinary layers, and configures tri-modal, bi-modal, and uni-modal scanning strategies for different layers. It also sets differentiated inventory frequencies and matching similarity thresholds, tilting limited scanning resources toward high-value assets. This achieves a dynamic balance between inventory accuracy and operational efficiency, avoiding the problems of resource waste and risk exposure of high-value assets that coexist in the traditional homogeneous inventory model.

[0028] The visual matching processing unit selects corresponding scan data based on asset level and compares it with asset benchmark features. It uses the matching similarity threshold corresponding to the level as the basis for anomaly judgment, realizing differentiated verification depth of assets at different value levels. When the matching similarity is lower than the corresponding threshold, an anomaly signal is output in a timely manner, improving the timeliness of inventory anomaly identification and the targeted handling of anomalies. Attached Figure Description

[0029] Figure 1 This is a structural block diagram of an asset tiered inventory system provided in some embodiments of the present invention.

[0030] Figure 2 The above is a structural block diagram of a master and auxiliary robot unit provided in some embodiments of the present invention.

[0031] Figure 3 This is a structural block diagram of a multimodal AI scanning unit provided in some embodiments of the present invention.

[0032] Figure 4 This is a structural block diagram of a visual matching processing unit provided in some embodiments of the present invention.

[0033] Figure 5 This is a structural block diagram of a cloud management platform provided in some embodiments of the present invention.

[0034] Component names and serial numbers in the diagram:

[0035] 1. Main and Auxiliary Robot Unit; 11. Main Robot Module; 12. Auxiliary Robot Module; 13. Autonomous Navigation Module; 14. Obstacle Avoidance Module; 15. Positioning Module; 2. Multimodal AI Scanning Unit; 21. Visual Image Scanning Module; 22. RFID Scanning Module; 23. Laser Contour Scanning Module; 24. Environmental Adaptive Adjustment Module; 3. Visual Matching Processing Unit; 31. Data Preprocessing Module; 32. Multimodal Feature Extraction Module; 33. Visual Matching Module; 34. Anomaly Recognition Module; 4. Asset Hierarchical Management Unit; 5. Data Interaction Unit; 6. Cloud Management Platform; 61. Task Management Module; 62. Real-time Monitoring Module; 63. Data Management Module; 64. Statistical Analysis Module; 65. Interface Integration Module; 7. Anomaly Handling Unit. Detailed Implementation

[0036] To make the above-mentioned objectives, features, and advantages of this application more apparent and understandable, the specific embodiments of this application are described in detail below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of this application. However, this application can be implemented in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of this application. Therefore, this application is not limited to the specific embodiments disclosed below.

[0037] Example

[0038] See Figures 1 to 5 As shown, the present invention provides a multimodal scanning-based master and auxiliary robot asset hierarchical inventory system, including a master and auxiliary robot unit 1, a multimodal AI scanning unit 2, a visual matching processing unit 3, an asset hierarchical management and control unit 4, a data interaction unit 5, and a cloud management platform 6.

[0039] The main and auxiliary robot unit 1 includes a main robot module 11 and at least one auxiliary robot module 12. The main robot module 11 is responsible for task allocation and inventory data aggregation for the overall inventory area, while the auxiliary robot module 12 is responsible for inventory operations in local areas, confined spaces, and high-rise assets. The two work in concert. Specifically, after receiving the inventory task from the cloud management platform 6, the main robot module 11 divides the inventory area into several sub-areas according to a preset map. Based on the spatial complexity, number of assets, and asset hierarchy of each sub-area, it allocates the inventory tasks for the corresponding sub-areas to each auxiliary robot module 12, and plans the operation path and sequence of each auxiliary robot module 12. During the operation, the main robot module 11 receives real-time reports of inventory progress and operating status from each auxiliary robot module 12. When an auxiliary robot module 12 experiences a task anomaly or inventory omission, the main robot module 11 dynamically dispatches the nearest auxiliary robot module 12 to verify the data. After all sub-areas are inventoried, the main robot module 11 aggregates the inventory data and anomaly handling records from each auxiliary robot module 12, verifies them, and then transmits them uniformly to the cloud management platform 6. This collaborative scheduling mechanism enables the main and auxiliary robot units 1 to complete the full-area inventory with optimal overall path planning and task allocation, avoiding duplicate and missed inventory checks and improving the overall system efficiency. This main-auxiliary collaborative architecture allows the system to simultaneously cover different spatial forms such as open areas and narrow passages, low-level shelves and high-level equipment cabinets, overcoming the bottleneck of a single robot's inability to cover the entire area due to size and mobility limitations, and significantly improving the overall progress efficiency of the inventory task.

[0040] The multimodal AI scanning unit 2 is integrated into the main and auxiliary robot units 1. In a preferred embodiment, the main robot module 11 and the auxiliary robot module 12 each carry a set of multimodal AI scanning units 2, serving their respective inventory areas and moving with the robot to each inventory location to perform scanning operations. The multimodal AI scanning unit 2 includes a visual image scanning module 21, an RFID scanning module 22, and a laser contour scanning module 23. The visual image scanning module 21 uses an industrial camera to collect features such as the appearance, nameplate markings, and barcode codes of the assets for identification and verification of the asset's appearance. The RFID scanning module 22 reads the unique identification code information stored in the asset's built-in RFID tag to achieve rapid and accurate verification of the asset's identity. The laser contour scanning module 23 uses a laser sensor to collect the asset's three-dimensional contour data for determining the integrity of the asset's shape. The data acquisition of the three scanning modules is independent and complementary, overcoming the risk of recognition failure in scenarios such as occlusion, uneven lighting, or damaged tags caused by a single sensing method.

[0041] The asset tiered management unit 4 divides all managed assets into three tiers: core, important, and ordinary, based on their value and importance. For core-tier assets, a three-modal simultaneous scanning strategy using a visual image scanning module 21, an RFID scanning module 22, and a laser contour scanning module 23 is implemented, with a high inventory frequency and a strict matching similarity threshold to maximize the inventory accuracy and verification reliability of high-value assets. For important-tier assets, a dual-modal scanning strategy using a visual image scanning module 21 and an RFID scanning module 22 is implemented, ensuring verification accuracy while appropriately reducing scanning resource consumption. For ordinary-tier assets, a single-modal scanning strategy using a visual image scanning module 21 is implemented, with corresponding inventory frequency and matching similarity thresholds set to further improve the robot's overall operational efficiency while meeting basic inventory requirements. This tiered configuration strategy allocates limited scanning resources to high-value assets, achieving a dynamic balance between inventory accuracy and operational efficiency, and avoiding the resource waste and risk exposure problems caused by the "one-size-fits-all" approach to high and low-value assets in traditional homogeneous inventory models.

[0042] The visual matching processing unit 3 is bidirectionally connected to both the multimodal AI scanning unit 2 and the asset hierarchical control unit 4. During inventory operations, the visual matching processing unit 3 selects corresponding scan data combinations from the scan data collected by the multimodal AI scanning unit 2 based on the asset hierarchy determined by the asset hierarchical control unit 4, and compares the selected scan data with the asset's baseline features pre-stored in the cloud management platform 6. When the calculated matching similarity is lower than the threshold corresponding to that hierarchy, the visual matching processing unit 3 outputs an anomaly signal to the asset hierarchical control unit 4 and the cloud management platform 6, triggering subsequent anomaly handling procedures. This "hierarchical-driven comparison" mechanism ensures that assets at different value levels differ in verification depth and threshold strictness, further enhancing the technical effectiveness of hierarchical control.

[0043] The data interaction unit 5 connects the main and auxiliary robot unit 1, the multimodal AI scanning unit 2, the visual matching processing unit 3, the asset hierarchical management unit 4, and the cloud management platform 6. It undertakes the bidirectional transmission of inventory data, control commands, and abnormal signals between these units, ensuring the coordinated operation of each unit throughout the entire inventory process. The cloud management platform 6 centrally stores the baseline characteristic information of all managed assets and issues inventory tasks to the main and auxiliary robot unit 1. Simultaneously, it stores and statistically analyzes the data generated during the inventory process, providing managers with visual management support for the overall inventory status.

[0044] See Figure 2 As shown, in some embodiments, the main and auxiliary robot unit 1 further includes an autonomous navigation module 13, an obstacle avoidance module 14, and a positioning module 15.

[0045] The autonomous navigation module 13 plans the inventory path based on SLAM (Simultaneous Localization and Mapping) technology. Specifically, when the robot first enters the inventory environment, the autonomous navigation module 13 uses LiDAR and cameras to collect real-time environmental data, simultaneously builds an environmental map, and determines the robot's own position. Based on this, and combined with the inventory task area issued by the cloud management platform 6, it plans the optimal path covering all inventory locations. When the inventory environment changes (such as shelf relocation, aisle closure, etc.), the autonomous navigation module 13 can update the map based on the real-time collected environmental data and replan the path to ensure the continuous progress of the inventory task.

[0046] The obstacle avoidance module 14 identifies obstacles and dynamically adjusts the path by linking a lidar and a vision sensor. The lidar is responsible for detecting static obstacles within a certain range along the robot's direction of travel, while the vision sensor is responsible for identifying dynamic obstacles such as people and forklifts. The two work together to form complementary perception, improving the comprehensiveness of obstacle recognition. When an obstacle is detected, the obstacle avoidance module 14 calculates the detour path in real time and instructs the robot to perform dynamic avoidance, effectively protecting the robot's hardware safety while ensuring that the inventory operation is not interrupted by obstacle interference.

[0047] The positioning module 15 employs UWB (Ultra-Wideband) positioning technology to achieve precise robot positioning. UWB positioning calculates the robot's real-time spatial coordinates by measuring the signal distance between several pre-deployed base stations in the inventory area and the UWB tags mounted on the robot, achieving centimeter-level positioning accuracy. This precise positioning data provides a reliable location benchmark for the autonomous navigation module 13's path planning and ensures that each inventory record includes accurate spatial coordinate information, facilitating the cloud management platform 6's accurate identification and tracing of asset location anomalies (such as assets deviating from their intended storage location).

[0048] The three modules work together to enable the main and auxiliary robot unit 1 to autonomously plan paths, avoid obstacles in real time, and accurately locate itself in complex warehouse environments, providing a basic guarantee for the multimodal AI scanning unit 2 to stably collect scanning data in the correct position.

[0049] In some embodiments, the auxiliary robot module 12 is a small tracked structure, and at least one auxiliary robot module 12 is equipped with a lifting mechanism with a lifting height range of 0.5m to 3m, which is used to complete the inventory operation of high-rise assets.

[0050] The tracked architecture gives the auxiliary robot module 12 strong terrain adaptability. Compared with the wheeled structure, the track has a larger ground contact area and a lower center of gravity, enabling it to move stably in narrow aisles between shelves, uneven ground areas, and environments with minor obstacles, effectively expanding the reach of the auxiliary robot module 12 for inventory checks.

[0051] The lifting mechanism is mounted on top of the auxiliary robot module 12. The multimodal AI scanning unit 2 rises and falls synchronously with the lifting mechanism, performing scanning operations on high-rise assets at different heights. The lifting height range is set from 0.5m to 3m, which covers the typical distribution height of assets in common industrial scenarios such as warehouse shelves and high-level equipment cabinets, and can meet the high-rise asset verification needs of most inventory sites. After reaching the target height, the lifting mechanism remains stable, providing a stable scanning platform for the multimodal AI scanning unit 2, avoiding image blurring or laser contour data distortion caused by machine shaking, and ensuring the quality of scanning data for high-rise assets.

[0052] The combination of the aforementioned small tracked structure and lifting mechanism enables the auxiliary robot module 12 to meet both the needs of navigating narrow spaces and scanning high-rise assets, forming an effective complementary capability with the main robot module 11. The two work together to achieve comprehensive coverage of assets in different spatial locations within the inventory area.

[0053] See Figure 3 As shown, in some embodiments, the multimodal AI scanning unit 2 further includes an environment adaptive adjustment module 24. The environment adaptive adjustment module 24 collects the light intensity, temperature and humidity parameters of the scene in real time, and automatically adjusts the exposure time of the visual image scanning module 21, the laser power of the laser contour scanning module 23 and the reading distance of the RFID scanning module 22 according to the collected parameters.

[0054] The environmental adaptive adjustment module 24 continuously monitors the environmental parameters at the inventory site through integrated light intensity sensor, temperature sensor and humidity sensor, and transmits the collected data to each scanning module in real time to drive the automatic adjustment of the corresponding parameters.

[0055] For the visual image scanning module 21, the environment adaptive adjustment module 24 automatically adjusts the exposure time according to the current light intensity. When the ambient light is strong, the exposure time is appropriately shortened to prevent overexposure of the image; when the ambient light is weak, the exposure time is appropriately extended to ensure image brightness and clarity, ensuring that the visual image scanning module 21 can acquire asset images of quality that meet the recognition requirements under different lighting conditions.

[0056] For the laser contour scanning module 23, the environmental adaptive adjustment module 24 dynamically adjusts the laser power based on temperature and humidity parameters. When the temperature or humidity increases, the scattering and absorption of the laser by the air are enhanced, and the environmental adaptive adjustment module 24 correspondingly increases the laser emission power to compensate for signal attenuation; when environmental conditions improve, the laser power is appropriately reduced to ensure the accuracy of three-dimensional contour data acquisition while avoiding unnecessary energy consumption.

[0057] For the RFID scanning module 22, the environmental adaptive adjustment module 24 dynamically adjusts the reading distance based on the current temperature and humidity conditions and the RFID tag recognition success rate. Specifically, the environmental adaptive adjustment module 24 achieves dynamic adjustment of the reading distance by controlling the radio frequency transmission power of the RFID scanning module 22: when the radio frequency transmission power is increased, the RFID signal coverage area expands, and the reading distance increases accordingly; when the radio frequency transmission power is decreased, the RFID signal coverage area narrows, and the reading distance shortens accordingly. In warehouse environments with dense metal shelving and strong electromagnetic interference, RFID signals attenuate significantly. The environmental adaptive adjustment module 24 monitors the RFID tag recognition success rate in real time. When the recognition success rate is lower than a preset threshold, it determines that the current environmental interference is strong and automatically reduces the radio frequency transmission power to shorten the reading distance. This allows the RFID scanning module 22 to communicate only with nearby tags, reducing crosstalk and signal aliasing among multiple tags, thereby improving the signal strength and recognition success rate of a single recognition. When the temperature and humidity are suitable, the electromagnetic environment is good, and the recognition success rate is consistently higher than the preset threshold, the environmental adaptive adjustment module 24 appropriately increases the radio frequency transmission power to increase the reading distance. This expands the signal coverage of the RFID scanning module 22, enabling the simultaneous reading of multiple RFID tags in a single scan, reducing the number of robot stops, and improving overall scanning efficiency.

[0058] Through the aforementioned adaptive adjustment mechanism, the multimodal AI scanning unit 2 can maintain stable scanning performance in actual warehousing environments with complex lighting conditions and large temperature and humidity fluctuations, avoiding the degradation of scanning data quality due to environmental changes, providing high-quality input data for the visual matching processing unit 3, and thus ensuring the accuracy of inventory results.

[0059] See Figure 4 As shown, in some embodiments, the visual matching processing unit 3 includes a data preprocessing module 31, a multimodal feature extraction module 32, a visual matching module 33, and an anomaly recognition module 34.

[0060] The data preprocessing module 31 cleans and standardizes the raw scanning data collected by the multimodal AI scanning unit 2, including denoising, brightness equalization and size normalization of visual images, point cloud filtering and coordinate alignment of laser contour data, and format verification and outlier removal of raw RFID coding data, providing reliable input data for subsequent feature extraction.

[0061] The multimodal feature extraction module 32 classifies and extracts features from the preprocessed multimodal data. For visual image data, the multimodal feature extraction module 32 uses a CNN convolutional neural network to extract deep visual features of the asset through multi-layer convolution and pooling operations, including high-dimensional semantic features such as texture, shape, and color distribution. Compared with traditional manual features, it has stronger expressive power and generalization ability. For laser contour data, the multimodal feature extraction module 32 first converts the laser contour data into a depth map, contour feature matrix, or other data format suitable for neural network processing, and then extracts the three-dimensional morphological features of the asset through a CNN network for verifying the integrity of the asset volume and the consistency of the contour. For RFID data, the multimodal feature extraction module 32 performs vectorization encoding on the unique identification code of the asset tag to form an identification feature vector. The above three types of feature vectors are merged into a unified multi-dimensional feature vector through a concatenation operation, which serves as the input for subsequent matching and comparison.

[0062] The visual matching module 33 employs a hierarchical matching strategy to compare scanned features with asset baseline features. First, the ORB algorithm performs rapid coarse matching of visual features. The ORB algorithm is computationally efficient and real-time, quickly filtering out candidate features with high similarity to the current scanned features from a large pool of candidate asset baseline features, thus narrowing the search range for fine matching. Second, within the candidate set, the SIFT algorithm optimizes the coarse matching results for fine matching. The SIFT algorithm is highly invariant to image rotation, scale changes, and illumination changes, further improving matching accuracy based on the ORB coarse matching. Finally, deep learning embedding matching technology maps multi-dimensional feature vectors to the corresponding asset baseline feature vectors stored in the cloud management platform 6 into a unified feature space, and performs matching calculations based on vector distance, cosine similarity, or other similarity evaluation methods, outputting the final matching similarity. This hierarchical processing flow—ORB coarse matching → SIFT fine matching → deep learning embedding matching—ensures both matching accuracy and processing efficiency, meeting the real-time requirements of inventory operations.

[0063] The anomaly identification module 34 compares the matching similarity output by the visual matching module 33 with the corresponding level threshold provided by the asset hierarchical management unit 4. When the matching similarity is lower than the corresponding level threshold, it is determined to be an inventory anomaly, and further, based on the deviation characteristics, the anomaly type is marked as one of missing inventory, wrong inventory, asset damage, or position deviation: if the matching similarity is extremely low and no corresponding benchmark feature can be found, it is marked as missing inventory; if the matched benchmark feature belongs to other assets, it is marked as wrong inventory; if the identity feature matches correctly but the visual appearance or laser contour feature shows obvious abnormalities, it is marked as asset damage; if the identity feature and appearance feature match correctly but there is a deviation in spatial position, it is marked as position deviation. The determination of position deviation is based on the spatial coordinates calculated by combining the robot's real-time coordinates provided by the positioning module 15 with the asset scanning position, and compared with the preset storage coordinates of the asset stored in the cloud management platform 6. If the deviation exceeds the preset range, it is determined to be position deviation. The preset range is preset according to the asset type, asset importance level, or storage area. After completing the anomaly type determination, the anomaly identification module 34 generates a corresponding anomaly signal. The anomaly type information, along with the corresponding asset number, anomaly location coordinates, and matching similarity value, is output by the visual matching processing unit 3 and simultaneously transmitted to the asset hierarchical control unit 4, the cloud management platform 6, and the anomaly processing unit 7, providing a basis for the subsequent classification and disposal by the anomaly processing unit 7.

[0064] In some embodiments, the visual matching module 33 also uses the RANSAC method to perform geometric verification on the matching results of visual features, and outputs the final matching result after eliminating mismatched feature point pairs.

[0065] Specifically, after ORB coarse matching and SIFT fine matching, the visual matching module 33 obtains a set of candidate feature point pairs. However, there may still be mismatched feature point pairs due to factors such as image noise, occlusion, or viewpoint changes. The RANSAC method verifies the geometric constraints of the above candidate feature point pairs through random sampling consistency: each time, several point pairs are randomly selected from the candidate feature point pairs, and a geometric transformation model between the current scanned image and the asset reference image is established based on the selected point pairs. The geometric transformation model includes one or more of homography matrix and fundamental matrix. Then, the geometric transformation model is used to verify all candidate feature point pairs. Feature point pairs that meet the geometric transformation constraints are marked as interior points, and feature point pairs that do not meet the constraints are marked as exterior points and discarded. The above random sampling and verification process is executed several times, and finally, the set of feature point pairs corresponding to the geometric transformation model with the most interior points is output as the valid matching result.

[0066] Through RANSAC geometric verification, the visual matching module 33 can effectively remove the mismatched feature point pairs remaining in the previous matching stage, significantly improve the geometric consistency and reliability of the final matching result, reduce the similarity calculation deviation caused by mismatch, thereby reducing the misjudgment rate of the anomaly identification module 34 and ensuring the accuracy of the inventory results.

[0067] In some embodiments, the asset hierarchical management unit 4 also has a hierarchical dynamic adjustment function, which updates the asset hierarchy according to changes in asset value and importance. The update results are synchronized to the cloud management platform 6 and the main and auxiliary robot units 1, so that the scanning strategy, inventory frequency and matching similarity threshold of the corresponding assets are switched synchronously with the hierarchical update.

[0068] Specifically, the asset tiered management unit 4 continuously receives asset status updates pushed by the cloud management platform 6, including changes in the asset's market value, changes in its service life, updates to maintenance records, and proactive adjustment instructions from management personnel. When an asset's value reaches the preset tier classification standard, or when the asset's importance meets the preset adjustment conditions, the asset tiered management unit 4 automatically performs a tier reassessment and regenerates the corresponding scanning strategy, inventory frequency, and matching similarity threshold parameters based on the new asset tier, adjusting the corresponding asset from the original tier to the new tier. For example, if a piece of equipment's book value falls below the important tier threshold due to depreciation, the asset tiered management unit 4 adjusts its tier from the important tier to the ordinary tier; conversely, if an asset is manually marked as a core asset by management personnel due to increased business importance, the asset tiered management unit 4 adjusts its tier from the original tier to the core tier.

[0069] After the hierarchical adjustment is completed, the update results are synchronized in real time to the cloud management platform 6 and the main and auxiliary robot units 1. The cloud management platform 6 updates the hierarchical record of the asset and the corresponding baseline feature storage index; after receiving the hierarchical update instruction, the main and auxiliary robot units 1 automatically switch the scanning strategy, inventory frequency, and matching similarity threshold of the corresponding asset during the next inventory task, without manual intervention. For example, assets adjusted to the core layer will automatically use a three-modal synchronous scanning strategy and a higher matching similarity threshold during the next inventory, while assets adjusted to the ordinary layer will switch to a single-modal scanning strategy and a basic threshold.

[0070] The aforementioned hierarchical dynamic adjustment function enables the asset hierarchical management unit 4 to continuously optimize the allocation strategy of inventory resources as the status changes throughout the asset's life cycle, avoiding the disconnect between the inventory strategy and actual management needs due to changes in asset value or importance, and ensuring the management accuracy and resource utilization efficiency of the system in the long-term operation process.

[0071] In some embodiments, the system further includes an anomaly handling unit 7, which is bidirectionally connected to the asset hierarchical control unit 4 and the cloud management platform 6. The anomaly handling unit 7 receives anomaly signals output by the visual matching processing unit 3, and simultaneously receives asset hierarchical information sent by the asset hierarchical control unit 4 and processing instructions or manual review results issued by the cloud management platform 6, and executes the corresponding processing flow according to the anomaly type and asset hierarchical level.

[0072] In response to missing inventory anomalies, the anomaly handling unit 7 extracts the preset storage location coordinates of the missing asset from the anomaly signal and issues a re-inspection command to the main and auxiliary robot units 1. The robot module closest to the corresponding area and currently idle is then dispatched to return to that location to re-execute the multimodal scan. After the re-inspection is completed, the visual matching processing unit 3 re-compares the newly acquired scan data. If the matching similarity reaches the corresponding level threshold, the missing inventory is determined to have been eliminated, and the inventory record is updated. If the corresponding asset is still not found after the re-inspection, the anomaly handling unit 7 marks the asset as potentially lost and reports the relevant information to the cloud management platform 6, prompting management personnel to conduct manual verification.

[0073] In response to misplaced assets, the anomaly handling unit 7 extracts the asset feature data actually scanned at the current scanning location and compares it item by item with the preset asset baseline information and the actual scanned asset baseline information stored in the cloud management platform 6. This generates a misplaced asset record containing the coordinates of the misplaced asset location, the preset asset number, the actual scanned asset number, and details of feature differences. This misplaced asset detail is then sent to the cloud management platform 6. The cloud management platform 6 generates a location update record to be confirmed based on the misplaced asset record and notifies the management personnel to manually return the incorrectly placed asset to its correct location. After confirmation by the management personnel, the location information of the corresponding asset is updated.

[0074] In response to asset damage anomalies, the anomaly handling unit 7 controls the main and auxiliary robot units 1 to acquire multi-angle images of the damaged asset at the current position, record the visual images and laser contour data of the damaged area, and calculate the area ratio of the damaged area, the degree of contour deviation or the degree of structural loss based on the acquired visual images and laser contour data, generate a damage assessment result, and simultaneously upload the acquired images of the damaged site, the damage assessment result and the asset number to the cloud management platform 6. At the same time, the status information of the asset is updated from normal to damaged and pending processing, triggering the cloud management platform 6 to push a repair or scrapping reminder to the management personnel.

[0075] In response to abnormal position offset, the anomaly handling unit 7 extracts the actual scanned position coordinates of the asset and the preset storage coordinates stored in the cloud management platform 6. The actual scanned position coordinates are determined by combining the real-time positioning coordinates of the robot provided by the positioning module 15 with the asset's scanned position. The offset direction and offset distance are calculated, and the main and auxiliary robot units 1 are controlled to collect on-site images of the asset's current position. The offset details and on-site images are uploaded to the cloud management platform 6, and the asset's storage location information is updated to the actual scanned position coordinates, which facilitates the management personnel's decision on whether to return the asset to its original position.

[0076] The above-mentioned classification and processing mechanism enables the anomaly handling unit 7 to take targeted measures for inventory anomalies of different natures. While automatically eliminating recoverable anomalies, it pushes anomalies requiring manual intervention to the cloud management platform 6 in a structured manner, realizing closed-loop management of inventory anomalies, effectively reducing the workload of manual investigation, and improving the overall reliability of inventory operations.

[0077] In some embodiments, the data interaction unit 5 adopts a dual-mode communication method of 5G and Wi-Fi and encrypts the transmitted data; the data interaction unit 5 also has an offline data caching function, which automatically stores inventory data and operation records when the network is interrupted, and synchronizes the offline inventory data to the cloud management platform 6 after the network is restored.

[0078] Specifically, the data interaction unit 5 integrates both a 5G communication module and a Wi-Fi communication module, allowing the two communication methods to operate in parallel. In the warehouse environment, where Wi-Fi network coverage is stable, Wi-Fi communication is prioritized for transmitting inventory data to reduce energy consumption. When the Wi-Fi signal strength falls below a preset threshold or the communication quality does not meet transmission requirements, the data interaction unit 5 automatically switches to 5G communication. In Wi-Fi coverage blind spots, the 5G communication link is maintained first, leveraging the wide coverage and low latency of the 5G network to ensure continuous data transmission. This automatic switching mechanism for dual-mode communication enables the data interaction unit 5 to always select the optimal communication link in complex warehouse environments, avoiding data transmission interruptions caused by unstable signals from a single communication method.

[0079] In terms of data security, the data interaction unit 5 encrypts all transmitted data, for example, by using one or more of the following methods: symmetric encryption, asymmetric encryption, or SSL / TLS secure transmission protocol, to ensure that inventory data, asset baseline characteristic information, and abnormal records are not intercepted or tampered with during transmission, thus ensuring the security of asset information.

[0080] Regarding offline caching, the data interaction unit 5 monitors the current network connection status in real time. When both 5G and Wi-Fi communication are interrupted, the data interaction unit 5 automatically activates the offline caching mode, storing the inventory data continuously generated by the main and auxiliary robot units 1, the matching results and abnormal signals output by the vision matching processing unit 3, and the robot's running trajectory and status records in timestamp order to the robot's local memory, industrial computer storage unit, or other non-volatile cache media, ensuring that inventory operations are not affected and data is not lost during network interruption. When the network connection is restored, the data interaction unit 5 automatically checks the integrity of the cached data and synchronizes all incremental inventory data accumulated during the offline period to the cloud management platform 6 in timestamp order. After synchronization is completed, the local cache is cleared, and the real-time transmission mode is restored.

[0081] The combination of the aforementioned dual-mode communication and offline caching mechanism enables the data interaction unit 5 to ensure the continuity and integrity of inventory data transmission in warehouse scenarios with complex network environments or signal blind spots, providing reliable data support for the cloud management platform 6.

[0082] See Figure 5 As shown, in some embodiments, the cloud management platform 6 further includes a task management module 61, a real-time monitoring module 62, a data management module 63, a statistical analysis module 64, and an interface docking module 65.

[0083] The task management module 61 is responsible for creating, splitting, and coordinating the scheduling of inventory tasks. After the administrator sets the inventory scope, inventory level, and inventory time plan through the task management module 61, the module automatically splits the overall inventory task into several sub-tasks based on the spatial distribution of the inventory area and the hierarchical distribution of assets. It then assigns each sub-task to the robot module whose current location is closest to the target area, whose remaining power meets the task requirements, and which is currently idle, based on the current position and remaining battery power of each robot module in the main and auxiliary robot units 1. During the inventory process, the task management module 61 tracks the progress of each sub-task in real time and automatically triggers task reassignment for sub-tasks that are not completed within the time limit, ensuring that the overall inventory task progresses as planned.

[0084] The real-time monitoring module 62 acquires real-time video streams of the inventory site through the vision sensors mounted on the main and auxiliary robot units 1. The current position, movement trajectory, and on-site view of each robot module are displayed on the monitoring interface of the cloud management platform 6, enabling managers to remotely monitor the real-time dynamics of the inventory site. When the vision matching processing unit 3 outputs an abnormal signal, the real-time monitoring module 62 automatically retrieves the on-site view of the corresponding robot module and pushes it to the managers, facilitating remote analysis and decision-making regarding the abnormal situation.

[0085] The data management module 63 centrally stores the baseline characteristic information of all managed assets, inventory data generated from each inventory count, and anomaly handling records. Asset baseline characteristic information includes the appearance image of each asset, RFID identification code, laser outline data or morphological feature vectors extracted from laser outline data, and preset storage coordinates, serving as the benchmark for feature comparison by the visual matching processing unit 3. Inventory data includes the scanning results, matching similarity, and inventory timestamps for each inventory count. Anomaly data includes anomaly type, anomaly location coordinates, on-site images, and processing result records. The above data is indexed by asset number and inventory batch, and a correlation is established between asset number, timestamp, and anomaly type, supporting managers in quickly retrieving and tracing historical inventory data.

[0086] The statistical analysis module 64 performs multi-dimensional statistical analysis on the inventory data stored in the data management module 63, including the inventory completion rate of assets at each level, the anomaly occurrence rate, the distribution patterns of various anomalies, and the trend comparison of inventory results over time. Based on the statistical results, it automatically generates a structured inventory report. The inventory report covers the overall overview of this inventory, details of the inventory in each area, a list of abnormal assets and their processing status, providing management personnel with a reference for asset control decisions.

[0087] The interface interface module 65 connects with external management systems such as the enterprise asset management system and financial management system through API interfaces, Web Service interfaces, database interfaces, or other standardized data exchange interfaces to achieve two-way synchronization of asset information. On the one hand, the interface interface module 65 pushes the latest inventory results, asset status updates, and anomaly handling records from the cloud management platform 6 to the external management system, ensuring that the asset ledger in the external system is consistent with the actual inventory results. On the other hand, changes in asset additions, scrapping, and transfers in the external management system are synchronized to the cloud management platform 6 through the interface interface module 65. When changes in asset additions, scrapping, or transfers affect the asset value or importance, the interface interface module 65 synchronizes the changes to the asset hierarchical control unit 4, triggering a reassessment of the corresponding asset's hierarchy, forming a data closed loop for the entire asset lifecycle management.

[0088] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

Claims

1. A hierarchical inventory system for master and auxiliary robot assets based on multimodal scanning, characterized in that, It includes main and auxiliary robot units, multimodal AI scanning units, visual matching and processing units, asset hierarchical management and control units, data interaction units, and cloud management platforms; The main and auxiliary robot unit includes a main robot module and at least one auxiliary robot module; The multimodal AI scanning unit is integrated into the main and auxiliary robot units and includes a visual image scanning module, an RFID scanning module, and a laser contour scanning module. The asset hierarchical management unit divides assets into a core layer, an important layer, and a normal layer. It configures tri-modal, bi-modal, and uni-modal scanning strategies for the three layers respectively, and sets differentiated inventory frequency and matching similarity thresholds. The visual matching processing unit selects corresponding scan data based on the asset level and compares it with the asset benchmark features stored in the cloud management platform. When the matching similarity is lower than the corresponding level threshold, an abnormal signal is output. The data interaction unit connects the aforementioned units to the cloud management platform; the cloud management platform stores the baseline characteristic information of each asset and is responsible for issuing inventory tasks.

2. The system according to claim 1, characterized in that, The main and auxiliary robot units also include an autonomous navigation module, an obstacle avoidance module, and a positioning module; the autonomous navigation module plans the inventory path based on SLAM (Simultaneous Localization and Mapping) technology; the obstacle avoidance module identifies obstacles and dynamically adjusts the path through the linkage of lidar and visual sensors; the positioning module uses UWB (Ultra-Wideband) positioning technology to achieve precise positioning of the robot.

3. The system according to claim 1, characterized in that, The auxiliary robot module is a small tracked structure, and at least one of the auxiliary robot modules is equipped with a lifting mechanism. The lifting height of the lifting mechanism is in the range of 0.5m to 3m, and it is used to complete the inventory operation of high-rise assets.

4. The system according to claim 1, characterized in that, The multimodal AI scanning unit also includes an environment adaptive adjustment module, which collects the light intensity, temperature and humidity parameters of the scene in real time, and automatically adjusts the exposure time of the visual image scanning module, the laser power of the laser contour scanning module and the reading distance of the RFID scanning module according to the collected parameters.

5. The system according to claim 1, characterized in that, The visual matching processing unit includes a data preprocessing module, a multimodal feature extraction module, a visual matching module, and an anomaly recognition module. The multimodal feature extraction module uses a CNN convolutional neural network to extract deep visual features from visual images and morphological features from laser contour data, and encodes RFID data to form identity feature vectors, generating multi-dimensional feature vectors. The visual matching module performs fast coarse matching of visual features using the ORB algorithm, then refines the matching results using the SIFT algorithm, and finally calculates the similarity between the multi-dimensional feature vectors and the asset baseline feature vectors using deep learning embedding matching technology. The anomaly recognition module compares the calculated matching similarity with the corresponding level threshold; if it is lower than the threshold, it is judged as an inventory anomaly. Anomaly types include missing inventory, incorrect inventory, asset damage, and position deviation.

6. The system according to claim 5, characterized in that, The visual matching module also uses the RANSAC method to perform geometric verification on the matching results of visual features, and outputs the final matching result after removing mismatched feature point pairs.

7. The system according to claim 1, characterized in that, The asset hierarchical management unit also has a hierarchical dynamic adjustment function, which updates the asset hierarchy according to changes in asset value and importance. The update results are synchronized to the cloud management platform and the main and auxiliary robot units, so that the scanning strategy, inventory frequency and matching similarity threshold of the corresponding assets are switched synchronously with the hierarchy update.

8. The system according to claim 1, characterized in that, The system also includes an anomaly handling unit, which is bidirectionally connected to the asset hierarchical management unit and the cloud management platform. The anomaly handling unit receives anomaly signals output by the visual matching processing unit. For missing inventory anomalies, it schedules the main and auxiliary robot units to return to the corresponding area for rescanning. For misplaced inventory anomalies, it compares the asset baseline information with the scan data and sends the misplaced inventory details to the cloud management platform. For asset damage or positional deviation anomalies, it controls the main and auxiliary robot units to collect images of the abnormal scene and update the status information of the corresponding asset.

9. The system according to claim 1, characterized in that, The data interaction unit adopts a dual-mode communication method of 5G and Wi-Fi and encrypts the transmitted data. The data interaction unit also has an offline data caching function, which automatically stores inventory data and operation records when the network is interrupted, and synchronizes the offline inventory data to the cloud management platform after the network is restored.

10. The system according to claim 1, characterized in that, The cloud management platform also includes a task management module, a real-time monitoring module, a data management module, a statistical analysis module, and an interface integration module; the task management module supports the splitting and collaborative scheduling of inventory tasks; The real-time monitoring module monitors the inventory site in real time through the vision sensors of the main and auxiliary robot units; the data management module stores asset baseline information, inventory data, and abnormal data; the statistical analysis module performs statistical analysis on the inventory data and generates an inventory report; and the interface docking module connects with external management systems through standardized interfaces to achieve two-way synchronization of asset information.

Images