Warehouse checking method and device based on machine vision

Abstract

The invention discloses a warehouse checking method and device based on machine vision, and relates to the technical field of warehouse checking. The method comprises the following steps: generating an optimal inventory path of a warehouse by using an improved swarm intelligence optimization algorithm; collecting multi-modal machine vision data of goods shelves in a warehouse; performing feature level fusion on the multi-modal machine vision data through a multi-modal fusion network to generate shelf digital representation; inputting the shelf digital representation into a commodity identification model to obtain a commodity identification result; carrying out manual re-checking on the commodity identification result; all commodity identification results are integrated to generate an electronic inventory list, and the electronic inventory list is compared with warehouse management system data; and regularly using a plurality of high-quality training samples, and carrying out online incremental learning on the multi-modal fusion network and / or the commodity recognition model. The problems that in the prior art, warehouse checking efficiency is low, accuracy is poor, path planning is not intelligent, and a model cannot be updated in a self-adaptive mode are solved.

Description

Technical Field

[0001] This invention relates to the field of warehouse inventory technology, and in particular to a warehouse inventory method and apparatus based on machine vision. Background Technology

[0002] With the rapid development of e-commerce and logistics, the efficiency and accuracy of warehouse management have become crucial. Warehouse inventory counting, as a core component of inventory management, has traditionally relied primarily on manual methods. Manual inventory counting is not only time-consuming and labor-intensive, but also prone to errors and omissions, leading to inaccurate inventory data and impacting normal business operations.

[0003] To address these issues, several automated inventory management solutions have emerged in the existing technology. For example, using QR codes or RFID tags for inventory management improves efficiency, but requires large-scale modifications to existing warehouses, has high labeling costs, and the tags are prone to damage and detachment, resulting in significant maintenance costs. Another solution is to use mobile robots equipped with a single camera for image recognition inventory management, but this method has limited accuracy and robustness in scenarios with complex lighting, stacked goods, and similar appearances.

[0004] Furthermore, path planning is crucial for the efficient operation of mobile inventory robots. Traditional path planning algorithms (such as A) Algorithms typically only consider the shortest distance, ignoring dynamic environmental factors within the warehouse (such as pedestrian and vehicle traffic) and task priorities. This results in low actual robot operating efficiency, potential delays due to congestion, or failure to prioritize high-value, fast-turnover product areas.

[0005] Finally, existing machine learning models are usually static after deployment and cannot adapt to new products, new packaging or environmental changes. The model performance will decline over time and requires regular offline retraining, which is cumbersome and has a slow response time.

[0006] Therefore, how to provide an efficient, accurate, and adaptive warehouse inventory solution is a technical problem that urgently needs to be solved in the field of smart warehousing. Summary of the Invention

[0007] This invention provides a warehouse inventory method and apparatus based on machine vision, which solves the problems of low efficiency, poor accuracy, unintelligent path planning, and inability of the model to adaptively update in existing technologies.

[0008] In a first aspect, embodiments of the present invention provide a warehouse inventory method based on machine vision, the method comprising: Based on warehouse map information, inventory task list and real-time environmental status, an improved swarm intelligence optimization algorithm is used to generate the optimal inventory path for the warehouse. The mobile inventory robot travels along the optimal inventory path and simultaneously collects multimodal machine vision data of the warehouse shelves. The multimodal machine vision data is spatiotemporally aligned and feature-level fusion is performed through a pre-trained multimodal fusion network to generate a digital representation of the warehouse shelves. The digital representation of the shelf is input into a pre-trained product recognition model for hierarchical recognition to obtain product recognition results; The product classification results of product instances whose uncertainty measure in the product identification results exceeds the preset measurement threshold are manually reviewed to obtain the product review classification result of the product instance, which is then used as a high-quality training sample. Integrate the product classification results and / or product verification classification results of all product instances in all product identification results, generate an electronic inventory list, compare it with the data of the warehouse management system, and generate a discrepancy report; Periodically use several high-quality training samples to perform online incremental learning on the multimodal fusion network and / or the product recognition model to obtain an updated multimodal fusion network and / or an updated product recognition model.

[0009] The technical solution provided in this application has at least the following beneficial effects: By introducing a hybrid enhanced ant colony algorithm that incorporates chaotic initialization, Levy flight, and PSO, the algorithm comprehensively considers path length, real-time congestion, task priority, and power constraints. This results in a more optimized inventory path that effectively avoids congestion, prioritizes high-value tasks, and significantly improves the actual operational efficiency of the inventory robot. Furthermore, the algorithm employs multimodal data fusion of 2D images, 3D point clouds, and thermal imaging. Through feature-level interaction via a multimodal fusion network, it generates digital representations of the shelves, overcoming the limitations of single-modal data. Combined with a hierarchical product recognition model (location-target detection-fine classification), it achieves coarse-to-fine recognition, greatly improving the accuracy and robustness of recognition in complex scenarios. Finally, through an uncertainty measurement mechanism, only low-confidence recognition results are manually reviewed, ensuring the accuracy of the final results while significantly reducing the cost of manual intervention. The high-quality samples generated from the review are used for online incremental learning of the model, forming a closed loop of "data-identification-review-optimization". This enables the system to adapt to new products and environmental changes and maintain high performance. The entire process, from path planning, data collection, intelligent identification to report generation and model self-optimization, is highly automated and systematic, significantly reducing manual intervention, lowering operating costs, and improving the level of intelligence in warehouse management.

[0010] In one alternative implementation, the improved swarm intelligence optimization algorithm is a hybrid enhanced ant colony algorithm that incorporates chaotic sequence initialization, the Levy flight mechanism, and the PSO algorithm.

[0011] In one alternative implementation, based on warehouse map information, inventory task list, and real-time environmental status, an improved swarm intelligence optimization algorithm is used to generate the optimal inventory path for the warehouse, including: Based on the cloud server and warehouse map information, a raster map of the warehouse is constructed, and the raster map is converted into a weighted directed graph. According to the real-time environmental status of the warehouse, each edge in the directed graph is assigned a corresponding dynamic weight. Set the key parameters and cost function of the hybrid enhanced ant colony algorithm; Obtain the inventory task list from the warehouse management system and assign corresponding priority weights to each shelf node in the directed graph; The chaotic sequence is generated using the Logistic mapping, and then mapped to the pheromone space to obtain the initial pheromone distribution. In each iteration, for each ant, starting from the beginning of the directed graph, and based on the initial or updated pheromone distribution and the mixed state transition probability formula, the next shelf node to be visited is selected until a path construction is completed, and the ant's construction path is obtained. Use a cost function to obtain the cost value of the construction path, and take the construction path with the lowest cost value as the local optimal path; If all ants complete one path construction, then update the pheromones on all constructed paths, obtain the updated pheromones, and proceed to the next iteration; Until the maximum number of iterations is reached or the local optimal path remains unchanged for several consecutive generations, the construction path with the lowest cost among all local optimal paths is taken as the global optimal path, which is the optimal inventory path for the warehouse.

[0012] In one alternative implementation, the multimodal machine vision data includes 2D image data, 3D point cloud data, and thermal imaging data; The product identification results include product classification results for several product instances; the product classification results include the location identifier, product SKU, and product quantity of the corresponding product instance.

[0013] In one optional implementation, the multimodal fusion network includes a 2D image feature extraction module based on the ResNet algorithm, a 3D point cloud feature extraction module based on the PointNet++ algorithm, a thermal imaging feature extraction module based on a lightweight CNN algorithm, and a feature fusion module based on the Transformer algorithm. The 2D image feature extraction module, the 3D point cloud feature extraction module, and the thermal imaging feature extraction module are all connected to the feature fusion module. The product recognition model includes a product location module based on the U-Net algorithm, a product target detection module based on the PointRCNN algorithm, and a product fine classification module based on the YOLO algorithm with a Triplet Loss function, which are connected in sequence.

[0014] In one optional implementation, the mobile inventory counting device is controlled to travel along the optimal inventory counting path and simultaneously collect multimodal machine vision data of the warehouse shelves, including: The optimal inventory path of the warehouse is sent to the mobile inventory gateway. Based on the mobile inventory gateway, the optimal inventory path and the shelf node sequence related to each mobile inventory robot in the mobile inventory device are extracted and sent to the corresponding mobile inventory robot. The mobile inventory robot travels along the optimal inventory sub-path composed of the shelf node sequence, and sends a synchronous trigger signal to the multimodal data acquisition device on the same machine when it reaches each shelf node. Based on the synchronization trigger signal, use multimodal data acquisition equipment to synchronously collect the raw data of the shelf node and upload the raw data to the mobile inventory gateway; Based on the mobile inventory gateway, the raw data uploaded by the mobile inventory robot is preprocessed to obtain multimodal machine vision data, which is then uploaded to the cloud server.

[0015] In one optional implementation, the multimodal machine vision data is spatiotemporally aligned, and feature-level fusion is performed through a pre-trained multimodal fusion network to generate a digital representation of the warehouse shelves, including: Based on a cloud server, it receives multimodal machine vision data uploaded by each mobile inventory robot; Based on the preset intrinsic parameters of the multimodal data acquisition device, the multimodal machine vision data is spatially calibrated and aligned to obtain aligned multimodal machine vision data, which is then input into the pre-trained multimodal fusion network. The 2D image feature extraction module, 3D point cloud feature extraction module, and thermal imaging feature extraction module of the multimodal fusion network are used to extract the 2D image feature map, 3D point cloud feature map, and thermal imaging feature map of the aligned multimodal machine vision data, respectively. The 2D image feature map, 3D point cloud feature map, and thermal imaging feature map are stitched together along the channel dimension to obtain the stitched feature map, which is then input into the feature fusion module of the multimodal fusion network. Based on the self-attention weights generated by the feature fusion module, the stitched feature maps are fused to obtain the digital representation of the warehouse shelves.

[0016] In one optional implementation, the digital representation of the shelf is input into a pre-trained product recognition model for hierarchical recognition to obtain product recognition results, including: The digital representation of the shelf is input into a pre-trained product recognition model; Based on the digital representation of the shelf, the location module of the product recognition model is used to locate the location and obtain the precise pixel mask and text content of the location identifier for each location area. Based on the precise pixel mask of each storage location area, the corresponding sub-shelf digital representation is extracted from the shelf digital representation and input into the product target detection module of the product recognition model; Based on the digital representation of the sub-shelf, the product target detection module is used to detect product targets, obtain the position and size of each product instance, and locate the corresponding 3D bounding box; Extract the fusion features from the 3D bounding box of each product instance and input them into the fine classification module of the product recognition model; Using the product fine classification module, the fused features are mapped to a high-dimensional space to obtain the corresponding high-dimensional feature vectors. The vectors are then searched for nearest neighbors with the pre-established SKU feature library to obtain the product SKU and product quantity for each product instance. By integrating the location identifier, product SKU, and product quantity of the same product instance, the corresponding product classification result is obtained. Then, by integrating the product classification results of all product instances, the product identification result is obtained.

[0017] In one optional implementation, the product classification results of product instances whose uncertainty measure in the product identification results exceeds a preset measurement threshold are manually reviewed to obtain the product review classification result of that product instance, which is then used as a high-quality training sample, including: Obtain the uncertainty measure of the product classification result for each product instance in the product identification results; If the uncertainty measure is greater than the preset measurement threshold, the multimodal machine vision data and product classification results of the product instance will be marked as samples to be reviewed. The sample to be reviewed is sent to the manual review terminal, which receives the correct label confirmed by the human. The correct label is combined with the multimodal machine vision data of the product instance to form the product review classification result, and stored as a high-quality training sample in the high-quality sample pool.

[0018] Secondly, embodiments of the present invention provide a machine vision-based warehouse inventory counting device for implementing a warehouse inventory counting method. The device includes: The path planning unit is used to generate the optimal inventory path for the warehouse based on warehouse map information, inventory task list and real-time environmental status, using an improved swarm intelligence optimization algorithm. The data acquisition unit is used to enable the mobile inventory robot to travel along the optimal inventory path and simultaneously collect multimodal machine vision data of the shelves in the warehouse. The feature fusion unit is used to perform spatiotemporal alignment of the multimodal machine vision data and to perform feature-level fusion through a pre-trained multimodal fusion network to generate a digital representation of the shelving in the warehouse. The product recognition unit is used to input the digital representation of the shelf into a pre-trained product recognition model, perform hierarchical recognition, and obtain product recognition results; The manual review unit is used to manually review the product classification results of product instances whose uncertainty measure in the product identification results exceeds a preset measurement threshold, obtain the product review classification result of the product instance, and use it as a high-quality training sample. The inventory generation unit is used to integrate the product classification results and / or product verification classification results of all product instances in all product identification results, generate an electronic inventory list, and compare it with the data of the warehouse management system to generate a discrepancy report; The continuous training unit is used to periodically perform online incremental learning on the multimodal fusion network and / or the product recognition model using a number of high-quality training samples, so as to obtain an updated multimodal fusion network and / or an updated product recognition model.

[0019] A third aspect of this invention provides an electronic device, which includes: At least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by at least one processor, such that the at least one processor can perform the method proposed in the first aspect of the present invention.

[0020] A fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method as described in the first aspect of the present invention. Attached Figure Description

[0021] Figure 1 This is a schematic diagram of the electronic device structure of the hardware operating environment involved in the embodiments of the present invention; Figure 2 This is a flowchart illustrating the steps of a machine vision-based warehouse inventory method provided in an embodiment of the present invention. Figure 3 This is a functional unit diagram of a machine vision-based warehouse inventory device provided in an embodiment of the present invention. Detailed Implementation

[0022] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0023] The present invention will be further described below with reference to the accompanying drawings.

[0024] Reference Figure 1 , Figure 1 This is a schematic diagram of the electronic device structure of the hardware operating environment involved in the embodiments of the present invention.

[0025] like Figure 1 As shown, the electronic device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen or an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be a high-speed random access memory (RAM) or a stable non-volatile memory (NVM), such as a disk drive. The memory 1005 may also optionally be a storage device independent of the aforementioned processor 1001.

[0026] Those skilled in the art will understand that Figure 1 The structure shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0027] like Figure 1 As shown, the memory 1005, which serves as a storage medium, may include an operating system, a data storage module, a network communication module, a user interface module, and an electronic program for a machine vision-based warehouse inventory device.

[0028] exist Figure 1In the electronic device shown, the network interface 1004 is mainly used for data communication with the network server; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and the memory 1005 in the electronic device of the present invention can be set in the electronic device. The electronic device calls the electronic program of the machine vision-based warehouse inventory device stored in the memory 1005 through the processor 1001 and executes the machine vision-based warehouse inventory method provided in the embodiment of the present invention.

[0029] Reference Figure 2 The present invention provides a machine vision-based warehouse inventory method, the method comprising: S201: Based on warehouse map information, inventory task list and real-time environmental status, use an improved swarm intelligence optimization algorithm to generate the optimal inventory path for the warehouse. S202: Make the mobile inventory robot travel along the optimal inventory path and simultaneously collect multimodal machine vision data of the shelves in the warehouse; S203: Spatiotemporally align the multimodal machine vision data and perform feature-level fusion through a pre-trained multimodal fusion network to generate a digital representation of the warehouse shelves. S204: Input the digital representation of the shelf into the pre-trained product recognition model, perform hierarchical recognition, and obtain the product recognition result; S205: Manually review the product classification results of product instances whose uncertainty measure in the product identification results exceeds the preset measurement threshold, obtain the product review classification result of the product instance, and use it as a high-quality training sample; S206: Integrate the product classification results and / or product verification classification results of all product instances in all product identification results, generate an electronic inventory list, compare it with the data of the warehouse management system, and generate a discrepancy report; S207: Periodically use several high-quality training samples to perform online incremental learning on the multimodal fusion network and / or the product recognition model to obtain an updated multimodal fusion network and / or an updated product recognition model.

[0030] The technical solution provided in this application has at least the following beneficial effects: By introducing a hybrid enhanced ant colony algorithm that incorporates chaotic initialization, Levy flight, and Particle Swarm Optimization (PSO), the algorithm comprehensively considers path length, real-time congestion, task priority, and power constraints. This results in a more optimized inventory path that effectively avoids congestion, prioritizes high-value tasks, and significantly improves the actual operational efficiency of the inventory robot. Furthermore, the algorithm employs multimodal data fusion of 2D images, 3D point clouds, and thermal imaging. Through feature-level interaction via a multimodal fusion network, it generates digital representations of the shelves, overcoming the limitations of single-modal data. Combined with a hierarchical product recognition model (location-target detection-fine classification), it achieves coarse-to-fine recognition, greatly improving the accuracy and robustness of recognition in complex scenarios. Finally, through an uncertainty measurement mechanism, only low-confidence recognition results are manually reviewed, ensuring the accuracy of the final results while significantly reducing the cost of manual intervention. The high-quality samples generated from the review are used for online incremental learning of the model, forming a closed loop of "data-identification-review-optimization". This enables the system to adapt to new products and environmental changes and maintain high performance. The entire process, from path planning, data collection, intelligent identification to report generation and model self-optimization, is highly automated and systematic, significantly reducing manual intervention, lowering operating costs, and improving the level of intelligence in warehouse management.

[0031] In one alternative implementation, the improved swarm intelligence optimization algorithm is a hybrid enhanced ant colony algorithm that incorporates chaotic sequence initialization, the Levy flight mechanism, and the PSO algorithm.

[0032] In one alternative implementation, based on warehouse map information, inventory task list, and real-time environmental status, an improved swarm intelligence optimization algorithm is used to generate the optimal inventory path for the warehouse, including: S2011: Based on a cloud server and warehouse map information, construct a raster map of the warehouse and convert the raster map into a weighted directed graph. And based on the real-time environmental status of the warehouse, assign corresponding dynamic weights to each edge in the directed graph, where, This is a set of nodes, representing all shelf locations, charging stations, and starting points that need to be inventoried. Let be the edge set, representing all feasible paths; The dynamic weights are calculated in real time from the following factors, using the following formula: In the formula, For the edge e Dynamic weights; The base distance is the physical length; For the edge eThe congestion coefficient is determined by communicating with the warehouse management system or other IoT devices within the warehouse (such as personnel positioning tags and forklift sensors) to obtain real-time pedestrian and vehicle traffic flow in that passage. The higher the traffic flow, the better. The higher the value; The passage difficulty coefficient is set based on static information such as passage width and ground flatness; These are preset empirical weighting coefficients; S2012: Set the key parameters and cost function of the hybrid enhanced ant colony algorithm; The key parameters include the number of ants, pheromone importance factor, heuristic function importance factor, pheromone volatility coefficient, maximum number of iterations, Levy flight control parameters, and PSO learning factor. The formula for the cost function is: In the formula, To build path The cost value; To build path Path length cost; To build path The cost of congestion; To build path Priority benefits; To build path The cost of electricity penalty; Weights for the cost function; In the formula, To the shelf node To the shelf node The physical length of the edge; For shelf node indication; This represents the total number of shelf nodes; this is the most basic cost, i.e., the total length of the path, which encourages the algorithm to find shorter physical paths. In the formula, To the shelf node To the shelf node The real-time congestion coefficient of the edge; this cost penalizes the behavior of passing through congested road sections, enabling route planning to "avoid peak hours" and guide the algorithm to choose channels with less traffic and improve actual operating efficiency; In the formula, For shelf nodes Priority weights; For shelf node indication; K To build path The total number of all shelf nodes requiring inventory checks; this component transforms high-priority tasks into "revenue," which is deducted from the total cost function. Accessing high-priority shelves can significantly reduce total cost, ensuring that the algorithm prioritizes high-value, fast-turnover, or soon-to-expire product areas; In the formula, Based on the cost, if the build path If the total length exceeds the maximum safe driving range of the device when fully charged, a huge penalty value is applied, causing the path to be eliminated. For dynamic penalties, the path construction is adjusted based on the device's remaining battery percentage at the start. A penalty will be imposed, with the penalty increasing as the remaining battery level decreases. S2013: Retrieve the inventory task list from the warehouse management system and assign corresponding priority weights to each shelf node in the directed graph. P ; In this embodiment, the inventory task list may contain the following information: Regional priority: such as Zone A (high value), Zone B (fast turnover), Zone C (normal); Product attributes: such as near-expiry products, promotional items; For example, shelves in the high-value goods section and the goods that are about to expire have a higher priority weight; S2014: Use Logistic mapping to generate chaotic sequences and map the chaotic sequences to the pheromone space to obtain the initial pheromone distribution; In this embodiment, to avoid the blindness of traditional random initialization, chaotic mapping (such as Logistic mapping) is used to generate the initial pheromone distribution to enhance ergodicity; The formula is: In the formula, For the first i+ 1. i One chaotic variable; This is the stability coefficient, typically 4; In the formula, For the first i The initial pheromone concentration at each shelf node; For the first i One chaotic variable; This sets the upper and lower limits for pheromone concentration, which makes the initial pheromone distribution more ergonomic and diverse, which is beneficial for subsequent global searches. i For shelf node indication; S2015: In each iteration, for each ant, starting from the starting point of the directed graph, and based on the initial or updated pheromone distribution and the mixed state transition probability formula, the next shelf node to be visited is selected until a path construction is completed, and the ant's construction path is obtained. In the formula, For the first t Located at the shelf node in the next iteration ants Select to move to the next neighbor shelf node. The transition probability; sub-path pheromones; For path Heuristic information; For ants The next set of shelf nodes that can be accessed; For shelf node indication; For transfer parameters; For ant indicators; To reach the shelf node Priority weights; For the first t The adaptive convergence factor for the next iteration; In the formula, sub-path Dynamic weights; In the formula, This represents the maximum number of iterations. , To adjust the parameters; It is the hyperbolic tangent function; This represents the maximum value of the convergence factor. This represents the minimum convergence factor. Levy flight mechanism: In order to escape local optima, ants will perform levy flight with a certain probability during the path construction process. When levy flight is triggered, the ant will not choose according to the above probability, but will make long-distance jumps. Lévy's flight stride L Generated by the Levy distribution, the formula is: In the formula, stride length for Levy flight ; Flight parameters that conform to a standard normal distribution; bThe Levy rating is typically set to 1.5. The ant randomly selects an unvisited node that is far from its current position as its next target based on its step size L, thereby enhancing the algorithm's global exploration capability. Constraint handling: During the path construction process, ensure that each shelf node is visited only once, and take into account the remaining battery power of the mobile inventory robot. The planned path must include or be close to a charging station. S2016: Use the cost function to obtain the cost value of the construction path, and take the construction path with the lowest cost value as the local optimal path; S2017: If all ants have completed one path construction, then update the pheromones on all constructed paths, obtain the updated pheromones, and proceed to the next iteration; The formula for pheromone evaporation is: In the formula, For the first t+ 1, t The pheromone of the next iteration; Volatilization rate; Pheromon release enhanced using the PSO algorithm: Pheromon release considers not only the path of the current iteration but also information from the historical best path, as shown in the formula: In the formula, For the release of pheromones; The additional pheromones released by the globally optimal path to date and the optimal path in this iteration; For ants Remaining in the sub-path in this iteration The amount of pheromone on a surface is inversely proportional to its path cost; m The total number of ants; In the formula, It is a constant; This serves as a learning factor, similar to the "individual cognition" and "social learning" components in PSO, guiding the algorithm to learn towards the historical best and the current iteration's best, respectively. This represents the total cost of the globally optimal path to date. This represents the total cost of the optimal path in this iteration; In the formula, For the first t+ 1, t Subpath of the next iteration The updated pheromones; after the update is complete, all ants return to the starting point, ready for the next iteration; S2018: Until the number of iterations reaches the maximum number of iterations or the local optimal path remains unchanged for several consecutive generations, the construction path with the lowest cost value among all local optimal paths is taken as the global optimal path, i.e., the optimal inventory path for the warehouse.

[0033] In one alternative implementation, the multimodal machine vision data includes 2D image data, 3D point cloud data, and thermal imaging data; The product identification results include product classification results for several product instances; the product classification results include the location identifier, stock keeping unit (SKU), and quantity of the corresponding product instance.

[0034] In one optional implementation, the multimodal fusion network includes a 2D image feature extraction module based on the ResNet algorithm, a 3D point cloud feature extraction module based on the PointNet++ algorithm, a thermal imaging feature extraction module based on the lightweight convolutional neural network (CNN) algorithm, and a feature fusion module based on the Transformer algorithm. The 2D image feature extraction module, the 3D point cloud feature extraction module, and the thermal imaging feature extraction module are all connected to the feature fusion module. The product recognition model includes a location module based on the U-Net algorithm, a product target detection module based on the Point Region-based Convolutional Neural Network (PointRCNN) algorithm, and a product fine classification module based on the YOLO algorithm with a Triplet Loss function, which are connected in sequence.

[0035] In one optional implementation, the mobile inventory counting device is controlled to travel along the optimal inventory counting path and simultaneously collect multimodal machine vision data of the warehouse shelves, including: S2021: Send the optimal inventory path of the warehouse to the mobile inventory gateway. Based on the mobile inventory gateway, extract the shelf node sequence related to each mobile inventory robot in the mobile inventory device and send it to the corresponding mobile inventory robot. In this embodiment, the mobile inventory gateway uses a load balancing strategy to split the global path into several sub-paths based on the number, location, and remaining battery power of the currently online mobile inventory robots; for example, the path is divided equally, or the nearest path segment is assigned based on the robot's location. S2022: The mobile inventory robot travels along the optimal inventory sub-path composed of the shelf node sequence, and sends a synchronous trigger signal to the multimodal data acquisition device on the same machine when each shelf node is reached. In this embodiment, after receiving the sub-path, the mobile robot calls its built-in Simultaneous Localization and Mapping (SLAM) navigation module and autonomously travels along the path. The robot estimates its own pose in real time through the Adaptive Monte Carlo Localization (AMCL) algorithm. When it detects that the coordinate error between its own position and the target shelf node in the sub-path is less than a threshold (such as 10cm), it determines that it has "arrived". After arrival, the robot's main control unit immediately sends a high-precision synchronization trigger signal (such as a PPS pulse or hardware interrupt signal) to its onboard multimodal data acquisition devices such as 2D camera, 3D LiDAR, and thermal imager through GPIO or internal bus. The synchronization trigger signal forces the three devices to start exposure and data acquisition at the same time (microsecond error). S2023: Based on the synchronization trigger signal, use multimodal data acquisition equipment to synchronously collect the raw data of the shelf node and upload the raw data to the mobile inventory gateway; S2024: Based on the mobile inventory gateway, perform data preprocessing on the raw data uploaded by the mobile inventory robot to obtain multimodal machine vision data, and upload the multimodal machine vision data to the cloud server; In this embodiment, a 2D camera (visible light camera) acquires a high-resolution 2D color image stream; 3D LiDAR (depth camera, such as structured light camera) collects 3D point cloud data streams, each point containing three-dimensional coordinates and reflection intensity; The thermal imager acquires a thermal imaging data stream, with each pixel representing a temperature value. The three data streams are transmitted in real time to the high-speed memory of the processing module for caching; Data Packaging: The robot packages multiple raw data files collected synchronously from the same shelf node, along with metadata such as node ID, timestamp, and robot pose, into a single data package; Data Upload: Upload data packets to the mobile inventory gateway via wireless network.

[0036] Gateway preprocessing: The mobile inventory gateway performs lightweight preprocessing tasks to reduce the burden on cloud servers. Data validation: Checks whether the data packet is complete; Format conversion: Convert 2D color images and thermal imaging data into standard formats (such as PNG or BIN format).

[0037] Preliminary noise reduction: Outlier removal from 3D point cloud data; Cloud transmission: The pre-processed multimodal machine vision data is stably transmitted to a cloud server for in-depth analysis; In one optional implementation, the multimodal machine vision data is spatiotemporally aligned, and feature-level fusion is performed through a pre-trained multimodal fusion network to generate a digital representation of the warehouse shelves, including: S2031: Based on a cloud server, receive multimodal machine vision data uploaded by each mobile inventory robot; S2032: Based on the preset intrinsic parameters of the multimodal data acquisition device, perform spatial calibration and alignment on the multimodal machine vision data to obtain aligned multimodal machine vision data, and input it into the pre-trained multimodal fusion network; In this embodiment, time alignment involves verifying and utilizing timestamps to ensure that data processed is from the same moment in time. Spatial calibration and alignment: using the sensor intrinsic and extrinsic parameter matrices obtained through prior calibration. External parameter calibration: Determine the relative pose (rotation matrix and translation vector) between the 2D camera, 3D LiDAR, and thermal imager. Coordinate transformation: Using the coordinate system of the 3D LiDAR as a reference, each pixel in the 2D image is back-projected to the 3D space through the PnP (Perspective-n-Point) algorithm, or the 3D point cloud is projected onto the 2D image plane; similarly, the thermal imaging data is also aligned to the 3D point cloud or 2D image. Data fusion: The final result is a unified data structure with multiple modal attributes. For example, each 3D point cloud point not only has three-dimensional coordinates, but also corresponding RGB color values ​​and thermal imaging temperature values. S2033: Using a multimodal fusion network, a 2D image feature extraction module, a 3D point cloud feature extraction module, and a thermal imaging feature extraction module, respectively extract 2D image feature maps (multi-scale texture, color, and semantic feature maps), 3D point cloud feature maps (local and global geometric structure features), and thermal imaging feature maps (temperature distribution and anomalous hot zone features) from the aligned multimodal machine vision data. S2034: The 2D image feature map, 3D point cloud feature map and thermal imaging feature map are stitched together in the channel dimension to obtain the stitched feature map, which is then input into the feature fusion module of the multimodal fusion network. In this embodiment, feature alignment: Since the feature maps output by different modules may have different sizes, it is necessary to unify them to the same size through upsampling or downsampling operations; Channel concatenation: The three feature maps are concatenated along the channel dimension to form a composite feature map that integrates multimodal information; S2035: Based on the self-attention weights generated by the feature fusion module, the stitched feature maps are fused to obtain the digital representation of the shelving in the warehouse. In this embodiment, the input is: flattening or dividing the spliced ​​composite feature map into blocks and inputting it into the encoder structure of the Transformer; Self-attention computation: Transformer uses a self-attention mechanism to calculate the dependency of each position (or token) in the feature map on all other positions; it can automatically learn, for example, "the abnormal high temperature in a certain area (thermal imaging feature) is highly correlated with the texture (2D feature) and shape (3D feature) of a certain product"; Feature fusion and output: After passing through multiple Transformer encoders, the output is a deeply fused and complementary digital representation of the shelf; this representation can be a global feature vector or a feature map that retains spatial information, which is more discriminative than any single-modality feature.

[0038] In one optional implementation, the digital representation of the shelf is input into a pre-trained product recognition model for hierarchical recognition to obtain product recognition results, including: S2041: Input the digital representation of the shelf into the pre-trained product recognition model; S2042: Based on the digital representation of the shelf, use the location positioning module of the product recognition model to locate the location and obtain the precise pixel mask and text content of the location identifier for each location area. In this embodiment, the digital representation of the shelf is first input into the U-Net network of the location module. U-Net is a classic semantic segmentation network that can output a mask image of the same size as the input. Location and Recognition: In the mask image, each storage location area is precisely segmented (pixel value 1, background value 0). Simultaneously, OCR (Optical Character Recognition) technology can be used to identify the text content on each storage location label (such as "..."). "A-01-03" is used as a cargo location identifier; S2043: Based on the precise pixel mask of each storage location area, extract the corresponding sub-shelf digital representation from the shelf digital representation and input it into the product target detection module of the product recognition model; In this embodiment, the area mapping is performed by locating the corresponding area on the original digital representation of the shelving based on the pixel mask of each storage location output by U-Net. Data cropping: The digital representation of each storage location area (including its multimodal fusion features) is cropped to form an independent "sub-shelf digital representation"; S2044: Based on the digital representation of the sub-shelf, use the product target detection module to perform product target detection, obtain the position and size of each product instance, and locate the corresponding 3D bounding box; In this embodiment, the input is: the digital representation of each sub-shelf is input into the PointRCNN network of the product target detection module; Detection: PointRCNN is an object detection algorithm designed specifically for 3D point clouds. It can directly generate candidate boxes in 3D space and regress the accurate 3D bounding box (center coordinates, length, width, height, and orientation) of each product instance. Output: For each storage location, output a list of 3D bounding boxes for one or more product instances; S2045: Extract the fusion features from the 3D bounding box of each product instance and input them into the fine classification module of the product recognition model; In this embodiment, the feature extraction within the bounding box is as follows: for each detected 3D bounding box, the corresponding fused feature region is "cut out" from the digital representation of the sub-shelf based on its coordinates and size. Feature aggregation: Aggregate the features within the bounding box (which may be a set of points or a feature map patch) into a single feature vector (such as through RoIPooling or average pooling) to represent the product instance; S2046: Using the product fine classification module, the fused features are mapped to a high-dimensional space to obtain the corresponding high-dimensional feature vectors, and the nearest neighbor search is performed with the pre-established SKU feature library to obtain the product SKU and product quantity of each product instance; In this embodiment, feature mapping: the feature vector of the instance is input into the YOLO based on Triplet Loss in the fine classification module of goods; The effect of Triplet Loss: During the training phase, Triplet Loss makes the feature vectors of products with the same SKU closer together in the metric space, and farther apart for products with different SKUs. YOLO Classification Header: This module maps the input vector to a high-dimensional feature space to obtain the feature vector of the instance; Nearest neighbor search: Perform a cosine similarity or Euclidean distance search on the feature vector and a pre-built "SKU feature library"; the SKU feature library stores the standard feature vectors of all known SKUs; Output results: Find the most similar SKU as the identification result (product SKU) for this product instance; at the same time, estimate the number of products based on the number of detected instances or the ratio of the volume of the 3D bounding box to the volume of the standard product. S2047: Integrate the location identifier, product SKU, and product quantity of the same product instance to obtain the corresponding product classification result, and integrate the product classification results of all product instances to obtain the product identification result; In this embodiment, instance integration involves binding the identification result (SKU, quantity) of each product instance to its corresponding storage location identifier. List generation: The results of all product instances on a shelf are summarized to form the preliminary product identification results for that shelf.

[0039] In one optional implementation, the product classification results of product instances whose uncertainty measure in the product identification results exceeds a preset measurement threshold are manually reviewed to obtain the product review classification result of that product instance, which is then used as a high-quality training sample, including: S2051: Obtain the uncertainty measure of the product classification result for each product instance in the product identification result; In this embodiment, the product fine classification module not only outputs the most similar SKU, but also outputs an uncertainty metric. Commonly used metrics include: The formula for classification probability entropy is: In the formula, For classification probability entropy; For prediction of the first The probability of SKU; The value indicates the quantity of the commodity; the higher the entropy value, the greater the uncertainty. Nearest neighbor distance: The ratio of the distance between the feature vector and its nearest neighbor to the distance between the feature vector and its second nearest neighbor; if the distances are very close, or the ratio is close to 1, it indicates that the model is uncertain. Confidence score: A confidence score between 0 and 1 directly output by the product fine classification module; S2052: If the uncertainty measure is greater than the preset measurement threshold (e.g., entropy > 0.8), then the multimodal machine vision data and product classification results of the product instance are marked as samples to be reviewed; S2053: Send the sample to be reviewed to the manual review terminal, receive the correct label confirmed by the human, combine the correct label with the multimodal machine vision data of the product instance to form the product review classification result, and store it as a high-quality training sample in the high-quality sample pool. In this embodiment, the sample to be reviewed is pushed to the warehouse manager's tablet or the review APP on the central monitoring station in real time through the communication module (5G / Wi-Fi); the APP interface displays the original multimodal data (such as 2D images as the main type, with 3D boxes and thermal images superimposed) in a user-friendly manner, and highlights the areas that the system is uncertain about. Human reviewers only need to perform simple interactive operations: If the system makes an incorrect identification, select the correct SKU from the drop-down list; If the system misses a product, manually select the product and its SKU. If the system recognizes it correctly, click the "Confirm" button; The correct labels confirmed by the reviewers, together with the original data, constitute a high-quality training sample with human confidence. All manually verified samples are stored in a high-quality sample pool; this sample pool is indexed and managed according to dimensions such as SKU, time, and uncertainty level.

[0040] In one optional implementation, the product classification results and / or product verification classification results of all product instances in all product identification results are integrated to generate an electronic inventory list, which is then compared with the data in the warehouse management system to generate a discrepancy report, including: S2061: Based on the cloud server, summarize the final product identification results of all shelves and / or after manual review and correction, and generate a complete electronic inventory list in a preset format (such as Excel or CSV). Each line records the information of one SKU, including: location identifier, product SKU, system record quantity, and inventory quantity. S2062: Retrieve current theoretical inventory data from the warehouse management system via API interface, automatically compare the "inventory quantity" in the electronic inventory list with the "system record quantity" in the warehouse management system, and mark any discrepancies such as quantity inconsistencies, misplaced goods, or SKU mismatches; S2063: Generate a detailed discrepancy report, including: Inventory Overview: Total SKU types, total number of storage locations, inventory time, etc. Discrepancy Details: List all product items with discrepancies, including location, SKU, system quantity, inventory quantity, discrepancy quantity, and discrepancy type (inventory surplus, inventory shortage, misplacement). Visual charts: such as pie charts showing the difference rate by region, bar charts showing the difference by SKU category, etc.; Discrepancy reports are automatically sent to inventory managers, finance department personnel, and other relevant personnel as a basis for inventory adjustments and problem analysis.

[0041] In one optional implementation, the multimodal fusion network and / or product recognition model are periodically subjected to online incremental learning using several high-quality training samples to obtain an updated multimodal fusion network and / or an updated product recognition model, including: S2071: Learning trigger conditions, setting the trigger mechanism for incremental learning, which can be: Time-triggered: at a fixed time each week or month; Quantity trigger: When the number of samples in the high-quality sample pool reaches a certain threshold (e.g., 500); Performance trigger: When the overall recognition accuracy of the model on recent data drops beyond a certain threshold; S2072: Perform online incremental learning on the multimodal fusion network and / or the product recognition model to obtain an updated multimodal fusion network and / or an updated product recognition model; Sample selection: Draw a batch of the latest samples from a high-quality sample pool. Random sampling can be used, or samples with the highest uncertainty can be selected first; Model fine-tuning: Load the weights of the currently running multimodal fusion network and / or product recognition model; The model is fine-tuned using selected high-quality samples; typically, a small learning rate and fewer training epochs are used to avoid "catastrophic forgetting" (i.e. forgetting old knowledge). You can fine-tune only the last few layers of the model, or use incremental learning algorithms such as Elastic Weight Consolidation (EWC) to preserve old knowledge; Model validation: Evaluate the performance of the fine-tuned model on an independent validation set to ensure that its accuracy has improved or at least not decreased; S2073: Deploy an updated multimodal fusion network and / or an updated product recognition model; Gray release: The updated model is first deployed to a small number of robots or processes for online testing to observe its performance; Full update: After confirming that the new model is stable and reliable, gradually replace all old versions of the model online to complete the iterative upgrade of the entire system; This step is crucial for the system to achieve self-evolution and continuous optimization, enabling the model to adapt to the ever-changing warehouse environment.

[0042] This invention also provides a warehouse inventory device based on machine vision, referring to... Figure 3 The diagram shows a functional unit diagram of a machine vision-based warehouse inventory device 300 according to the present invention. The device may include the following units: The path planning unit 301 is used to generate the optimal inventory path for the warehouse based on warehouse map information, inventory task list and real-time environmental status using an improved swarm intelligence optimization algorithm. The data acquisition unit 302 is used to enable the mobile inventory robot to travel along the optimal inventory path and simultaneously collect multimodal machine vision data of the shelves in the warehouse. The feature fusion unit 303 is used to perform spatiotemporal alignment of the multimodal machine vision data and perform feature-level fusion through a pre-trained multimodal fusion network to generate a digital representation of the shelving in the warehouse. The product recognition unit 304 is used to input the digital representation of the shelf into a pre-trained product recognition model, perform hierarchical recognition, and obtain product recognition results; The manual review unit 305 is used to manually review the product classification results of product instances whose uncertainty measurement in the product identification results exceeds a preset measurement threshold, obtain the product review classification result of the product instance, and use it as a high-quality training sample. The list generation unit 306 is used to integrate the product classification results and / or product verification classification results of all product instances in all product identification results, generate an electronic inventory list, and compare it with the data of the warehouse management system to generate a discrepancy report; The continuous training unit 307 is used to periodically use several high-quality training samples to perform online incremental learning on the multimodal fusion network and / or the product recognition model to obtain an updated multimodal fusion network and / or an updated product recognition model.

[0043] Based on the same inventive concept, another embodiment of the present invention provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus. Memory, used to store computer programs; The processor, when executing a program stored in memory, implements the machine vision-based warehouse inventory method of the present invention.

[0044] The communication bus mentioned above can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of representation, only one thick line is used in the diagram, but this does not indicate that there is only one bus or one type of bus. The communication interface is used for communication between the aforementioned terminal and other devices. The memory can include Random Access Memory (RAM) or non-volatile memory, such as at least one disk storage device. Optionally, the memory can also be at least one storage device located remotely from the aforementioned processor.

[0045] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0046] Furthermore, to achieve the above objectives, embodiments of the present invention also propose a computer-readable storage medium storing a computer program that, when executed by a processor, implements the machine vision-based warehouse inventory method of the embodiments of the present invention.

[0047] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, apparatus, or computer program products. Therefore, embodiments of the present invention can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of the present invention can take the form of computer program products implemented on one or more computer-usable hardware devices (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0048] The embodiments of the present invention are described with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (apparatus), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0049] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0050] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0051] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. "And / or" indicates that either one or both can be chosen. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes the element.

[0052] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A warehouse inventory method based on machine vision, characterized in that, The method includes: Based on warehouse map information, inventory task list and real-time environmental status, an improved swarm intelligence optimization algorithm is used to generate the optimal inventory path for the warehouse. The mobile inventory robot travels along the optimal inventory path and simultaneously collects multimodal machine vision data of the warehouse shelves. The multimodal machine vision data is spatiotemporally aligned and feature-level fusion is performed through a pre-trained multimodal fusion network to generate a digital representation of the warehouse shelves. The digital representation of the shelf is input into a pre-trained product recognition model for hierarchical recognition to obtain product recognition results; The product classification results of product instances whose uncertainty measure in the product identification results exceeds the preset measurement threshold are manually reviewed to obtain the product review classification result of the product instance, which is then used as a high-quality training sample. Integrate the product classification results and / or product verification classification results of all product instances in all product identification results, generate an electronic inventory list, compare it with the data of the warehouse management system, and generate a discrepancy report; Periodically use several high-quality training samples to perform online incremental learning on the multimodal fusion network and / or the product recognition model to obtain an updated multimodal fusion network and / or an updated product recognition model.

2. The warehouse inventory method based on machine vision according to claim 1, characterized in that, The improved swarm intelligence optimization algorithm is a hybrid enhanced ant colony algorithm that incorporates chaotic sequence initialization, Levy flight mechanism, and PSO algorithm.

3. The warehouse inventory method based on machine vision according to claim 2, characterized in that, Based on warehouse map information, inventory task list, and real-time environmental status, an improved swarm intelligence optimization algorithm is used to generate the optimal inventory path for the warehouse, including: Based on the cloud server and warehouse map information, a raster map of the warehouse is constructed, and the raster map is converted into a weighted directed graph. According to the real-time environmental status of the warehouse, each edge in the directed graph is assigned a corresponding dynamic weight. Set the key parameters and cost function of the hybrid enhanced ant colony algorithm; Obtain the inventory task list from the warehouse management system and assign corresponding priority weights to each shelf node in the directed graph; The chaotic sequence is generated using the Logistic mapping, and then mapped to the pheromone space to obtain the initial pheromone distribution. In each iteration, for each ant, starting from the beginning of the directed graph, and based on the initial or updated pheromone distribution and the mixed state transition probability formula, the next shelf node to be visited is selected until a path construction is completed, and the ant's construction path is obtained. Use a cost function to obtain the cost value of the construction path, and take the construction path with the lowest cost value as the local optimal path; If all ants complete one path construction, then update the pheromones on all constructed paths, obtain the updated pheromones, and proceed to the next iteration; Until the maximum number of iterations is reached or the local optimal path remains unchanged for several consecutive generations, the construction path with the lowest cost among all local optimal paths is taken as the global optimal path, which is the optimal inventory path for the warehouse.

4. The warehouse inventory method based on machine vision according to claim 3, characterized in that, The multimodal machine vision data includes 2D image data, 3D point cloud data, and thermal imaging data; The product identification results include product classification results for several product instances; the product classification results include the location identifier, product SKU, and product quantity of the corresponding product instance.

5. The warehouse inventory method based on machine vision according to claim 4, characterized in that, The multimodal fusion network includes a 2D image feature extraction module based on the ResNet algorithm, a 3D point cloud feature extraction module based on the PointNet++ algorithm, a thermal imaging feature extraction module based on a lightweight CNN algorithm, and a feature fusion module based on the Transformer algorithm. The 2D image feature extraction module, the 3D point cloud feature extraction module, and the thermal imaging feature extraction module are all connected to the feature fusion module. The product recognition model includes a product location module based on the U-Net algorithm, a product target detection module based on the PointRCNN algorithm, and a product fine classification module based on the YOLO algorithm with a Triplet Loss function, which are connected in sequence.

6. The warehouse inventory method based on machine vision according to claim 5, characterized in that, Control the mobile inventory counting equipment to travel along the optimal inventory counting path and simultaneously collect multimodal machine vision data of the warehouse shelves, including: The optimal inventory path of the warehouse is sent to the mobile inventory gateway. Based on the mobile inventory gateway, the optimal inventory path and the shelf node sequence related to each mobile inventory robot in the mobile inventory device are extracted and sent to the corresponding mobile inventory robot. The mobile inventory robot travels along the optimal inventory sub-path composed of the shelf node sequence, and sends a synchronous trigger signal to the multimodal data acquisition device on the same machine when it reaches each shelf node. Based on the synchronization trigger signal, use multimodal data acquisition equipment to synchronously collect the raw data of the shelf node and upload the raw data to the mobile inventory gateway; Based on the mobile inventory gateway, the raw data uploaded by the mobile inventory robot is preprocessed to obtain multimodal machine vision data, which is then uploaded to the cloud server.

7. The warehouse inventory method based on machine vision according to claim 6, characterized in that, The multimodal machine vision data is spatiotemporally aligned, and feature-level fusion is performed through a pre-trained multimodal fusion network to generate a digital representation of the warehouse shelves, including: Based on a cloud server, it receives multimodal machine vision data uploaded by each mobile inventory robot; Based on the preset intrinsic parameters of the multimodal data acquisition device, the multimodal machine vision data is spatially calibrated and aligned to obtain aligned multimodal machine vision data, which is then input into the pre-trained multimodal fusion network. The 2D image feature extraction module, 3D point cloud feature extraction module, and thermal imaging feature extraction module of the multimodal fusion network are used to extract the 2D image feature map, 3D point cloud feature map, and thermal imaging feature map of the aligned multimodal machine vision data, respectively. The 2D image feature map, 3D point cloud feature map, and thermal imaging feature map are stitched together along the channel dimension to obtain the stitched feature map, which is then input into the feature fusion module of the multimodal fusion network. Based on the self-attention weights generated by the feature fusion module, the stitched feature maps are fused to obtain the digital representation of the warehouse shelves.

8. The warehouse inventory method based on machine vision according to claim 7, characterized in that, The digital representation of the shelf is input into a pre-trained product recognition model for hierarchical recognition to obtain product recognition results, including: The digital representation of the shelf is input into a pre-trained product recognition model; Based on the digital representation of the shelf, the location module of the product recognition model is used to locate the location and obtain the precise pixel mask and text content of the location identifier for each location area. Based on the precise pixel mask of each storage location area, the corresponding sub-shelf digital representation is extracted from the shelf digital representation and input into the product target detection module of the product recognition model; Based on the digital representation of the sub-shelf, the product target detection module is used to detect product targets, obtain the position and size of each product instance, and locate the corresponding 3D bounding box; Extract the fusion features from the 3D bounding box of each product instance and input them into the fine classification module of the product recognition model; Using the product fine classification module, the fused features are mapped to a high-dimensional space to obtain the corresponding high-dimensional feature vectors. The vectors are then searched for nearest neighbors with the pre-established SKU feature library to obtain the product SKU and product quantity for each product instance. By integrating the location identifier, product SKU, and product quantity of the same product instance, the corresponding product classification result is obtained. Then, by integrating the product classification results of all product instances, the product identification result is obtained.

9. The warehouse inventory method based on machine vision according to claim 8, characterized in that, For product instances where the uncertainty measure in the product identification results exceeds a preset threshold, the product classification results are manually reviewed to obtain the reviewed product classification results for those instances. These reviewed results are then used as high-quality training samples, including: Obtain the uncertainty measure of the product classification result for each product instance in the product identification results; If the uncertainty measure is greater than the preset measurement threshold, the multimodal machine vision data and product classification results of the product instance will be marked as samples to be reviewed. The sample to be reviewed is sent to the manual review terminal, which receives the correct label confirmed by the human. The correct label is combined with the multimodal machine vision data of the product instance to form the product review classification result, and stored as a high-quality training sample in the high-quality sample pool.

10. A machine vision-based warehouse inventory device for implementing the warehouse inventory method as described in any one of claims 1-9, characterized in that, The device includes: The path planning unit is used to generate the optimal inventory path for the warehouse based on warehouse map information, inventory task list and real-time environmental status, using an improved swarm intelligence optimization algorithm. The data acquisition unit is used to enable the mobile inventory robot to travel along the optimal inventory path and simultaneously collect multimodal machine vision data of the shelves in the warehouse. The feature fusion unit is used to perform spatiotemporal alignment of the multimodal machine vision data and to perform feature-level fusion through a pre-trained multimodal fusion network to generate a digital representation of the shelving in the warehouse. The product recognition unit is used to input the digital representation of the shelf into a pre-trained product recognition model, perform hierarchical recognition, and obtain product recognition results; The manual review unit is used to manually review the product classification results of product instances whose uncertainty measure in the product identification results exceeds a preset measurement threshold, obtain the product review classification result of the product instance, and use it as a high-quality training sample. The inventory generation unit is used to integrate the product classification results and / or product verification classification results of all product instances in all product identification results, generate an electronic inventory list, and compare it with the data of the warehouse management system to generate a discrepancy report; The continuous training unit is used to periodically perform online incremental learning on the multimodal fusion network and / or the product recognition model using a number of high-quality training samples, so as to obtain an updated multimodal fusion network and / or an updated product recognition model.

Images