Intelligent unmanned warehouse visual processing method based on AIoT

Abstract

The invention discloses an AIoT-based intelligent unmanned warehouse visual processing method. The method comprises the steps of collecting visual, depth and environmental data of a storage environment through a multi-mode sensor network; an improved deep learning model is adopted to realize cargo identification and tracking in a complex scene; edge computing nodes are deployed for data preprocessing and feature compression; sub-centimeter-level cargo positioning is realized based on a feature point matching algorithm; and a digital twin platform is constructed to coordinate multi-device collaborative operation. According to the method, the shielding scene recognition rate is remarkably improved to 95% or above, the network transmission delay is reduced by 40%, unmanned intelligent management of the whole process of warehousing operation is achieved, and the method is suitable for large-scale warehousing scenes such as power grid materials and e-commerce logistics.

Description

Technical Field

[0001] The present invention relates to the field of smart logistics technology, and specifically to a smart unmanned warehouse visual processing method based on AIoT. Background Art

[0002] Traditional warehouse management has long relied on manual operations, resulting in low efficiency, inventory discrepancies, high security risks, and data silos. For example, manual goods registration and location allocation can have an error rate of up to 20%, and lacks real-time dynamic monitoring capabilities. With the development of e-commerce and smart manufacturing, warehouses must cope with challenges such as a surge in SKU variety and order fragmentation. Existing technologies, such as RFID tags and single-source visual recognition systems, have the following limitations:

[0003] Complex cargo appearance recognition is difficult: In real-world warehouse environments, goods vary widely in type, appearance, and packaging materials and colors. Some goods may share similar appearances, or even suffer packaging damage or deformation during transportation, posing significant challenges for visual recognition algorithms. For example, the packaging of some electronic products may only have subtle differences in model identification. Existing visual recognition technology struggles to accurately distinguish these similar products, leading to incorrect cargo identification and impacting subsequent sorting and delivery processes.

[0004] Lighting conditions affect recognition: The lighting environment within a warehouse is complex and changeable, and the light intensity and color temperature may vary significantly across different areas. Furthermore, the placement and angle of goods can lead to uneven lighting on their surfaces, resulting in reflections, shadows, and other phenomena. These lighting factors can severely affect the quality of images captured by the camera, blurring the features of the goods in the image and reducing the accuracy of the visual recognition algorithm. For example, in darker areas such as the bottom of a shelf, or when the surface of the goods is made of highly reflective material (such as metal packaging), existing visual recognition technologies often struggle to accurately extract the features of the goods, leading to recognition failures or errors.

[0005] Inadequate recognition capabilities in occluded and stacked situations: During warehousing, goods are often stacked, which can cause some goods to be obscured by other goods. Furthermore, during handling and sorting, goods may obstruct each other. Existing visual recognition technologies often have difficulty identifying goods in occluded and stacked situations. For example, when a piece of goods is partially obscured by multiple other items, the visual recognition algorithm may not be able to accurately identify the entire obscured item, resulting in incorrect or unrecognized recognition.

[0006] AIoT (Artificial Intelligence Internet of Things) technology offers a breakthrough for addressing these challenges. By integrating visual processing, edge computing, and multimodal data collaboration, a fully intelligent, unmanned warehouse management system can be built.

[0007] In summary, there is an urgent need to adopt automated equipment and new-generation information technology to rebuild warehousing business processes, improve warehousing management efficiency, ensure the quality of material supply, reduce management costs, and meet the new requirements for fast and high-quality material supply. Summary of the Invention

[0008] The present invention aims to solve at least one of the technical problems existing in the prior art. To this end, one purpose of the present invention is to propose an AIoT-based smart unmanned warehouse visual processing method to solve the problems of low recognition accuracy, poor real-time performance, insufficient coordination with other equipment, and incomplete monitoring of the storage environment in the prior art smart unmanned warehouse visual processing, thereby achieving accurate identification, positioning, and tracking of goods in the smart unmanned warehouse, as well as intelligent monitoring and management of the storage environment, and improving the operational efficiency and intelligence level of the smart unmanned warehouse.

[0009] In order to solve the above problems, the present invention provides an AIoT-based intelligent unmanned warehouse visual processing method, which is characterized by comprising the following steps:

[0010] S1. Deploy a multimodal sensor network to collect multi-source data in the storage environment, including visual images, depth information, infrared signals, and environmental parameters;

[0011] S2. Joint analysis of multimodal data based on deep learning models, including fine-grained cargo classification, pose estimation, and target tracking;

[0012] S3, uses edge computing nodes to pre-process data in real time and optimizes data transmission efficiency through adaptive transmission protocols;

[0013] S4, through the feature point matching algorithm to fuse visual and depth data, to achieve three-dimensional spatial positioning and dynamic trajectory tracking of cargo;

[0014] S5. Build a collaborative control interface based on the digital twin platform to dispatch AGVs, robotic arms, and three-dimensional shelves to complete warehousing tasks.

[0015] Preferably, the multimodal sensor network in step S1 includes:

[0016] a. High-definition cameras and depth cameras deployed in the shelf area to collect RGB-D data of goods;

[0017] b. Infrared sensor arrays installed in the passageways to detect movement of people and equipment;

[0018] c. Environmental sensor groups distributed throughout the storage space monitor temperature, humidity, and gas concentration in real time.

[0019] Preferably, the deep learning model in S2 adopts an improved YOLOv7 architecture, which integrates the attention mechanism and multi-scale feature pyramid to achieve cargo recognition in occluded scenes. The model training data includes:

[0020] a. 100,000 images of goods in different lighting, stacking, and damage states;

[0021] b. Associated depth map and infrared thermal imaging data.

[0022] Preferably, the edge computing node in S3 uses an NVIDIA JetsonAGX Xavier module and performs the following processing:

[0023] a. De-mosaic and dynamic range compression of visual data;

[0024] b. Use lightweight MobileNetV3 model for real-time target detection;

[0025] c. Generate compressed feature vectors and transmit them to the cloud server.

[0026] Preferably, the feature point matching algorithm in S4 includes:

[0027] a. Use SURF algorithm to extract key points on the surface of cargo;

[0028] b. Combine RANSAC algorithm to eliminate mismatched points;

[0029] c. The cargo position is calculated using the PnP algorithm, with a positioning accuracy of ±2cm.

[0030] Preferably, the digital twin platform in S5 implements the following functions:

[0031] a. Build a 3D virtual image of the warehouse based on the Unity3D engine;

[0032] b. Real-time synchronization of physical space equipment status data;

[0033] c. Use genetic algorithm to optimize multi-device task scheduling paths.

[0034] The advantages of the present invention compared with the prior art are:

[0035] (1) The present invention can effectively improve the accuracy of visual recognition: through multimodal data collection, it effectively solves the problems of difficulty in recognizing the appearance of complex goods, the influence of lighting conditions on recognition effects, and insufficient recognition capabilities under occlusion and stacking conditions, and significantly improves the accuracy of visual recognition of goods. After training with a large number of cargo image samples covering different scenes, it can accurately learn the characteristic patterns of various types of goods. Even when faced with goods with similar appearance, the model can accurately distinguish the types of goods through deep feature extraction and analysis. Compared with traditional methods, it greatly reduces the misjudgment and missed judgment caused by environmental interference and cargo similarity.

[0036] (2) The present invention can effectively improve real-time performance: by adopting edge computing, cloud computing and distributed computing architecture, as well as optimized data transmission protocols and adaptive network transmission strategies, the speed of data processing and transmission is greatly improved, and the system response delay is reduced. In scenarios with high real-time requirements such as cargo sorting, the system can quickly identify cargo and control equipment to operate, shortening the response time, effectively meeting the real-time requirements of efficient logistics, and improving the overall operational efficiency of the warehouse. In processes such as cargo storage, handling and sorting, each device can work closely together to avoid conflicts and interference between devices and improve the efficiency of collaborative work.

[0037] (3) The present invention can effectively achieve precise inventory management: the positioning algorithm based on feature point matching and the Kalman filter tracking algorithm can accurately determine the location and movement trajectory of goods in a complex warehouse environment, even if the goods are partially blocked or there is image noise. The system automatically and in real time updates inventory information based on the entry and exit of goods, avoiding errors and delays in manual inventory. For example, when goods are transported in the warehouse, the system can track their position changes in real time and synchronously update the storage location information in the inventory list; when goods are in and out of the warehouse, through precise identification and positioning, the changes in the quantity of goods are accurately recorded to ensure that the inventory data is completely consistent with the actual situation of the goods. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or technical descriptions. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0039] Figure 1 Schematic diagram of the visual processing principle of the intelligent unmanned warehouse of the present invention. DETAILED DESCRIPTION

[0040] The following describes in detail embodiments of the present application. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application and are not to be construed as limiting the present application.

[0041] Combined with the figure, the present invention is a smart unmanned warehouse visual processing method based on AIoT

[0042] The purpose of the present invention is to provide an AIoT-based smart unmanned warehouse visual processing method to solve the problems of low recognition accuracy, poor real-time performance, insufficient coordination with other equipment, and incomplete monitoring of the storage environment in the existing smart unmanned warehouse visual processing technology, so as to achieve accurate identification, positioning, and tracking of goods in the smart unmanned warehouse, as well as intelligent monitoring and management of the storage environment, thereby improving the operational efficiency and intelligence level of the smart unmanned warehouse.

[0043] Combining computer vision, deep learning, and multi-sensor fusion technologies, we perform 3D reconstruction of the physical warehouse space. Leveraging AI algorithms, we analyze dynamic data about people, goods, and the field in real time, enabling digital and automated warehouse operations. We accurately identify personnel picking behavior and provide a new warehouse management model where customers can "grab and go" with no need for any input.

[0044] (1) Multimodal visual data collection of stored power materials

[0045] Various types of sensors, including high-definition cameras, depth cameras, infrared sensors, temperature and humidity sensors, and gas sensors, are deployed within the smart unmanned warehouse to create a multimodal data collection system. High-definition cameras capture color images of the cargo and the environment, while depth cameras acquire depth information. The collaborative operation of these sensors enables the acquisition of more comprehensive and rich cargo and environmental data, providing ample data support for subsequent visual processing and intelligent analysis. Based on Internet of Things (IoT) technology, all sensors are connected into a unified network, enabling real-time transmission and sharing of sensor data. Low-power, highly reliable communication protocols are employed to ensure stable and rapid transmission of sensor data to the data processing center. Sensor data is also encrypted to ensure data security.

[0046] (2) Visual recognition and analysis based on deep learning

[0047] Deep learning algorithms are used to perform in-depth analysis and processing of collected multimodal data. A specialized cargo recognition model is built for cargo identification tasks. This model is trained using a large amount of cargo image data, enabling it to accurately identify various types of cargo, including fine-grained classification of similar-looking goods. Useful features are extracted from images and effectively described. These features can be low-level features such as color, texture, and shape, or higher-level features derived through deep learning. Common feature extraction algorithms include SIFT, SURF, and ORB.

[0048] Deep learning algorithms are used to estimate and locate cargo poses. Combining depth information captured by a depth camera with image data from a high-definition camera, a trained pose estimation model accurately calculates the cargo's position, orientation, and pose in three-dimensional space. A target tracking algorithm, the SORT (Simple Online and Realtime Tracking) algorithm based on deep learning, is introduced to track cargo's movement in real time during storage. By continuously monitoring cargo position changes, abnormal cargo movement, such as theft or mishandling, can be detected and alerted promptly.

[0049] (3) Real-time data processing and transmission optimization

[0050] In the data processing center, a high-performance graphics processing unit (GPU) cluster and distributed computing architecture are used to parallelize and process large amounts of sensor data, accelerating data processing speed. To reduce data transmission latency, edge computing technology is used to preprocess and analyze some data on edge devices close to the sensors. Edge computing nodes are deployed near the cameras to perform preliminary processing such as denoising, enhancement, and object detection on the collected image data, transmitting key detection results and feature information to the data processing center.

[0051] Optimize data transmission protocols and adopt adaptive local area network transmission strategies. Dynamically adjust the data transmission rate and format based on real-time changes in the local area network. Also, introduce data caching and retransmission mechanisms to ensure data integrity and accuracy during brief network outages or packet loss.

[0052] (4) Cargo positioning and tracking

[0053] Feature point extraction and matching positioning: A positioning algorithm based on feature point matching is used to determine the location of the goods in the warehouse. First, feature point extraction algorithms such as SIFT (Scale-Invariant Feature Transform) or SURF (Speeded Robust Features) are used in the preprocessed image to extract the feature points of the goods. These feature points are usually located at locations with unique characteristics such as corners and edge turning points of the goods. At the same time, an accurate map of the warehouse is pre-established, and feature points at key locations are marked on the map. During actual positioning, the feature points of the goods extracted from the image are matched with the feature points in the warehouse map. The similarity between the feature points is calculated using metrics such as Euclidean distance and Hamming distance to find the most matching feature point pair. Using the matched feature point pairs, algorithms such as triangulation and least squares are used to calculate the coordinate position of the goods relative to the map, thereby determining the specific location of the goods in the warehouse.

[0054] (5) Collaborative control of the vision system and other equipment

[0055] Establish unified equipment communication protocols and data interface standards to ensure smooth information exchange between the vision system and other equipment such as three-dimensional shelves, AGVs, and robotic arms. Establish a digital twin platform to virtually reconstruct all elements of the unmanned warehouse through the digital twin system, enabling online real-time monitoring and control. Once the vision system identifies the location and posture information of the goods, it can accurately transmit this information to the AGV through the communication interface. The AGV plans the optimal driving path based on the received information and proceeds to the designated location to transport the goods. Based on the storage, handling, and sorting tasks required for the goods, the working order and time of each device are reasonably scheduled through collaborative control algorithms to avoid conflicts and interference between devices.

[0056] In order to more clearly illustrate the specific embodiment of the present invention, an embodiment is provided below:

[0057] The unmanned AI intelligent warehouse is based on 3D machine vision, AIoT, gravity sensing and authorization authentication technology, forming a closed place that integrates efficiency and control. Relying on 3D machine vision, AI learning and other technologies, by building facial recognition gates, smart shelves / boards, and creating a 24-hour unmanned intelligent warehouse, it realizes the application of smart scenarios such as "one-keyboard click" for physical storage, in-stock material supervision, paperless entry and exit of personnel, automatic verification of material collection, and asset visualization. The business principle diagram is as follows Figure 1 shown

[0058] Example 1

[0059] (1) Multimodal data acquisition

[0060] Installed at key locations such as warehouse entrances and exits, and connecting passages between warehouses and sorting areas, they are used to detect the entry, exit, and movement of personnel, equipment, and goods, providing real-time dynamic trigger signals for the system. Gates or access control, card swiping, and facial recognition allow for rapid entry into the warehouse. Smart cameras use skeletal imaging technology to capture the paths and movements of personnel in all scenarios. Smart shelves accurately identify personnel's pick-up and placement actions and the real-time quantity of goods on display. When the infrared sensor detects a person entering the monitoring area, it immediately triggers nearby cameras and depth cameras to collect data for dynamic tracking and identification of goods. The system uses a command trigger mechanism to actively control relevant sensors for data collection based on the requirements of the task. For example, when taking inventory of goods, it triggers sensors in all shelf areas for a full-scale scan.

[0061] (2) Implementation of visual recognition and analysis based on deep learning

[0062] We collect massive amounts of cargo images and multimodal data, including samples of goods of different brands, models, and packaging forms, as well as samples under various lighting, occlusion, and stacking conditions. Data collection covers both actual warehousing scenarios and simulated experimental scenarios, ensuring the diversity and authenticity of the dataset.

[0063] The collected data is annotated using professional annotation tools, including information such as cargo category, bounding boxes, and key points (for pose estimation). To improve annotation accuracy, multi-person cross-validation is employed, and the annotation results are quality-assessed to eliminate incorrectly annotated data. Data is also preprocessed, including image scaling, cropping, normalization, and depth data smoothing, to meet the input requirements of deep learning models.

[0064] (3) Real-time data processing and transmission optimization implementation

[0065] Edge computing nodes are distributed throughout the warehouse, deployed in locations such as shelf areas, sorting areas, and entrances and exits based on sensor distribution and business needs. Each edge computing node is equipped with an NVIDIA Jetson AGXXavier or a computing module with similar performance, and tasks are rationally allocated to the edge computing nodes: For image data captured by the camera, preliminary target detection and feature extraction are performed at the edge node, and only the detected cargo target information (such as category and location) is transmitted to the data processing center. For temperature, humidity, and gas sensor data, pre-processing such as data filtering and outlier detection is performed at the edge node to reduce invalid data transmission. This approach can reduce data transmission volume by 70%-80%, effectively reducing network pressure.

[0066] (4) Unified communication protocols and interface standards

[0067] Establish unified equipment communication protocols and interface standards, and adopt the OPC UA protocol as the basic protocol for information exchange between devices. Define standardized data models, including equipment status information (such as AGV location, robotic arm working status), task instruction information (such as cargo handling tasks, sorting targets), visual inspection information (such as cargo identification results, location coordinates), etc., to achieve seamless docking between equipment of different brands and models, and ensure accurate and timely transmission of information. Assign a reasonable time window to each device's operating task to avoid conflicts and waiting between devices. By predicting the equipment's operating time and combining task priorities, dynamically adjust the equipment's execution plan. For example, when the AGV goes to a shelf to transport goods, plan the robotic arm's grasping time in advance to ensure that the robotic arm can operate immediately when the AGV arrives, reducing equipment idle time and improving overall operational efficiency.

[0068] (5) Collaborative control of the vision system and other equipment

[0069] A virtual simulation environment for a smart unmanned warehouse is constructed based on digital twin technology. Models of vision systems, AGVs, robotic arms, and sorting equipment are imported into the simulation platform to simulate real-world warehouse operation scenarios. By setting different task scenarios and parameters, the device collaborative control algorithm is simulated and tested to observe the collaborative effect between devices and detect conflicts and collisions. Based on the simulation results, the collaborative control algorithm is optimized and adjusted. For example, by adjusting parameters such as the time window size and task priority weight, the efficiency of device collaboration can be improved. At the same time, stress testing is performed in the simulation environment to simulate high-concurrency operation scenarios, verifying the stability and reliability of the system under complex conditions and ensuring efficient operation in real-world applications.

[0070] The above description of the present invention and its embodiments is non-limiting. The drawings illustrate only one embodiment of the present invention, and the actual structure is not limited thereto. In short, if a person skilled in the art is inspired by the above, and does not deviate from the purpose of the present invention, without inventive design, a structure and embodiment similar to the technical solution should fall within the scope of protection of the present invention.

Claims

1. A vision processing method for intelligent unmanned warehouse based on AIoT, characterized by: The following steps are involved: S1. Deploy a multimodal sensor network to collect multi-source data in the storage environment, including visual images, depth information, infrared signals, and environmental parameters; S2. Joint analysis of multimodal data based on deep learning models, including fine-grained cargo classification, pose estimation, and target tracking; S3, uses edge computing nodes to pre-process data in real time and optimizes data transmission efficiency through adaptive transmission protocols; S4, through the feature point matching algorithm to integrate visual and depth data, to achieve three-dimensional spatial positioning and dynamic trajectory tracking of cargo; S5. Build a collaborative control interface based on the digital twin platform to dispatch AGVs, robotic arms, and three-dimensional shelves to complete warehousing tasks.

2. The AIoT-based intelligent unmanned warehouse visual processing method according to claim 1 is characterized by: The multimodal sensor network in step S1 includes: a. High-definition cameras and depth cameras deployed in the shelf area to collect RGB-D data of goods; b. Infrared sensor arrays installed in the passageways to detect movement of people and equipment; c. Environmental sensor groups distributed throughout the storage space monitor temperature, humidity, and gas concentration in real time.

3. The AIoT-based intelligent unmanned warehouse visual processing method according to claim 1 is characterized by: The deep learning model in S2 uses an improved YOLOv7 architecture and integrates the attention mechanism with a multi-scale feature pyramid to achieve cargo recognition in occluded scenes. The model training data includes: a. 100,000 images of goods in different lighting, stacking, and damage states; b. Associated depth map and infrared thermal imaging data.

4. The AIoT-based intelligent unmanned warehouse visual processing method according to claim 1 is characterized by: The edge computing node in S3 uses the NVIDIA JetsonAGX Xavier module and performs the following processing: a. De-mosaic and dynamic range compression of visual data; b. Use lightweight MobileNetV3 model for real-time target detection; c. Generate compressed feature vectors and transmit them to the cloud server.

5. The AIoT-based intelligent unmanned warehouse visual processing method according to claim 1 is characterized by: The feature point matching algorithm in S4 includes: a. Use SURF algorithm to extract key points on the surface of cargo; b. Combine RANSAC algorithm to eliminate mismatched points; c. The cargo position is calculated using the PnP algorithm, with a positioning accuracy of ±2cm.

6. The AIoT-based intelligent unmanned warehouse visual processing method according to claim 1 is characterized by: The digital twin platform described in S5 implements the following functions: a. Build a 3D virtual image of the warehouse based on the Unity3D engine; b. Real-time synchronization of physical space equipment status data; c. Use genetic algorithm to optimize multi-device task scheduling paths.

Images