Warehouse goods location state monitoring method and system based on deep learning and aerial view vision

Abstract

The invention relates to the field of intelligent storage, and particularly discloses a storage goods location state monitoring method and system based on deep learning and aerial view vision, and the method comprises the steps: collecting a storage region image through a single camera, carrying out the distortion removal and aerial view transformation through a pre-calibrated parameter, and generating an aerial view image; adopting a pre-trained deep learning target detection model to identify a goods location state in the image; the identified goods allocation state is matched with the virtual goods allocation grid space position, the state information of each virtual goods allocation is determined or updated according to the matching result, and the system comprises an image acquisition module, an edge calculation unit and a configuration storage module; according to the method, the model training difficulty is remarkably reduced, and the system robustness is improved; software-defined goods allocation layout is realized through virtual grid configuration, and the flexibility and expandability of the system are greatly improved; a low-cost visual scheme is used for replacing a high-cost sensor array, and real-time, accurate and automatic monitoring of the goods allocation state of a large-scale storage area is achieved.

Description

Technical Field

[0001] This invention relates to the field of smart warehousing technology, specifically to a method and system for monitoring the status of warehouse storage locations based on deep learning and bird's-eye view. Background Technology

[0002] In modern intelligent warehousing and logistics management, real-time, accurate, and automatic acquisition of the occupancy status of storage locations is the core foundation for improving warehousing efficiency, achieving refined inventory management, and automated scheduling. This information directly serves the Warehouse Management System (WMS) in inventory counting, inbound location allocation, picking route optimization, and automated guided vehicle (AGV) scheduling decisions.

[0003] Currently, the mainstream technical solutions for obtaining cargo location status in the industry have many limitations: Manual inspection and inventory: This method relies on operators to periodically check and record on-site. It is inefficient, lacks real-time performance, has high labor costs, and is prone to errors due to fatigue or negligence. Barcode / Radio Frequency Identification (RFID) technology: This method involves deploying physical tags on goods and storage locations, and then using handheld or vehicle-mounted scanning devices to associate and read the information. While this method improves data accuracy, it requires labeling a massive number of goods units, resulting in high material and management costs. Furthermore, in complex scenarios such as dense stacking or metal obstructions, there is a risk of missed or misread tags. Independent sensor array solution: Infrared beam, ultrasonic, or pressure sensors are installed at the bottom or side of each storage location to directly detect the presence of goods. This solution can achieve near real-time monitoring, but for large warehouses with hundreds or thousands of storage locations, the purchase cost of the sensors themselves, the complex wiring project, and the workload of subsequent maintenance are huge. The system has poor scalability, and the sensors are susceptible to interference from industrial environmental factors such as dust, temperature, and humidity, which can cause them to fail. Traditional image processing-based identification methods use ordinary cameras to capture images of shelves and then use algorithms such as background subtraction, edge detection, and color segmentation to determine the presence of goods. These methods heavily rely on stable lighting conditions and fixed goods appearances, making them extremely unsuitable for the complex and ever-changing lighting, goods shapes, colors, and stacking postures in actual warehousing. They lack robustness, have high false alarm and false false alarm rates, and are difficult to implement in actual production applications.

[0004] Therefore, how to achieve real-time and automated monitoring of the status of cargo locations throughout the entire warehouse area in a low-cost, high-precision, and highly reliable manner is a technical challenge that urgently needs to be solved in the field of smart warehousing.

[0005] To address the existing problems, this invention provides a method and system for monitoring the status of warehouse storage locations based on deep learning and bird's-eye view, which can effectively solve the problems mentioned in the background art.

[0006] To solve the above problems, the present invention adopts the following technical solution: In a first aspect, the present invention provides a method for monitoring the status of warehouse storage locations based on deep learning and bird's-eye view, comprising the following steps: S1. System Configuration: Acquire and store the camera's intrinsic parameters and distortion correction parameters, and the view transformation mapping parameters used to generate the bird's-eye view. Define and store the virtual cargo space grid configuration information corresponding to the physical cargo space. The configuration information includes the identifier of each cargo space and its area definition in the bird's-eye view coordinate system. Load the deep learning target detection model used to detect empty cargo spaces. S2. Image Acquisition and Bird's-eye View Generation: Acquire original images of the warehouse area, process the original images based on the correction parameters and viewpoint transformation mapping parameters, and generate a standard bird's-eye view image; S3. Empty cargo space target detection: Input the standard bird's-eye view image into the deep learning target detection model for inference and identify the cargo space status in the image; S4. Cargo Location Status Mapping: Match the identified cargo location status with the virtual cargo location grid, and determine or update the status information of each virtual cargo location based on the matching result. S5. Status Information Output: Output the final status information of all virtual storage locations.

[0007] As a further embodiment of the present invention: the forward inference process of the deep learning object detection model in step S3 is expressed as: Output = F θ (I topview ) Among them, F θ For a deep learning model with parameter θ, I topview The input is a standard bird's-eye view image, and the output is a set of predicted bounding boxes output by the model.

[0008] As a further embodiment of the present invention: step S3 further includes a post-processing procedure, specifically including: Confidence filtering: retain confidence scores s i For predicted bounding boxes that exceed the preset threshold Tconf, a filtered set of bounding boxes B is obtained. filtered ={bi∣s i >T conf}; Non-maximum suppression: Sort the predicted boxes in Bfiltered by confidence and calculate the crossover ratio (CROR) between the boxes. Remove boxes with IoU exceeding the threshold T from high-confidence bounding boxes. nms The repeated bounding boxes are used to obtain the final set of detection boxes B. final .

[0009] As a further embodiment of the present invention: the spatial location matching in step S4 specifically involves: for the final detection box set B final Each detection box b in j Calculate the center pixel coordinates (cx) j cy j ), where cx j =x j ×W img cy j =y j ×H img , (x j y j w j h j W represents the normalized coordinates of the detection box. img H img The dimensions are for the bird's-eye view image; determine whether the center point falls on the virtual storage location V. k Within the rectangular area: (x k ≤cx j ≤x k +w k AND(y) k ≤cy j ≤y k +h k If true, then the storage location V will be... k The status has been updated to idle.

[0010] As a further embodiment of the present invention: the status information in step S5 is encapsulated in JSON format and provided to the outside world through an HTTP RESTful API interface.

[0011] As a further embodiment of the present invention, it also includes step S6: overlaying and rendering the standard bird's-eye view image with the status information of the virtual cargo location to generate a visual monitoring image.

[0012] As a further aspect of this invention: the deep learning object detection model is a binary classification model trained based on the YOLO architecture, with the objective of detecting empty cargo spaces. Its training process minimizes the loss function L(θ) = λ. coord L coord +λ obj L obj +λ cls L cls To optimize the model parameters θ.

[0013] In a first aspect, the present invention also provides a warehouse location status monitoring system for implementing the above method, comprising: The image acquisition module is used to acquire video or image data of the warehouse area; An edge computing unit is communicatively connected to the image acquisition module, and the edge computing unit is configured with: The image preprocessing submodule is used to call pre-stored calibration parameters to perform distortion correction and perspective transformation on the original image and generate a bird's-eye view. The deep learning inference submodule is used to load and run the empty cargo space detection model, perform intelligent analysis on the bird's-eye view, and output the empty cargo space detection results. The status mapping and management submodule is used to read the virtual storage location grid configuration, spatially match the detection results with the virtual storage locations, and manage and update the status of each storage location. The data service interface submodule is used to provide a standardized data access interface for external systems to query the status of cargo locations. A configuration storage module is provided, which is connected to the edge computing unit, for storing the camera calibration parameters, virtual storage location configuration information, and deep learning models required by the system.

[0014] As a further embodiment of the present invention, it also includes a visualization module for receiving cargo location status data and bird's-eye view images, generating visualization images with status annotations, and outputting them for display.

[0015] As a further embodiment of the present invention: the image acquisition module is one of a wide-angle RGB camera, an infrared thermal imaging camera, or a time-of-flight depth camera.

[0016] Compared with existing technologies, the beneficial effects of this invention are as follows: This invention replaces high-cost sensor solutions with low-cost vision solutions, realizing unmanned and automated inventory management of warehouse locations, improving the efficiency of warehouse management, and achieving the goal of cost reduction and efficiency improvement. It eliminates perspective distortion through bird's-eye view transformation and accurately identifies locations through deep learning models. The system has high recognition accuracy and stable and reliable operation. The warehouse location layout is defined through configuration files. When the warehouse layout changes, only the configuration files and calibration parameters need to be updated, without modifying the hardware. The system has strong adaptability. Real-time and accurate warehouse location status data can be directly used to optimize inbound strategies and schedule AGV operations. It is an important foundation for realizing intelligent warehouse management and provides strong support for intelligent decision-making.

[0017] This invention employs reverse detection logic, simplifying the complex problem of cargo identification into empty storage location detection. This significantly reduces the difficulty of model training and improves system robustness. By configuring a virtual grid to achieve software-defined storage location layout, it greatly enhances system flexibility and scalability. Replacing high-cost sensor arrays with a low-cost vision solution, it enables real-time, accurate, and automated monitoring of storage location status in large-scale warehouse areas. Attached Figure Description

[0018] Figure 1This is a flowchart illustrating a method for monitoring the status of warehouse storage locations based on deep learning and bird's-eye view. Figure 2 This is a system connection diagram for a warehouse storage location status monitoring system; Figure 3 This is a schematic diagram illustrating the effect of performing intrinsic parameter correction and distortion correction on the raw images captured by the camera. Figure 4 To Figure 3 A schematic diagram of a bird's-eye view image generated after perspective transformation of the image shown. Figure 5 In order to be in Figure 4 A visual representation of the empty cargo space detection and status mapping performed on an aerial view.

[0019] In the diagram: 1. Image acquisition module; 2. Edge computing unit; 3. Image preprocessing submodule; 4. Deep learning inference submodule; 5. State mapping and management submodule; 6. Data service interface submodule; 7. Configuration storage module; 8. Visualization module. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. Example

[0021] Combination Figure 1 This embodiment describes a method for monitoring the status of warehouse storage locations based on deep learning and bird's-eye view, including the following steps: S1. System Configuration: Acquire and store the camera's intrinsic parameters and distortion correction parameters, and the view transformation mapping parameters used to generate the bird's-eye view. Define and store the virtual cargo space grid configuration information corresponding to the physical cargo space. The configuration information includes the identifier of each cargo space and its area definition in the bird's-eye view coordinate system. Load the deep learning target detection model used to detect empty cargo spaces. Real-time monitoring and status assessment: S2. Image Acquisition and Bird's-eye View Generation: Acquire original images of the warehouse area, process the original images based on the correction parameters and viewpoint transformation mapping parameters, and generate a standard bird's-eye view image; S3. Empty cargo space target detection: Input the standard bird's-eye view image into the deep learning target detection model for inference, and identify the cargo space status in the image, such as goods or empty cargo spaces; S4. Cargo Location Status Mapping: Match the identified cargo location status with the virtual cargo location grid, and determine or update the status information of each virtual cargo location based on the matching result. S5. Status Information Output: Outputs the final status information of all virtual storage locations; S6. Overlay the standard bird's-eye view image with the status information of the virtual cargo location to generate a visual monitoring image.

[0022] Furthermore, the forward inference process of the deep learning object detection model described in step S3 is expressed as: Output = F θ (I topview ) Among them, F θ For a deep learning model with parameter θ, I topview The input is a standard bird's-eye view image, and the output is a set of predicted bounding boxes output by the model.

[0023] Step S3 also includes post-processing, specifically including: Confidence filtering: retain confidence scores s i For predicted bounding boxes that exceed the preset threshold Tconf, a filtered set of bounding boxes B is obtained. filtered ={bi∣s i >T conf}; Non-maximum suppression: Sort the predicted boxes in Bfiltered by confidence and calculate the crossover ratio (CROR) between the boxes. Remove boxes with IoU exceeding the threshold T from high-confidence bounding boxes. nms The repeated bounding boxes are used to obtain the final set of detection boxes B. final .

[0024] Furthermore, in step S4, spatial location matching involves calculating the coordinates of the center point of the identified empty storage area and determining whether these center point coordinates are located within a certain virtual storage area. Specifically, for the final set of detection boxes B... final Each detection box b in j Calculate the center pixel coordinates (cx) j cy j ), where cx j =x j ×W img cy j =y j ×H img , (x j y j w j h j W represents the normalized coordinates of the detection box. img H imgThe dimensions are for the bird's-eye view image; determine whether the center point falls on the virtual storage location V. k Within the rectangular area: (x k ≤cx j ≤x k +w k AND (y) k ≤cy j ≤y k +h k If true, then the storage location V will be... k The status has been updated to idle.

[0025] Furthermore, in step S5, the status information is encapsulated in JSON format and provided externally through an HTTP RESTful API interface.

[0026] Furthermore, the deep learning object detection model is a binary classification model trained based on the YOLO architecture, with the goal of detecting empty cargo spaces. Its training process minimizes the loss function L(θ)=λ. coord L coord +λ obj L obj +λ cls L cls To optimize the model parameters θ. Example

[0027] like Figure 2 As shown, this embodiment provides a warehouse location status monitoring system for implementing the above method, including: Image acquisition module 1 is used to acquire video or image data of the warehouse area. Image acquisition module 1 is one of a wide-angle RGB camera, an infrared thermal imaging camera, or a time-of-flight depth camera. Edge computing unit 2 is communicatively connected to the image acquisition module 1, and the edge computing unit 2 is configured with: Image preprocessing submodule 3 is used to call pre-stored calibration parameters to perform distortion correction and perspective transformation on the original image and generate a bird's-eye view; Deep learning inference submodule 4 is used to load and run the empty cargo space detection model, perform intelligent analysis on the bird's-eye view, and output the empty cargo space detection results. The status mapping and management submodule 5 is used to read the virtual storage location grid configuration, spatially match the detection results with the virtual storage locations, and manage and update the status of each storage location. Data service interface submodule 6 is used to provide a standardized data access interface for external systems to query the status of cargo locations; The configuration storage module 7 is connected to the edge computing unit 2 and is used to store the camera calibration parameters, virtual storage location configuration information and deep learning model required by the system.

[0028] It also includes a visualization module 8, which receives cargo location status data and bird's-eye view images, generates visualization images with status labels, and outputs them for display. Example

[0029] like Figure 3-5 As shown, this embodiment uses an HTTP API-triggered warehouse location monitoring system to monitor an existing warehouse location. Specifically, the system hardware includes a wide-angle network camera installed on the warehouse beam, which is connected to an industrial control computer via a network cable. The industrial control computer is equipped with a solid-state drive.

[0030] The implementation process is as follows: 1. Hardware Calibration: Using calibration methods such as checkerboard patterns, calculate the camera's intrinsic parameter matrix (cameraMatrix) and distortion coefficients (distCoeffs). Select four vertices of a rectangular region in the physical world (such as the entire warehouse area), obtain their pixel coordinates in the original image and their actual coordinates in the physical world (or pixel coordinates in the target bird's-eye view), and calculate the homography matrix (homography) used for perspective transformation. 2. Storage location layout definition: In a configuration file (e.g., locations.yaml), define a unique ID for each storage location and specify its region of interest (ROI) in the generated bird's-eye view, which is usually a rectangle (x, y, width, height).

[0031] 3. Model Loading: Load a pre-trained deep learning object detection model specifically designed for detecting "empty cargo space" features (e.g., the empty.onnx model in ONNX format). The camera intrinsic parameter matrix and distortion coefficients are calculated using a checkerboard calibration board, the homography matrix of the bird's-eye view transformation is calculated, a virtual cargo space grid configuration file in YAML format is created, and a pre-trained YOLO empty cargo space detection model (ONNX format) is loaded.

[0032] 4. Real-time monitoring and data processing: The HTTP server receives a status query request from the WMS. The current frame image is obtained through the RTSP protocol, and distortion correction and bird's-eye view transformation are performed. The bird's-eye view is input into a deep learning model for inference, and the predicted bounding box is output. Confidence filtering of the predicted bounding boxes (T) conf =0.25) and NMS processing (T nms =0.45); Calculate the center point of each detection frame, match it with the virtual storage location grid, and update the storage location status; The status information is encapsulated in JSON format and returned via an HTTP response.

[0033] 3. Visual Debugging: Display the location status as rectangles of different colors overlaid on the bird's-eye view. For example, use green rectangles to mark locations judged as "vacant" and red rectangles to mark locations "occupied," and display the location ID and status text. Finally, display or save the image.

[0034] The system is set up as an independent background service, automatically executing a complete monitoring process every 30 seconds. The results are published to a specified topic via the MQTT protocol or written directly to the database for subscription by AGV scheduling systems and other systems.

[0035] Of course, the model can be expanded into a multi-category detector to identify specific cargo types and its applications can be migrated to scenarios such as intelligent parking lot space management and production line material monitoring. In special environments, infrared or ToF cameras can be used instead of RGB cameras.

[0036] The working principle of this invention is to replace the high-cost sensor solution with a low-cost vision solution, thereby achieving unmanned and automated inventory management of warehouse locations, improving the efficiency of warehouse management, and achieving the goal of cost reduction and efficiency improvement. Perspective distortion is eliminated by transforming the bird's-eye view, and accurate recognition is achieved through a deep learning model. The system has a high recognition accuracy rate and stable and reliable operation. The warehouse location layout is defined through a configuration file. When the warehouse layout changes, only the configuration file and calibration parameters need to be updated, without modifying the hardware. The system has strong adaptability, and real-time and accurate warehouse location status data can be directly used to optimize the inbound strategy and schedule AGV operations. It is an important foundation for realizing intelligent warehouse management and provides strong support for intelligent decision-making.

[0037] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0038] Although embodiments of the present invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for monitoring the status of warehouse storage locations based on deep learning and bird's-eye view, characterized in that, Includes the following steps: S1. System Configuration: Acquire and store the camera's intrinsic parameters and distortion correction parameters, and the view transformation mapping parameters used to generate the bird's-eye view. Define and store the virtual cargo space grid configuration information corresponding to the physical cargo space. The configuration information includes the identifier of each cargo space and its area definition in the bird's-eye view coordinate system. Load the deep learning target detection model used to detect empty cargo spaces. S2. Image Acquisition and Bird's-eye View Generation: Acquire original images of the warehouse area, process the original images based on correction parameters and viewpoint transformation mapping parameters, and generate standard bird's-eye view images; S3. Empty cargo space target detection: Input the standard bird's-eye view image into a deep learning target detection model for inference to identify the cargo space status in the image; S4. Cargo Location Status Mapping: Match the identified cargo location status with the virtual cargo location grid, and determine or update the status information of each virtual cargo location based on the matching result. S5. Status Information Output: Output the final status information of all virtual storage locations.

2. The warehouse location status monitoring method based on deep learning and bird's-eye view according to claim 1, characterized in that, The forward inference process of the deep learning object detection model in step S3 is represented as: Output = F θ (I topview ) Among them, F θ For a deep learning model with parameter θ, I topview The input is a standard bird's-eye view image, and the output is a set of predicted bounding boxes output by the model.

3. The warehouse location status monitoring method based on deep learning and bird's-eye view according to claim 2, characterized in that, Step S3 also includes a post-processing procedure, specifically including: Confidence filtering: retain confidence scores s i For predicted bounding boxes that exceed the preset threshold Tconf, a filtered set of bounding boxes B is obtained. filtered ={bi∣s i >T conf }; Non-maximum suppression: Sort the predicted boxes in Bfiltered by confidence and calculate the crossover ratio (CROR) between the boxes. Remove boxes with IoU exceeding the threshold T from high-confidence bounding boxes. nms The repeated bounding boxes are used to obtain the final set of detection boxes B. final .

4. The warehouse location status monitoring method based on deep learning and bird's-eye view according to claim 1, characterized in that, The spatial location matching in step S4 specifically involves: for the final detection box set B... final Each detection box b in j Calculate the center pixel coordinates (cx) j cy j ), where cx j =x j ×W img cy j =y j ×H img , (x j y j w j h j W represents the normalized coordinates of the detection box. img H img The dimensions are for the bird's-eye view image; determine whether the center point falls on the virtual storage location V. k Within the rectangular area: (x k ≤cx j ≤x k +w k AND(y) k ≤cy j ≤y k +h k If true, then the storage location V will be... k The status has been updated to idle.

5. The warehouse location status monitoring method based on deep learning and bird's-eye view according to claim 1, characterized in that, In step S5, the status information is encapsulated in JSON format and provided externally through an HTTP RESTful API interface.

6. The warehouse location status monitoring method based on deep learning and bird's-eye view according to claim 1, characterized in that, It also includes step S6: overlaying and rendering the standard bird's-eye view image with the status information of the virtual cargo location to generate a visual monitoring image.

7. The warehouse location status monitoring method based on deep learning and bird's-eye view according to claim 1, characterized in that, The deep learning object detection model is a binary classification model trained based on the YOLO architecture, with the goal of detecting empty cargo spaces. Its training process minimizes the loss function L(θ)=λ. coord L coord +λ obj L obj +λ cls L cls To optimize the model parameters θ.

8. A warehouse storage location status monitoring system for implementing any one of the methods of claims 1-7, characterized in that, include: The image acquisition module is used to acquire video or image data of the warehouse area; An edge computing unit is communicatively connected to the image acquisition module, and the edge computing unit is configured with: The image preprocessing submodule is used to call pre-stored calibration parameters to perform distortion correction and perspective transformation on the original image and generate a bird's-eye view. The deep learning inference submodule is used to load and run the empty cargo space detection model, perform intelligent analysis on the bird's-eye view, and output the empty cargo space detection results. The status mapping and management submodule is used to read the virtual storage location grid configuration, spatially match the detection results with the virtual storage locations, and manage and update the status of each storage location. The data service interface submodule is used to provide a standardized data access interface for external systems to query the status of cargo locations. A configuration storage module is provided, which is connected to the edge computing unit, for storing the camera calibration parameters, virtual storage location configuration information, and deep learning models required by the system.

9. The warehouse location status monitoring system according to claim 8, characterized in that, It also includes a visualization module, which receives cargo location status data and bird's-eye view images, generates visualization images with status labels, and outputs them for display.

10. The warehouse location status monitoring system according to claim 8, characterized in that, The image acquisition module is one of a wide-angle RGB camera, an infrared thermal imaging camera, or a time-of-flight depth camera.

Images