A method and system for inventory checking of a three-dimensional warehouse based on an unmanned aerial vehicle

Abstract

The present application relates to a method and system for inventory checking of a three-dimensional warehouse based on an unmanned aerial vehicle. The method includes: receiving a goods inventory checking task and obtaining the layout information of the three-dimensional warehouse; determining the position of the inventory checking storage location in the goods inventory checking task in the three-dimensional warehouse; based on the layout information of the three-dimensional warehouse and the position of the inventory checking storage location in the three-dimensional warehouse, determining the safe flight area and hovering points of the unmanned aerial vehicle based on a path planning algorithm, and generating a flight task; sending the flight task to the unmanned aerial vehicle, and calculating the pose information of the unmanned aerial vehicle in real time during the execution of the flight task through an inertial measurement unit; based on the observed image of the unmanned aerial vehicle vision system, correcting the cumulative error of the pose information through a Kalman filtering algorithm; obtaining in real time the RGB color image and depth image collected by the unmanned aerial vehicle at each inventory checking storage location; determining whether the box stack is in an abnormal working condition according to the depth image; for the box stack not in an abnormal working condition, determining the quantity of goods in the box stack according to the RGB color image.

Description

Technical Field

[0001] This application relates to the technical field of warehouse inventory, and particularly to a three-dimensional warehouse inventory method and system based on unmanned aerial vehicles (UAVs). Background Art

[0002] In warehouse management, especially for the management of automated three-dimensional warehouses, the inventory work is an essential basic link to ensure efficient operation. Regular, daily, and irregular inventory modes enable enterprises to monitor the inventory status, assist in business decision-making, and optimize the overall warehouse management process. To achieve efficient inventory management, the inventory system should have the characteristics of high precision, non-continuous operation, and low time occupancy rate to ensure that even during the inventory period, the impact on regular operations can be minimized, while maintaining a high degree of flexibility and system integration.

[0003] In the prior art, the inventory of automated three-dimensional warehouses is generally achieved through manual visual inventory and RFID (Radio Frequency Identification) inventory. Manual visual inventory refers to directly observing and recording the status and quantity of the inventory in the warehouse by warehouse management personnel to ensure the accuracy of inventory data. RFID inventory refers to automatically reading the RFID tag information attached to items through radio frequency identification technology.

[0004] Manual visual inventory is inefficient and consumes a large amount of time and human resources. Since it relies on manual operations, it is prone to misreading, missing records, or writing errors, affecting the accuracy of inventory data. At the same time, because the inventory process takes a long time, it cannot provide instant updates of the inventory status, which may lead to delayed management decisions. Moreover, frequent manual inventories will increase labor costs, and the warehouse operation may need to be suspended during the inventory period, indirectly increasing the operating costs.

[0005] Although RFID inventory improves efficiency and accuracy, it also has some disadvantages. For example, the initial system construction and tag costs are relatively high, the reading of items made of special materials such as metals or liquids may be unstable, and signal interference may occur in a dense storage environment, resulting in reading errors. In addition, the effective operation of the RFID system depends on good technical support and maintenance. Once a technical failure occurs, it may affect the smooth progress of the entire inventory process. Moreover, if the tag is damaged or falls off, the commodity information cannot be correctly identified. Summary of the Invention

[0006] To at least partly overcome the problems of high cost, high maintenance difficulty, and low system stability of the automated three-dimensional warehouse inventory method in the related art, this application provides a three-dimensional warehouse inventory method and system based on unmanned aerial vehicles (UAVs).

[0007] The solution of this application is as follows:

[0008] According to the first aspect of the embodiments of the present application, a method for inventory checking of a three-dimensional warehouse based on an unmanned aerial vehicle is provided, including:

[0009] Receiving a goods inventory checking task and obtaining the layout information of the three-dimensional warehouse;

[0010] Determining the position of the inventory-checking storage locations in the goods inventory checking task in the three-dimensional warehouse;

[0011] According to the layout information of the three-dimensional warehouse and the position of the inventory-checking storage locations in the three-dimensional warehouse, based on a path planning algorithm, determining the safe flight area and hovering points of the unmanned aerial vehicle and generating a flight task;

[0012] Sending the flight task to the unmanned aerial vehicle and calculating in real time the pose information of the unmanned aerial vehicle when executing the flight task through an inertial measurement unit;

[0013] Based on the observed images of the unmanned aerial vehicle vision system, correcting the cumulative error of the pose information through a Kalman filtering algorithm;

[0014] Obtaining in real time the RGB color images and depth images collected by the unmanned aerial vehicle at each inventory-checking storage location;

[0015] Determining whether the box stack is in an abnormal working condition according to the depth image;

[0016] For the box stacks not in an abnormal working condition, determining the quantity of goods in the box stacks according to the RGB color images.

[0017] Preferably, obtaining the layout information of the three-dimensional warehouse includes:

[0018] Obtaining the three-dimensional drawing of the three-dimensional warehouse and determining the number of shelves in the three-dimensional warehouse, the volume parameters of each shelf, the position of the stacker crane track, the width of the shelf channel, and the specific position of the warehousing and outbound ports based on the three-dimensional drawing of the three-dimensional warehouse;

[0019] Obtaining the on-site investigation and verification results and verifying the three-dimensional drawing of the three-dimensional warehouse according to the on-site investigation and verification results.

[0020] Preferably, the method further includes:

[0021] Correspondingly binding each shelf channel in the three-dimensional warehouse with each unmanned aerial vehicle;

[0022] According to the layout information of the shelf channel and the position of the inventory-checking storage location in the shelf channel, based on a path planning algorithm, determining the safe flight area and hovering points of the unmanned aerial vehicle in its corresponding shelf channel and generating a flight task for the unmanned aerial vehicle.

[0023] Preferably, the method further includes:

[0024] When the UAV is performing a flight mission, the stacker in the corresponding shelf channel of the UAV is controlled to withdraw to the entrance and exit and suspend operation.

[0025] Preferably, based on the observation image of the UAV vision system, the cumulative error of the pose information is corrected by the Kalman filtering algorithm, including:

[0026] Construct a state covariance matrix, and predict the error covariance of the pose information through the Kalman filtering algorithm;

[0027] Determine whether the current observation image of the UAV vision system is a key frame;

[0028] If it is a key frame, expand the state covariance matrix, track and remove the feature points in the key frame, and extract new feature points;

[0029] Determine whether the tracking of the feature points is completed;

[0030] If the tracking of the feature points has ended, compare the length of the tracking frame with the minimum tracking threshold;

[0031] If the length of the tracking frame is greater than the minimum tracking threshold, calculate the three-dimensional coordinates of the feature points, calculate the reprojection error of the feature points according to the three-dimensional coordinates of the feature points to obtain the observation information, and correct the cumulative error of the pose information according to the observation information;

[0032] If the tracking of the feature points has not ended, compare the length of the tracking frame with the maximum tracking threshold;

[0033] If the length of the tracking frame is greater than the maximum tracking threshold, calculate the three-dimensional coordinates of the feature points, screen the tracking frame, calculate the reprojection error of the feature points according to the three-dimensional coordinates of the feature points in the screened tracking frame to obtain the observation information, and correct the cumulative error of the pose information according to the observation information.

[0034] Preferably, the method further includes:

[0035] Construct a state update estimation equation from time k - 1 to k:

[0036] ;

[0037] Wherein, represents the system state estimation at time k; represents the system state estimation at time k - 1; represents the gain matrix at time k, represents the measurement value or residual at time k; represents the measurement model matrix at time k;

[0038] The state vector Decomposed into the inertial measurement unit vector part and the UAV vision system vector part, the state vector matrix is expressed as:

[0039] ;

[0040] Wherein, represents the state vector of the entire system; represents the inertial measurement unit vector part; represents the UAV vision system vector part;

[0041] The inertial measurement unit vector is expressed as:

[0042] ;

[0043] Wherein; represents the carrier attitude quaternion; represents the velocity vector; represents the position vector; represents the gyroscope zero bias; represents the accelerometer zero bias; represents the installation deviation angle quaternion between the UAV vision system and the inertial measurement unit; represents the arm vector between the UAV vision system and the inertial measurement unit;

[0044] The error state vector of the inertial measurement unit is expressed as:

[0045] ;

[0046] Wherein; represents the rotation angle error of the inertial measurement unit relative to the world coordinate system; represents the angular velocity error of the inertial measurement unit relative to the world coordinate system; represents the translation error from the world coordinate system to the inertial measurement unit; represents the gyroscope bias of the inertial measurement unit; represents the accelerometer bias of the inertial measurement unit; represents the rotation angle error of the UAV vision system relative to the inertial measurement unit; represents the translation error from the inertial measurement unit to the UAV vision system;

[0047] The angular velocity error of the inertial measurement unit relative to the world coordinate system is expressed as:

[0048] ;

[0049] Wherein, and represent the rotation matrix between the world coordinate system W and the inertial measurement unit; Represents the velocity vector;

[0050] The state prediction model of the Kalman filter is expressed as:

[0051] ;

[0052] Where; Represents the increment of the change rate of the inertial measurement unit state; Represents the state transition matrix; Represents the increment of the state variable; Represents the control input matrix; Represents the process noise;

[0053] State transition matrix Is expressed as:

[0054] ;

[0055] Where; Represents the angular velocity of the Earth's rotation; Represents the direction cosine matrix from the vehicle coordinate system to the world coordinate system; Represents the gravitational acceleration in the world coordinate system;

[0056] The gravitational acceleration conversion formula is expressed as:

[0057] ;

[0058] Direction cosine matrix from the Earth coordinate system to the world coordinate system Is expressed as:

[0059] ;

[0060] Where; Represents the gravitational acceleration in the Earth coordinate system; Represents the latitude of the world coordinate system relative to the Earth coordinate system; Represents the longitude of the world coordinate system relative to the Earth coordinate system;

[0061] Inertial measurement unit noise transfer matrix Is expressed as:

[0062] ;

[0063] White noise vector Is expressed as:

[0064] ;

[0065] Where; Represents the white noise of the gyroscope; Represents the white noise of the accelerometer; Represents the white noise of the gyroscope bias; Represents the white noise of the accelerometer bias; Represents the white noise of the attitude angle; Represents the white noise of the position;

[0066] The vector representation of the UAV vision system is:

[0067] ;

[0068] Where Represents the quaternion rotation attitude from the world coordinate system to the UAV vision system coordinate system at the first frame; Represents the position of the UAV vision system in the world coordinate system at the first frame; Represents the quaternion rotation attitude from the world coordinate system to the UAV vision system coordinate system at the Nth frame; Represents the position of the UAV vision system in the world coordinate system at the Nth frame;

[0069] The error state vector related to the UAV vision system is represented as:

[0070] ;

[0071] Where and Represent the attitude error and position error of the UAV vision system at the first frame; Represents the rotation error from the world coordinate system to the UAV vision system coordinate system at the Nth frame; Represents the position error of the UAV vision system in the world coordinate system at the Nth frame;

[0072] The state error vector of the (N + 1)th frame is:

[0073] ;

[0074] ;

[0075] Where Represents the rotation error from the world coordinate system to the UAV vision system coordinate system at the (N + 1)th frame; Represents the position error of the UAV vision system in the world coordinate system at the (N + 1)th frame; Represents the 3×3 identity matrix; Represents the 3×3 identity matrix; Represents the rotation matrix from the inertial measurement unit coordinate system to the world coordinate system; Represents the position of the UAV vision system in the inertial measurement unit coordinate system, Denotes the operation of converting a vector into an anti-symmetric matrix;

[0076] The augmented state covariance matrix Is expressed as:

[0077] ;

[0078] ;

[0079] ;

[0080] Wherein, Denotes the augmented state covariance matrix; Denotes the identity matrix of order 6N + 15; Denotes the Jacobian matrix, which is used to describe the relationship between state error vectors; Denotes the original state covariance matrix; Denotes the covariance between the state error of the UAV vision system in the (N + 1)-th frame and the conversion error from the inertial measurement unit coordinate system to the UAV vision system coordinate system; Denotes the covariance of the state error of the UAV vision system in the (N + 1)-th frame itself;

[0081] Calculate the three-dimensional coordinate estimation value of the j-th feature point in the UAV vision system coordinate system in the i-th frame image:

[0082] ;

[0083] Wherein, Denotes the three-dimensional coordinate estimation value of the j-th feature point in the UAV vision system coordinate system in the i-th frame image; Denotes the X-axis coordinate of the feature point in the UAV vision system coordinate system; Denotes the Y-axis coordinate of the feature point in the UAV vision system coordinate system; Denotes the Z-axis coordinate of the feature point in the UAV vision system coordinate system; Denotes the rotation matrix from the world coordinate system to the UAV vision system coordinate system in the i-th frame; Denotes the three-dimensional coordinate of the j-th feature point in the world coordinate system; Denotes the position of the UAV vision system in the i-th frame in the world coordinate system;

[0084] Calculate the two-dimensional coordinate estimation value of the j-th feature point in the UAV vision system coordinate system in the i-th frame image:

[0085] ;

[0086] Wherein, Denote the two-dimensional coordinate estimation value of the $j$-th feature point in the $i$-th frame of the UAV vision system coordinate system; Denote the depth of the $j$-th feature point in the $i$-th frame of the UAV vision system coordinate system; Denote the $X$-axis coordinate of the $j$-th feature point in the $i$-th frame of the UAV vision system coordinate system; Denote the $Y$-axis coordinate of the $j$-th feature point in the $i$-th frame of the UAV vision system coordinate system;

[0087] Linearly represent the observation model with the system error state and the feature point position error:

[0088] ;

[0089] wherein, Denote the residual of the $i$-th observation value in the $j$-th frame; Denote the influence of the system error state in the $j$-th frame on the $i$-th observation value; Denote the system error state vector; Denote the influence of the feature point position error in the $j$-th frame on the $i$-th observation value; Denote the position error vector of the feature point in the world coordinate system, Denote the noise term of the $i$-th observation value in the $j$-th frame;

[0090] wherein, Denote as:

[0091] ;

[0092] ;

[0093] Convert the observation model to:

[0094] .

[0095] Preferably, the method further includes:

[0096] Obtain historical bin images, perform target detection frame annotation on the front end face, upper top face of the whole stack of boxes, and the front end face of the pallet in the historical bin images, and use the annotation data as training data;

[0097] Train a target detection model based on the training data.

[0098] Preferably, the method further includes:

[0099] Perform confidence screening on the front end face, upper top face of the box stack, and the front end face of the pallet in the RGB color image based on the target detection model;

[0100] Perform validity screening on the pallet of this bin position based on the preset pallet region of interest;

[0101] The working conditions of the screened effective pallets are divided into: no pallet and no goods, pallet but no goods, and pallet with goods;

[0102] The quantity of goods directly output from effective pallets for the working conditions of no pallet and no goods or pallet and no goods is zero;

[0103] Based on the pre-set cargo area of interest and the depth image of the valid pallet with pallets and goods, the abnormal working condition of the box stack is identified;

[0104] The front end face and upper top face of the abnormal box stack are removed, and the detection frames of the side stack and rear stack of the abnormal box stack are removed.

[0105] Preferably, determining the quantity of goods in the box stack according to the RGB color image comprises:

[0106] The whole stack is layered according to the detection frame on the front end of the box stack;

[0107] Determine whether each layer is full and calculate the quantity of goods on each layer except the top layer of the pallet;

[0108] Determine the effective front face of the box stack;

[0109] According to the effective front end surface of the box stack, the effective upper top surface of the box stack is determined, and the quantity of goods in the effective upper top surface of the box stack is identified;

[0110] Calculate the sum of the quantity of goods in each layer of the stack except the top layer and the quantity of goods in the effective upper top surface of the stack to obtain the quantity of goods in the stack.

[0111] According to a second aspect of an embodiment of the present application, a drone-based three-dimensional warehouse inventory system is provided, comprising:

[0112] Processor and memory;

[0113] The processor and the memory are connected via a communication bus:

[0114] Wherein, the processor is used to call and execute the program stored in the memory;

[0115] The memory is used to store a program, and the program is at least used to execute a drone-based three-dimensional warehouse inventory method as described in any one of the above items.

[0116] The technical solution provided by this application may have the following beneficial effects:

[0117] Compared with manual visual inventory, the drone can quickly reach the designated location without personnel entering the warehouse for inventory, greatly reducing the inventory time. At the same time, the RGBD vision device carried by the drone realizes accurate inventory counting by combining deep learning object detection and visual counting technology, reducing the safety risk of operators.

[0118] Compared with RFID inventory, this technical solution does not require each item to be equipped with an RFID tag, saving the cost of tags and the time for tag installation and maintenance. The image information provided by the RGBD vision device is not limited to simple presence detection and can provide more information such as the abnormal status of storage locations. RFID technology may be affected by the environment, resulting in reading failures. In contrast, the vision-based inventory method is not restricted by these factors and can work stably.

[0119] In summary, this technical solution is superior to traditional manual visual inventory and RFID inventory in terms of efficiency, accuracy, safety, and cost-effectiveness, and is particularly suitable for large and complex warehousing environments.

[0120] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and do not limit this application. BRIEF DESCRIPTION OF THE DRAWINGS

[0121] The accompanying drawings herein are incorporated into the specification and form a part of this specification, showing embodiments consistent with this application, and are used together with the specification to explain the principles of this application.

[0122] Figure 1 is a flowchart showing a method for inventory of a three-dimensional warehouse based on a drone provided by an embodiment of this application;

[0123] Figure 2 is a flowchart showing a process of correcting the cumulative error of pose information through the Kalman filter algorithm for an observation image provided by an embodiment of this application based on a drone vision system;

[0124] Figure 3 is a flowchart showing a process of determining whether a pallet is in an abnormal working condition according to a depth image provided by an embodiment of this application;

[0125] Figure 4 is a flowchart showing a process of determining the number of items in a pallet according to an RGB color image provided by an embodiment of this application;

[0126] Figure 5 is a schematic structural diagram of a three-dimensional warehouse inventory system based on a drone provided by an embodiment of this application.

[0127] Reference numerals: Processor - 51; Memory - 52. Detailed implementation manners

[0128] Here, exemplary embodiments will be described in detail, and examples thereof are shown in the accompanying drawings. When the following description refers to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. On the contrary, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0129] Embodiment 1

[0130] Figure 1 is a schematic flowchart of a three-dimensional warehouse inventory method based on an unmanned aerial vehicle provided by an embodiment of the present application. Referring to Figure 1 , a three-dimensional warehouse inventory method based on an unmanned aerial vehicle includes:

[0131] S11: Receive a goods inventory task and obtain the layout information of the three-dimensional warehouse;

[0132] The operator selects the inventory locations to be inventoried through the WMS (Warehouse Management System), and the WMS automatically generates an inventory task list and sends it to the WCS (Warehouse Control System) at the same time. The warehouse control system (i.e., this system) further processes it after receiving the goods inventory task.

[0133] It should be noted that obtaining the layout information of the three-dimensional warehouse includes:

[0134] Obtain the three-dimensional drawing of the three-dimensional warehouse, and determine the number of shelves in the three-dimensional warehouse, the volume parameters of each shelf, the position of the stacker crane track, the width of the shelf aisle, and the specific position of the entrance and exit based on the three-dimensional drawing of the three-dimensional warehouse;

[0135] Obtain the on-site inspection and verification results, and verify the three-dimensional drawing of the three-dimensional warehouse according to the on-site inspection and verification results.

[0136] To ensure that the unmanned aerial vehicle can execute the inventory task for the specified location efficiently and accurately, it is first necessary to obtain the detailed layout information of the automated warehouse. This includes accurate three-dimensional drawings covering the number of shelves in the three-dimensional warehouse, the volume parameters of each shelf, the position of the stacker crane track, the width of the shelf aisle, and the specific position of the entrance and exit. Then, it is also necessary to verify the accuracy of the existing drawings through the on-site inspection and verification results, and promptly correct the discrepancies with the actual environment.

[0137] S12: Determine the position of the inventory locations to be inventoried in the goods inventory task in the three-dimensional warehouse;

[0138] S13: Based on the layout information of the automated storage and retrieval system (AS / RS) and the position of the inventory locations to be counted in the AS / RS, determine the safe flight area and hovering points of the unmanned aerial vehicle (UAV) based on a path planning algorithm, and generate a flight mission.

[0139] It should be noted that the method further includes:

[0140] Correspondingly bind each rack aisle in the AS / RS to each UAV one by one.

[0141] Based on the layout information of the rack aisle and the position of the inventory locations to be counted in the rack aisle, determine the safe flight area and hovering points of the UAV in its corresponding rack aisle based on a path planning algorithm, and generate the flight mission of the UAV.

[0142] It should be noted that in this embodiment, a single-aisle configuration of UAVs is adopted. A dedicated UAV is equipped for each aisle, and each UAV is responsible for the aerial photography and inventory counting tasks of a single aisle. This can reduce the difficulty of obstacle avoidance and improve work efficiency.

[0143] It should be noted that the method further includes:

[0144] When the UAV executes the flight mission, control the stacker in the rack aisle corresponding to the UAV to retreat to the entrance / exit and pause working.

[0145] During the period when the UAV executes the inventory counting task, the stacker in that aisle should pause working and retreat to the entrance / exit end to avoid mutual interference and ensure safe operation.

[0146] In specific practice, according to the specific inventory counting tasks of each aisle, use a path planning algorithm combined with the three-dimensional information of the AS / RS to calculate and generate the flight path of the UAV. This process takes into account factors such as the position of the inventory locations to be counted, the positions of obstacles such as racks and other fixed structures. The overall process of path planning: Starting from the fixed starting point position at the UAV storage end, go to each inventory location to be counted in turn. (Such as inventory location 1 → inventory location 2 →... → inventory location N), and finally return to the fixed starting point position of the UAV.

[0147] S14: Send the flight mission to the UAV, and calculate the pose information of the UAV in real time during the execution of the flight mission through an inertial measurement unit;

[0148] Calculate the attitude, position, and velocity information of the UAV in real time during the execution of the flight mission through an inertial measurement unit as the pose information of the UAV.

[0149] S15: Based on the observed images of the UAV vision system, correct the cumulative error of the pose information through a Kalman filtering algorithm;

[0150] S16: Obtain the RGB color images and depth images collected by the drone at each inventory location to be counted in real time;

[0151] S17: Determine whether the pallet is in an abnormal working condition according to the depth image;

[0152] S18: For the pallets not in an abnormal working condition, determine the quantity of goods in the pallet according to the RGB color images.

[0153] It should be noted that drones usually rely on GPS (Global Positioning System) for navigation. However, in the environment of an automated storage and retrieval system (AS / RS), this dependence encounters challenges. In the AS / RS, there are densely arranged and very high shelves, and high-density goods are stored. These factors will not only block or reflect GPS signals, resulting in signal attenuation, but also produce multipath effects, seriously affecting the positioning accuracy. Therefore, when performing flight operations in such an environment, the drone may not be able to achieve precise navigation and positioning relying on GPS.

[0154] Therefore, the technical problems to be solved by this technical solution are mainly divided into two parts. One is the precise positioning of the drone, and the other is the inventory count in the warehouse.

[0155] In this technical solution, the precise positioning of the drone is achieved by inertial navigation plus visual positioning. Among them, the inertial measurement unit (accelerometer and gyroscope) is used as the main positioning sensor to calculate the pose information of the drone in real time, and the drone vision system is used as the observed quantity to correct the cumulative error of inertial navigation through the Kalman filter algorithm.

[0156] The inventory count in the warehouse can achieve efficient counting of the entire stack of goods at the specified storage location through advanced visual recognition technology. This process combines RGB color images and depth images and uses computer vision algorithms to complete the automatic counting of pallets. Specifically, the RGB color images are mainly used for visual counting; while the depth images confirm whether the pallet is in an abnormal working condition through three-dimensional space data.

[0157] Compared with manual visual inventory counting, the drone in this technical solution can quickly reach the specified location without personnel entering the warehouse for inventory counting, greatly reducing the inventory counting time. At the same time, the RGBD vision device carried by the drone realizes precise inventory counting by combining deep learning object detection and visual counting technology, reducing the safety risks of operators.

[0158] Compared with RFID inventory, this technical solution does not require equipping each item with an RFID tag, saving the tag cost and the time for tag installation and maintenance. The image information provided by the RGBD vision device is not limited to simple presence detection, but can also provide more information such as abnormal status of storage locations. RFID technology may be affected by the environment, resulting in reading failures. However, the vision-based inventory method is not restricted by these factors and can work stably.

[0159] In summary, this technical solution is superior to traditional manual visual inventory and RFID inventory in terms of efficiency, accuracy, security, and cost-effectiveness, and is especially suitable for large and complex warehousing environments.

[0160] Embodiment 2

[0161] In this embodiment, an explanation is given on how to achieve precise positioning of the unmanned aerial vehicle (UAV).

[0162] Refer to Figure 2 , based on the observed images of the UAV vision system, the cumulative error of the pose information is corrected through the Kalman filtering algorithm, including:

[0163] S21: Construct a state covariance matrix, and predict the error covariance of the pose information through the Kalman filtering algorithm;

[0164] S22: Determine whether the current observed image of the UAV vision system is a key frame;

[0165] S23: If it is a key frame, expand the state covariance matrix, track and remove the feature points in the key frame, and extract new feature points;

[0166] S24: Determine whether the tracking of the feature points has ended;

[0167] S25: If the tracking of the feature points has ended, compare the length of the tracking frame with the minimum tracking threshold;

[0168] S26: If the length of the tracking frame is greater than the minimum tracking threshold, calculate the three-dimensional coordinates of the feature points, calculate the reprojection error of the feature points based on the three-dimensional coordinates to obtain the observation information, and correct the cumulative error of the pose information according to the observation information;

[0169] S27: If the tracking of the feature points has not ended, compare the length of the tracking frame with the maximum tracking threshold;

[0170] S28: If the length of the tracking frame is greater than the maximum tracking threshold, calculate the three-dimensional coordinates of the feature points, screen the tracking frame, calculate the reprojection error of the feature points based on the three-dimensional coordinates in the screened tracking frame to obtain the observation information, and correct the cumulative error of the pose information according to the observation information.

[0171] It should be noted that the method further includes:

[0172] Construct a state update estimation equation from the (k - 1)th moment to the kth moment:

[0173] ;

[0174] where represents the system state estimation at the kth moment; represents the system state estimation at the (k - 1)th moment; represents the gain matrix at the kth moment, represents the measurement value or residual at the kth moment; represents the measurement model matrix at the kth moment;

[0175] Decompose the state vector into an inertial measurement unit vector part and a UAV vision system vector part, then the state vector matrix is expressed as:

[0176] ;

[0177] where represents the state vector of the entire system; represents the inertial measurement unit vector part; represents the UAV vision system vector part;

[0178] The inertial measurement unit vector is expressed as:

[0179] ;

[0180] where; represents the carrier attitude quaternion; represents the velocity vector; represents the position vector; represents the gyroscope zero bias; represents the accelerometer zero bias; represents the installation deviation angle quaternion between the UAV vision system and the inertial measurement unit; represents the arm vector between the UAV vision system and the inertial measurement unit;

[0181] The error state vector of the inertial measurement unit is expressed as:

[0182] ;

[0183] where; represents the rotation angle error of the inertial measurement unit relative to the world coordinate system; represents the angular velocity error of the inertial measurement unit relative to the world coordinate system; represents the translation error from the world coordinate system to the inertial measurement unit; Represents the gyroscope bias of the inertial measurement unit; Represents the accelerometer bias of the inertial measurement unit; Represents the rotational angle error of the UAV vision system relative to the inertial measurement unit; Represents the translational error from the inertial measurement unit to the UAV vision system;

[0184] The angular velocity error of the inertial measurement unit relative to the world coordinate system is expressed as:

[0185] ;

[0186] Among them, and Represent the rotation matrix between the world coordinate system W and the inertial measurement unit; Represents the velocity vector;

[0187] The state prediction model of the Kalman filter is expressed as:

[0188] ;

[0189] Among them; Represents the increment of the state change rate of the inertial measurement unit; Represents the state transition matrix; Represents the increment of the state variable; Represents the control input matrix; Represents the process noise;

[0190] The state transition matrix Is expressed as:

[0191] ;

[0192] Among them; Represents the angular velocity of the earth's rotation; Represents the direction cosine matrix from the vehicle coordinate system to the world coordinate system; Represents the gravitational acceleration in the world coordinate system;

[0193] The gravitational acceleration conversion formula is expressed as:

[0194] ;

[0195] The direction cosine matrix from the earth coordinate system to the world coordinate system Is expressed as:

[0196] ;

[0197] Among them; Represents the gravitational acceleration in the earth coordinate system; Represents the latitude of the world coordinate system relative to the earth coordinate system; Represents the longitude of the world coordinate system relative to the earth coordinate system;

[0198] Inertial measurement unit noise transfer matrix Is expressed as:

[0199] ;

[0200] White noise vector Is expressed as:

[0201] ;

[0202] Where; Represents the white noise of the gyroscope; Represents the white noise of the accelerometer; Represents the white noise of the gyroscope bias; Represents the white noise of the accelerometer bias; Represents the white noise of the attitude angle; Represents the white noise of the position;

[0203] The UAV vision system vector is expressed as:

[0204] ;

[0205] Where, Represents the quaternion rotation attitude from the world coordinate system to the UAV vision system coordinate system at the first frame; Represents the position of the UAV vision system in the world coordinate system at the first frame; Represents the quaternion rotation attitude from the world coordinate system to the UAV vision system coordinate system at the Nth frame; Represents the position of the UAV vision system in the world coordinate system at the Nth frame;

[0206] The error state vector related to the UAV vision system is expressed as:

[0207] ;

[0208] Where, and Represent the attitude error and position error of the UAV vision system at the first frame; Represents the rotation error from the world coordinate system to the UAV vision system coordinate system at the Nth frame; Represents the position error of the UAV vision system in the world coordinate system at the Nth frame;

[0209] It should be noted that whenever the UAV vision system calculates the pose measurement value for a key frame, the new key frame camera pose state needs to be added to the original state vector, and the state covariance matrix of the Kalman filter needs to be augmented.

[0210] The state error vector of the (N + 1)-th frame is:

[0211] ;

[0212] ;

[0213] where represents the rotation error from the world coordinate system to the UAV vision system coordinate system at the (N + 1)-th frame; represents the position error of the UAV vision system in the world coordinate system at the (N + 1)-th frame; represents the 3×3 identity matrix; represents the 3×3 identity matrix; represents the rotation matrix from the inertial measurement unit coordinate system to the world coordinate system; represents the position of the UAV vision system in the inertial measurement unit coordinate system, represents the operation of converting a vector into an anti-symmetric matrix;

[0214] The augmented state covariance matrix is expressed as:

[0215] ;

[0216] ;

[0217] ;

[0218] where represents the augmented state covariance matrix; represents the identity matrix of order 6N + 15; represents the Jacobian matrix, which is used to describe the relationship between state error vectors; represents the original state covariance matrix; represents the covariance between the state error of the UAV vision system at the (N + 1)-th frame and the conversion error from the inertial measurement unit coordinate system to the UAV vision system coordinate system; represents the covariance of the state error of the UAV vision system itself at the (N + 1)-th frame;

[0219] In this integrated navigation, the UAV vision system serves as an observation sensor, and its observation model Defined as the reprojection error of the key - frame feature points of the UAV vision system, that is, the error obtained by comparing the pixel coordinates (the projected position of the key points) with the position obtained by projecting the 3D points according to the currently estimated pose.

[0220] Calculate the estimated three - dimensional coordinates of the j - th feature point in the coordinate system of the UAV vision system in the i - th frame image:

[0221] ;

[0222] Among them, represents the estimated three - dimensional coordinates of the j - th feature point in the coordinate system of the UAV vision system in the i - th frame image; represents the X - axis coordinate of the feature point in the coordinate system of the UAV vision system; represents the Y - axis coordinate of the feature point in the coordinate system of the UAV vision system; represents the Z - axis coordinate of the feature point in the coordinate system of the UAV vision system; represents the rotation matrix from the world coordinate system to the coordinate system of the i - th frame UAV vision system; represents the three - dimensional coordinates of the j - th feature point in the world coordinate system; represents the position of the i - th frame UAV vision system in the world coordinate system;

[0223] Calculate the estimated two - dimensional coordinates of the j - th feature point in the coordinate system of the UAV vision system in the i - th frame image:

[0224] ;

[0225] Among them, represents the estimated two - dimensional coordinates of the j - th feature point in the coordinate system of the UAV vision system in the i - th frame image; represents the depth of the j - th feature point in the coordinate system of the i - th frame UAV vision system; represents the X - axis coordinate of the j - th feature point in the coordinate system of the i - th frame UAV vision system; represents the Y - axis coordinate of the j - th feature point in the coordinate system of the i - th frame UAV vision system;

[0226] Linearly represent the observation model with the system error state and the feature point position error:

[0227] ;

[0228] Among them, represents the residual of the i - th observation value in the j - th frame; represents the influence of the system error state in the j - th frame on the i - th observation value; represents the system error state vector; Indicates the influence of the feature point position error in the j-th frame on the i-th observation value; Indicates the position error vector of the feature point in the world coordinate system, Indicates the noise term of the i-th observation value in the j-th frame;

[0229] Among them, It is expressed as:

[0230] ;

[0231] ;

[0232] Convert the observation model to:

[0233] .

[0234] After having the state model, the observation model and the covariance matrix, the Kalman gain can be solved and the state update of the UAV pose can be performed.

[0235] It should be noted that the above visual navigation feature point detection and pairing algorithm can apply the FAST (Features from Accelerated Segment Test) corner detection algorithm; the feature point tracking of sequential frame images can use the KLT (Kanade Lucas Tomasi) algorithm; for the corner points of moving objects in the image, outliers need to be removed, and the two-point RANSAC algorithm assisted by the inertial measurement unit can be used to achieve this. The above three algorithms are all common algorithms and will not be elaborated here.

[0236] Embodiment 3

[0237] In this embodiment, how to implement the in-library inventory is described.

[0238] In this technical solution, the object of the three-dimensional library inventory mainly focuses on the box-pallet three-dimensional library. At the same time, before performing the box-pallet visual counting, it is necessary to obtain the full-stack stack type information of the library location, including the full-stack quantity and the quantity of each layer of the full-stack

[0239] It should be noted that the method further includes:

[0240] Obtain historical library location images, perform target detection frame annotation on the front end face, upper top face of the whole stack of box pallets, and the front end face of the pallet in the historical library location images, and use the annotation data as training data;

[0241] Train a target detection model based on the training data.

[0242] In the preparation stage, it is necessary to train a target detection model through historical library location images.

[0243] Preferably, the YOLO target detection model is selected as the target detection model.

[0244] After the target detection model is trained, refer to Figure 3 , the method further comprises:

[0245] S31: Confidence screening of the front face of the box stack, the top face and the front face of the pallet in the RGB color image based on the target detection model;

[0246] First, confidence screening is performed on the front face of the box stack, the top face, and the front face of the pallet to remove detection frames with poor results.

[0247] S32: screening the effectiveness of the pallets in the current cargo location based on the pre-set pallet interest area;

[0248] Since the storage location images taken by the drone in the early stage are at a specified height, the pallet will appear in a fixed position, so the pallet ROI (region of interest) is set to ensure that the rear pallet will not enter the recognition process.

[0249] S33: classifying the working conditions of the screened valid pallets; the pallet working conditions include: no pallet and no goods, pallet but no goods, and pallet with goods;

[0250] According to the number of valid pallets obtained, the working conditions of the screened valid pallets are divided into three working conditions: no pallet and no goods, pallet but no goods, and pallet with goods (normal pallet counting).

[0251] S34: The quantity of goods directly output from effective pallets for the working conditions of no pallet and no goods or pallet and no goods is zero;

[0252] There is no need to count valid pallets with no goods or pallets with no goods, and the results can be returned directly.

[0253] S35: Based on the preset cargo location interest area and the depth image of the valid pallet with pallets and goods, identify the abnormal working condition of the box stack;

[0254] In view of the abnormal situations that whole pallets of goods may be scattered or skewed in actual engineering situations, the acquired depth map is used to generate a point cloud and set the cargo location ROI. If it exceeds the range, it is judged as an abnormal state.

[0255] S36: removing the front end face and the upper top face of the abnormal box stack, and removing the detection frames of the side stack and the rear stack of the abnormal box stack.

[0256] The detection frames of the side and rear stacks are removed. The removal principle depends on the spatial coordinates of the pallet and the spatial relationship between the front end and top surface of the box stack.

[0257] It should be noted that, refer to Figure 4, determining the quantity of goods in the pallet according to the RGB color image, including:

[0258] S41: performing whole-pallet layering based on the detection frame of the front face of the pallet;

[0259] Whole-pallet layering: performing whole-pallet layering based on the detection frame of the front face of the pallet. The condition for judging as the same layer is:

[0260] ;

[0261] Among them, represents the layer distance threshold for judging the same layer on the front face, and represent the minimum and maximum values in the y direction of the detection frame of the front face, and represent the minimum and maximum values in the x direction of the detection frame of the front face.

[0262] S42: judging and identifying whether each layer is a full layer, and calculating the quantity of goods in each layer of the pallet except the top layer;

[0263] Due to the parallax caused by the shooting angle, it is necessary to judge and identify whether each layer is a full layer or a non-full layer.

[0264] The specific judgment conditions are:

[0265] Layer width + perspective compensation value > pallet width Lower boundary of the layer > all top surface frames .

[0266] Among them, the perspective compensation value can be set by oneself.

[0267] All top surface frames represent the maximum value in the y direction of all top surface frames.

[0268] S43: determining the effective front face of the pallet;

[0269] Further screening out the effective front face according to the front face involved in the non-full layer. The effective front face is the basis for selecting the upper top surface.

[0270] The screening conditions for the effective front face are:

[0271] ;

[0272] Among them, the y-direction threshold and the x-direction threshold can be set by oneself. and represent the distances in the y direction and the x direction of the front face detection frame.

[0273] Front face frame represents the minimum value in the y direction of the front face frame;

[0274] Front end face frame Represents the minimum value of the front end face frame in the x direction;

[0275] Front end face frame Represents the maximum value of the front end face frame in the x direction;

[0276] Upper top face frame Represents the maximum value of the upper top face frame in the y direction;

[0277] Upper top face frame Represents the minimum value of the upper top face frame in the x direction;

[0278] Upper top face frame Represents the maximum value of the upper top face frame in the x direction.

[0279] S44: Determine the effective upper top face of the pallet stack according to the effective front end face of the pallet stack, and identify the number of goods in the effective upper top face of the pallet stack;

[0280] During the process of counting the top face frames, the top face frames exposed in the lower layer will also be detected, and now count the effective top face detection frames.

[0281] The specific judgment condition is:

[0282] ;

[0283] Top face frame Represents the central value of the top face frame in the y direction;

[0284] Effective front end face frame Represents the minimum value of the effective front end face frame in the y direction;

[0285] Effective front end face frame Represents the minimum value of the effective front end face frame in the x direction;

[0286] Top face frame Represents the central value of the top face frame in the x direction;

[0287] Effective front end face frame Represents the maximum value of the effective front end face frame in the x direction.

[0288] S45: Calculate the sum of the number of goods in each layer of the pallet stack except the top layer and the number of goods in the effective upper top face of the pallet stack to obtain the number of goods in the pallet stack.

[0289] The final number of goods in the pallet stack is the sum of the number of goods in each layer of the pallet stack except the top layer and the number of goods in the effective upper top face of the pallet stack.

[0290] Embodiment 4

[0291] A three-dimensional warehouse inventory system based on an unmanned aerial vehicle, referring to Figure 5, including:

[0292] a processor 51 and a memory 52;

[0293] The processor 51 and the memory 52 are connected through a communication bus:

[0294] Among them, the processor 51 is used to call and execute the program stored in the memory 52;

[0295] The memory 52 is used to store the program, and the program is at least used to execute one of the above-described embodiments of the three-dimensional warehouse inventory method based on the unmanned aerial vehicle.

[0296] It can be understood that the same or similar parts in the above embodiments can be referred to each other, and the content not detailed in some embodiments can be seen in the same or similar content in other embodiments.

[0297] It should be noted that in the description of the present application, the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. In addition, in the description of the present application, unless otherwise specified, the meaning of "a plurality" means at least two.

[0298] Any process or method description shown in the flowchart or described in other ways herein can be understood as representing a module, segment, or part of code including one or more executable instructions for implementing a specific logical function or process, and the scope of the preferred embodiments of the present application includes additional implementations, where the functions can be executed in a substantially simultaneous manner or in a reverse order according to the involved functions, rather than in the order shown or discussed, which should be understood by those skilled in the technical field of the embodiments of the present application.

[0299] It should be understood that each part of the present application can be implemented by hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in the memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, any one of the following well-known technologies in the art or a combination thereof can be used: discrete logic circuits having logic gate circuits for implementing logical functions on data signals, application-specific integrated circuits having appropriate combinational logic gate circuits, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0300] Those of ordinary skill in the art of this technology can understand that all or part of the steps carried by the method of the above embodiments can be completed by instructing relevant hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiments.

[0301] In addition, each functional unit in various embodiments of the present application may be integrated into one processing module, or each unit may exist physically alone, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. When the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0302] The above-mentioned storage medium may be a read-only memory, a magnetic disk, an optical disc, etc.

[0303] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples", etc. means that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0304] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and should not be construed as limiting the present application. Those of ordinary skill in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present application.

Claims

1. A method for inventory taking of a three-dimensional warehouse based on an unmanned aerial vehicle, characterized in that, Including: Receiving a goods inventory task and obtaining the layout information of the stereoscopic warehouse; Determining the position of the goods location to be inventoried in the goods inventory task in the stereoscopic warehouse; Based on the layout information of the stereoscopic warehouse and the position of the goods location to be inventoried in the stereoscopic warehouse, determining the safe flight area and hovering points of the unmanned aerial vehicle (UAV) based on a path planning algorithm and generating a flight task; Sending the flight task to the UAV and calculating the pose information of the UAV in real time during the execution of the flight task through an inertial measurement unit; Based on the observed image of the UAV vision system, correcting the cumulative error of the pose information through a Kalman filtering algorithm; Obtaining the RGB color image and depth image collected by the UAV at each goods location to be inventoried in real time; Determining whether the box stack is in an abnormal working condition according to the depth image; For the box stacks not in an abnormal working condition, determining the quantity of goods in the box stack according to the RGB color image; Based on the observed image of the UAV vision system, correcting the cumulative error of the pose information through a Kalman filtering algorithm, including: Constructing a state covariance matrix and predicting the error covariance of the pose information through a Kalman filtering algorithm; Judging whether the current observed image of the UAV vision system is a key frame; If it is a key frame, expanding the state covariance matrix, tracking and removing the feature points in the key frame, and extracting new feature points; Judging whether the tracking of the feature points is ended; If the tracking of the feature points has ended, comparing the length of the tracking frame with the minimum tracking threshold; If the length of the tracking frame is greater than the minimum tracking threshold, calculating the three-dimensional coordinates of the feature points, calculating the reprojection error of the feature points according to the three-dimensional coordinates of the feature points to obtain the observed information, and correcting the cumulative error of the pose information according to the observed information; If the tracking of the feature points has not ended, comparing the length of the tracking frame with the maximum tracking threshold; If the length of the tracking frame is greater than the maximum tracking threshold, calculating the three-dimensional coordinates of the feature points, screening the tracking frame, calculating the reprojection error of the feature points according to the three-dimensional coordinates of the feature points in the screened tracking frame to obtain the observed information, and correcting the cumulative error of the pose information according to the observed information.

2. The method according to claim 1, characterized in that Obtaining the layout information of the stereoscopic warehouse, including: Obtaining the three-dimensional drawing of the stereoscopic warehouse and determining the number of shelves in the stereoscopic warehouse, the volume parameters of each shelf, the position of the stacker crane track, the width of the shelf aisle, and the specific position of the entrance and exit according to the three-dimensional drawing of the stereoscopic warehouse; Obtaining the on-site investigation and verification results and verifying the three-dimensional drawing of the stereoscopic warehouse according to the on-site investigation and verification results.

3. The method according to claim 1, wherein The method further includes: Correspondingly binding each shelf aisle in the stereoscopic warehouse with each UAV one by one; Based on the layout information of the shelf aisle and the position of the goods location to be inventoried in the shelf aisle, determining the safe flight area and hovering points of the UAV in its corresponding shelf aisle based on a path planning algorithm and generating the flight task of the UAV.

4. The method according to claim 1, wherein The method further includes: When the UAV executes the flight task, controlling the stacker crane in the shelf aisle corresponding to the UAV to withdraw to the entrance and exit and suspend working.

5. The method according to claim 1, characterized in that, The method further includes: Constructing a state update and estimation equation from time k - 1 to time k: ； Among them, represents the system state estimate at time k; represents the system state estimate at time k-1; represents the gain matrix at time k, represents the measurement value or residual at time k; represents the measurement model matrix at time k; Decompose the state vector into the inertial measurement unit vector part and the UAV vision system vector part, and the state vector matrix is expressed as: ； Among them, represents the state vector of the entire system; represents the inertial measurement unit vector part; represents the UAV vision system vector part; The inertial measurement unit vector is expressed as: ； Among them; represents the carrier attitude quaternion; represents the velocity vector; represents the position vector; represents the gyroscope zero bias; represents the accelerometer zero bias; represents the installation deviation angle quaternion of the UAV vision system and the inertial measurement unit; represents the arm vector of the UAV vision system and the inertial measurement unit; The error state vector of the inertial measurement unit is expressed as: ； Wherein; represents the rotation angle error of the inertial measurement unit relative to the world coordinate system; represents the angular velocity error of the inertial measurement unit relative to the world coordinate system; represents the translation error from the world coordinate system to the inertial measurement unit; represents the gyroscope bias of the inertial measurement unit; represents the accelerometer bias of the inertial measurement unit; represents the rotation angle error of the UAV vision system relative to the inertial measurement unit; represents the translation error from the inertial measurement unit to the UAV vision system; The angular velocity error of the inertial measurement unit relative to the world coordinate system is expressed as: ； Among them, and represent the rotation matrix between the world coordinate system W and the inertial measurement unit; represents the velocity vector; The state prediction model of the Kalman filter is expressed as: ； Wherein; represents the increment of the change rate of the inertial measurement unit state; represents the state transition matrix; represents the increment of the state variable; represents the control input matrix; represents the process noise; State transition matrix It is expressed as: ； Wherein; represents the angular velocity of the Earth's rotation; represents the direction cosine matrix from the vehicle coordinate system to the world coordinate system; represents the gravitational acceleration in the world coordinate system; The gravitational acceleration conversion formula is expressed as: ； Direction cosine matrix from the Earth coordinate system to the world coordinate system It is expressed as: ； Wherein; represents the gravitational acceleration in the earth coordinate system; represents the latitude of the world coordinate system relative to the earth coordinate system; represents the longitude of the world coordinate system relative to the earth coordinate system; Inertial Measurement Unit Noise Transfer Matrix It is expressed as: ； White noise vector It is expressed as: ； Wherein; represents the white noise of the gyroscope; represents the white noise of the accelerometer; represents the white noise of the gyroscope bias; represents the white noise of the accelerometer bias; represents the white noise of the attitude angle; represents the white noise of the position; The vector of the UAV vision system is expressed as: ； Among them, represents the quaternion rotation attitude from the world coordinate system to the UAV vision system coordinate system at the first frame; represents the position of the UAV vision system in the world coordinate system at the first frame; represents the quaternion rotation attitude from the world coordinate system to the UAV vision system coordinate system at the Nth frame; represents the position of the UAV vision system in the world coordinate system at the Nth frame; The error state vector related to the UAV vision system is expressed as: ； Among them, and represent the attitude error and position error of the UAV vision system in the first frame; represents the rotation error from the world coordinate system to the UAV vision system coordinate system at the Nth frame; represents the position error of the UAV vision system in the world coordinate system at the Nth frame; The state error vector of the (N + 1)-th frame is: ； ； Among them, represents the rotation error from the world coordinate system to the UAV vision system coordinate system at the (N + 1)-th frame; represents the position error of the UAV vision system in the world coordinate system at the (N + 1)-th frame; represents a 3×3 identity matrix; represents a 3×3 identity matrix; represents the rotation matrix from the inertial measurement unit coordinate system to the world coordinate system; represents the position of the UAV vision system in the inertial measurement unit coordinate system, represents the operation of converting a vector into an anti-symmetric matrix; The amplified state covariance matrix is expressed as: ； ； ； Among them, represents the state covariance matrix after amplification; represents the identity matrix of order 6N + 15; represents the Jacobian matrix, which is used to describe the relationship between state error vectors; represents the original state covariance matrix; represents the covariance between the state error of the drone vision system in the (N + 1)-th frame and the conversion error from the inertial measurement unit coordinate system to the drone vision system coordinate system; represents the covariance of the state error of the drone vision system in the (N + 1)-th frame itself; Calculate the three-dimensional coordinate estimate of the j-th feature point in the UAV vision system coordinate system in the i-th frame image: ； Among them, represents the estimated three-dimensional coordinate value of the j-th feature point in the UAV vision system coordinate system of the i-th frame image; represents the X-axis coordinate of the feature point in the UAV vision system coordinate system; represents the Y-axis coordinate of the feature point in the UAV vision system coordinate system; represents the Z-axis coordinate of the feature point in the UAV vision system coordinate system; represents the rotation matrix from the world coordinate system to the i-th frame UAV vision system coordinate system; represents the three-dimensional coordinate of the j-th feature point in the world coordinate system; represents the position of the i-th frame UAV vision system in the world coordinate system; Calculate the two-dimensional coordinate estimate of the j-th feature point in the UAV vision system coordinate system in the i-th frame image: ； wherein, represents the two-dimensional coordinate estimation value of the j-th feature point in the UAV vision system coordinate system of the i-th frame image; represents the depth of the j-th feature point in the UAV vision system coordinate system of the i-th frame; represents the X-axis coordinate of the j-th feature point in the UAV vision system coordinate system of the i-th frame; represents the Y-axis coordinate of the j-th feature point in the UAV vision system coordinate system of the i-th frame; Linearly represent the observation model with the system error state and the feature point position error: ； Among them, represents the residual of the i-th observation value in the j-th frame; represents the influence of the system error state on the i-th observation value in the j-th frame; represents the system error state vector; represents the influence of the feature point position error on the i-th observation value in the j-th frame; represents the position error vector of the feature point in the world coordinate system, represents the noise term of the i-th observation value in the j-th frame; Among them, It is expressed as: ； ； Convert the observation model to: 。 6. The method according to claim 1, wherein The method further includes: Obtain historical bin images, perform target detection box annotation on the front end face, upper top face of the whole stack of boxes, and the front end face of the pallet in the historical bin images, and use the annotation data as training data; Train a target detection model based on the training data.

7. The method according to claim 6, wherein The method further includes: Perform confidence screening on the front end face, upper top face, and front end face of the pallet in the RGB color image based on the target detection model; Perform validity screening on the pallet of the current bin based on the preset region of interest of the pallet; Perform working condition classification on the screened valid pallets; The pallet working conditions include: no pallet and no goods, there is a pallet but no goods, and there is a pallet and there are goods; Directly output that the quantity of goods is zero for the valid pallets in the working conditions of no pallet and no goods and there is a pallet but no goods; Based on the preset region of interest of the bin and the depth image of the valid pallet with a pallet and there are goods, perform abnormal working condition identification of the box stack; Remove the front end face and upper top face of the abnormal box stack, and remove the detection frames of the side stack and the rear stack of the abnormal box stack.

8. The method according to claim 7, characterized in that, Determine the quantity of goods in the box stack according to the RGB color image, including: Perform whole-stack layering according to the detection frame of the front end face of the box stack; Judge and identify whether each layer is a full layer, and calculate the quantity of goods in each layer of the box stack except the top layer; Determine the valid front end face of the box stack; According to the valid front end face of the box stack, determine the valid upper top face of the box stack, and identify the quantity of goods in the valid upper top face of the box stack; Calculate the sum of the quantity of goods in each layer of the box stack except the top layer and the quantity of goods in the valid upper top face of the box stack to obtain the quantity of goods in the box stack.

9. A three-dimensional warehouse inventory system based on drones, characterized in that, Includes: A processor and a memory; The processor is connected to the memory through a communication bus: Wherein, the processor is used to call and execute the program stored in the memory; The memory is used to store the program, and the program is at least used to execute a method for inventory checking of a three-dimensional warehouse based on a UAV according to any one of claims 1-8.

Images