Digital twin-based rotor unmanned aerial vehicle inventory counting method and apparatus, and storage medium

Abstract

The present invention relates to a digital twin-based rotor unmanned aerial vehicle inventory counting method and apparatus, and a storage medium, applicable to the technical field of logistics warehousing. The method comprises: constructing a digital twin model of a logistics warehouse; synchronizing a warehouse exit and warehouse entry task of the warehouse in the digital twin model to realize virtuality and reality synchronization of goods; simultaneously controlling a virtual unmanned aerial vehicle and a physical unmanned aerial vehicle to advance according to a planned path; taking photos of the goods on cells by means of cameras; carrying out virtuality and reality comparison, thereby determining the accuracy of the goods in each cell; in the cruising processes of the unmanned aerial vehicles, determining the coordinate deviations of the virtual unmanned aerial vehicle and the physical unmanned aerial vehicle by means of an epipolar geometry algorithm; and calculating the position of the physical unmanned aerial vehicle in real time according to the position of the virtual unmanned aerial vehicle. According to the present invention, the physical unmanned aerial vehicle can be prevented from gradually deviating from an originally planned path due to external interference, and the real-time performance of the path of the unmanned aerial vehicle is improved. The present application does not need manual intervention, the number of participating sensors is small, energy consumption is low, a photo is independently taken for each cell, and the definition of the photo is high.

Description

Rotary-wing UAV disk library method, device and storage medium based on digital twin Technical Field

[0001] The present invention relates to the field of logistics and warehousing technology, and in particular to a rotary-wing drone inventory method, device, and storage medium based on digital twins. Background Art

[0002] Automated three-dimensional warehousing systems are used to store goods. The accuracy of cargo accounts is crucial to warehouse management. Due to equipment failures such as stackers, manual entry and exit, and other reasons, there are often discrepancies between the system records and actual inventory. In such cases, inventory checks are necessary to verify the inventory accounts, confirm the actual inventory quantity, and correct any discrepancies.

[0003] At present, the main methods for inventory counting in automated warehouses include manual inventory counting, stacker crane video inventory counting and other methods;

[0004] There are two methods of manual inventory: one is that workers sit on the stacker loading platform and check the information and verify the cargo location information as the loading platform approaches the goods; the other is to use the stacker to pick up the goods from the designated cargo location and bring them to the outbound platform, where workers check the cargo information next to the platform.

[0005] The stacker crane video inventory method involves placing an industrial camera on the stacker crane's loading platform, traversing the entire three-dimensional warehouse, taking photos or videos of the goods and transmitting them to the management system. Workers then compare the inventory information with the information presented in the actual photos, or with the inventory database and the inventory database to complete the inventory operation. For example, patent No. 202110687757.2, an intelligent inventory method and system for goods in unmanned warehouses based on environmental perception, uses a stacker crane equipped with a multi-eye panoramic camera for inventory counting. The stacker crane needs to run through the entire three-dimensional warehouse to collect video streams to form an inventory database.

[0006] Drone inventory method: Application No. 202010213428.X discloses a drone-based RFID warehousing logistics method and system, Application No. 202210344350.4 discloses a drone inventory system, and Application No. 202010513538.8 discloses a drone indoor warehouse autonomous inventory system and method;

[0007] Manual inventory method: In the first manual inventory method, workers need to sit on the loading platform to get close to the goods, and then need to check the goods information. There is a great risk when running with the stacker between high-rise shelves, and the workload is large, the efficiency is low, and the labor cost is high. In the second manual inventory method, the stacker is required to complete the outbound and inbound operations when completing each cargo verification work. The operation time is long, the workload is large, the inventory efficiency is low, the energy consumption is high, the service life of the stacker is reduced, and the full life cycle cost is increased.

[0008] Stacker video inventory method: The stacker is required to carry a camera to traverse the entire warehouse to collect video streams, form an inventory database and compare it with the inventory database to complete the inventory. This method requires the stacker to run in two directions to traverse the entire warehouse, which has high energy consumption, long operation time and low efficiency. In addition, when inventorying the double-extended warehouse, it is difficult to fully capture the goods at the far end of the stacker, affecting the accuracy of the inventory.

[0009] Drone inventory method: Application No. 202010213428.X discloses a drone-based RFID warehousing logistics method and system. The system needs to be equipped with RFID readers, image processing systems, positioning systems, height sensors, etc., and labels must be placed on shelves and goods. The system is redundant, costly, and multiple sensors and data acquisition equipment lead to high energy consumption. The RFID reader has a high bit error rate, and the drone inventory path has poor real-time update performance. When encountering obstacles such as stackers or other equipment, collisions and other safety issues are prone to occur; Application No. 202210344350.4 discloses a drone inventory system. The system needs to be equipped with scanning cameras, visual cameras, laser rangefinders and UWB positioning tags. Color labels need to be arranged on environmental hardware such as shelves. The system is complicated, costly, and the multiple sensors and data acquisition equipment result in high energy consumption. The UWB positioning technology has high energy consumption and poor accuracy, and the real-time update of the drone's inventory path is poor. When encountering obstacles such as stackers or other equipment, it is easy to cause collisions and other safety issues; Application No. 202010513538.8 discloses a drone indoor warehouse autonomous inventory system and method. The system needs to be equipped with image collectors, temperature and humidity collectors, WUB locators, RFID readers, etc., and labels must be arranged on shelves and goods. The system is complicated, costly, and the multiple sensors and data acquisition equipment result in high energy consumption. The RFID reader has a high bit error rate, and the real-time update of the drone's inventory path is poor. When encountering obstacles such as stackers or other equipment, it is easy to cause collisions and other safety issues.

[0010] Summary of the Invention

[0011] In view of this, the purpose of the present invention is to provide a rotary-wing drone inventory method, device and storage medium based on digital twins, so as to solve the problems in the prior art that the manual inventory method has a large workload, low efficiency and high labor cost, and the stacker video inventory method has high energy consumption, long operation time and low efficiency, and when inventorying a double-extending stereoscopic warehouse, it is difficult to fully capture the goods at the far end of the stacker, which affects the accuracy of the inventory. At the same time, it solves the problems of the existing drone inventory method such as system complexity, high cost, high energy consumption caused by multiple sensors and data acquisition equipment, and high energy consumption and poor accuracy of UWB positioning technology.

[0012] According to a first aspect of an embodiment of the present invention, a rotary-wing drone inventory method based on digital twins is provided, comprising:

[0013] Constructing a digital twin model of a logistics warehouse, wherein the digital twin model of the logistics warehouse includes a virtual logistics warehouse, a virtual drone, a virtual shelf, and virtual goods;

[0014] Synchronize the inbound and outbound tasks to the digital twin model of the logistics warehouse, and synchronously update the virtual goods on the virtual shelves according to the inbound and outbound tasks;

[0015] Determine an area of ​​a warehouse to be inspected, and generate a drone cruising path in the area of ​​the warehouse to be inspected according to a preset route generation rule;

[0016] The physical drone and the virtual drone start cruising along the drone cruising path from an initial position of the drone cruising path;

[0017] The physical drone takes photos of the physical cargo every time it flies a distance of one cargo grid; the virtual drone takes photos of the virtual cargo every time it flies a distance of one virtual cargo grid;

[0018] Calculating the physical cargo photo and the virtual cargo photo using an epipolar geometry algorithm to determine whether there is a position offset between the physical drone and the virtual drone;

[0019] If it does not exist, the physical cargo photo and the virtual cargo photo are similarly calculated, the calculation result is compared with the preset similarity threshold, and the inventory record of the cargo compartment is generated based on the comparison result, and the process continues to the next cargo compartment;

[0020] If so, the physical drone position is adjusted according to the virtual drone position. After adjusting the position, the physical drone takes a photo of the physical cargo again, calculates the similarity between the photo and the virtual cargo photo, generates a stocktaking record for the cargo compartment, and then continues to cruise to the next cargo compartment.

[0021] After the physical drone and the virtual drone complete their cruise along the drone cruise path, they obtain inventory records of the goods on all cargo cells in the area to be inventoryed.

[0022] Preferably,

[0023] The calculating the physical cargo photo and the virtual cargo photo using an epipolar geometry algorithm to determine whether there is a position offset between the physical drone and the virtual drone includes:

[0024] Denoising the virtual goods photo and the physical goods photo respectively by a mean filtering operation, and then performing a grayscale operation on the denoised photos;

[0025] constructing multi-scale spaces for the virtual goods photo and the physical goods photo, respectively, calculating extreme values ​​of pixels of the virtual goods photo and the physical goods photo at each scale, and selecting the maximum extreme value of the virtual goods photo and the physical goods photo in each scale space as a feature point;

[0026] A continuous curve is used to fit the multi-scale spatial extreme value to determine the exact position and scale of each feature point. The gradient information of the feature point and the neighboring pixels is used as the direction parameter of the feature point to obtain the virtual feature point set and the real feature point set respectively.

[0027] Each feature point is described using a 128-dimensional feature vector. A virtual feature point from the virtual feature point set is selected. The entire real feature point set is traversed to obtain the two real feature points in the real feature point set that are closest to the selected virtual feature point in Euclidean distance. The ratio of the Euclidean distance between the closest real feature point and the second closest real feature point and the selected virtual feature point is calculated. If the ratio is less than the preset Euclidean distance threshold, the match is successful, and the selected virtual feature point and the real feature point with the closest Euclidean distance are taken as a pair of matching feature points.

[0028] Then select another virtual feature point from the virtual feature point set and repeat the above process until all virtual feature points in the virtual feature point set are matched with the real feature point set to obtain multiple pairs of matching feature points;

[0029] According to the mapping relationship of the camera, multiple pairs of matching feature points in the obtained image coordinate system are converted into the image pixel coordinate system;

[0030] Obtaining a basic matrix based on the association relationship between multiple pairs of matching feature points of the virtual goods photo and the physical goods photo in the image pixel coordinate system;

[0031] According to the relationship between the basic matrix and the essential matrix, the essential matrix is ​​obtained, and the rotation matrix R and translation matrix t of the physical camera relative to the virtual camera are obtained according to the essential matrix;

[0032] The rotation matrix R and the translation matrix t are used to calculate whether there is a position offset between the physical drone and the virtual drone.

[0033] Preferably,

[0034] Calculating the similarity between the physical goods photo and the virtual goods photo includes:

[0035] grayscale the physical goods photo and the virtual goods photo;

[0036] A Gaussian weighting function with a preset standard deviation is used as a weighting window to calculate the pixel points in the grayscaled physical goods photo and the virtual goods photo to obtain an SSIM index mapping matrix composed of local SSIM indices. The average SSIM index of the SSIM index mapping matrix is ​​calculated, and the average SSIM index of the SSIM index mapping matrix is ​​used as the similarity.

[0037] Preferably,

[0038] In the area of ​​the waiting warehouse, generating a drone cruising path according to a preset route generation rule includes:

[0039] The intersection of the lines connecting the four corner points of each cargo grid is used as the base point, and the base points of each cargo grid are connected from low to high and from the storage end to the storage end, and the line connecting the base points is used as the cruising path of the drone.

[0040] Preferably,

[0041] The digital twin model of the logistics warehouse also includes:

[0042] Virtual stacker crane and virtual shuttle;

[0043] The digital twin model of the logistics warehouse obtains PLC motion data of the physical stacker and the physical shuttle, and drives the virtual stacker and the virtual shuttle to move in the virtual logistics warehouse according to the PLC motion data;

[0044] During the cruising of the virtual drone along the drone cruising path, when the digital twin model of the logistics warehouse determines that the cruising path of the virtual drone to the next cruising base point conflicts with the motion path of the virtual stacker or virtual shuttle, the digital twin model of the logistics warehouse issues a collision alarm.

[0045] Preferably, it also includes:

[0046] Setting multiple safety stops on the drone’s cruising path, when the digital twin model of the logistics warehouse issues a collision warning;

[0047] Control the virtual drone to fly to the safe stop closest to the current location, and simultaneously control the physical drone to fly to the corresponding safe stop;

[0048] When the digital twin model of the logistics warehouse determines that the virtual stacker or the virtual shuttle has passed the next cruising base point of the virtual drone, it controls the virtual drone to return to the original cruising position and continue to cruise to the next cruising base point according to the drone cruising path, and synchronously controls the physical drone to return to the original cruising position and continue to cruise to the next cruising base point according to the drone cruising path.

[0049] According to a second aspect of an embodiment of the present invention, a rotary-wing drone storage device based on digital twin is provided, the device comprising:

[0050] Model building module: Build a digital twin model of the logistics warehouse, which includes a virtual logistics warehouse, virtual drones, virtual shelves, and virtual goods;

[0051] Virtual-reality synchronization module: used to synchronize the inbound and outbound tasks to the digital twin model of the logistics warehouse, and synchronously update the virtual goods on the virtual shelves according to the inbound and outbound tasks;

[0052] Path generation module: used to determine the area of ​​the waiting warehouse, and generate the UAV cruising path in the area of ​​the waiting warehouse according to the preset route generation rules;

[0053] Cruise module: used for the physical drone and the virtual drone to start cruising along the drone cruising path from the initial position of the drone cruising path;

[0054] Posture calibration module: used for taking photos of physical cargo every time the physical drone flies a cargo grid; and taking photos of virtual cargo every time the virtual drone flies a virtual cargo grid;

[0055] Calculating the physical cargo photo and the virtual cargo photo using an epipolar geometry algorithm to determine whether there is a position offset between the physical drone and the virtual drone;

[0056] If it does not exist, the physical goods photo and the virtual goods photo are similarly calculated, the calculation result is compared with the preset similarity threshold, and the inventory record of the cargo compartment is generated based on the comparison result, and the process continues to the next cargo compartment;

[0057] If so, the physical drone position is adjusted according to the virtual drone position. After adjusting the position, the physical drone takes a photo of the physical cargo again, calculates the similarity between the photo and the virtual cargo photo, generates a stocktaking record for the cargo compartment, and then continues to cruise to the next cargo compartment.

[0058] The inventory record generation module is used to obtain the inventory records of the goods on all cargo compartments in the area to be inventoryed after the physical drone and the virtual drone complete their cruise along the drone cruise path.

[0059] According to a third aspect of an embodiment of the present invention, a storage medium is provided, which stores a computer program. When the computer program is executed by a main controller, it implements each step of the logistics equipment redesign method based on digital twins.

[0060] The technical solutions provided by the embodiments of the present invention may have the following beneficial effects:

[0061] This application constructs a digital twin model of the logistics warehouse, synchronizes the warehouse in and out tasks in the digital twin model, realizes the synchronization of virtual and real goods, and controls the virtual and real drones to move along the planned route. During the drone cruising process, the camera takes photos of the goods on the cargo grid and compares the virtual and real to determine the accuracy of the goods in the cargo grid, avoiding the problem of incorrect stacking of goods caused by human operational errors in real warehouses. At the same time, during the drone cruising process, the coordinate offset of the virtual and real drones is determined by the epipolar geometry algorithm, and the position of the physical drone is calibrated in real time through the position of the virtual drone, which can prevent the physical drone from gradually deviating from the original planned path due to external interference, thereby improving the real-time performance of the drone path. This application does not require human participation, the number of sensors involved is small, the energy consumption is low, and photos of each cargo grid are taken separately, so the photos are clear.

[0062] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0064] FIG1 is a flow chart of a method for storing a rotary-wing UAV disk based on digital twins according to an exemplary embodiment;

[0065] FIG2 is a schematic diagram of a UAV cruise path planning according to another exemplary embodiment;

[0066] FIG3 is a schematic diagram showing the principle of UAV position calibration according to another exemplary embodiment;

[0067] FIG4 is a system schematic diagram of a rotary-wing UAV storage device based on digital twins according to another exemplary embodiment;

[0068] In the attached figure: 1-model building module, 2-virtual-reality synchronization module, 3-path generation module, 4-cruise module, 5-posture calibration module, 6-disk library record generation module. DETAILED DESCRIPTION

[0069] Exemplary embodiments will be described in detail herein, examples of which are illustrated in the accompanying drawings. In the following description, when referring to the drawings, like numbers in different figures represent like or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all possible embodiments consistent with the present invention. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present invention, as detailed in the appended claims.

[0070] Example 1

[0071] FIG1 is a flow chart of a method for storing a rotary-wing drone disk based on a digital twin according to an exemplary embodiment. As shown in FIG1 , the method includes:

[0072] S1, constructing a digital twin model of a logistics warehouse, wherein the digital twin model of the logistics warehouse includes a virtual logistics warehouse, a virtual drone, a virtual shelf, and virtual goods;

[0073] S2, synchronizing the inbound and outbound tasks to the digital twin model of the logistics warehouse, and synchronously updating the virtual goods on the virtual shelves according to the inbound and outbound tasks;

[0074] S3, determining an area of ​​a warehouse to be inspected, and generating a drone cruising path in the area of ​​the warehouse to be inspected according to a preset route generation rule;

[0075] S4, the physical drone and the virtual drone start cruising along the drone cruising path from the initial position of the drone cruising path;

[0076] S5, the physical drone takes a photo of the physical cargo every time it flies a distance of one cargo grid; the virtual drone takes a photo of the virtual cargo every time it flies a distance of one virtual cargo grid;

[0077] Calculating the physical cargo photo and the virtual cargo photo using an epipolar geometry algorithm to determine whether there is a position offset between the physical drone and the virtual drone;

[0078] If it does not exist, the physical cargo photo and the virtual cargo photo are similarly calculated, the calculation result is compared with the preset similarity threshold, and the inventory record of the cargo compartment is generated based on the comparison result, and the process continues to the next cargo compartment;

[0079] If so, the physical drone position is adjusted according to the virtual drone position. After adjusting the position, the physical drone takes a photo of the physical cargo again, calculates the similarity between the photo and the virtual cargo photo, generates a stocktaking record for the cargo compartment, and then continues to cruise to the next cargo compartment.

[0080] S6, after the physical drone and the virtual drone complete their patrols along the drone patrol route, obtain inventory records of the goods on all cargo compartments in the area to be patrolled;

[0081] It is understandable that the geometric models of logistics warehouses, drones, shelves, goods, etc. are built in a 1:1 ratio through 3D modeling software, and the appearance of materials, colors, skins, textures, etc. are given. The drone geometric model includes an industrial camera virtual model. The virtual camera focal length, aperture and other parameters are consistent with the drone physical camera parameters. A virtual model of virtual lighting equipment is established, and the parameters such as luminous flux, light intensity, and illumination are adjusted to be consistent with the actual factory lighting parameters. Different identification marks are arranged on the shelf columns according to different cargo compartments, and the same marks are arranged on the virtual shelves at the corresponding positions of the physical shelves for cargo compartment identification and feature recognition. It is used for later photo shooting to identify different cargo boxes, facilitate inventory of goods on each cargo box, and then synchronize the inbound and outbound tasks. The goods pass through the inbound platform and are photographed by an industrial camera to update the cargo virtual model map to ensure that the virtual model of the incoming goods has the same appearance as the physical goods. The area to be inventoryed is determined, and the digital twin system forms a cruising path according to the distribution of cargo boxes to determine whether the inventory drone is in the initial position. If not, it automatically cruises to the initial position and starts cruising according to the drone's cruising path. The physical drone automatically cruises to the first cargo box according to the cruising path (the subsequent cycle corresponds to the next position ) and takes photos of the cargo. The virtual drone cruises to the first virtual cargo compartment along the cruise path and takes photos of the virtual cargo. The photos are uniformly sent to the image processing module via the local area network. The epipolar geometry algorithm is used to calculate the offset of the physical drone relative to the virtual drone using the virtual and real object photos. (Since the physical drone may be subject to various interferences when moving in real space and may deviate from the preset cruise trajectory, the virtual drone will strictly follow the preset cruise path because there is no interference. Therefore, after each cruise, the position of the physical drone needs to be calibrated using the position of the virtual drone.) It is also determined whether there is relative motion (rotation and translation). If not, a similarity calculation is performed on the physical cargo photo and the virtual cargo photo, and the calculation result is compared with a preset similarity threshold. If it is less than the preset similarity threshold, a "no" is recorded; otherwise, a "yes" is recorded and a stock inventory record is generated. The flight continues to the next cargo compartment. If relative motion exists, the position of the physical drone is updated according to the position of the virtual drone. After adjusting its position, the physical drone takes another photo of the physical cargo and performs a similarity calculation with the virtual cargo photo to generate a stock inventory record for the cargo compartment. The flight continues to the next cargo compartment until the entire cruise path is completed.This application builds a digital twin model of a logistics warehouse, synchronizing warehouse inbound and outbound tasks within the digital twin model to achieve virtual and real cargo synchronization. Simultaneously, it controls a virtual and real drone to follow a planned route. During the drone's patrol, it uses a camera to take photos of the cargo on the cargo compartments and compares the virtual and real images to determine the accuracy of the cargo in the compartments, avoiding the problem of incorrect cargo placement caused by human error in real warehouses. Furthermore, during the drone's patrol, it uses an epipolar geometry algorithm to determine the coordinate offset of the virtual and real drones. The position of the physical drone is then calibrated in real time based on the virtual drone's position. This prevents the physical drone from gradually deviating from the planned path due to external interference, improving the real-time performance of the drone's route. This application requires no human intervention, uses a small number of sensors, and consumes low energy. Furthermore, it takes individual photos of each cargo compartment, resulting in high-definition photos.

[0082] Preferably,

[0083] The calculating the physical cargo photo and the virtual cargo photo using an epipolar geometry algorithm to determine whether there is a position offset between the physical drone and the virtual drone includes:

[0084] Denoising the virtual goods photo and the physical goods photo respectively by a mean filtering operation, and then performing a grayscale operation on the denoised photos;

[0085] constructing multi-scale spaces for the virtual goods photo and the physical goods photo, respectively, calculating extreme values ​​of pixels of the virtual goods photo and the physical goods photo at each scale, and selecting the maximum extreme value of the virtual goods photo and the physical goods photo in each scale space as a feature point;

[0086] A continuous curve is used to fit the multi-scale spatial extreme value to determine the exact position and scale of each feature point. The gradient information of the feature point and the neighboring pixels is used as the direction parameter of the feature point to obtain the virtual feature point set and the real feature point set respectively.

[0087] Each feature point is described using a 128-dimensional feature vector. A virtual feature point from the virtual feature point set is selected. The entire real feature point set is traversed to obtain the two real feature points in the real feature point set that are closest to the selected virtual feature point in Euclidean distance. The ratio of the Euclidean distance between the closest real feature point and the second closest real feature point and the selected virtual feature point is calculated. If the ratio is less than the preset Euclidean distance threshold, the match is successful, and the selected virtual feature point and the real feature point with the closest Euclidean distance are taken as a pair of matching feature points.

[0088] Then select another virtual feature point from the virtual feature point set and repeat the above process until all virtual feature points in the virtual feature point set are matched with the real feature point set to obtain multiple pairs of matching feature points;

[0089] According to the mapping relationship of the camera, multiple pairs of matching feature points in the obtained image coordinate system are converted into the image pixel coordinate system;

[0090] Obtaining a basic matrix based on the association relationship between multiple pairs of matching feature points of the virtual goods photo and the physical goods photo in the image pixel coordinate system;

[0091] According to the relationship between the basic matrix and the essential matrix, the essential matrix is ​​obtained, and the rotation matrix R and translation matrix t of the physical camera relative to the virtual camera are obtained according to the essential matrix;

[0092] Calculate whether there is a position offset between the physical drone and the virtual drone using the rotation matrix R and the translation matrix t;

[0093] It is understandable that establishing a world coordinate system ( w X w Y w Z w ); Camera coordinate system (O c X c Y c Z c ), the origin of the camera coordinate system is located at the camera optical center O c , O c Z c Parallel to the optical axis of the industrial camera and perpendicular to the image plane, the intersection is the origin of the physical imaging plane coordinate, X c and Y c , parallel to the x-axis and y-axis of the image coordinate system, O c O p is the equivalent camera focal length; image coordinate system (O p xy), where O p is the intersection of the camera optical axis and the image; the image pixel coordinate system (O u uv), denoise the virtual and real images by mean filtering and convert them to grayscale; write the SIFT algorithm in Python and embed it into the Unity digital twin system to compare the images taken by the physical camera and the images taken by the virtual camera: construct the scale space L(x,y,σ)=G(x,y,σ)*I(x,y) of the virtual and real images I(x,y) respectively (where σ is the scale space factor, (x, y) is the pixel coordinate in the image), calculate the extreme value DoG at each scale (D(x, y, σ) = L(x, y, kσ) - L(x, y, σ), K is the factor of difference in the same order of scale space), and compare the extreme values ​​of each pixel in each scale image to obtain the extreme maximum value in the scale space and the two-dimensional space of the image (each scale space has an extreme maximum value, and because it is a multi-scale space, there are multiple extreme maximum values). A continuous curve is used to fit the scale space DoG value to determine the exact position and scale of the feature point, and the gradient information of the feature point and the neighboring pixels is used as the direction parameter of the feature point.

[0094] The gradient is:

[0095] Direction: θ(x,y)=tan -1 ((L(x,y+1)-L(x,y - 1)) / (L(x+1,y)-L(x-1,y)))),

[0096] Get the information of the feature point (x, y, σ, θ), where θ represents the direction of the feature point, and finally form a virtual feature point set and a real feature point set; take the 16×16 pixels around the feature point as the sampling area, divide it into 4×4 sampling units on average, and calculate the gradient and direction information of the sampling unit in 8 directions (every 45 degrees is a direction), then each feature point can be described by a 4×4×8=128-dimensional feature vector; take the feature point in the virtual image, and find the two feature points with the closest Euclidean distance to it in the real image by traversing (Euclidean distance (where l and h are the feature vector descriptors of each feature point)), calculate the ratio of the nearest value to the second nearest value, if the ratio is less than the threshold value of 0.8, the match is successful, and the coordinates of the matching feature points in the virtual and real images in the image coordinate system (x m ,y m )、(x m' ,y m' );

[0097] The pixel coordinates of the matching feature points are solved by the following formula to solve the coordinates of the matching feature points in the image pixel coordinate system (u m ,v m )、(u m' ,v m' ):

[0098] Where dx and dy are the physical lengths of each pixel in the x-axis and y-axis directions, and u0 and v0 are the pixel coordinates of the origin of the image coordinate system;

[0099] The calibration principle is: in the actual implementation process, since the virtual camera cruises along the inventory path, the world coordinates of the virtual camera can be determined according to the endurance path, (g0+g1+...+0.5g y ,c0+c1+...+0.5c x ,s), where g represents the layer height, c represents the length of each row, and s represents the distance between the camera and the shelf. Each time the physical camera posture is calibrated, the world coordinate system is established on the virtual camera coordinate system. Let R and t matrices be the angle conversion matrix and displacement transformation matrix between the physical camera coordinate system and the virtual camera coordinate system. As shown in Figure 3, the virtual model completely restores the real world in 1:1, so the two are exactly the same. When calibrating the physical camera (inventory drone), the real entity is overlapped with the virtual entity. O and O' are the optical centers of the virtual camera and the physical camera respectively. m and m' are on the epipolar plane, respectively OM The projection points of the ray and the O'M ray and the imaging plane; the OM ray is on the π plane, and the projection line l' in the second imaging plane is the polar line. All three-dimensional space points M projected to point m must exist on the OM ray, that is, the imaging point m of point M on the first view corresponds to the matching imaging point m' on the second view and must also be on l'. Similarly, the imaging point of m' in the first view must be on l; the OO' line is the baseline, and the intersection points of the baseline with the virtual and real images are the poles e and e' respectively. The plane OO'M is the polar plane π. O' and O have a rotation matrix R and a displacement transformation t in the world coordinate system;

[0100] After obtaining the coordinates of the best feature point pair in the image pixel coordinate system, according to the formula

[0101] Among them, m i and m′ i is the homogeneous coordinate of the best feature point pair in the image pixel coordinate system, T is the matrix transpose calculation, and F is the basic matrix. Since F (basic matrix) is a 3×3 homogeneous matrix with a rank of 2 and 7 degrees of freedom, 7 groups of matching feature points can be randomly selected and F can be solved using the 7-point method. The correctness of the matching feature points can be verified through the basic matrix, and the incorrect matching points of the original matching points can be eliminated;

[0102] The essential matrix E and the basic matrix F have the relationship F = K -T EK -1 , where E = [t] × R, using SVD decomposition, E has one singular value of zero, and the other two singular values ​​are equal, that is, E = Udiag(k,k,0)V T , (k is a singular value);

[0103] in:

[0104] Then, the rotation matrix R and translation matrix t of the physical camera relative to the virtual camera can be obtained, and then the physical camera pose can be corrected based on the virtual camera position.

[0105] Preferably,

[0106] Calculating the similarity between the physical goods photo and the virtual goods photo includes:

[0107] grayscale the physical goods photo and the virtual goods photo;

[0108] Using a Gaussian weighting function with a preset standard deviation as a weighting window, calculating the pixel points in the grayscaled physical goods photo and the virtual goods photo to obtain an SSIM index mapping matrix composed of local SSIM indices, calculating the average SSIM index of the SSIM index mapping matrix, and using the average SSIM index of the SSIM index mapping matrix as the similarity;

[0109] It can be understood that the physical image and the virtual image taken after the physical drone position is adjusted are grayed, and a Gaussian weighting function with a standard deviation of 1.5 is used as a weighting window. Each step is calculated based on the pixels in the window to obtain an SSIM index mapping matrix composed of local SSIM indices, and the average SSIM index is calculated. The SSIM index calculation formula is:

[0110] Where a and b represent virtual goods photos and physical goods photos respectively, μ a is the average value of each pixel in image a, μ b is the average value of each pixel in the b image, is the variance of the pixels in the a image, is the variance of the pixels in the b image, σ ab is the covariance of a and b, c1=(k1L) 2 and c2=(k2L) 2 It is a constant used to maintain stability, L is the value range of pixel values, k1=0.01, k2=0.03, and the calculation result is used as the similarity between the virtual and real photos. The similarity is compared with the preset similarity threshold. The threshold is set to 0.9 and can be adjusted according to the actual use effect. When the similarity is greater than 0.9, "yes" is recorded, and when the similarity is less than or equal to 0.9, "no" is recorded.

[0111] Preferably,

[0112] In the area of ​​the waiting warehouse, generating a drone cruising path according to a preset route generation rule includes:

[0113] The intersection of the lines connecting the four corner points of each cargo compartment is used as the base point, and the base points of each cargo compartment are connected from low to high and from the inbound end to the outbound end, and the line connecting the base points is used as the cruising path of the drone;

[0114] It can be understood that, as shown in Figure 2, the shelf is a rectangular three-dimensional shelf, and there are multiple cargo compartments on each shelf, such as cargo compartment 1, cargo compartment 2, etc. As shown in cargo compartment 1 in Figure 2, the intersection of the four corner points is used as the base point, that is, the midpoint of the cargo compartment, and the base point of each cargo compartment is connected from low to high and from the entry end to the exit end. The drone stops at the base point position of each cargo compartment and takes photos of the goods in this cargo compartment, and identifies the cargo compartment through the logo on each cargo compartment.

[0115] Preferably,

[0116] The digital twin model of the logistics warehouse also includes:

[0117] Virtual stacker crane and virtual shuttle;

[0118] The digital twin model of the logistics warehouse obtains PLC motion data of the physical stacker and the physical shuttle, and drives the virtual stacker and the virtual shuttle to move in the virtual logistics warehouse according to the PLC motion data;

[0119] During the cruising of the virtual drone along the drone cruising path, when the digital twin model of the logistics warehouse determines that the cruising path of the virtual drone to the next cruising base point conflicts with the motion path of the virtual stacker or the virtual shuttle, the digital twin model of the logistics warehouse issues a collision alarm;

[0120] It is understandable that the digital twin system connects to the PLCs of motion equipment such as stackers and shuttles through S7.net, reads motion data, and drives the movement of virtual models such as stackers and shuttles. When the route of the next cruising base point of the drone during inventory processing conflicts with the motion path of equipment such as stackers, the digital twin system issues a collision risk alarm.

[0121] Preferably, it also includes:

[0122] Setting multiple safety stops on the drone’s cruising path, when the digital twin model of the logistics warehouse issues a collision warning;

[0123] Control the virtual drone to fly to the safe stop closest to the current location, and simultaneously control the physical drone to fly to the corresponding safe stop;

[0124] When the digital twin model of the logistics warehouse determines that the virtual stacker or the virtual shuttle has passed the next cruising base point of the virtual drone, the virtual drone is controlled to return to the original cruising position and continue to cruise to the next cruising base point according to the drone cruising path, and the physical drone is simultaneously controlled to return to the original cruising position and continue to cruise to the next cruising base point according to the drone cruising path;

[0125] It can be understood that, as shown in Figure 2, a safety stop is set under each shelf. When the digital twin system of the logistics warehouse issues a collision alarm, it controls the virtual drone to fly to the safety stop nearest to the current position, and synchronously controls the physical drone to fly to the corresponding safety stop. In simple terms, when the drone is at the base point of cargo grid 1 or halfway from cargo grid 1 to the base point of cargo grid 2, the digital twin system issues a collision alarm. At this time, the digital twin system determines whether the drone is closer to safety stop 1 or safety stop 2, and chooses a closer safety stop to avoid until the virtual stacker or the virtual shuttle has passed the base point of cargo grid 2. At this time, the drone returns to the original cruising path and continues to go to the base point of cargo grid 2.

[0126] Example 2

[0127] This embodiment also discloses a system diagram of a multi-motor coordinated control device based on digital twins, as shown in FIG4 , including:

[0128] Model building module 1: Constructing a digital twin model of the logistics warehouse, which includes a virtual logistics warehouse, virtual drones, virtual shelves, and virtual goods;

[0129] Virtual-reality synchronization module 2: used to synchronize the inbound and outbound tasks to the digital twin model of the logistics warehouse, and synchronously update the virtual goods on the virtual shelves according to the inbound and outbound tasks;

[0130] Path generation module 3: used to determine the area of ​​the waiting warehouse, and generate the drone cruising path in the area of ​​the waiting warehouse according to the preset route generation rules;

[0131] Cruise module 4: used for the physical drone and the virtual drone to start cruising along the drone cruising path from the initial position of the drone cruising path;

[0132] Posture calibration module 5: used for taking photos of physical cargo every time the physical drone flies a distance of one cargo grid; and taking photos of virtual cargo every time the virtual drone flies a distance of one virtual cargo grid;

[0133] Calculating the physical cargo photo and the virtual cargo photo using an epipolar geometry algorithm to determine whether there is a position offset between the physical drone and the virtual drone;

[0134] If it does not exist, the physical cargo photo and the virtual cargo photo are similarly calculated, the calculation result is compared with the preset similarity threshold, and the inventory record of the cargo compartment is generated based on the comparison result, and the process continues to the next cargo compartment;

[0135] If so, the physical drone position is adjusted according to the virtual drone position. After adjusting the position, the physical drone takes a photo of the physical cargo again, calculates the similarity between the photo and the virtual cargo photo, generates a stocktaking record for the cargo compartment, and then continues to cruise to the next cargo compartment.

[0136] Inventory record generation module 6: used for obtaining inventory records of goods on all cargo compartments in the area to be inventoried after the physical drone and the virtual drone complete their cruise along the drone's cruise path;

[0137] It can be understood that the present application constructs a digital twin model of the logistics warehouse through the model building module 1, and the digital twin model of the logistics warehouse includes a virtual logistics warehouse, a virtual drone, a virtual shelf and virtual goods; the virtual-real synchronization module 2 is used to synchronize the inbound and outbound tasks to the digital twin model of the logistics warehouse, and the virtual goods on the virtual shelf are synchronously updated according to the inbound and outbound tasks; the path generation module 3 is used to determine the area of ​​the warehouse to be inspected, and in the area of ​​the warehouse to be inspected, the drone cruise path is generated according to the preset route generation rules; the cruise module 4 is used for the physical drone and the virtual drone to move from the drone patrol The initial position of the flight path starts cruising according to the drone's cruising path; the posture calibration module 5 is used to take a photo of the physical cargo every time the physical drone flies a distance of one cargo grid; the virtual drone takes a photo of the virtual cargo every time it flies a distance of one virtual cargo grid; the epipolar geometry algorithm is used to calculate the physical cargo photos and the virtual cargo photos to determine whether there is a position offset between the physical drone and the virtual drone; if not, the physical cargo photos and the virtual cargo photos are calculated for similarity, the calculation result is compared with the preset similarity threshold, and the inventory record of the cargo grid is generated according to the comparison result, and the flight continues to the next cargo grid; if it exists, the position of the physical drone is adjusted according to the position of the virtual drone. After the physical drone adjusts its position, it takes a photo of the physical goods again, calculates the similarity with the virtual goods photo, generates a stocktaking record for the cargo grid, and then continues to cruise to the next cargo grid; the stocktaking record generation module 6 is used for the physical drone and the virtual drone to obtain the stocktaking record of the goods on all cargo grids in the area to be inventoryed after the cruise is completed according to the drone cruise path; this application realizes the synchronization of virtual and real goods by constructing a digital twin model of the logistics warehouse, synchronizing the warehouse in and out tasks in the digital twin model, and controlling the virtual and real drones to move goods according to the plan As the drone moves forward along the route, it uses a camera to take photos of the goods on the cargo grid during its cruising process, and compares the real and virtual images to determine the accuracy of the goods in the grid, thus avoiding the problem of incorrect stacking of goods caused by human error in real warehouses. At the same time, during the drone cruising process, the coordinate offset of the virtual and real drones is determined by the epipolar geometry algorithm, and the position of the physical drone is calibrated in real time through the position of the virtual drone. This can prevent the physical drone from gradually deviating from the original planned path due to external interference, thereby improving the real-time performance of the drone path. This application does not require human participation, requires a small number of sensors, has low energy consumption, and takes photos of each cargo grid separately, so the photos are of high clarity.

[0138] Example 3:

[0139] This embodiment provides a storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by a host controller, each step in the above method is implemented;

[0140] It is understandable that the storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc.

[0141] It can be understood that the same or similar parts of the above embodiments can be referenced to each other, and the contents not described in detail in some embodiments can refer to the same or similar contents in other embodiments.

[0142] It should be noted that, in the description of the present invention, the terms "first", "second", etc. are used for descriptive purposes only and should not be understood as indicating or implying relative importance. In addition, in the description of the present invention, unless otherwise specified, the meaning of "plurality" is at least two.

[0143] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code comprising one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may be performed out of the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present invention pertain.

[0144] It should be understood that various parts of the present invention can be implemented using hardware, software, firmware, or a combination thereof. In the above-described embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0145] Those skilled in the art will understand that all or part of the steps in the method of the above embodiment can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiment.

[0146] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing module, or each unit may exist physically separately, or two or more units may be integrated into a single module. The aforementioned integrated modules may be implemented in the form of hardware or in the form of software functional modules. If the integrated modules are implemented in the form of software functional modules and sold or used as independent products, they may also be stored in a computer-readable storage medium.

[0147] The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc.

[0148] Throughout this specification, reference to terms such as "one embodiment," "some embodiments," "examples," "specific examples," or "some examples" means that a specific feature, structure, material, or characteristic described in conjunction with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0149] Although the embodiments of the present invention have been shown and described above, it will be understood that the above embodiments are illustrative and are not to be construed as limitations on the present invention. A person skilled in the art may change, modify, replace and modify the above embodiments within the scope of the present invention.

Claims

1. A rotary-wing UAV disk library method based on digital twins, characterized in that: include: Constructing a digital twin model of a logistics warehouse, wherein the digital twin model of the logistics warehouse includes a virtual logistics warehouse, a virtual drone, a virtual shelf, and virtual goods; Synchronize the inbound and outbound tasks to the digital twin model of the logistics warehouse, and synchronously update the virtual goods on the virtual shelves according to the inbound and outbound tasks; Determine an area of ​​a waiting warehouse, and generate a drone cruising path in the area of ​​the waiting warehouse according to a preset route generation rule; The physical drone and the virtual drone start cruising along the drone cruising path from an initial position of the drone cruising path; The physical drone takes photos of physical goods every time it flies a distance of one cargo grid; the virtual drone takes photos of virtual goods every time it flies a distance of one virtual cargo grid; The epipolar geometry algorithm is used to calculate the physical cargo photo and the virtual cargo photo to determine whether there is a position offset between the physical drone and the virtual drone; If it does not exist, the physical cargo photo and the virtual cargo photo are similarly calculated, the calculation result is compared with the preset similarity threshold, and the inventory record of the cargo compartment is generated according to the comparison result, and the navigation continues to the next cargo compartment; If there is, the position of the physical drone is adjusted according to the position of the virtual drone. After the physical drone adjusts its position, it takes photos of the physical goods again, calculates the similarity with the virtual goods photos, generates a stocktaking record of the cargo compartment, and then continues to cruise to the next cargo compartment; After the physical drone and the virtual drone complete their cruise along the drone cruise path, the inventory records of the goods on all cargo cells in the area to be inventoryed are obtained.

2. The method according to claim 1, characterized in that The using of the epipolar geometry algorithm to calculate the physical cargo photo and the virtual cargo photo to determine whether there is a position offset between the physical drone and the virtual drone includes: The virtual goods photo and the physical goods photo are denoised by a mean filtering operation. Processing: grayscale the denoised photos; Constructing multi-scale spaces of the virtual goods photo and the physical goods photo respectively, calculating extreme values ​​of pixel points of the virtual goods photo and the physical goods photo at each scale, and selecting the maximum extreme value of the virtual goods photo and the physical goods photo in each scale space as a feature point; A continuous curve is used to fit the multi-scale spatial extreme value to determine the exact position and scale of each feature point. The gradient information of the feature point and the neighboring pixels is used as the direction parameter of the feature point to obtain the virtual feature point set and the real feature point set respectively. Each feature point is described using a 128-dimensional feature vector, a virtual feature point in the virtual feature point set is selected, the entire real feature point set is traversed, the two real feature points in the real feature point set whose Euclidean distance to the selected virtual feature point is closest are obtained, and the ratio of the Euclidean distance between the nearest real feature point and the second nearest real feature point and the selected virtual feature point is calculated. If the ratio is less than the preset Euclidean distance threshold, the match is successful, and the selected virtual feature point and the real feature point with the closest Euclidean distance are taken as a pair of matching feature points; Then select another virtual feature point from the virtual feature point set, and repeat the above process until all virtual feature points in the virtual feature point set are matched with the real feature point set, and multiple pairs of matching feature points are obtained; According to the mapping relationship of the camera, multiple pairs of matching feature points in the obtained image coordinate system are converted into the image pixel coordinate system; Obtaining a basic matrix according to the association relationship between multiple pairs of matching feature points of the virtual goods photo and the physical goods photo in the image pixel coordinate system; According to the correlation between the basic matrix and the essential matrix, the essential matrix is ​​obtained, and the rotation matrix R and translation matrix t of the physical camera relative to the virtual camera are obtained according to the essential matrix; The rotation matrix R and the translation matrix t are used to calculate whether there is a position offset between the physical drone and the virtual drone.

3. The method according to claim 1, characterized in that The similarity calculation of the physical goods photo and the virtual goods photo includes: Grayscale the physical goods photo and the virtual goods photo; A Gaussian weighting function with a preset standard deviation is used as a weighting window to The pixel points in the physical goods photo and the virtual goods photo are calculated to obtain an SSIM index mapping matrix composed of local SSIM indices, the average SSIM index of the SSIM index mapping matrix is ​​calculated, and the average SSIM index of the SSIM index mapping matrix is ​​used as the similarity.

4. The method according to claim 1, characterized in that: In the area of ​​the waiting warehouse, generating a drone cruising path according to a preset route generation rule includes: The intersection of the lines connecting the four corner points of each cargo grid is used as the base point, and the base points of each cargo grid are connected from low to high and from the entry end to the exit end, and the line connecting the base points is used as the cruising path of the drone.

5. The method according to claim 4, characterized in that The digital twin model of the logistics warehouse also includes: Virtual stacker and virtual shuttle; The digital twin model of the logistics warehouse obtains PLC motion data of the physical stacker and the physical shuttle, and drives the virtual stacker and the virtual shuttle to move in the virtual logistics warehouse according to the PLC motion data; During the cruising of the virtual drone along the drone cruising path, when the digital twin model of the logistics warehouse determines that the cruising path of the virtual drone to the next cruising base point conflicts with the motion path of the virtual stacker or the virtual shuttle, the digital twin model of the logistics warehouse issues a collision alarm.

6. The method according to claim 5, characterized in that Also includes: Setting a plurality of safety stops on the cruise path of the drone, when the digital twin model of the logistics warehouse issues a collision warning; Control the virtual drone to fly to the safe stop point closest to the current location, and simultaneously control the physical drone to fly to the corresponding safe stop point; When the digital twin model of the logistics warehouse determines that the virtual stacker or the virtual shuttle has passed the next cruising base point of the virtual drone, the virtual drone is controlled to return to the original cruising position. The physical drone is placed in the original cruising position and continues to cruise to the next cruising base point according to the drone cruising path, and the physical drone is synchronously controlled to return to the original cruising position and continues to cruise to the next cruising base point according to the drone cruising path.

7. A rotary-wing drone storage device based on digital twins, characterized in that: The device comprises: Model building module: building a digital twin model of a logistics warehouse, wherein the digital twin model of the logistics warehouse includes a virtual logistics warehouse, a virtual drone, a virtual shelf, and virtual goods; Virtual-reality synchronization module: used to synchronize the inbound and outbound tasks to the digital twin model of the logistics warehouse, and synchronously update the virtual goods on the virtual shelves according to the inbound and outbound tasks; Path generation module: used to determine the area of ​​the warehouse to be inspected, and generate the UAV cruising path in the area of ​​the warehouse to be inspected according to the preset route generation rules; Cruise module: used for the physical drone and the virtual drone to start cruising along the drone cruising path from the initial position of the drone cruising path; Posture calibration module: used for taking photos of physical goods every time the physical drone flies a distance of one cargo grid; and taking photos of virtual goods every time the virtual drone flies a distance of one virtual cargo grid; The epipolar geometry algorithm is used to calculate the physical cargo photo and the virtual cargo photo to determine whether there is a position offset between the physical drone and the virtual drone; If it does not exist, the physical cargo photo and the virtual cargo photo are similarly calculated, the calculation result is compared with the preset similarity threshold, and the inventory record of the cargo compartment is generated according to the comparison result, and the navigation continues to the next cargo compartment; If there is, the position of the physical drone is adjusted according to the position of the virtual drone. After the physical drone adjusts its position, it takes photos of the physical goods again, calculates the similarity with the virtual goods photos, generates a stocktaking record of the cargo compartment, and then continues to cruise to the next cargo compartment; Inventory record generation module: used for obtaining inventory records of goods on all cargo compartments in the area to be inventoried after the physical drone and the virtual drone complete their cruise along the drone cruise path.

8. A storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by the main controller, each step of the rotary-wing drone inventory method based on digital twins as described in any one of claims 1 to 6 is implemented.

Images