Warehouse inspection and management method, system and equipment based on unmanned aerial vehicle

Abstract

The invention provides a warehouse inspection and management method, system and device based on an unmanned aerial vehicle, and belongs to the technical field of unmanned aerial vehicles. According to the method, an initial warehouse inspection path can be generated based on shelf layout information input by a user terminal; the shelf layout information at least comprises shelf height, shelf spacing, goods category and barrier distribution; the initial warehouse inspection path is sent to an unmanned aerial vehicle cluster, so that corresponding unmanned aerial vehicles collect inspection shelf data, and a warehouse three-dimensional model is constructed; wherein the warehouse three-dimensional model at least comprises associated semantic information between the shelf and the goods. Based on a preset anomaly detection model, determining an anomaly detection result of the inspection shelf in real time, and dynamically adjusting a warehouse inspection path of the corresponding unmanned aerial vehicle according to the anomaly detection result and a preset inspection path optimization condition; and synchronously sending the warehouse three-dimensional model and the anomaly detection result to a cloud digital twin platform so as to display the warehouse state to a user terminal.

Description

Technical Field

[0001] The present application relates to the technical field of unmanned aerial vehicles, and in particular, to a method, a system and a device for cargo warehouse inspection and management based on unmanned aerial vehicles. Background Art

[0002] With the rapid development of the logistics industry, the number of large-scale cargo warehouses is increasing day by day, which puts forward higher requirements for the internal cargo display, dynamic layout and abnormal management of the cargo warehouse.

[0003] At present, the management of cargo warehouses mainly relies on manual labor or fixed cameras, which has low efficiency, incomplete coverage and strong subjectivity in judging cargo abnormalities, relying too much on manual experience; secondly, the perspective of fixed cameras is easily limited, there are monitoring blind spots, and there is a lack of perspective for detecting abnormalities inside the cargo warehouse, making it difficult to achieve dynamic cargo warehouse management. At the same time, there are also deficiencies in the multi-machine collaboration and data processing of cameras. Summary of the Invention

[0004] To solve the above problems, the embodiments of the present application provide a method, a system and a device for cargo warehouse inspection and management based on unmanned aerial vehicles, which are used to complete cargo warehouse inspection and management through multi-machine collaboration, and realize the intelligence and high efficiency of cargo warehouse management.

[0005] On the one hand, the embodiments of the present application provide a method for cargo warehouse inspection and management based on unmanned aerial vehicles, and the method includes: Generating an initial cargo warehouse inspection path based on the shelf layout information input by the user terminal; the shelf layout information at least includes shelf height, shelf spacing, goods categories and obstacle distribution; Sending the initial cargo warehouse inspection path to the unmanned aerial vehicle cluster, so that the corresponding unmanned aerial vehicles collect inspection shelf data and construct a three-dimensional model of the cargo warehouse; wherein, the three-dimensional model of the cargo warehouse at least includes the associated semantic information between the shelves and the goods; Based on a preset anomaly detection model, determining the anomaly detection result of the inspected shelf in real time, and dynamically adjusting the cargo warehouse inspection path of the corresponding unmanned aerial vehicle according to the anomaly detection result and the preset inspection path optimization condition; Synchronously sending the three-dimensional model of the cargo warehouse and the anomaly detection result to the cloud digital twin platform, so as to display the state of the cargo warehouse to the user terminal.

[0006] In an implementation manner of the present application, generating an initial cargo warehouse inspection path based on the shelf layout information input by the user terminal specifically includes: Determining the shelf priority corresponding to each shelf according to the goods categories in the shelf layout information; Determining the corresponding obstacle projection area according to the obstacle distribution; Input the shelf priority, the projected area of the obstacle, and the preset warehouse information into the path cost function of the A* algorithm for path optimization, so as to determine the initial warehouse inspection path according to the function value of the path cost function; the preset warehouse information at least includes the diagonal length of the warehouse, the priority threshold, and the area of the preset inspection grid area; the area of the preset inspection grid area is the area of the grid area obtained by pre-dividing the total warehouse area according to the shelf height, the shelf spacing, and the product category.

[0007] In an implementation manner of the present application, the drone is equipped with a multi-modal sensor module for synchronously collecting inspection shelf data; the multi-modal sensor module at least includes a lidar, a 3D camera, and an RFID reader; the RFID reader is used to read the RFID tags carrying storage attributes on each shelf; the storage attributes at least include the product category, storage conditions, and inspection record timestamp.

[0008] In an implementation manner of the present application, send the initial warehouse inspection path to the drone cluster, so that the corresponding drones collect inspection shelf data and build a 3D model of the warehouse, specifically including: Along the initial warehouse inspection path, perform clustering processing on the corresponding grid areas to obtain a plurality of clustering task assignment areas; one clustering task assignment area corresponds to a section of the warehouse inspection path; each warehouse inspection path is spliced to obtain the initial warehouse inspection path; According to the task execution power of each drone in the drone cluster and the estimated inspection energy consumption of each warehouse inspection path, allocate each drone to the corresponding warehouse inspection path, so as to make the drone collect the inspection shelf data along the assigned warehouse inspection path according to the allocation result; According to the inspection shelf data, construct the association relationship between the shelf and the product, and generate a local 3D map, so as to establish the 3D model of the warehouse according to each association relationship and each local 3D map.

[0009] In an implementation manner of the present application, according to the task execution power of each drone in the drone cluster and the estimated inspection energy consumption of each warehouse inspection path, allocate each drone to the corresponding warehouse inspection path, specifically including: When the task execution power of any drone is less than the estimated inspection energy consumption of the warehouse inspection path, match the joint inspection drone group in the drone cluster according to the estimated inspection energy consumption; the sum of the task execution powers of each drone in the joint inspection drone group is not less than the estimated inspection energy consumption.

[0010] In one implementation of the present application, based on a preset anomaly detection model, the anomaly detection result of the inspected shelf is determined in real time, specifically including: Input the real-time inspected shelf data from the drone into the preset anomaly detection model to determine whether there is an inspection anomaly according to the model output result; the anomaly types of the inspection anomaly at least include shelf structure anomaly and / or inventory anomaly; the shelf structure anomaly is obtained based on the shelf normal vector offset angle; If so, add the anomaly type corresponding to the inspection anomaly to the anomaly detection result.

[0011] In one implementation of the present application, according to the anomaly detection result and the preset inspection path optimization conditions, the cargo hold inspection path of the corresponding drone is dynamically adjusted, specifically including: Match the anomaly detection result with the preset inspection path optimization conditions to determine the dynamic adjustment rule according to the optimization condition matching result; wherein, the dynamic adjustment rule at least includes one or more of the following: generating a detour path, generating a re-inspection path; Update the cargo hold inspection path according to the dynamic adjustment rule.

[0012] In one implementation of the present application, after synchronously sending the three-dimensional model of the cargo hold and the anomaly detection result to the cloud digital twin platform for displaying the state of the cargo hold to the user terminal, the method further includes: Receive a viewing instruction from the user terminal and determine the three-dimensional coordinates of the cargo hold corresponding to the viewing instruction; Determine the timestamp of the inspected cargo hold inspection path where the three-dimensional coordinates of the cargo hold are located; When the time interval between the inspected timestamp and the current time is less than a predetermined value, magnify and display the cargo hold inspection path where the three-dimensional coordinates of the cargo hold are located through the cloud digital twin platform; When the time interval between the inspected timestamp and the current time is greater than or equal to the predetermined value, generate an inspection execution instruction to call the corresponding drone to inspect the cargo hold inspection path where the three-dimensional coordinates of the cargo hold are located through the inspection execution instruction, and magnify and display the inspected cargo hold inspection path.

[0013] On the other hand, the embodiment of the present application further provides a cargo hold inspection and management system based on a drone, and the system includes: A generation module, configured to generate an initial cargo hold inspection path based on the shelf layout information input by the user terminal; the shelf layout information at least includes shelf height, shelf spacing, goods category, and obstacle distribution; A sending module, configured to send the initial warehouse inspection path to a drone cluster, so that corresponding drones collect inspection shelf data and construct a three-dimensional model of the warehouse; wherein, the three-dimensional model of the warehouse at least includes the associated semantic information between the shelves and the goods. A determining module, configured to determine the anomaly detection result of the inspected shelves in real time based on a preset anomaly detection model, and dynamically adjust the warehouse inspection path of the corresponding drones according to the anomaly detection result and preset inspection path optimization conditions. A sending and displaying module, configured to synchronously send the three-dimensional model of the warehouse and the anomaly detection result to a cloud digital twin platform, so as to display the warehouse status to the user terminal.

[0014] On the other hand, an embodiment of the present application further provides a drone-based warehouse inspection and management device, and the device includes: At least one processor; and a memory communicatively connected to the at least one processor; wherein, the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor, so that the at least one processor can execute a drone-based warehouse inspection and management method as described above.

[0015] Compared with the prior art, the present application has the following remarkable effects: Through the above technical solution, using drones to complete warehouse inspection and management can liberate manpower, improve the efficiency of warehouse inspection, and can flexibly collect perspectives that cannot be captured by fixed cameras, improve the coverage rate of warehouse inspection, so as to realize dynamic warehouse management. And using a drone cluster to realize the inspection of a large warehouse realizes multi-aircraft cooperation to improve the efficiency of warehouse management. At the same time, anomaly detection of the inspected shelves is also carried out, and the warehouse inspection path is further corrected, so that the drones can autonomously complete adaptive inspection in a large warehouse. In addition, by using digital twin technology to display the warehouse status, the visualization and intelligent level of warehouse management are further improved. Description of the Drawings

[0016] The drawings described herein are used to provide a further understanding of the present application, and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation to the present application. In the drawings: Figure 1 It is a schematic flowchart of a drone-based warehouse inspection and management method in an embodiment of the present application; Figure 2 It is a schematic structural diagram of a drone-based warehouse inspection and management system in an embodiment of the present application; Figure 3 It is a schematic structural diagram of a drone-based warehouse inspection and management device in an embodiment of the present application. Specific Embodiments

[0017] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments of this application and the corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in this application without creative efforts shall fall within the scope of protection of this application.

[0018] The embodiments of this application provide a method, system, and device for warehouse inspection and management based on unmanned aerial vehicles (UAVs) to solve the problems of low efficiency and incomplete coverage in current warehouse management, reliance on manual experience, as well as insufficient dynamic warehouse management capabilities and low intelligence levels.

[0019] The following will describe each embodiment of this application in detail with reference to the drawings.

[0020] The embodiments of this application provide a method for warehouse inspection and management based on UAVs, as Figure 1 shown, this method may include steps S101 - S104: S101. The microcontroller generates an initial warehouse inspection path based on the shelf layout information input by the user terminal.

[0021] Among them, the shelf layout information at least includes shelf height, shelf spacing, product categories, and obstacle distribution.

[0022] It should be noted that the microcontroller, as the execution entity of the method for warehouse inspection and management based on UAVs, can be set in devices such as edge computing nodes and servers, and this application does not make specific limitations on this.

[0023] The user terminal can be devices such as the mobile phone or computer of warehouse management personnel, and this application does not make specific limitations on this. It is used to input warehouse layout information and receive interaction instructions, and the user terminal can also display the three - dimensional model of the warehouse on the cloud digital twin platform.

[0024] In the embodiments of this application, the generation of the initial warehouse inspection path based on the shelf layout information input by the user terminal specifically includes: Determine the shelf priority corresponding to each shelf according to the goods category in the shelf layout information. Determine the corresponding obstacle projection area according to the obstacle distribution. Input the shelf priority, the obstacle projection area, and the preset warehouse information into the path cost function of the A* algorithm for path optimization, so as to determine the initial warehouse inspection path according to the function value of the path cost function. The preset warehouse information includes at least the diagonal length of the warehouse, the priority threshold, and the area of the preset inspection grid area. The area of the preset inspection grid area is the area of the grid area obtained by dividing the total warehouse area in advance according to the shelf height, the shelf spacing, and the goods category.

[0025] In other words, the microcontroller can obtain the shelf priority corresponding to each shelf in the shelf layout information through the goods category priority stored in the connected memory, and the goods category priority corresponds to the shelf priority. At the same time, the obstacle distribution is also used, such as the position distribution of obstacles such as ladders and columns, and the obstacles are projected from top to bottom to obtain the obstacle projection area. Input the shelf priority, the obstacle projection area, and the preset warehouse information into the path cost function of the following A* algorithm (A* search algorithm), continuously perform path optimization, generate several paths, and calculate the path cost according to the length of each path. According to the comparison result of the path cost, obtain the final initial warehouse inspection path, such as taking the path with the minimum path cost as the initial warehouse inspection path. The path cost function is as follows:

[0026] Wherein, represents the path cost of the th path; , , are weight coefficients respectively, which can be set by the user according to the actual usage scenario, ; is the path length of the th path; is the diagonal length of the warehouse; is the maximum value of the warehouse priority in the th path; is the preset priority threshold, which is set by the developer or user and is not specifically limited here; is the sum of the obstacle projection areas in the th path; is the area of the preset inspection grid area in the th path.

[0027] S102. The microcontroller sends the initial warehouse inspection path to the UAV cluster so that the corresponding UAVs can collect inspection shelf data and construct a 3D model of the warehouse.

[0028] Among them, the three-dimensional model of the warehouse at least includes the associated semantic information between the shelves and the goods.

[0029] In the embodiment of the present application, the above-mentioned unmanned aerial vehicle is equipped with a multi-modal sensor module for synchronously collecting inspection shelf data. The multi-modal sensor module at least includes a lidar, a 3D camera, and a radio frequency identification (RFID) reader-writer. The RFID reader-writer is used to read the RFID tags carrying storage attributes on each shelf. The storage attributes at least include the goods category, storage conditions, and inspection record timestamps.

[0030] In the embodiment of the present application, the above-mentioned step of sending the initial warehouse inspection path to the UAV cluster so that the corresponding UAVs collect inspection shelf data and construct a three-dimensional model of the warehouse specifically includes: Along the initial warehouse inspection path, perform clustering processing on the corresponding grid regions to obtain multiple clustering task assignment regions. One clustering task assignment region corresponds to a section of the warehouse inspection path. The inspection paths of each warehouse are spliced to obtain the initial warehouse inspection path. According to the task execution power of each UAV in the UAV cluster and the estimated inspection energy consumption of each warehouse inspection path, allocate each UAV to the corresponding warehouse inspection path, so that the UAVs collect inspection shelf data along the assigned warehouse inspection paths according to the allocation results. According to the inspection shelf data, establish the association relationship between the shelves and the goods, and generate a local three-dimensional map, so as to establish a three-dimensional model of the warehouse based on each association relationship and each local three-dimensional map.

[0031] That is to say, the present application can first perform clustering processing on the grid regions obtained by dividing the total area of the warehouse according to the initial warehouse inspection path. Among them, the grid regions can be divided according to the shelf height, shelf spacing, and goods category. For example, by performing dimensionless processing on the shelf height and shelf spacing, and encoding different goods categories as numbers, where the numbers correspond to the required storage space size of the goods category, and using a1 * shelf height + a2 * shelf spacing + a3 * goods category, the calculated value is used as the area of the grid region. After performing clustering processing on the grid regions of the initial warehouse inspection path, multiple clustering task assignment regions can be generated. The clustering processing can be performed by a density-based clustering algorithm (Density-Based Spatial Clustering of Applications with Noise, DBSCAN), or other clustering algorithms can also be used. The present application does not make specific limitations in this regard. According to the task execution power of each UAV in the UAV cluster that can be used for inspection and each warehouse inspection path, perform task allocation for the UAVs, so as to allocate one or more UAVs to each warehouse inspection path.

[0032] Meanwhile, the microcontroller will also use the inspection shelf data collected by the drone during the inspection task to bind the association relationship between the shelf and the goods, such as Shelf A - Third Layer - Electronic Products - Inventory Quantity = 20, and establish the associated semantic information of the shelf and the goods. It will also generate a local 3D map by combining the point cloud and visual data collected by the drone, and establish a 3D model of the warehouse in combination with the associated semantic information to display the relevant information of the shelves, goods, and passages in the warehouse. Among them, the above-mentioned inspection shelf data at least includes the images of the shelf and the goods, the point cloud data, and the storage attributes corresponding to the RFID tags.

[0033] In addition, when the drone is performing the inspection task, there may be a situation where the inspection task takes a long time or consumes a lot of energy. However, relying on a single drone to perform the inspection may not be able to complete the task in time. Therefore, according to the task execution power of each drone in the drone cluster and the estimated inspection energy consumption of each warehouse inspection path, each drone is assigned to the corresponding warehouse inspection path. Specifically, it further includes: When the task execution power of any drone is less than the estimated inspection energy consumption of the warehouse inspection path, a joint inspection drone group in the drone cluster is matched according to the estimated inspection energy consumption. The sum of the task execution powers of each drone in the joint inspection drone group is not less than the estimated inspection energy consumption.

[0034] In other words, when assigning the drone to the warehouse inspection path, the microcontroller will compare the task execution power of the drone with the estimated inspection energy consumption of the warehouse inspection path. If there is no drone whose power can meet the estimated inspection energy consumption, it will traverse the task execution powers of each drone in the drone cluster according to the estimated inspection energy consumption and form a joint inspection drone group. The sum of the task execution powers of each drone in the joint inspection drone group is at least equal to or greater than the estimated inspection energy consumption. That is, each drone in the joint inspection drone group separately inspects a sub-path in the warehouse inspection path, and the sub-paths responsible for inspection by each drone in the joint inspection drone group constitute the warehouse inspection path. The path length of each sub-path is positively correlated with the task execution power of the drone in the joint inspection drone group. Thus, the warehouse inspection path is further segmented, effectively scheduling the drone to complete the warehouse inspection and efficiently completing the warehouse management.

[0035] S103, the microcontroller based on a preset anomaly detection model, determines the anomaly detection result of the inspection shelf in real time, and dynamically adjusts the warehouse inspection path of the corresponding drone according to the anomaly detection result and the preset inspection path optimization conditions.

[0036] In the embodiment of the present application, the above-mentioned real-time determination of the anomaly detection result of the inspection shelf based on the preset anomaly detection model specifically includes: Input the real-time inspection shelf data from the drone into a preset anomaly detection model to determine whether there is an inspection anomaly according to the model output result. The anomaly types of the inspection anomaly include at least shelf structure anomaly and / or inventory anomaly. The shelf structure anomaly is obtained based on the shelf normal vector deviation angle. If so, add the anomaly type corresponding to the inspection anomaly to the anomaly detection result.

[0037] In other words, the present application is pre-trained with a preset anomaly detection model, which can be a neural network model and is trained by a number of inspection shelf data samples and inspection anomaly labels. The microcontroller inputs the real-time inspection shelf data from the drone into the preset anomaly detection model, so as to judge whether there is an inspection anomaly according to the model output result of the preset anomaly detection model. Among them, the anomaly detection includes structure anomaly detection, which can fit the shelf plane through the PointNet++ network and calculate the normal vector deviation angle θ. If θ>5°, it is determined that the shelf structure is abnormal; for inventory anomaly detection, the YOLOv7 model can be used to identify the misalignment of goods and combine the RFID missed reading event to determine the inventory anomaly. When there is an inspection anomaly in the model output result, the microcontroller will generate an anomaly detection result at this time. For example, the anomaly detection result is "There is a shelf structure anomaly in Shelf A".

[0038] Furthermore, in the embodiment of the present application, the above-mentioned dynamically adjusts the cargo hold inspection path of the corresponding drone according to the anomaly detection result and the preset inspection path optimization condition, which specifically includes: Match the anomaly detection result with the preset inspection path optimization condition to determine the dynamic adjustment rule according to the optimization condition matching result. Among them, the dynamic adjustment rule includes at least one or more of the following: generating a bypass path, generating a review inspection path. Update the cargo hold inspection path according to the dynamic adjustment rule.

[0039] That is to say, the preset inspection path optimization conditions are stored in the memory connected to the microcontroller. This condition records the corresponding relationship between different anomaly detection results and different dynamic adjustment rules. By matching the anomaly detection result with the preset inspection path optimization condition, the dynamic adjustment rule for this anomaly detection result is obtained. The dynamic adjustment rule is used to update the inspection path of the drone, including generating a bypass path and generating a review inspection path for the position where the anomaly detection result is located in the cargo hold inspection path, so as to update the cargo hold inspection path of the drone.

[0040] S104, the microcontroller synchronously sends the three-dimensional model of the cargo hold and the anomaly detection result to the cloud digital twin platform to display the status of the cargo hold to the user terminal.

[0041] The microcontroller can synchronize the three-dimensional model of the warehouse and the anomaly detection results to the cloud digital twin platform, so as to display the digital space of the internal state of the warehouse expressed digitally. This digital space includes the internal state of the warehouse such as the shelf positions, the storage positions of goods, the simulation of the movement and storage process of goods, etc. inside the warehouse.

[0042] In the embodiment of the present application, after the above-mentioned three-dimensional model of the warehouse and the anomaly detection results are synchronously sent to the cloud digital twin platform to display the warehouse state to the user terminal, the method further includes: Receiving a viewing instruction from the user terminal and determining the three-dimensional coordinates of the warehouse corresponding to the viewing instruction. Determining the timestamp of the inspected path of the warehouse where the three-dimensional coordinates of the warehouse are located. When the time interval between the inspected timestamp and the current time is less than a predetermined value, the inspected path of the warehouse where the three-dimensional coordinates of the warehouse are located is enlarged and displayed through the cloud digital twin platform. When the time interval between the inspected timestamp and the current time is greater than or equal to the predetermined value, a patrol execution instruction is generated to call the corresponding drone to patrol the inspected path of the warehouse where the three-dimensional coordinates of the warehouse are located through the patrol execution instruction, and the inspected path of the warehouse after the patrol is enlarged and displayed.

[0043] In other words, the user can operate the user terminal to view the digital twin model on the cloud digital twin platform. The microcontroller can generate a viewing instruction for viewing the selected area by analyzing the area of the three-dimensional model of the warehouse selected by the user on the user terminal. At the same time, it is judged the timestamp of the last inspection of the inspected path of the warehouse where the selected area is located. If the inspection interval is relatively long, greater than the predetermined value preset by the user, re-inspection is performed to obtain the latest inspection data for the user to view. If the inspection interval is not greater than the predetermined value, the three-dimensional image of the selected area in the three-dimensional model of the warehouse can be directly displayed to the user. In addition, the present application can also enlarge the displayed inspected path of the warehouse to more clearly display the internal information of the warehouse on the screen, such as shelves, the goods carried by the shelves, the placement positions of the goods, etc.

[0044] Through the above technical solution, using drones to complete warehouse inspection and management can liberate manpower, improve the efficiency of warehouse inspection, and can flexibly collect perspectives that cannot be captured by fixed cameras, improve the coverage rate of warehouse inspection, so as to achieve dynamic warehouse management. And using a drone swarm to achieve the inspection of large warehouses realizes multi-aircraft cooperation to improve the efficiency of warehouse management. At the same time, anomaly detection of the inspected shelves is also carried out to further correct the inspected path of the warehouse, enabling the drone to autonomously complete adaptive inspection in the large warehouse. In addition, by using digital twin technology to display the warehouse state, the visualization and intelligent level of warehouse management are further improved.

[0045] Figure 2The structural schematic diagram of a warehouse inspection and management system based on an unmanned aerial vehicle provided by an embodiment of the present application is as follows Figure 2 As shown, the warehouse inspection and management system 200 based on an unmanned aerial vehicle includes: A generation module 201, configured to generate an initial warehouse inspection path based on the shelf layout information input by a user terminal. The shelf layout information includes at least shelf height, shelf spacing, goods categories, and obstacle distribution. A sending module 202, configured to send the initial warehouse inspection path to a drone cluster, so that the corresponding drones collect inspection shelf data and construct a three-dimensional model of the warehouse. Among them, the three-dimensional model of the warehouse includes at least the associated semantic information between the shelves and the goods. A determination module 203, configured to determine the anomaly detection result of the inspected shelves in real time based on a preset anomaly detection model, and dynamically adjust the warehouse inspection path of the corresponding drone according to the anomaly detection result and the preset inspection path optimization conditions. A sending and display module 204, configured to synchronously send the three-dimensional model of the warehouse and the anomaly detection result to a cloud digital twin platform, so as to display the warehouse status to the user terminal.

[0046] Figure 3 The structural schematic diagram of a warehouse inspection and management device based on an unmanned aerial vehicle provided by an embodiment of the present application is as follows Figure 3 As shown, the device includes: At least one processor; and a memory communicatively connected to the at least one processor. Among them, the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor, so that the at least one processor can: Generate an initial warehouse inspection path based on the shelf layout information input by a user terminal. The shelf layout information includes at least shelf height, shelf spacing, goods categories, and obstacle distribution. Send the initial warehouse inspection path to a drone cluster, so that the corresponding drones collect inspection shelf data and construct a three-dimensional model of the warehouse. Among them, the three-dimensional model of the warehouse includes at least the associated semantic information between the shelves and the goods. Determine the anomaly detection result of the inspected shelves in real time based on a preset anomaly detection model, and dynamically adjust the warehouse inspection path of the corresponding drone according to the anomaly detection result and the preset inspection path optimization conditions. Synchronously send the three-dimensional model of the warehouse and the anomaly detection result to a cloud digital twin platform, so as to display the warehouse status to the user terminal.

[0047] Each embodiment in the present application is described in a progressive manner. The same or similar parts among the embodiments can be referred to each other, and the differences between each embodiment and other embodiments are emphasized. In particular, for the system and device embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0048] The system, device, and method provided by the embodiments of the present application correspond one by one. Therefore, the system and device also have beneficial technical effects similar to those of their corresponding methods. Since the beneficial technical effects of the method have been described in detail above, the beneficial technical effects of the system and device will not be elaborated here.

[0049] It should also be noted that the term "including", "comprising", or any other variant thereof is intended to cover non-exclusive inclusion, such that a process, method, commodity, or device that includes a series of elements includes not only those elements but also other elements that are not explicitly listed, or elements that are inherent to such process, method, commodity, or device. Without further limitation, an element defined by the statement "including one..." does not exclude the existence of additional identical elements in the process, method, commodity, or device that includes the said element.

[0050] The above description is only for the embodiments of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included within the scope of the claims of the present application.

Claims

1. A method for inspection and management of a cargo warehouse based on an unmanned aerial vehicle, characterized in that, The method includes: Generating an initial warehouse inspection path based on the shelf layout information input by the user terminal; the shelf layout information at least includes shelf height, shelf spacing, item categories, and obstacle distribution; Sending the initial warehouse inspection path to the UAV cluster so that the corresponding UAVs collect inspection shelf data and construct a three-dimensional model of the warehouse; wherein, the three-dimensional model of the warehouse at least includes the associated semantic information between the shelves and the items; Based on a preset anomaly detection model, determining the anomaly detection result of the inspected shelves in real time, and dynamically adjusting the warehouse inspection path of the corresponding UAV according to the anomaly detection result and the preset inspection path optimization conditions; Synchronously sending the three-dimensional model of the warehouse and the anomaly detection result to the cloud digital twin platform to display the warehouse status to the user terminal.

2. The method for inspecting and managing a cargo warehouse based on a drone according to claim 1, characterized in that, Generating an initial warehouse inspection path based on the shelf layout information input by the user terminal specifically includes: Determining the shelf priority corresponding to each shelf according to the item categories in the shelf layout information; Determining the corresponding obstacle projection area according to the obstacle distribution; Inputting the shelf priority, the obstacle projection area, and the preset warehouse information into the path cost function of the A* algorithm for path optimization to determine the initial warehouse inspection path according to the function value of the path cost function; the preset warehouse information at least includes the diagonal length of the warehouse, the priority threshold, and the area of the preset inspection grid area; the area of the preset inspection grid area is the area of the grid area obtained by pre-dividing the total area of the warehouse according to the shelf height, the shelf spacing, and the item categories.

3. The method for inspection and management of a cargo warehouse based on an unmanned aerial vehicle according to claim 1, wherein The UAV is equipped with a multi-modal sensor module for synchronously collecting inspection shelf data; the multi-modal sensor module at least includes a lidar, a 3D camera, and an RFID reader; the RFID reader is used to read the RFID tags carrying storage attributes on each shelf; the storage attributes at least include the item categories, storage conditions, and inspection record timestamps.

4. The method for inspection and management of a cargo warehouse based on an unmanned aerial vehicle according to claim 2, characterized in that, Sending the initial warehouse inspection path to the UAV cluster so that the corresponding UAVs collect inspection shelf data and construct a three-dimensional model of the warehouse specifically includes: Performing clustering processing on the corresponding grid areas along the initial warehouse inspection path to obtain multiple clustering task allocation areas; one clustering task allocation area corresponds to a section of the warehouse inspection path; each section of the warehouse inspection path is spliced to obtain the initial warehouse inspection path; According to the task execution power of each UAV in the UAV cluster and the estimated inspection energy consumption of each warehouse inspection path, allocating each UAV to the corresponding warehouse inspection path so that the UAVs collect the inspection shelf data along the allocated warehouse inspection path; According to the inspection shelf data, constructing the association relationship between the shelves and the items, and generating a local three-dimensional map, and establishing the three-dimensional model of the warehouse according to each association relationship and each local three-dimensional map.

5. The method for inspecting and managing a cargo warehouse based on a drone according to claim 4, wherein, According to the task execution power of each drone in the drone cluster and the estimated inspection energy consumption of each cargo warehouse inspection path, allocate each drone to the corresponding cargo warehouse inspection path, specifically including: When the task execution power of any drone is less than the estimated inspection energy consumption of the cargo warehouse inspection path, match the joint inspection drone group in the drone cluster according to the estimated inspection energy consumption; the sum of the task execution powers of the drones in the joint inspection drone group is not less than the estimated inspection energy consumption.

6. A method for cargo hold inspection and management based on an unmanned aerial vehicle according to claim 1, characterized in that, Based on a preset anomaly detection model, determine the anomaly detection result of the inspected shelf in real time, specifically including: Input the real-time inspected shelf data from the drone into the preset anomaly detection model to determine whether there is an inspection anomaly according to the model output result; the anomaly types of the inspection anomaly at least include shelf structure anomaly and / or inventory anomaly; the shelf structure anomaly is obtained based on the shelf normal vector offset angle. If so, add the anomaly type corresponding to the inspection anomaly to the anomaly detection result.

7. A method for inspection and management of a cargo warehouse based on an unmanned aerial vehicle according to claim 1, characterized in that Dynamically adjust the cargo warehouse inspection path of the corresponding drone according to the anomaly detection result and the preset inspection path optimization conditions, specifically including: Match the anomaly detection result with the preset inspection path optimization conditions to determine the dynamic adjustment rule according to the optimization condition matching result; wherein, the dynamic adjustment rule at least includes one or more of the following: generating a detour path, generating a re-inspection path. Update the cargo warehouse inspection path according to the dynamic adjustment rule.

8. A method for inspection and management of a cargo warehouse based on an unmanned aerial vehicle according to claim 1, characterized in that, After synchronously sending the three-dimensional cargo warehouse model and the anomaly detection result to the cloud digital twin platform for displaying the cargo warehouse status to the user terminal, the method further includes: Receive a viewing instruction from the user terminal and determine the three-dimensional cargo coordinates corresponding to the viewing instruction. Determine the inspected timestamp of the cargo warehouse inspection path where the three-dimensional cargo coordinates are located. In the case that the time interval between the inspected timestamp and the current time is less than a predetermined value, magnify and display the cargo warehouse inspection path where the three-dimensional cargo coordinates are located through the cloud digital twin platform. In the case that the time interval between the inspected timestamp and the current time is greater than or equal to the predetermined value, generate an inspection execution instruction to call the corresponding drone to inspect the cargo warehouse inspection path where the three-dimensional cargo coordinates are located through the inspection execution instruction, and magnify and display the inspected cargo warehouse inspection path.

9. An unmanned aerial vehicle-based cargo warehouse inspection and management system, characterized in that, The system includes: A generation module, configured to generate an initial cargo warehouse inspection path based on the shelf layout information input by the user terminal; the shelf layout information at least includes shelf height, shelf spacing, goods category, and obstacle distribution. A sending module, configured to send the initial cargo warehouse inspection path to the drone cluster, so that the corresponding drone collects inspected shelf data and constructs a three-dimensional cargo warehouse model; wherein, the three-dimensional cargo warehouse model at least includes the associated semantic information between the shelf and the goods. A determination module, configured to determine in real time an anomaly detection result of an inspection shelf based on a preset anomaly detection model, and dynamically adjust a cargo hold inspection path of a corresponding unmanned aerial vehicle according to the anomaly detection result and preset inspection path optimization conditions; A sending and display module, configured to synchronously send the three-dimensional model of the cargo hold and the anomaly detection result to a cloud digital twin platform, so as to display the status of the cargo hold to the user terminal.

10. An unmanned aerial vehicle-based cargo warehouse inspection and management device, characterized in that, The device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor, so that the at least one processor can execute a method for cargo hold inspection and management based on an unmanned aerial vehicle according to any one of claims 1-8 above.

Images