Electric power material unmanned aerial vehicle checking method and system based on electronic map, terminal and storage medium

Abstract

The invention relates to the technical field of server and switch products, and particularly provides an electric power material unmanned aerial vehicle checking method and system based on an electronic map, a terminal and a storage medium, and the method comprises the steps: a handheld device is used for issuing an unmanned aerial vehicle checking task, and a checking server starts the image live broadcast of an unmanned aerial vehicle visual angle in a pre-established 3D warehouse electronic map; the unmanned aerial vehicle starts from an initial site, flies based on a preset inventory route, shoots electric power materials in an inventory area in real time in the flight process, sends a shot video to the edge computing device, and the inventory server displays a shot picture of the unmanned aerial vehicle in a pre-established 3D warehouse electronic map in real time; the edge computing device intercepts an image in the video, sends the image to a trained electric power material identification model for identification, obtains the number and the type of the identified electric power materials, obtains an inventory result, and displays the inventory result in a 3D warehouse electronic map; and after checking is finished, the unmanned aerial vehicle flies back to the initial site position. The transparency and the operation efficiency of the checking process are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of cargo inventory, and in particular relates to an electronic map-based electric power material drone inventory method, system, terminal and storage medium. Background Art

[0002] The outdoor storage yard for power materials is an important node in the supply chain of power engineering materials. It is mainly used to store large-volume, multi-variety, and high-value equipment and materials. Outdoor storage yards are usually large in scale, with complex layouts, many types of materials, and different stacking heights.

[0003] Existing technologies mainly rely on manual inspections or semi-automatic equipment (such as handheld scanners) to complete inventory. This method requires operators to confirm the types and quantities of materials one by one, which is not only time-consuming and labor-intensive, but also inefficient. In particular, it is difficult to update inventory data in a timely manner in emergency situations, which affects the response speed of management decisions. In addition, materials are exposed to the outdoors for a long time, affected by dust, rain, rust, etc., and the surface labels may be blurred or damaged, resulting in frequent omissions or misreadings during information reading. At the same time, materials stacked high or obscured are difficult to find during manual inspections, further affecting the comprehensiveness and accuracy of inventory data. In addition, the inventory results of manual inspections are usually presented in static tables or data, lacking dynamic visualization capabilities. It is difficult for managers to intuitively understand the distribution of materials in the yard and the progress of inventory through these results. Especially in high-density stacking scenarios, it is difficult to locate specific materials or identify abnormal areas in a timely manner. Summary of the invention

[0004] In view of the problem in the prior art that when manual inspection or semi-automatic equipment is used to conduct inventory of power materials, materials piled high or obscured are difficult to find during manual inspection and inventory data is difficult to update in time, the present invention provides an electronic map-based power material drone inventory method, system, terminal and storage medium to solve the above technical problems.

[0005] In a first aspect, the present invention provides a method for inventorying electric power materials by drone based on an electronic map, comprising: Use a handheld device to issue a drone inventory task, and the inventory server starts live broadcasting of the drone's perspective in the pre-established 3D warehouse electronic map; The drone starts from the initial location and flies based on the preset inventory route. During the flight, it takes real-time photos of the power materials in the inventory area and sends the video to the edge computing device. The inventory server displays the drone's footage in real time on the pre-established 3D warehouse electronic map. The edge computing device captures images in the video and sends them to the trained power material recognition model for recognition, obtains the quantity and type of the recognized power materials, obtains the inventory results, and displays the inventory results in the 3D warehouse electronic map; After the inventory is completed, the drone flies back to the initial location.

[0006] Furthermore, a drone inventory task is issued using a handheld device, and the inventory server starts live broadcasting of the drone's perspective in the pre-established 3D warehouse electronic map, including: When the drone initiates an inventory, it creates and starts an inventory task through the PDA. The electronic map sends the route task and transmits it to the cloud API through MQTT data.

[0007] Furthermore, during the flight, the power materials in the inventory area are photographed in real time and the video is sent to the edge computing device, including: The video stream shot by the drone is transmitted to the edge computing device using the RTMP protocol; The edge computing device receives the video stream through the streaming interface.

[0008] Furthermore, the method for establishing the electric power material identification model includes: Obtain sample images of all power supplies in the inventory area; Use Labelimg to label the targets in the sample image to obtain a labeled data set; Find the model configuration file in the YOLO models folder, set the number of classes in the model configuration file to the preset parameters, set the total number of rounds of model training to n, and set the batch size to m; Find the coco.yaml file in the YOLO data folder and name it as a custom data configuration file. Open the custom data configuration file, modify the paths of train and val, and list the category names of all power materials in name. After the settings are completed, click to start training. After the training is completed, an exp folder is generated under the runs folder. The exp folder includes the training results and weights, and a trained power material recognition model is obtained.

[0009] In a second aspect, the present invention provides an electronic map-based power material drone inventory system, comprising: The inventory task issuing module is used to issue drone inventory tasks using handheld devices. The inventory server starts live broadcast of the drone's perspective in the pre-established 3D warehouse electronic map; The inventory start module is used for the drone to start from the initial location and fly based on the preset inventory route. During the flight, the drone takes real-time photos of the power materials in the inventory area and sends the captured videos to the edge computing device. The inventory server then displays the drone’s captured images in real time on the pre-established 3D warehouse electronic map. The inventory recognition module is used for edge computing devices to capture images in the video, send the images to the trained power material recognition model for recognition, obtain the quantity and type of the recognized power materials, obtain the inventory results, and display the inventory results in the 3D warehouse electronic map; The inventory completion module is used to make the drone fly back to the initial location after the inventory is completed.

[0010] Furthermore, the inventory task issuing module includes: When the drone initiates an inventory, it creates and starts an inventory task through the PDA. The electronic map sends the route task and transmits it to the cloud API through MQTT data.

[0011] Furthermore, the inventory start module includes: Video transmission unit: the video stream shot by the drone is transmitted to the edge computing device using the RTMP protocol; Video receiving unit,The edge computing device receives the video stream through the streaming interface.

[0012] Furthermore, it also includes: a model building unit, including: An image acquisition unit, used to acquire sample images of all power materials in the inventory area; A data labeling unit, used to label the target in the sample image using Labelimg to obtain a labeled data set; The first setting unit is used to find the model configuration file in the models folder of YOLO, set the number of classes in the model configuration file to the preset parameters, set the total rounds of model training to n, and set the batch size to m; The second setting unit is used to find the coco.yaml file in the YOLO data folder and name it as a custom data configuration file, open the custom data configuration file, modify the paths of train and val, and list the category names of all power materials in name; The model generation unit is used to start training after the settings are completed. After the training is completed, an exp folder is generated under the runs folder. The exp folder includes the training results and weights to obtain a trained power material identification model.

[0013] In a third aspect, a terminal is provided, including: processor, memory, wherein: The memory is used to store computer programs. The processor is used to call and run the computer program from the memory, so that the terminal executes the above-mentioned terminal method.

[0014] According to a fourth aspect, a computer storage medium is provided, wherein the computer-readable storage medium stores instructions, and when the instructions are executed on a computer, the computer executes the methods described in the above aspects.

[0015] The beneficial effect of the present invention is that the method, system, terminal and storage medium for the electronic map-based power material drone inventory provided by the present invention first issue the drone inventory task through a handheld device, and realize the live broadcast of the picture in combination with the pre-established 3D warehouse electronic map, which not only enhances the convenience of task issuance, but also provides an intuitive real-time monitoring interface for operators, and improves the transparency and operation efficiency of the inventory process. Secondly, by flying the drone along the preset route and shooting the power materials in the inventory area in real time, it can greatly reduce manual participation, reduce the workload and error rate of manual inventory. The method of transmitting the video stream to the edge computing device using the RTMP protocol and receiving the video stream through the streaming media interface ensures the efficiency and stability of data transmission, and adapts to the needs of large-scale power material inventory. In addition, the edge computing device uses the trained power material recognition model to process the video image in real time, which can quickly and accurately identify the quantity and type of materials, and directly display the inventory results in the 3D warehouse electronic map. This recognition method combined with the YOLO deep learning algorithm greatly improves the accuracy and reliability of recognition through precise image segmentation and high-confidence detection, and significantly reduces the data processing time. Finally, the drone automatically returns to its initial location, which not only reflects the intelligence of the system but also ensures the safe recovery of the drone. This method integrates multiple advanced technologies such as drone technology, edge computing, deep learning, and 3D electronic maps, providing a comprehensive, intelligent, and low-cost solution for the efficient inventory of power materials, which has broad application prospects.

[0016] In addition, the invention has a reliable design principle, a simple structure and a very broad application prospect. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0018] Figure 1 is a schematic flow chart of a method according to an embodiment of the present invention.

[0019] Figure 2 is a schematic block diagram of a system according to an embodiment of the present invention.

[0020] Figure 3 A schematic diagram of the structure of a terminal provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0021] In order to enable those skilled in the art to better understand the technical solutions in the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0022] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the specification of the present invention herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention.

[0023] The method for counting electric power materials by drone based on electronic maps provided in the embodiment of the present invention is executed by a computer device, and accordingly, the system for counting electric power materials by drone based on electronic maps runs in the computer device.

[0024] Figure 1 is a schematic flow chart of a method according to an embodiment of the present invention. Figure 1 The execution subject can be an electric power material drone inventory system based on an electronic map. According to different needs, the order of the steps in the flowchart can be changed, and some can be omitted.

[0025] To facilitate understanding of the present invention, the following is a further description of the electric power material drone inventory method based on an electronic map provided by the present invention based on the principle of the electric power material drone inventory method based on an electronic map, combined with the process of managing pluggable module materials in the embodiment.

[0026] Specifically, Figure 1 As shown, the method for counting power materials by drone based on electronic map includes: S1. Use a handheld device to issue a drone inventory task, and the inventory server starts live broadcasting of the drone's perspective in the pre-established 3D warehouse electronic map.

[0027] When the drone initiates an inventory, it creates and starts an inventory task through the PDA. The electronic map sends the route task and transmits it to the cloud API through MQTT data.

[0028] Specifically, in the process of implementing drone inventory, this project applied the private deployment of DJI's cloud API technology to realize the basic business management of the airport. Based on the research on the RTMP protocol, the FLV streaming media playback technology of HTML 5 was applied to realize the real-time playback of the inventory video. The Unity3D digital twin modeling technology was used to realize the digital twin modeling of the airport and warehouse. Finally, based on the recognition and application technology of YOLO's large model visual training, the inventory of materials was realized through continuous pre-training models.

[0029] (1) Private deployment of DJI’s cloud API technology In order to effectively manage the internal network of DJI Airport, the basic business management of the airport is achieved by using the private deployment of DJI Cloud API, including airport and aircraft status monitoring, live broadcast, mission flight and other functions. DJI Cloud API is a protocol designed specifically for data interaction with DJI devices. It allows third-party cloud platforms to exchange information with DJI devices (such as drones, airports, remote controls, etc.). In the absence of Cloud API, DJI drones can only transmit data with remote controls or airports through private image transmission links, and cannot directly exchange information with network devices. The emergence of Cloud API solves this problem, allowing third-party devices to obtain information from DJI devices (such as the latitude and longitude, altitude, battery information, etc. of the drone) and transmit instructions to DJI devices.

[0030] The cloud API is deployed using Docker container technology, and redis is used as a cache database to store real-time status information of the airport and drones, ensuring fast reading and writing of status data. The MySQL database is used to store and manage various types of airport data, including flight records, equipment information, user operation logs, etc., ensuring persistent storage and security of data. The EMQX message server is deployed to achieve communication capabilities with external systems, and the cloud API is interacted with by writing MQTT messages to send inventory instructions. After receiving the instructions, the drone performs flight inventory according to the preset route and feeds back the inventory results to the management system in real time.

[0031] (2) FLV streaming media playback technology based on HTML 5 The full inventory server receives the airport RTMP video stream by building an SRS streaming media server, and converts the RTMP stream to FLV video stream through FFmpeg. The electronic map builds a multimedia player by using the FLV.js tool and Html 5 video component to play FLV videos.

[0032] The RTMP protocol is a network protocol for real-time data communication. It is mainly used for audio, video and data communication between the Flash / Air platform and the streaming media / interactive server that supports the RTMP protocol, so it can be used to support live broadcasting. The advantage of the RTMP protocol is that it has good real-time performance and a small data transmission delay, so it is widely used in the field of streaming live data transmission. However, RTMP is also controversial due to some disadvantages. First, the RTMP protocol was proposed by Adobe and was originally designed to support data transmission between the server and the Adobe Flash Player player, but it cannot adapt to other playback environments and has poor client compatibility. Secondly, the fields specified in the RTMP protocol are very complex, and there are many redundant fields and they are cumbersome, which affects the transmission efficiency to a certain extent. At the same time, as a typical representative of streaming data transmission protocols, the server needs strong computing power when transmitting based on the RTMP protocol, and its performance is easily limited in high-visit scenarios. For this reason, this project uses the FLV streaming media playback technology based on Html 5 for inventory and display.

[0033] FLV (Flash Video) is a media container format released by Adobe for live streaming and on-demand video. FLV is known for its extremely simple structure, which mainly consists of a file header (FLV Header) and multiple FLV Tags. Among them, FLV Tags can be divided into three categories: audio tags (Audio Tag), video tags (Video Tag) and script tags (Script Tag), also known as (Meat Data Tag), which are used to store audio data, video data and script metadata respectively.

[0034] The HTML 5 streaming media player is a streaming media playback platform that runs in the HTML 5 browser environment. The HTML 5 streaming media player provides services for live broadcast viewers, is responsible for processing various interactive operations of users on the playback platform, sends data requests to the streaming media server application, and displays the real-time media stream decoding and playback from the server on the user interface.

[0035] The four major modules on the HTML 5 streaming media server (player control module, live data request module, live data processing module, and playback content management module) will work together to complete the tasks of requesting streaming media data, receiving and processing data, and performing streaming media live broadcast.

[0036] The process of FLV streaming media playback technology based on HTML 5 is as follows: 1) The live streaming data collection source pushes the live streaming data to the streaming media server application in the form of RTMP packets.

[0037] 2) The streaming media server uses the live data receiving module to receive RTMP data packets, extracts the actual transmission data from the data packets according to the format of the data packets, and outputs it to the data push storage module. At the same time, the live data receiving module records the data push status of the live collection source through the "heartbeat detection" mechanism, which serves as the basis for the data request processing module to respond to the live data request from the streaming media player.

[0038] 3) After receiving the live data segment output by the data push receiving module, the data push storage module stores the audio and video tag data starting from the latest key frame in the live buffer data queue, and stores the FVHeader, script tag, and the first group of audio and video tags and other pre-data in the live metadata queue.

[0039] 4) After the user issues a live broadcast instruction to the streaming media player, the streaming media player sends a live broadcast data request to the streaming media server application via the HTTP protocol; 5) The streaming media server data request processing module receives the live data request and determines whether to perform live data transmission based on the data push status provided by the data push receiving module. If the data push from the live acquisition source is "in progress", the "streaming start" instruction is output to the streaming management module; 6) After receiving the "streaming start" instruction, the streaming management module establishes a streaming connection between the server and the browser, first transmitting the front FIV data in the live metadata queue, and then starts to transmit the FLV audio and video Tag data in the live buffer data queue, and transmits the streaming media data required for live broadcast to the streaming media player.

[0040] The pre-built 3D warehouse electronic map is as follows: The electronic map system for electric power materials is connected to the intelligent warehousing and dispatching system, integrating cutting-edge technologies such as IOT perception, visual conversion, and digital twins, and innovating the fusion of the physical and digital worlds to achieve remote issuance of commands, automatic execution of operations, online monitoring of status, and real-time feedback of information.

[0041] The electronic map is deployed in the local computer room of the warehouse and applied locally in the warehouse. It can access each other with various IoT devices and operating equipment, and uses local computing resource model rendering to tightly integrate various operating equipment, IoT sensing devices, and business operation data in the warehouse to achieve applications such as visual job scheduling, one-click smart inventory, operation process monitoring and tracing, and intelligent robot operation command.

[0042] Digital Twin: Digital Twin technology is an innovative digital transformation method that uses multi-source heterogeneous information such as physical models, sensor networks, and historical operation data to build a simulation environment that is multidisciplinary, multi-physical quantity coupled, and multi-scale integrated. In this environment, digital twin technology achieves accurate mapping of physical space in digital space, and fully reflects the entire life cycle of the corresponding physical equipment and products, including design, manufacturing, logistics, maintenance, and other stages.

[0043] During the implementation process, we conducted on-site surveys using tools such as laser rangefinders, drones, cameras, calipers, and tape measures to measure the real dimensions of the park, warehouse, and shelves, and designed CAD drawings. Based on CAD dimensions and on-site photos, we split the elements in the warehouse, and used 3DMax tools to perform high-precision physical modeling of entities such as the park, warehouse, storage space, materials, and automated equipment in the physical world to obtain a single model. Using three.js technology, we merged the single model into a layout to form a digital twin scene. Using 3D digital modeling technology, we simulated the environment in the park in real time, replicated the status of equipment and materials simultaneously, connected the physical world with the digital world, and achieved a complete mapping of all elements of warehouse operation information, so that the park's dynamics can be grasped in real time and visualized online in all directions.

[0044] Data presentation: Through sensing terminals, including portable data collection terminals (PDA), RFID tags, on-site operation robots, etc., a series of warehouse management operations such as outbound, inbound, transfer, inventory, return, inspection, and transfer are supported, which realizes accurate monitoring and tracking of material status and ensures the accuracy of data collection in business processes. Through full coding, unique identification codes are generated for materials and warehouse locations, and efficient scanning operations are performed, laying the foundation for building basic data for digital twins.

[0045] The Internet of Things (IoT) technology is used to integrate automatic identification equipment to capture the operational data generated on-site in real time and realize real-time data collection. The collected data includes operation logs, model data, full coding data, etc., covering information such as warehouse equipment, materials and business. Through comprehensive processing and analysis of the data, the collected data is presented in real time in the digital twin model.

[0046] Digital space: In the construction of digital space, relying on digital twin (Digital Twin) technology and digital label (Digital Labeling), containerization technology and orchestration tools are used to realize the modularization and service-oriented of the system, and to build an electronic map of physical resources with spatial positioning and geographic information display capabilities, so as to realize the "online" of warehouse materials, equipment, and business throughout the process, and the real-time "control" of operation progress, inventory location and other status, so as to realize the full visualization of physical resources and operation scenarios.

[0047] Through the Internet of Things (IoT) and Radio Frequency Identification (RFID) technologies, the operation process is intelligently tracked, and the dynamic position of the materials is monitored and located through real-time data streams. Java and SpringCloud are used to process business logic and data exchange. NoSQL database management systems are used to store and manage massive physical resource data. HTML5, CSS3, JavaScript and Vue.js frameworks are used to build user interfaces and interactive experiences, and a full-scale, multi-dimensional, dynamically updated panoramic business roaming and processing is carried out.

[0048] 2. Key technologies In the electronic map system, the bidirectional real-time mapping of digital space to physical space is the basic goal of the application of digital twin technology. To achieve this goal, the key to system construction is to establish an information connection and driving relationship between bidirectional real mapping and real-time interaction, and to ensure that the digital twin and the physical entity are synchronized in state. Digital twin modeling technology, RocketMQ real-time data transmission technology, and multi-source heterogeneous data processing technology are applied, and innovative algorithms for warehouse space coordinate analysis and physical information visual conversion are proposed.

[0049] Three.js digital twin modeling technology When constructing a visual twin virtual scene, the electric power material storage electronic map system uses Three.js for development. Three.js is a lightweight, cross-platform JavaScript library that is a 3D engine based on native WebGL encapsulation and runs in the browser. By combining with technologies such as Canvas and SVG in HTML5, 3D scenes and animations are created and displayed in the browser page. In the scene, the parameters of the model can be manipulated to modify the model's material, size, position and other information, and tools are provided to optimize the model rendering speed. By importing the 3D model produced by 3DMax into the scene for loading and rendering, high-precision, high-simulation restoration of the physical world's parks, warehouses, storage areas, storage locations, materials, automation equipment and other physical elements is generated.

[0050] RocketMQ real-time data transmission technology RocketMQ is a distributed messaging middleware developed in Java. It is mainly composed of four modules: Producer, Consumer, NameServer and Broker. In data synchronization based on RocketMQ technology, the RocketMQ message queue server is mainly responsible for the storage, forwarding and query of Topic messages.

[0051] In the process of RocketMQ data synchronization, the intelligent warehouse scheduling system acts as the message producer, and other business systems act as consumers. Netty is used to communicate between the Broker and Consumer / Producer in RocketMQ. Therefore, whether the Consumer calls the Broker to go offline normally or the machine goes offline abnormally, the Broker can perceive it in real time through Netty's communication mechanism.

[0052] The electronic map system for electric power material storage needs to achieve real-time mapping of digital space and physical space, so the timeliness of data transmission is very important. When building the system, RocketMQ technology was applied to decouple various business systems, eliminate the unavailability of direct contact or direct data call between business systems, ensure the high availability of data synchronization, and ensure the real-time nature of data while ensuring that data is not lost during data synchronization.

[0053] Multi-source heterogeneous data processing technology Multi-source heterogeneous data refers to data from different sources and different types, which may have different structural formats, semantics and uses. Multi-source heterogeneous data can generally be divided into three types: structured data, semi-structured data and unstructured data. In the process of building warehouse digital twins, solving the access of multiple types of equipment and systems from various manufacturers, multi-data type parsing and format unification have become basic issues.

[0054] During the development and construction of electronic maps, when connecting with the business system, the data is processed by accessing the data center and using http requests. When connecting with the intelligent warehousing and scheduling system, the view sharing method is used to directly access the database and use SQL to query and clean the data, thereby achieving unified management of multi-source heterogeneous data.

[0055] Through warehouse model construction, the warehouse is twinned and a digital warehouse space is created. This provides an interactive application with "first-person immersive experience and control", which allows users to roam the warehouse freely from any perspective and intuitively experience the warehouse structure, physical resources, and storage space usage.

[0056] S2. The drone starts from the initial location and flies based on the preset inventory route. During the flight, it takes real-time photos of the power materials in the inventory area and sends the captured video to the edge computing device. The inventory server displays the drone's captured images in real time on the pre-established 3D warehouse electronic map.

[0057] The video stream captured by the drone is transmitted to the edge computing device using the RTMP protocol, and the edge computing device receives the video stream through the streaming interface.

[0058] Specifically, when the PDA sends an inventory task, the full inventory server will send a websocket notification to the electronic map page, start the live broadcast of the cameras inside and outside the airport, push the stream to the SRS service, and return the stream address to the electronic map page; after receiving the notification, the electronic map page enters the airport inventory model page, calls the player to pull the stream, realizes the live broadcast of the airport screen, and starts the mqtt monitoring of the airport and the drone. When it is detected that the airport is turned on, the webgl interface is called to open the lid of the airport model, and the camera is pulled in to the sky above the airport for a bird's-eye view, and the drone model rotation animation is turned on; when the drone mqtt monitoring detects that the drone is turned on, it starts to live broadcast the drone and push the stream to SRS, obtains the stream address, displays the drone's shooting screen to the page, and converts the real-time coordinates of the drone through a proportional algorithm, calls the webgl interface to realize the position control of the drone, and controls the camera lens to follow the movement of the drone.

[0059] The high-performance small unmanned platform DJI Airport 2, equipped with the proprietary Matrice 3TD drone, is used for drone inventory in outdoor yards. It is not only lightweight and easy to deploy, but also has more powerful operating capabilities and cloud-based intelligent functions, greatly reducing the threshold for unmanned operations, operating efficiency and quality.

[0060] The high-performance drone Matrice 3TD is equipped with a 1 / 1.32-inch CMOS, equivalent focal length of 24 mm, a wide-angle camera with 480,000 effective pixels, a 1 / 2-inch CMOS, equivalent focal length of 162 mm, a telephoto camera with 120,000 effective pixels, and an equivalent focal length of 4 mm. Normal mode: 64×512@3fps, super-resolution mode: 128×124@3fps (after turning on the infrared super-resolution function, the aircraft can automatically turn on or off the super-resolution mode according to the ambient light brightness), and an infrared camera with 28x digital zoom, which can intuitively present visible light and thermal imaging images, and is suitable for security, inspection, and inventory operations.

[0061] S3. The edge computing device captures images in the video and sends the images to the trained power material identification model for identification, obtains the quantity and type of the identified power materials, obtains the inventory results, and displays the inventory results in the 3D warehouse electronic map.

[0062] Get sample images of all power materials in the inventory area. Use Labelimg to annotate the targets in the sample images to obtain annotated data sets. Find the model configuration file in the models folder of YOLO, set the number of classes in the model configuration file to the preset parameters, set the total rounds of model training to n, and set the batch size to m. Find the coco.yaml file in the data folder of YOLO, name it a custom data configuration file, open the custom data configuration file, modify the paths of train and val, and list the category names of all power materials in name. After the settings are completed, click to start training. After the training is completed, an exp folder is generated in the runs folder. The exp folder includes the training results and weights, and a trained power material recognition model is obtained.

[0063] S4. After the inventory is completed, the drone flies back to the initial location.

[0064] In some embodiments, the electronic map-based power material drone inventory system may include multiple functional modules composed of computer program segments. The computer program of each program segment in the electronic map-based power material drone inventory system may be stored in the memory of a computer device and executed by at least one processor to execute (see Figure 1 Description) The function of drone inventory of power materials based on electronic maps.

[0065] In this embodiment, the electronic map-based power material drone inventory system can be divided into multiple functional modules according to the functions it performs, such as Figure 2 As shown. The functional modules of the system 200 may include: an inventory task issuing module 210, an inventory starting module 220, an inventory identification module 230, and an inventory ending module 240. The module referred to in the present invention refers to a series of computer program segments that can be executed by at least one processor and can complete fixed functions, which are stored in a memory. In this embodiment, the functions of each module will be described in detail in subsequent embodiments.

[0066] The inventory task issuing module is used to issue drone inventory tasks using handheld devices. The inventory server starts live broadcast of the drone's perspective in the pre-established 3D warehouse electronic map; The inventory start module is used for the drone to start from the initial location and fly based on the preset inventory route. During the flight, the drone takes real-time photos of the power materials in the inventory area and sends the captured videos to the edge computing device. The inventory server then displays the drone’s captured images in real time on the pre-established 3D warehouse electronic map. The inventory recognition module is used for edge computing devices to capture images in the video, send the images to the trained power material recognition model for recognition, obtain the quantity and type of the recognized power materials, obtain the inventory results, and display the inventory results in the 3D warehouse electronic map; The inventory completion module is used to make the drone fly back to the initial location after the inventory is completed.

[0067] Optionally, as an embodiment of the present invention, the inventory task issuing module includes: When the drone initiates an inventory, it creates and starts an inventory task through the PDA. The electronic map sends the route task and transmits it to the cloud API through MQTT data.

[0068] Optionally, as an embodiment of the present invention, the inventory start module includes: Video transmission unit: the video stream shot by the drone is transmitted to the edge computing device using the RTMP protocol; Video receiving unit,The edge computing device receives the video stream through the streaming interface.

[0069] Optionally, as an embodiment of the present invention, it further includes: a model building unit, including: An image acquisition unit, used to acquire sample images of all power materials in the inventory area; A data labeling unit, used to label the target in the sample image using Labelimg to obtain a labeled data set; The first setting unit is used to find the model configuration file in the models folder of YOLO, set the number of classes in the model configuration file to the preset parameters, set the total rounds of model training to n, and set the batch size to m; The second setting unit is used to find the coco.yaml file in the YOLO data folder and name it as a custom data configuration file, open the custom data configuration file, modify the paths of train and val, and list the category names of all power materials in name; The model generation unit is used to start training after the settings are completed. After the training is completed, an exp folder is generated under the runs folder. The exp folder includes the training results and weights to obtain a trained power material identification model.

[0070] Figure 3 The present invention provides a schematic diagram of the structure of a terminal 300 provided in an embodiment of the present invention. The terminal 300 can be used to execute the electric power material drone inventory method based on electronic map provided in an embodiment of the present invention.

[0071] The terminal 300 may include: a processor 310, a memory 320 and a communication unit 330. These components communicate via one or more buses. Those skilled in the art will appreciate that the server structure shown in the figure does not limit the present invention, and it may be a bus structure or a star structure, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0072] The memory 320 can be used to store the execution instructions of the processor 310, and the memory 320 can be implemented by any type of volatile or non-volatile storage terminal or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. When the execution instructions in the memory 320 are executed by the processor 310, the terminal 300 can perform some or all of the steps in the following method embodiments.

[0073] The processor 310 is the control center of the storage terminal, and uses various interfaces and lines to connect various parts of the entire electronic terminal. It runs or executes software programs and / or modules stored in the memory 320, and calls data stored in the memory to perform various functions of the electronic terminal and / or process data. The processor can be composed of an integrated circuit (IC), for example, it can be composed of a single packaged IC, or it can be composed of multiple packaged ICs with the same or different functions. For example, the processor 310 can only include a central processing unit (CPU). In the embodiment of the present invention, the CPU can be a single computing core or multiple computing cores.

[0074] The communication unit 330 is used to establish a communication channel so that the storage terminal can communicate with other terminals, receive user data sent by other terminals or send user data to other terminals.

[0075] The present invention also provides a computer storage medium, wherein the computer storage medium may store a program, and when the program is executed, the program may include some or all of the steps in each embodiment provided by the present invention. The storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM).

[0076] Therefore, the present invention first issues the drone inventory task through a handheld device, and realizes live broadcast of the screen in combination with the pre-established 3D warehouse electronic map, which not only enhances the convenience of task issuance, but also provides an intuitive real-time monitoring interface for operators, and improves the transparency and operation efficiency of the inventory process. Secondly, by flying the drone along the preset route and shooting the power materials in the inventory area in real time, it can greatly reduce manual participation, reduce the workload and error rate of manual inventory. The method of transmitting the video stream to the edge computing device using the RTMP protocol and receiving the video stream through the streaming media interface ensures the efficiency and stability of data transmission, and adapts to the needs of large-scale power material inventory. In addition, the edge computing device uses the trained power material recognition model to process the video image in real time, which can quickly and accurately identify the quantity and type of materials, and display the inventory results directly in the 3D warehouse electronic map. This recognition method combined with the YOLO deep learning algorithm greatly improves the accuracy and reliability of recognition through precise image segmentation and high-confidence detection, and significantly reduces the data processing time. Finally, the drone automatically returns to the initial location, which not only reflects the intelligence of the system, but also ensures the safe recovery of the drone. This method integrates multiple advanced technologies such as drone technology, edge computing, deep learning, and 3D electronic maps, providing a comprehensive, intelligent, and low-cost solution for the efficient inventory of power materials. It has broad application prospects. The technical effects that can be achieved in this embodiment can be found in the description above and will not be repeated here.

[0077] Those skilled in the art can clearly understand that the technology in the embodiments of the present invention can be implemented by means of software plus a necessary general hardware platform. Based on this understanding, the technical solution in the embodiments of the present invention, in essence or in other words, the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, and other media that can store program codes, including several instructions for enabling a computer terminal (which can be a personal computer, a server, or a second terminal, a network terminal, etc.) to execute all or part of the steps of the methods described in each embodiment of the present invention.

[0078] In this specification, the same or similar parts between the various embodiments can be referred to each other. In particular, for the terminal embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the description in the method embodiment.

[0079] In the several embodiments provided by the present invention, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are only schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of systems or modules, which can be electrical, mechanical or other forms.

[0080] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0081] In addition, each functional module in each embodiment of the present invention may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0082] Although the present invention has been described in detail by referring to the accompanying drawings and in combination with the preferred embodiments, the present invention is not limited thereto. Without departing from the spirit and essence of the present invention, a person of ordinary skill in the art may make various equivalent modifications or substitutions to the embodiments of the present invention, and these modifications or substitutions shall be within the scope of the present invention. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed by the present invention, and all of these shall be within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be subject to the scope of protection of the claims.

Claims

1. A method for inventorying electric power materials by drone based on electronic maps, characterized in that: include: Use a handheld device to issue a drone inventory task, and the inventory server starts live broadcasting of the drone's perspective in the pre-established 3D warehouse electronic map; The drone starts from the initial location and flies based on the preset inventory route. During the flight, it takes real-time photos of the power materials in the inventory area and sends the video to the edge computing device. The inventory server displays the drone's footage in real time on the pre-established 3D warehouse electronic map. The edge computing device captures images in the video and sends them to the trained power material recognition model for recognition, obtains the quantity and type of the recognized power materials, obtains the inventory results, and displays the inventory results in the 3D warehouse electronic map; After the inventory is completed, the drone flies back to the initial location.

2. The method according to claim 1, characterized in that Use a handheld device to issue a drone inventory task, and the inventory server will start live broadcasting of the drone's perspective in the pre-established 3D warehouse electronic map, including: When the drone initiates an inventory, it creates and starts an inventory task through the PDA. The electronic map sends the route task and transmits it to the cloud API through MQTT data.

3. The method according to claim 1, characterized in that During the flight, the power supplies in the inventory area are photographed in real time and the video is sent to the edge computing device, including: The video stream shot by the drone is transmitted to the edge computing device using the RTMP protocol; The edge computing device receives the video stream through the streaming interface.

4. The method according to claim 1, characterized in that The method for establishing the power material identification model includes: Obtain sample images of all power supplies in the inventory area; Use Labelimg to label the targets in the sample image to obtain a labeled data set; Find the model configuration file in the YOLO models folder, set the number of classes in the model configuration file to the preset parameters, set the total number of rounds of model training to n, and set the batch size to m; Find the coco.yaml file in the YOLO data folder and name it as a custom data configuration file. Open the custom data configuration file, modify the paths of train and val, and list the category names of all power materials in name. After the settings are completed, click to start training. After the training is completed, an exp folder is generated under the runs folder. The exp folder includes the training results and weights, and a trained power material recognition model is obtained.

5. An electronic map-based power material drone inventory system, characterized in that: include: The inventory task issuing module is used to issue drone inventory tasks using handheld devices. The inventory server starts live broadcast of the drone's perspective in the pre-established 3D warehouse electronic map; The inventory start module is used for the drone to start from the initial location and fly based on the preset inventory route. During the flight, the drone takes real-time photos of the power materials in the inventory area and sends the captured videos to the edge computing device. The inventory server then displays the drone’s captured images in real time on the pre-established 3D warehouse electronic map. The inventory recognition module is used for edge computing devices to capture images in the video, send the images to the trained power material recognition model for recognition, obtain the quantity and type of the recognized power materials, obtain the inventory results, and display the inventory results in the 3D warehouse electronic map; The inventory completion module is used to make the drone fly back to the initial location after the inventory is completed.

6. The system according to claim 1, characterized in that The inventory task issuing module includes: When the drone initiates an inventory, it creates and starts an inventory task through the PDA. The electronic map sends the route task and transmits it to the cloud API through MQTT data.

7. The system according to claim 1, characterized in that The inventory start module includes: Video transmission unit: the video stream shot by the drone is transmitted to the edge computing device using the RTMP protocol; Video receiving unit,The edge computing device receives the video stream through the streaming interface.

8. The system according to claim 1, characterized in that Also includes: The model building unit includes: An image acquisition unit, used to acquire sample images of all power materials in the inventory area; A data labeling unit, used to label the target in the sample image using Labelimg to obtain a labeled data set; The first setting unit is used to find the model configuration file in the models folder of YOLO, set the number of classes in the model configuration file to the preset parameters, set the total rounds of model training to n, and set the batch size to m; The second setting unit is used to find the coco.yaml file in the YOLO data folder and name it as a custom data configuration file, open the custom data configuration file, modify the paths of train and val, and list the category names of all power materials in name; The model generation unit is used to start training after the settings are completed. After the training is completed, an exp folder is generated under the runs folder. The exp folder includes the training results and weights to obtain a trained power material identification model.

9. A terminal, characterized in that: include: A memory, used for storing an electric power material drone inventory program based on an electronic map; A processor is used to implement the steps of the electric power material drone inventory method based on an electronic map as described in any one of claims 1 to 4 when executing the electric power material drone inventory program based on an electronic map.

10. A computer-readable storage medium storing a computer program, characterized in that: The readable storage medium stores an electric power material drone inventory program based on an electronic map, and when the electric power material drone inventory program based on an electronic map is executed by a processor, the steps of the electric power material drone inventory method based on an electronic map as described in any one of claims 1-4 are implemented.

Images