Inspection method and device

Abstract

According to the method, through combined use of an unmanned aerial vehicle and a mechanical dog, the advantages of the unmanned aerial vehicle and the mechanical dog are fully exerted, the inspection efficiency and comprehensiveness are greatly improved, intelligent inspection of different heights and areas of a target area (such as a construction site) is achieved, the accuracy and speed of inspection result judgment are improved, and the inspection efficiency is improved. And the inspection process is more intelligent. Besides, reasonable inspection strategies are respectively formulated for high-altitude inspection objects and low-altitude inspection objects, inspection results are analyzed and reasoned, the safety level is timely and accurately judged, and when potential safety hazards are found, an alarm can be quickly given and inspection alarm information is transmitted back to a control center, so that measures can be conveniently taken in time for processing, and the inspection efficiency is improved. Therefore, the safety of the target area (such as a construction site) is effectively improved, and the possibility of accidents is reduced.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular, to an inspection method and device. Background Art

[0002] In the field of construction engineering, the inspection work on construction sites is crucial for ensuring construction safety, quality, and the smooth progress of projects. Currently, most construction sites still rely on manual inspection methods. However, in large-scale or complex-structured construction projects, many problems have emerged with this traditional method. On the one hand, manual inspection requires a large amount of time and labor costs, and it is difficult to achieve high-frequency and high-coverage inspection tasks. On the other hand, the efficiency of manual inspection is limited by the energy and experience level of personnel, and it is prone to missed inspections, resulting in potential safety hazards not being discovered in a timely manner.

[0003] At the same time, with the development of robot technology, inspection robots are gradually being applied to construction sites. However, the existing robot inspection path planning mainly relies on manual operation. Due to the complex and ever-changing construction site environment, factors such as the continuous change of building structures, frequent material handling, and personnel flow make it difficult for the manually planned robot inspection path to adapt to the dynamically changing construction site scenario. This may cause some areas or risk points to be overlooked, and the inspection quality cannot be effectively guaranteed, and the advantages of robots in efficient and accurate inspection cannot be fully utilized. Therefore, there is an urgent need for an inspection method that can adapt to the complex construction site environment and is efficient and intelligent to solve the deficiencies of the existing technology. Summary of the Invention

[0004] This application provides an inspection method and device, which can formulate reasonable inspection strategies for high-altitude inspection objects and low-altitude inspection objects respectively, analyze and infer the inspection results, accurately judge the safety level in a timely manner, quickly alarm when a safety hazard is found, and transmit the inspection alarm information back, facilitating timely measures to be taken for handling, thereby effectively improving the safety of the target area (such as a construction site) and reducing the possibility of accidents.

[0005] In a first aspect, this application provides an inspection method, and the method includes:

[0006] Obtain an orthophoto map and determine the target area image in the orthophoto map;

[0007] Classify the objects in the target area image to obtain high-altitude inspection objects and low-altitude inspection objects;

[0008] Determine the high-altitude inspection route according to the northeast celestial coordinates of the high-altitude inspection object and the northeast celestial coordinates of the flight starting point of the unmanned aerial vehicle; and use the unmanned aerial vehicle to perform inspections along the high-altitude inspection route to obtain the inspection results of the high-altitude inspection route;

[0009] Determine the low-altitude inspection area and the inspection sequence of the low-altitude inspection area according to the image mask of the low-altitude inspection object and the northeast celestial coordinates of the starting position of the robotic dog; and, use the robotic dog and the drone to perform inspections based on the low-altitude inspection area and the inspection sequence of the low-altitude inspection area to obtain the inspection results of the low-altitude inspection route;

[0010] Determine the inspection alarm information corresponding to the target area image according to the inspection results of the high-altitude inspection route and the inspection results of the low-altitude inspection route.

[0011] In a second aspect, the present application provides an inspection device, and the device includes:

[0012] A first unit, configured to obtain an orthophoto map and determine a target area image in the orthophoto map;

[0013] A second unit, configured to classify the objects in the target area image to obtain high-altitude inspection objects and low-altitude inspection objects;

[0014] A third unit, configured to determine a high-altitude inspection route according to the northeast celestial coordinates of the high-altitude inspection object and the northeast celestial coordinates of the flight starting point of the drone; and, use the drone to perform inspections along the high-altitude inspection route to obtain the inspection results of the high-altitude inspection route;

[0015] A fourth unit, configured to determine the low-altitude inspection area and the inspection sequence of the low-altitude inspection area according to the image mask of the low-altitude inspection object and the northeast celestial coordinates of the starting position of the robotic dog; and, use the robotic dog and the drone to perform inspections based on the low-altitude inspection area and the inspection sequence of the low-altitude inspection area to obtain the inspection results of the low-altitude inspection route;

[0016] A fifth unit, configured to determine the inspection alarm information corresponding to the target area image according to the inspection results of the high-altitude inspection route and the inspection results of the low-altitude inspection route.

[0017] In a third aspect, the present application provides a readable medium, including execution instructions, and when a processor of an electronic device executes the execution instructions, the electronic device executes the method described in any one of the first aspects.

[0018] In a fourth aspect, the present application provides an electronic device, including a processor and a memory storing execution instructions, and when the processor executes the execution instructions stored in the memory, the processor executes the method described in any one of the first aspects.

[0019] As can be seen from the above technical solutions, the method provided by this application can classify the objects in the target area image to obtain high-altitude inspection objects and low-altitude inspection objects, and adopt different inspection methods for high-altitude inspection objects and low-altitude inspection objects respectively. For example, use a drone to inspect high-altitude inspection objects along the high-altitude inspection route, and use a robotic dog and a drone to inspect low-altitude inspection objects based on the low-altitude inspection area and the inspection sequence of the low-altitude inspection area, and can determine the inspection alarm information corresponding to the target area image according to the inspection results of the high-altitude inspection route and the inspection results of the low-altitude inspection route. In this way, through the combined use of drones and robotic dogs, the advantages of both are fully utilized. The drone can conduct a comprehensive inspection of high-altitude inspection objects at high altitude, quickly cover a large area, and obtain overall target area (such as a construction site) information; the robotic dog can move flexibly in the low-altitude area, cooperate with the drone, and conduct a detailed inspection of low-altitude inspection objects, greatly improving the efficiency and comprehensiveness of the inspection, and realizing intelligent inspection of different heights and areas of the target area (such as a construction site). This application can automatically process and analyze the inspection results, reduce manual intervention, improve the accuracy and speed of judging the inspection results, and make the inspection process more intelligent. In addition, the combined inspection method of drones and robotic dogs can better cope with the complex environment where the objects in the target area (such as buildings, materials, and personnel in a construction site) change dynamically and frequently. The robotic dog can autonomously navigate to a suitable inspection point to release the drone according to the actual situation on site, and the drone can autonomously plan the inspection times and routes according to the real-time situation in the inspection area (such as the target position and its own battery power), avoiding inspection loopholes caused by environmental changes, effectively improving the inspection quality, and ensuring the timely discovery and handling of potential risks. And this application formulates reasonable inspection strategies for high-altitude inspection objects and low-altitude inspection objects respectively, analyzes and reasons the inspection results, accurately judges the safety level in a timely manner, can quickly alarm when discovering potential safety hazards, and transmit the inspection alarm information back to the control center for timely measures to be taken for handling, thus effectively improving the safety of the target area (such as a construction site) and reducing the possibility of accidents.

[0020] The further effects of the above non-conventional preferred methods will be described in conjunction with specific embodiments below. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the embodiments of the present application or the existing technical solutions, the following will briefly introduce the drawings required for the description of the embodiments or the existing technical solutions. Obviously, the drawings described below are only some embodiments recorded in the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0022] Figure 1 A schematic flow chart of an inspection method provided by an embodiment of the present application;

[0023] Figure 2 An orthophoto map of a target area provided by an embodiment of the present application;

[0024] Figure 3 A schematic structural diagram of an inspection device provided by an embodiment of the present application;

[0025] Figure 4 A schematic structural diagram of an electronic device provided by an embodiment of the present application. Detailed implementation manners

[0026] To make the objectives, technical solutions and advantages of the present application clearer, the technical solutions of the present application will be clearly and completely described below in conjunction with specific embodiments and the corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0027] The following will describe in detail various non-restrictive implementation manners of the present application in conjunction with the drawings.

[0028] Through research by the inventors, it is found that in the field of construction engineering, the inspection work on construction sites is crucial for ensuring construction safety, quality, and the smooth progress of projects. Currently, most construction sites still rely on manual inspection methods. However, in large-scale or complex-structured construction projects, many problems have emerged in this traditional method. On the one hand, manual inspection requires a large amount of time and labor costs and is difficult to achieve high-frequency and high-coverage inspection tasks. On the other hand, the efficiency of manual inspection is limited by the energy and experience levels of personnel, and it is easy to miss inspections, resulting in potential safety hazards not being discovered in a timely manner.

[0029] At the same time, with the development of robot technology, inspection robots have gradually been applied to construction sites. However, the existing robot inspection path planning mainly relies on manual operation. Due to the complex and ever-changing construction site environment, factors such as the continuous change of building structures, frequent material handling, and personnel flow make it difficult for the manually planned robot inspection paths to adapt to the dynamically changing construction site scenarios. This may cause some areas or risk points to be overlooked, and it is difficult to effectively guarantee the inspection quality, and the advantages of robots in efficient and accurate inspection cannot be fully utilized. Therefore, there is an urgent need for an inspection method that can adapt to the complex construction site environment and is efficient and intelligent to solve the deficiencies of the existing technology.

[0030] To solve the above-mentioned problems of the existing technology, the present application provides an inspection method. See Figure 1, which shows an inspection method in an embodiment of the present application. In this embodiment, the method may, for example, include the following steps:

[0031] S101: Obtain an orthophoto map and determine the target area image in the orthophoto map.

[0032] In the present application, an orthophoto map including the target area can be obtained first. Specifically, a drone can be used to collect several area images of the target area, and then, the several area images of the target area can be stitched into an orthophoto map. For example, the orthophoto map (including altitude and longitude and latitude) can be stitched from several area images corresponding to the target area using software, and at the same time, a PNG image can be exported, where the pixel coordinates in the PNG picture correspond one-to-one with the longitude and latitude of the orthophoto map. As Figure 2 shown, Figure 2 is an orthophoto map including the target area when the target area is a construction site.

[0033] After obtaining the orthophoto map, the target area image in the orthophoto map can be determined. Specifically, the orthophoto map can be input into a trained regional semantic segmentation model first to obtain a semantic segmentation result; wherein, the semantic segmentation result can include each object in the orthophoto map and the height of each object from the ground in the orthophoto map. In one implementation, a vision-based model can be used as a feature extractor, and construction site picture data can be collected to train the model to form a regional semantic segmentation model. For example, on the basis of Ground SAM, the construction site picture dataset can be fine-tuned to form a regional semantic segmentation model that can segment the objects (i.e., objects) on the construction site (i.e., the target area). That is to say, the input of the regional semantic segmentation model is a picture, and the output is an image mask of specific category objects in the picture. It can be understood that the image mask is an element in the semantic segmentation result.

[0034] Then, according to the semantic segmentation result, the boundary between the target area and the non-target area in the orthophoto map can be determined. Next, according to the boundary, the target area image corresponding to the target area in the orthophoto map can be determined.

[0035] As an example, assuming that the target area is a construction site, the present application can use a drone to collect construction site area images and stitch them into an orthophoto map, use the regional semantic segmentation model to find the boundary between the construction site area and the non-construction site area, and calculate the geodetic coordinate system coordinates of the construction site. Input the orthophoto map into the regional semantic segmentation model, and classify the construction site pictures according to the semantic segmentation result.

[0036] In this embodiment, after obtaining the semantic segmentation result, the northeast celestial coordinates of the object in the target region image can also be determined. It should be noted that the northeast celestial coordinates of the object are the coordinates of the object in the northeast celestial coordinate system. The local-level coordinate system (or user coordinate system or northeast celestial coordinate system) is a coordinate system with the user's location as the coordinate origin, and the coordinate axes are respectively perpendicular to the east, north, and sky directions. It can be transformed into the geocentric rectangular coordinate system through translation and rotation. The east, north, and sky coordinates of the local-level coordinate system are represented by e, n, and u respectively. For the geocentric rectangular coordinate system, its origin is the centroid of the earth, the x-axis extends through the intersection of the prime meridian (0 degrees longitude) and the equator (0 degrees latitude). The z-axis extends through the North Pole (i.e., coincides with the earth's rotation axis). The rectangular coordinates x, y, z are represented by x, y, z respectively. For the geodetic coordinate system, the geodetic latitude, geodetic longitude, and elevation in the longitude and latitude coordinates are represented by ψ, λ, h respectively.

[0037] In this embodiment, the method for determining the northeast celestial coordinates of the object in the target region image is as follows:

[0038] Assume that the longitude and latitude coordinates of the camera for the UAV to collect images are (ψ 0 , λ 0 , h 0 ), and the geocentric rectangular coordinate system coordinates (x 0 , y 0 , z 0 ) of the camera can be obtained by the following formula (1):

[0039]

[0040] Where:

[0041] f is the flattening of the reference ellipsoid, and a is the semi-major axis of the reference ellipsoid.

[0042] For any point in the orthophoto, if the longitude and latitude coordinates (ψ 1 , λ 1 , h 1 ) correspond to the geocentric rectangular coordinate system coordinates (x 1 , y 1 , z 1 ) obtained by formula (1), then the northeast celestial coordinates (e 0 , n 0 , u 0 ) of this point with the geocentric rectangular coordinate system of the camera (x 1 , y 1 , z 1 ) as the origin can be obtained by the following formula (2):

[0043]

[0044] Where:

[0045]

[0046] S102: Classify the objects in the target area image to obtain high-altitude inspection objects and low-altitude inspection objects.

[0047] In this embodiment, the semantic segmentation result includes the ground height of each object in the orthophoto. Objects in the target area image with a ground height greater than a preset height threshold can be used as high-altitude inspection objects, and objects in the target area image with a ground height less than or equal to the preset height threshold can be used as low-altitude inspection objects. For example, inspection objects with a ground height higher than 6 meters, such as buildings, tower cranes, etc., can be used as high-altitude inspection objects; inspection objects with a ground height lower than or equal to 6 meters, such as protective fences, material stacking areas, etc., can be used as low-altitude inspection objects.

[0048] It can be understood that after the orthophoto image is segmented, a mask for each category is obtained. The objects in the target area image can be divided according to the ground height into:

[0049] High-altitude inspection object set: where N F is the number of high-altitude inspection objects.

[0050] High-altitude inspection object mask set: is the image mask set of the i-th object.

[0051] High-altitude inspection coordinate set: where is the northeast celestial coordinate corresponding to the centroid of the orthophoto image of the inspection object of the inspection object.

[0052] Low-altitude inspection object set: where, N U is the number of low-altitude inspection objects.

[0053] Low-altitude inspection object mask set: is the image mask set of the i-th inspection object.

[0054] S103: Determine the high-altitude inspection route according to the northeast celestial coordinates of the high-altitude inspection object and the northeast celestial coordinates of the flight starting point of the unmanned aerial vehicle; and use the unmanned aerial vehicle to perform inspections along the high-altitude inspection route to obtain the inspection results of the high-altitude inspection route.

[0055] In this embodiment, for the area of the high-altitude inspection object, the UAV inspection path can be planned by an algorithm and then regular inspections can be carried out. First, the high-altitude inspection route can be determined according to the northeast celestial coordinates of the high-altitude inspection object and the northeast celestial coordinates of the preset UAV flight starting point. As an example, a first extended inspection coordinate set can be constructed based on the northeast celestial coordinates of the high-altitude inspection object and the northeast celestial coordinates of the UAV flight starting point.

[0056] Then, an optimization function can be established based on the first extended inspection coordinate set; wherein, the optimization function aims to minimize the total inspection path, and the constraint conditions of the optimization function include that each high-altitude inspection object is only visited once, starting from the flight starting point and returning to the flight starting point, and ensuring the effectiveness of the path. For example, the optimization function is as

[0057]

[0058] wherein, c ij represents the distance between the northeast celestial coordinates of the i-th and j-th high-altitude inspection objects; x ij is an integer variable of 0 or 1, and when it is taken as 1, it means that the edge is in the obtained solution; N is the number of high-altitude inspection objects; P F is the high-altitude inspection coordinate set, S is a subset of P F ; |S| is the number of elements in S.

[0059] Next, the optimization function can be solved to obtain the high-altitude inspection target sequence corresponding to the high-altitude inspection object; the high-altitude inspection target sequence can be understood as the inspection order of the high-altitude inspection object. Immediately afterwards, taking the UAV flight starting point as the starting point, the high-altitude inspection target sequence corresponding to the high-altitude inspection object can be rearranged to obtain a rearranged high-altitude inspection target sequence; and for each high-altitude inspection object in the rearranged high-altitude inspection target sequence, a path covering the masked area of the high-altitude inspection object is generated; according to the paths of each high-altitude inspection object in the rearranged high-altitude inspection target sequence, the high-altitude inspection route is determined, that is, according to the paths of each high-altitude inspection object in the rearranged high-altitude inspection target sequence, the paths of all high-altitude inspection objects in the rearranged high-altitude inspection target sequence are arranged in the order of the rearranged high-altitude inspection target sequence to obtain the high-altitude inspection route. Finally, the UAV can be used to perform inspections along the high-altitude inspection route to obtain the inspection results of the high-altitude inspection route. Among them, the inspection results include inspection position information and inspection pictures.

[0060] Taking the target area as the construction site area as an example, for the construction site area of the high-altitude inspection object, after using the algorithm to plan the UAV inspection path, regular inspections are carried out. For the high-altitude inspection target (i.e., the high-altitude inspection object), construct an extended inspection coordinate set: where is the northeast celestial coordinate of the UAV flight starting point. Establish the following optimization function (derived from the traveling salesman problem, given a series of inspection targets and the distances between each pair of inspection targets, solve for the shortest circuit that visits each inspection target once and returns to the starting inspection target), with the goal of the shortest total inspection path, and find the inspection order:

[0061]

[0062] where c ij represents the distance between the northeast celestial coordinates of the i-th and j-th high-altitude inspection objects; x ij is an integer variable of 0 or 1, and when it is taken as 1, it means the edge is in the obtained solution; N is the number of high-altitude inspection objects; P F is the high-altitude inspection coordinate set, S is a subset of P F ; |S| is the number of elements in S. Starting from the UAV starting point, along the edges formed by the solution, form a re-ordered high-altitude inspection target, and its subscripts are

[0063] For each re-ordered high-altitude inspection target generate a path W i to cover the masked area of the high-altitude inspection target The entire inspection path is After obtaining the inspection path, the UAV automatically executes the flight in the inspection area to obtain inspection results such as inspection pictures,

[0064] S104: Determine the low-altitude inspection area and the inspection order of the low-altitude inspection area according to the image mask of the low-altitude inspection object and the northeast celestial coordinate of the starting position of the robotic dog; and, use the robotic dog and the UAV to perform inspections based on the low-altitude inspection area and the inspection order of the low-altitude inspection area to obtain the inspection results of the low-altitude inspection route.

[0065] For low-altitude inspection objects, based on the obtained set of low-altitude inspection objects, a mechanical dog can be used to carry a drone for joint inspection. Plan the two-dimensional inspection route of the mechanical dog according to the inspection location and divide the safe inspection area of the drone; the mechanical dog autonomously navigates to the inspection point and releases the drone; the drone autonomously plans the number of inspections and the path according to the targets and power constraints within the inspection area, and transmits the inspection pictures back to the mechanical dog. That is to say, for low-altitude inspection objects, joint inspection is adopted: the mechanical dog serves as the drone carrier and can supply power to the drone; it can be understood that the drone provides navigation planning for the mechanical dog and can autonomously inspect and locate within the construction site.

[0066] Specifically, the low-altitude inspection area and the inspection order of the low-altitude inspection area can be determined first according to the image mask of the low-altitude inspection object and the northeast celestial coordinates of the starting position of the mechanical dog.

[0067] As an example, the low-altitude inspection object can be processed to determine the mask center corresponding to the low-altitude inspection object, and, taking the mask center corresponding to the low-altitude inspection object as a sample, use the Euclidean clustering algorithm for clustering to obtain the low-altitude inspection clustering. Determine the two-dimensional minimum convex hull of the low-altitude inspection objects in each low-altitude inspection clustering, determine the low-altitude inspection area according to the two-dimensional minimum convex hull, and use the center of the two-dimensional minimum convex hull as the northeast celestial coordinates of the low-altitude inspection area. That is to say, for the set of low-altitude inspection objects: U=(u 1 , …, u i , …, u M ), find the center of the mask M U corresponding to the low-altitude inspection object, and take the center as a sample, use the Euclidean clustering algorithm for clustering to obtain the low-altitude inspection clustering C={c 1 ,..., c i ,..., c K}, where K is the number of clusters, find the two-dimensional minimum convex hull of the low-altitude inspection objects in each cluster, and use the convex hull center as the northeast celestial coordinates of the low-altitude inspection area. Among them, the two-dimensional minimum convex hull is: if a two-dimensional polygon is convex, then any line segment connected by two points on its side falls inside the surface. Given the inspection clustering c i , the set with the smallest area that contains all convex points is called the two-dimensional minimum convex hull of the low-altitude inspection object.

[0068] Construct a second extended inspection coordinate set based on the northeast celestial coordinates of the low-altitude inspection area and the northeast celestial coordinates of the starting position of the robotic dog. Based on the second extended inspection coordinate set, establish an optimization function; wherein, the optimization function aims to minimize the total inspection path, and the constraint conditions of the optimization function include that each low-altitude inspection object is only visited once, starting from the starting position and returning to the starting position, and ensuring the validity of the path. Solve the optimization function to obtain the inspection order of the low-altitude inspection area. It can be understood that the northeast celestial coordinates p of the starting position of the robotic dog D , and the northeast celestial coordinates of the low-altitude inspection area are According to formula (3), determine the low-altitude inspection area and the inspection order of the low-altitude inspection area.

[0069] After determining the low-altitude inspection area and the inspection order of the low-altitude inspection area, the robotic dog and the drone can be used to conduct inspections based on the low-altitude inspection area and the inspection order of the low-altitude inspection area to obtain the inspection results of the low-altitude inspection route.

[0070] As an example, the robotic dog sequentially travels to the low-altitude inspection area according to the inspection order of the low-altitude inspection area and avoids obstacles autonomously according to the on-site environment. Among them, the condition for the robotic dog to reach the low-altitude inspection area is that the distance between the current position of the robotic dog and the central coordinates of the low-altitude inspection area is less than the set distance threshold. It can be understood that the robotic dog can avoid obstacles autonomously according to the on-site environment and conduct inspections on the low-altitude inspection area in sequence according to the obtained inspection order. The condition for reaching the inspection area is:

[0071]

[0072] Among them, is the northeast celestial coordinate when inspecting the i-th low-altitude inspection area, is the central coordinate of the i-th low-altitude inspection area, and D th is the distance threshold.

[0073] For each low-altitude inspection area, use the drone to determine the order of the low-altitude inspection objects in the low-altitude inspection area. Among them, using the drone to determine the order of the low-altitude inspection objects in the low-altitude inspection area includes: constructing a third extended inspection coordinate set based on the northeast celestial coordinates of the low-altitude inspection objects and the flight starting point of the drone in the low-altitude inspection area; establishing an optimization function based on the third extended inspection coordinate set; where the optimization function aims to minimize the total inspection path, and the constraint conditions of the optimization function include that each low-altitude inspection object in the low-altitude inspection area is only visited once, starting from the flight starting point and returning to the flight starting point, and ensuring the effectiveness of the path; solving the optimization function to obtain the order of the low-altitude inspection objects in the low-altitude inspection area. It can be understood that assuming the i-th low-altitude inspection area c i , the set of northeast celestial coordinates of the low-altitude inspection objects it contains is Among them, is the northeast celestial coordinate corresponding to the mask center of the low-altitude inspection object. Calculate the order of the low-altitude inspection objects in the low-altitude inspection area according to formula (3) for P U

[0074] After using the drone to determine the order of the low-altitude inspection objects in the low-altitude inspection area, the mechanical dog and the drone can be used to inspect the low-altitude inspection objects in the low-altitude inspection area based on the order of the low-altitude inspection objects in the low-altitude inspection area to obtain the inspection result of the low-altitude inspection area. That is, the mechanical dog and the drone are used in cooperation to inspect the low-altitude inspection objects in the low-altitude inspection area to obtain the inspection result of the low-altitude inspection area.

[0075] Among them, the method of using the drone to inspect the low-altitude inspection objects in the low-altitude inspection area based on the order of the low-altitude inspection objects in the low-altitude inspection area to obtain the inspection result of the low-altitude inspection area may include the following steps. For each low-altitude inspection object in the low-altitude inspection area, generate a coverage path at a preset height from the highest point of the low-altitude inspection object, calculate the distance of the coverage path, and determine the starting point and ending point of the coverage path. The total inspection distance within the low-altitude inspection area can be calculated according to the distances of the coverage paths corresponding to each low-altitude inspection object. The inspection speed of the drone can be determined according to the total inspection distance within the low-altitude inspection area, the starting and ending points of each coverage path, and the self-power and flight energy consumption of the drone; control the drone to inspect the low-altitude inspection objects in the low-altitude inspection area based on the inspection speed and the order of the low-altitude inspection objects in the low-altitude inspection area to obtain the inspection result of the low-altitude inspection area. It can be understood that the inspection objects in the cluster (i.e., the low-altitude inspection area) c i can be sorted in this order. After sorting, the cluster​ Among them, o j ∈U represents the j-th inspection object starting in the i-th low-altitude inspection area, and N i is the number of objects in the i-th cluster. Each low-altitude inspection object O j , at a distance from O j at the highest point H, generates a coverage path, and calculates the distance of its coverage path as d j , and records the starting point and ending point of the coverage path as Assume that the takeoff coordinate of the UAV is s, then the inspection distance within the i-th low-altitude inspection area

[0076] is:

[0077]

[0078] The total distance can be obtained as follows:

[0079]

[0080] Then, when the UAV patrols at a speed of V according to the patrol order within the obtained low-altitude inspection area, its constraint condition is: Among them, E is the battery capacity of the UAV, P(V) is the power of the UAV, is the flight time.

[0081] Finally, the inspection results of all low-altitude inspection areas can be used as the inspection results of the low-altitude inspection route.

[0082] S105: Determine the inspection alarm information corresponding to the target area image according to the inspection results of the high-altitude inspection route and the inspection results of the low-altitude inspection route

[0083] In this embodiment, the inspection results include inspection location information and inspection pictures. Specifically, the inspection results of the high-altitude inspection route and the inspection results of the low-altitude inspection route can be input into a pre-trained vision-language large model first to obtain the text descriptions of the inspection results of each inspection result; the text descriptions of the inspection results of each inspection result are respectively input into a pre-trained safety large model to obtain the safety levels of each inspection result. Among them, the generation process of the safety large model includes: obtaining the construction site safety management regulations and constructing a safety rule knowledge graph based on the construction site safety management regulations; using the safety rule knowledge graph as the retrieval enhancement of a preset large language model to form the safety large model; it should be emphasized that the safety large model belongs to the Think-on-Graph reasoning paradigm, and according to the picture description, the safety large model can search and reason step by step on the entity nodes of the knowledge graph until the final answer is deduced; the safety large model participates in every step of the reasoning of the knowledge graph in person to achieve information complementarity, thereby improving the overall reasoning ability. The generation process of the vision-language large model includes: establishing the language description of the picture data set of the target area and using this data set to fine-tune the vision-language large model to form the vision-language large model; for example, after giving the language description of the construction site image, fine-tune the vision-language large model BLIP2 to form the construction site vision-language large model (that is, given a picture, output the description of the construction site picture).

[0084] If the safety level of the inspection result meets the preset risk condition, the inspection result and the safety risk information of the inspection result are used as the inspection information to be alarmed; all the inspection information to be alarmed is used as the inspection alarm information corresponding to the target area image.

[0085] It is understandable that when the target area is a construction site, the visual-language large model of the construction site can be used to obtain the text description of the inspection pictures in the inspection results and input it into the safety large model of the construction site located in the control center. If an alarm is required, save the location of the inspection pictures, alarm information, original pictures, and detected pictures in the inspection results. For the construction site area of the high-altitude inspection object, the inspection path of the drone can be planned using an algorithm and then regular inspections can be carried out; use the visual-language large model of the construction site to describe the inspection pictures in text, and use the safety large model to determine whether to generate an alarm based on the inferred safety level. For the low-altitude inspection object, according to the obtained set of low-altitude inspection objects, a mechanical dog can be used to carry a drone for joint inspections; plan the two-dimensional inspection route of the mechanical dog according to the inspection location and divide the safe inspection area of the drone; the mechanical dog autonomously navigates to the inspection point and releases the drone; the drone autonomously plans the number of inspections and the path according to the targets and power constraints within the inspection area and transmits the inspection results including the inspection pictures back to the mechanical dog. The mechanical dog uses the safety large model on the edge side to infer the safety level of the inspection pictures and decides whether to generate an alarm; the mechanical dog can pack the inspection pictures that need to be alarmed in the format of location, alarm information, original pictures, and detected pictures and transmit them back to the control center.

[0086] As can be seen from the above technical solution, the method provided by this application can classify the objects in the target area image to obtain high-altitude inspection objects and low-altitude inspection objects, and adopt different inspection methods for high-altitude inspection objects and low-altitude inspection objects respectively. For example, use a drone to inspect high-altitude inspection objects along the high-altitude inspection route, and use a robotic dog and a drone to inspect low-altitude inspection objects based on the low-altitude inspection area and the inspection sequence of the low-altitude inspection area, and can determine the inspection alarm information corresponding to the target area image according to the inspection results of the high-altitude inspection route and the inspection results of the low-altitude inspection route. In this way, through the combined use of drones and robotic dogs, the advantages of both are fully utilized. The drone can conduct a comprehensive inspection of high-altitude inspection objects at high altitude, quickly cover a large area, and obtain the overall information of the target area (such as a construction site); the robotic dog can move flexibly in the low-altitude area, cooperate with the drone, and conduct a detailed inspection of low-altitude inspection objects, greatly improving the efficiency and comprehensiveness of the inspection, and realizing intelligent inspection of different heights and areas of the target area (such as a construction site). By using technical means such as a construction site semantic segmentation model, a vision-language large model, and a safety large model, the inspection pictures can be automatically processed and analyzed, reducing manual intervention, improving the accuracy and speed of judging inspection results, and making the inspection process more intelligent. In addition, the combined inspection method of drones and robotic dogs can better cope with the complex environment where the objects in the target area (such as buildings, materials, and personnel in a construction site) change dynamically and frequently. The robotic dog can autonomously navigate to a suitable inspection point to release the drone according to the actual situation on site, and the drone can autonomously plan the inspection times and paths according to the real-time situation in the inspection area (such as the target position and its own power), avoiding inspection loopholes caused by environmental changes, effectively improving the inspection quality, and ensuring the timely discovery and handling of potential risks. And this application formulates reasonable inspection strategies for high-altitude inspection objects and low-altitude inspection objects respectively, analyzes and reasons the inspection results, judges the safety level accurately and timely, can quickly alarm when discovering potential safety hazards and transmit the inspection alarm information back to the control center, facilitating timely measures to be taken for handling, thereby effectively improving the safety of the target area (such as a construction site) and reducing the possibility of accidents.

[0087] That is to say, this application proposes a method for jointly inspecting a target area (such as a construction site) based on a drone and a robotic dog. By classifying different inspection objects in the target area (such as a construction site), a reasonable inspection method and inspection path can be arranged for the target area (such as a construction site). The robotic dog, as a drone carrier, can not only perform reasoning on the inspection results as edge computing power, but also supply power to the drone. And the drone can not only conduct inspections on the target area (such as a construction site) in all directions, but also perform autonomous inspection positioning within the target area (such as a construction site).

[0088] It can be understood that the solution provided by this application has the following advantages:

[0089] 1. Efficient and intelligent inspection;

[0090] By jointly using drones and robotic dogs, the advantages of both are fully utilized. Drones can conduct all-round inspections of the construction site at high altitudes, quickly cover large areas, and obtain overall construction site information; robotic dogs can move flexibly in low-altitude areas, cooperate with drones, and conduct detailed inspections on low-altitude inspection objects, greatly improving the efficiency and comprehensiveness of inspections, and realizing intelligent inspections of different heights and areas of the construction site.

[0091] By using technical means such as the construction site semantic segmentation model, vision-language large model, and safety large model, it is possible to automatically process and analyze inspection pictures, reduce manual intervention, improve the accuracy and speed of judging inspection results, and make the inspection process more intelligent.

[0092] 2. Adapt to complex environments;

[0093] The joint inspection method of drones and robotic dogs can better cope with the complex environment in the construction site where buildings, materials, and personnel are dynamically changing frequently. The robotic dog can autonomously navigate to a suitable inspection point to release the drone according to the actual situation on site, and the drone can autonomously plan the inspection times and routes according to the real-time situation in the inspection area (such as the target position and its own battery power), avoiding inspection loopholes caused by environmental changes, effectively improving the inspection quality, and ensuring the timely discovery and handling of potential risks.

[0094] 3. Improve safety;

[0095] Reasonable inspection strategies are formulated for high-altitude and low-altitude inspection objects respectively, and the inspection pictures are analyzed and inferred through a multi-modal large model to accurately judge the safety level in a timely manner. When safety hazards are found, it can quickly alarm and transmit relevant information back to the control center, facilitating timely measures to be taken for handling, thereby effectively improving the safety of the construction site and reducing the possibility of accidents.

[0096] 4. Optimize resource allocation;

[0097] The robotic dog serves as a drone carrier to supply power to the drone, reasonably utilizes the resources of both, extends the endurance of the drone, and reduces the problems of inspection interruption or incompleteness caused by insufficient battery power. At the same time, by reasonably planning the inspection path and sequence, the operating efficiency of the equipment is improved, unnecessary energy consumption and equipment wear are reduced, and the optimization of resource allocation is achieved.

[0098] So far, this embodiment has implemented the processing process of the inspection method in combination with a specific application scenario. Of course, it should be considered that the above scenario is only an exemplary scenario and does not limit the method provided in this application. The method provided in this application can be extended and applied to the processing process of other inspection methods with the same principle.

[0099] As Figure 3 shown, this is a specific embodiment of the inspection device described in this application. The device in this embodiment is the physical device for executing the method described in the above embodiment. Its technical solution is essentially the same as that of the above embodiment, and the corresponding descriptions in the above embodiment also apply to this embodiment. In this embodiment, the device includes:

[0100] The first unit 301 is used to obtain an orthophoto map and determine the target area image in the orthophoto map;

[0101] The second unit 302 is used to classify the objects in the target area image to obtain high-altitude inspection objects and low-altitude inspection objects;

[0102] The third unit 303 is used to determine the high-altitude inspection route according to the northeast celestial coordinates of the high-altitude inspection object and the northeast celestial coordinates of the flight starting point of the unmanned aerial vehicle; and, use the unmanned aerial vehicle to perform inspections along the high-altitude inspection route to obtain the inspection results of the high-altitude inspection route;

[0103] The fourth unit 304 is used to determine the low-altitude inspection area and the inspection order of the low-altitude inspection area according to the image mask of the low-altitude inspection object and the northeast celestial coordinates of the starting position of the robotic dog; and, use the robotic dog and the unmanned aerial vehicle to perform inspections based on the low-altitude inspection area and the inspection order of the low-altitude inspection area to obtain the inspection results of the low-altitude inspection route;

[0104] The fifth unit 305 is used to determine the inspection alarm information corresponding to the target area image according to the inspection results of the high-altitude inspection route and the inspection results of the low-altitude inspection route.

[0105] Optionally, the first unit 301 is used to:

[0106] Use an unmanned aerial vehicle to collect several area images of the target area;

[0107] Stitch the several area images of the target area into an orthophoto map.

[0108] Optionally, the first unit 301 is used to:

[0109] Input the orthophoto map into a trained regional semantic segmentation model to obtain a semantic segmentation result;

[0110] Determine the boundary between the target area and the non-target area in the orthophoto map according to the semantic segmentation result;

[0111] Determine the target area image corresponding to the target area in the orthophoto map according to the boundary.

[0112] Optionally, the semantic segmentation result includes the height above the ground of each object in the orthophoto map; the second unit 302 is used for:

[0113] Take the objects with a height above the ground greater than a preset height threshold in the target area image as high-altitude inspection objects;

[0114] Take the objects with a height above the ground less than or equal to the preset height threshold in the target area image as low-altitude inspection objects.

[0115] Optionally, the third unit 303 is used for:

[0116] Construct a first extended inspection coordinate set based on the northeast-sky coordinates of the high-altitude inspection objects and the northeast-sky coordinates of the flight starting point of the unmanned aerial vehicle;

[0117] Establish an optimization function based on the first extended inspection coordinate set; wherein, the optimization function aims to minimize the total inspection path, and the constraint conditions of the optimization function include that each high-altitude inspection object is only visited once, starting from the flight starting point and returning to the flight starting point, and ensuring the validity of the path;

[0118] Solve the optimization function to obtain the high-altitude inspection target sequence corresponding to the high-altitude inspection objects;

[0119] Take the flight starting point of the unmanned aerial vehicle as the starting point, and re-arrange the high-altitude inspection target sequence corresponding to the high-altitude inspection objects to obtain a re-ordered high-altitude inspection target sequence;

[0120] For each high-altitude inspection object in the re-ordered high-altitude inspection target sequence, generate a path covering the mask area of the high-altitude inspection object;

[0121] Determine the high-altitude inspection route according to the paths of each high-altitude inspection object in the re-ordered high-altitude inspection target sequence.

[0122] Optionally, the fourth unit 304 is used for:

[0123] Process the low-altitude inspection objects to determine the mask centers corresponding to the low-altitude inspection objects;

[0124] Use the Euclidean clustering algorithm to cluster with the mask centers corresponding to the low-altitude inspection objects as samples to obtain low-altitude inspection clusters;

[0125] Determine the two-dimensional minimum convex hull of the low-altitude inspection objects in each low-altitude inspection cluster, determine the low-altitude inspection area according to the two-dimensional minimum convex hull, and use the center of the two-dimensional minimum convex hull as the northeast celestial coordinate of the low-altitude inspection area;

[0126] Construct a second extended inspection coordinate set based on the northeast celestial coordinate of the low-altitude inspection area and the northeast celestial coordinate of the starting position of the robotic dog;

[0127] Establish an optimization function based on the second extended inspection coordinate set; wherein, the optimization function aims to minimize the total inspection path, and the constraint conditions of the optimization function include that each low-altitude inspection object is only visited once, starting from the starting position and returning to the starting position, and ensuring the effectiveness of the path;

[0128] Solve the optimization function to obtain the inspection order of the low-altitude inspection area.

[0129] Optionally, the fourth unit 304 is used for:

[0130] The robotic dog sequentially travels to the low-altitude inspection area according to the inspection order of the low-altitude inspection area, and autonomously avoids obstacles according to the on-site environment; wherein, the condition for the robotic dog to reach the low-altitude inspection area is that the distance between the current position of the robotic dog and the center coordinate of the low-altitude inspection area is less than a set distance threshold;

[0131] For each low-altitude inspection area, use the drone to determine the order of the low-altitude inspection objects in the low-altitude inspection area, and use the robotic dog and the drone to inspect the low-altitude inspection objects in the low-altitude inspection area based on the order of the low-altitude inspection objects in the low-altitude inspection area to obtain the inspection result of the low-altitude inspection area;

[0132] Use the inspection results of all low-altitude inspection areas as the inspection result of the low-altitude inspection route.

[0133] Optionally, the fourth unit 304 is used for:

[0134] Construct a third extended inspection coordinate set based on the northeast celestial coordinate of the low-altitude inspection objects in the low-altitude inspection area and the starting point of the drone flight;

[0135] Establish an optimization function based on the third extended inspection coordinate set; wherein, the optimization function aims to minimize the total inspection path, and the constraint conditions of the optimization function include that each low-altitude inspection object in the low-altitude inspection area is only visited once, starting from the starting point of flight and returning to the starting point of flight, and ensuring the effectiveness of the path;

[0136] Solve the optimization function to obtain the order of the low-altitude inspection objects in the low-altitude inspection area.

[0137] Optionally, the fourth unit 304 is used for:

[0138] For each low-altitude inspection object in the low-altitude inspection area, generate a coverage path at a preset height from the highest point of the low-altitude inspection object, calculate the distance of the coverage path, and determine the starting point and ending point of the coverage path;

[0139] Calculate the total inspection distance within the low-altitude inspection area based on the distances of the coverage paths corresponding to each low-altitude inspection object;

[0140] Determine the inspection speed of the drone based on the total inspection distance within the low-altitude inspection area, the starting and ending points of each coverage path, and the drone's own battery power and flight energy consumption;

[0141] Control the drone to inspect the low-altitude inspection objects in the low-altitude inspection area based on the inspection speed and the order of the low-altitude inspection objects in the low-altitude inspection area to obtain the inspection results of the low-altitude inspection area.

[0142] Optionally, the inspection results include inspection location information and inspection pictures; the fifth unit 305 is used for:

[0143] Input the inspection results of the high-altitude inspection route and the inspection results of the low-altitude inspection route into the trained vision-language large model to obtain the text descriptions of the inspection results of each inspection result;

[0144] Input the text descriptions of the inspection results of each inspection result into the trained safety large model to obtain the safety levels of each inspection result;

[0145] If the safety level of the inspection result meets the preset risk conditions, use the inspection result and the safety risk information of the inspection result as the inspection information to be alarmed;

[0146] Use all the inspection information to be alarmed as the inspection alarm information corresponding to the target area image;

[0147] Among them, the generation process of the safety large model includes:

[0148] Obtain the construction site safety management regulations and construct a safety rule knowledge graph based on the construction site safety management regulations;

[0149] Use the safety rule knowledge graph as the retrieval enhancement of the preset large language model to form the safety large model.

[0150] Figure 4It is a schematic structural diagram of an electronic device provided by an embodiment of the present application. At the hardware level, the electronic device includes a processor, and optionally also includes an internal bus, a network interface, and a memory. Among them, the memory may include a memory, such as a high-speed random access memory (Random-Access Memory, RAM), and may also include a non-volatile memory, such as at least one disk memory, etc. Of course, the electronic device may also include other hardware required for other services.

[0151] The processor, network interface, and memory can be interconnected through an internal bus, and the internal bus can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the sake of representation, Figure 4 only a bidirectional arrow is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0152] The memory is used to store executable instructions. Specifically, the executable instructions are computer programs that can be executed. The memory can include a memory and a non-volatile memory, and provide executable instructions and data to the processor.

[0153] In a possible implementation manner, the processor reads the corresponding executable instructions from the non-volatile memory into the memory and then runs, or can also obtain the corresponding executable instructions from other devices to form an inspection device at the logical level. The processor executes the executable instructions stored in the memory to implement the inspection method provided in any embodiment of the present application through the executed executable instructions.

[0154] As described above in the present application Figure 1The method executed by the inspection device provided in the illustrated embodiment can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by the integrated logic circuit in the hardware of the processor or by instructions in software form. The above-mentioned processor may be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it may also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. It can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0155] The steps of the method disclosed in combination with the embodiments of the present application can be directly embodied as being executed and completed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software module may be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. This storage medium is located in the memory, and the processor reads the information in the memory and combines its hardware to complete the steps of the above method.

[0156] The embodiments of the present application also propose a readable medium. When the execution instructions stored in the readable storage medium are executed by the processor of the electronic device, the electronic device can execute the inspection method provided in any embodiment of the present application, and is specifically used to execute the method described in the above data query.

[0157] The electronic device described in each of the foregoing embodiments may be a computer.

[0158] Those skilled in the art should understand that the embodiments of the present application may be provided as a method or a computer program product. Therefore, the present application may adopt a completely hardware embodiment, a completely software embodiment, or a form combining software and hardware.

[0159] Each embodiment in this application is described in a progressive manner. For the same or similar parts among the embodiments, reference can be made to each other. Each embodiment focuses on the differences from other embodiments. In particular, for the device embodiments, since they are basically similar to the method embodiments, the description is relatively simple. For the relevant parts, reference can be made to the corresponding descriptions in the method embodiments.

[0160] It should also be noted that the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or further includes elements inherent to such process, method, commodity or device. Without further limitation, the element defined by the statement "including one..." does not exclude the existence of another identical element in the process, method, commodity or device including the said element.

[0161] The above are only the embodiments of this application and are not intended to limit this application. For those skilled in the art, various changes and modifications can be made to this application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this application shall be included within the scope of the claims of this application.

Claims

1. A patrol inspection method, characterized in that: The method comprises: Acquire an orthophoto image, and determine an image of a target area in the orthophoto image; Classifying objects in the target area image to obtain high-altitude inspection objects and low-altitude inspection objects; Determine a high-altitude inspection route according to the northeast celestial coordinates of the high-altitude inspection object and the northeast celestial coordinates of the flight starting point of the UAV; and use the UAV to inspect along the high-altitude inspection route to obtain an inspection result of the high-altitude inspection route; Determine the low-altitude inspection area and the inspection order of the low-altitude inspection area according to the image mask of the low-altitude inspection object and the northeast celestial coordinate of the starting position of the mechanical dog; and, use the mechanical dog and the drone to perform inspections based on the low-altitude inspection area and the inspection order of the low-altitude inspection area to obtain the inspection result of the low-altitude inspection route; Inspection alarm information corresponding to the target area image is determined according to the inspection results of the high-altitude inspection route and the inspection results of the low-altitude inspection route.

2. The method according to claim 1, characterized in that The obtaining of orthophoto images comprises: Using drones to collect images of several areas of the target area; Several regional images of the target area are stitched together into an orthophoto map.

3. The method according to claim 2, characterized in that The method according to claim 1, characterized in that determining the target area image in the orthophoto image comprises: Inputting the orthophoto into a trained regional semantic segmentation model to obtain a semantic segmentation result; Determine the boundary between the target area and the non-target area in the orthophoto according to the semantic segmentation result; According to the boundary, a target area image corresponding to the target area in the orthophoto image is determined.

4. The method according to claim 3, characterized in that The semantic segmentation result includes the height above the ground of each object in the orthophoto image; the objects in the target area image are classified to obtain high-altitude inspection objects and low-altitude inspection objects, including: Taking objects in the target area image whose height above the ground is greater than a preset height threshold as high-altitude inspection objects; The objects in the target area image whose height above the ground is less than or equal to the preset height threshold are inspected at low altitude.

5. The method according to claim 1, characterized in that Determining a high-altitude inspection route according to the northeast celestial coordinates of the high-altitude inspection object and the northeast celestial coordinates of the flight starting point of the UAV includes: Constructing a first extended inspection coordinate set based on the northeast celestial coordinate of the high-altitude inspection object and the northeast celestial coordinate of the flight starting point of the UAV; Based on the first extended inspection coordinate set, an optimization function is established; wherein the optimization function takes the shortest total inspection path as a goal, and the constraints of the optimization function include that each of the high-altitude inspection objects is visited only once, starting from the flight starting point and returning to the flight starting point, and ensuring the validity of the path; Solving the optimization function to obtain a high-altitude inspection target sequence corresponding to the high-altitude inspection object; Taking the flight starting point of the UAV as the starting point, rearranging the high-altitude inspection target sequence corresponding to the high-altitude inspection object to obtain a reordered high-altitude inspection target sequence; For each high-altitude inspection object in the reordered high-altitude inspection target sequence, generating a path covering a mask area of ​​the high-altitude inspection object; A high-altitude inspection route is determined according to the path of each high-altitude inspection object in the reordered high-altitude inspection target sequence.

6. The method according to claim 1, characterized in that The method of determining the low-altitude inspection area and the inspection order of the low-altitude inspection area according to the image mask of the low-altitude inspection object and the northeast celestial coordinate of the starting position of the mechanical dog includes: Processing the low-altitude inspection object to determine a mask center corresponding to the low-altitude inspection object; Taking the mask center corresponding to the low-altitude inspection object as a sample, clustering is performed using the Euclidean clustering algorithm to obtain a low-altitude inspection cluster; Determine the two-dimensional minimum convex hull of the low-altitude inspection object in each low-altitude inspection cluster, determine the low-altitude inspection area according to the two-dimensional minimum convex hull, and use the center of the two-dimensional minimum convex hull as the northeast celestial coordinate of the low-altitude inspection area; Constructing a second extended inspection coordinate set based on the northeast celestial coordinate of the low-altitude inspection area and the northeast celestial coordinate of the starting position of the robot dog; Based on the second extended inspection coordinate set, an optimization function is established; wherein the optimization function takes the shortest total inspection path as a goal, and the constraints of the optimization function include that each of the low-altitude inspection objects is visited only once, starting from the starting position and returning to the starting position, and ensuring the validity of the path; The optimization function is solved to obtain the inspection sequence of the low-altitude inspection area.

7. The method according to claim 6, characterized in that The method of using the mechanical dog and the drone to perform inspection based on the low-altitude inspection area and the inspection order of the low-altitude inspection area to obtain the inspection result of the low-altitude inspection route includes: The robot dog goes to the low-altitude inspection area in turn according to the inspection order of the low-altitude inspection area, and autonomously avoids obstacles according to the on-site environment; wherein the condition for the robot dog to reach the low-altitude inspection area is that the distance between the current position of the robot dog and the center coordinates of the low-altitude inspection area is less than a set distance threshold; For each low-altitude inspection area, the drone is used to determine the order of low-altitude inspection objects in the low-altitude inspection area, and the mechanical dog and the drone are used to inspect the low-altitude inspection objects in the low-altitude inspection area based on the order of the low-altitude inspection objects in the low-altitude inspection area to obtain the inspection result of the low-altitude inspection area; The inspection results of all low-altitude inspection areas are used as the inspection results of the low-altitude inspection route.

8. The method according to claim 7, characterized in that The determining the order of low-altitude inspection objects in the low-altitude inspection area by using the drone includes: Constructing a third extended inspection coordinate set based on the northeast celestial coordinates of the low-altitude inspection object and the flight starting point of the UAV in the low-altitude inspection area; Based on the third extended inspection coordinate set, an optimization function is established; wherein the optimization function takes the shortest total inspection path as a goal, and the constraints of the optimization function include that each low-altitude inspection object in the low-altitude inspection area is visited only once, starting from the flight starting point and returning to the flight starting point, and ensuring the validity of the path; The optimization function is solved to obtain the order of low-altitude inspection objects in the low-altitude inspection area.

9. The method according to claim 7 or 8, characterized in that: Using the drone to inspect the low-altitude inspection objects in the low-altitude inspection area based on the order of the low-altitude inspection objects in the low-altitude inspection area to obtain the inspection result of the low-altitude inspection area, including: For each low-altitude inspection object in the low-altitude inspection area, a coverage path is generated at a preset height from the highest point of the low-altitude inspection object, a distance of the coverage path is calculated, and a starting point and an end point of the coverage path are determined; Calculate the total inspection distance in the low-altitude inspection area according to the distance of the coverage path corresponding to each low-altitude inspection object; Determine the inspection speed of the drone according to the total inspection distance in the low-altitude inspection area, the starting point and the end point of each coverage path, the drone's own power, and the flight energy consumption; The drone is controlled to inspect the low-altitude inspection objects in the low-altitude inspection area based on the inspection speed and the order of the low-altitude inspection objects in the low-altitude inspection area to obtain an inspection result of the low-altitude inspection area.

10. The method according to claim 1, characterized in that The inspection results include inspection location information and inspection pictures; The step of determining inspection alarm information corresponding to the target area image according to the inspection result of the high-altitude inspection route and the inspection result of the low-altitude inspection route includes: Inputting the inspection results of the high-altitude inspection route and the inspection results of the low-altitude inspection route into the trained visual-language large model to obtain a text description of the inspection results of each inspection result; Inputting the text descriptions of the inspection results of each inspection result into the trained security model to obtain the security level of each inspection result; If the security level of the inspection result meets the preset risk condition, the inspection result and the security risk information of the inspection result are used as the inspection information to be alarmed; Using all patrol inspection information to be alarmed as patrol inspection alarm information corresponding to the target area image; The generation process of the security model includes: Obtaining construction site safety management regulations, and building a safety rule knowledge graph based on the construction site safety management regulations; The security rule knowledge graph is used as a retrieval enhancement of a preset large language model to form the security large model.

11. A patrol inspection device, characterized in that: The device comprises: The first unit is used to obtain an orthophoto map and determine a target area image in the orthophoto map; The second unit is used to classify the objects in the target area image to obtain high-altitude inspection objects and low-altitude inspection objects; The third unit is used to determine a high-altitude inspection route according to the northeast celestial coordinates of the high-altitude inspection object and the northeast celestial coordinates of the flight starting point of the UAV; and to use the UAV to inspect along the high-altitude inspection route to obtain an inspection result of the high-altitude inspection route; The fourth unit is used to determine the low-altitude inspection area and the inspection order of the low-altitude inspection area according to the image mask of the low-altitude inspection object and the northeast celestial coordinate of the starting position of the mechanical dog; and to use the mechanical dog and the drone to perform inspections based on the low-altitude inspection area and the inspection order of the low-altitude inspection area to obtain the inspection result of the low-altitude inspection route; The fifth unit is used to determine the inspection alarm information corresponding to the target area image according to the inspection results of the high-altitude inspection route and the inspection results of the low-altitude inspection route.

Images