A method and system for joint inspection control of a drone and a robot dog

Abstract

The application provides a kind of unmanned aerial vehicle and robot dog's joint inspection control method and system, first acquisition inspection reference model and unmanned aerial vehicle inspection data, and using comparison strategy obtains initial task data, based on initial task obtains task cluster data, so that macroscopic abnormality found by unmanned aerial vehicle can be perceived in real time and converted into the task required by robot dog subsequently, then through task cluster data, robot dog waits data is obtained, while constructing solving scheduling objective function, task instruction data is obtained, so that robot dog and unmanned aerial vehicle can execute task instruction data, when multiple points are concurrent, the optimal task allocation of multi-machine cooperation is realized, the utilization rate of inspection resource is improved, then through robot dog and unmanned aerial vehicle, update data is collected, and enhanced data is obtained, finally, the inspection reference can be updated by enhanced update data, to ensure that the real-time updated inspection reference model is provided, to provide more reliable and comprehensive basis for operation and maintenance personnel, to save subsequent operation and maintenance resources.

Description

Technical Field

[0001] This invention relates to the field of inspection control technology, and more specifically, to a joint inspection control method and system of a drone and a robot dog. Background Technology

[0002] In the modern operation and maintenance system of large-scale infrastructure such as industrial parks, automated and intelligent inspections have become the routine for ensuring safety. Among them, drones, with their aerial mobility and wide-area coverage capabilities, are widely used for rapid inspections and preliminary identification of macroscopic anomalies. Robot dogs, with their stable ground mobility and close-range perception capabilities, are used for in-depth and detailed inspections. Theoretically, by using drones and robot dogs to work together, an inspection mode of aerial general survey and ground detailed survey can be formed.

[0003] However, the mainstream approach often involves scanning with drones along fixed routes, followed by manual image analysis before assigning tasks to the robot dog. This inspection method is prone to omissions or misjudgments by maintenance personnel. Furthermore, when the robot dog executes tasks manually generated by maintenance personnel, the missing or incorrect tasks can cause a chain reaction of missing or incorrect inspections, ultimately resulting in the generated inspection benchmark model failing to provide an effective and comprehensive reference, leading to a waste of maintenance resources.

[0004] Therefore, there is an urgent need for a collaborative inspection control method that can automatically generate precise tasks based on data collected by drones, with robot dogs cooperating in task collection and updating the inspection model. Summary of the Invention

[0005] In view of the aforementioned problems, and in conjunction with the first aspect of the present invention, embodiments of the present invention provide a joint inspection and control method for a drone and a robot dog, comprising the following steps: X1: Obtain the inspection baseline model and UAV inspection data, obtain the initial task data based on the inspection baseline model and UAV inspection data and adopt a comparison strategy, and obtain the task cluster data based on the initial task data; X2: Obtain robot dog waiting data through task cluster data, construct scheduling objective function and obtain task assignment data through robot dog waiting data and task cluster data, obtain task instruction data through task assignment data, and robot dog and drone execute task instruction data; X3: Collects and updates data through robot dogs and drones, and evaluates and optimizes the updated data during the collection process to obtain enhanced updated data; X4: Updates the inspection baseline model by enhancing and updating data.

[0006] As a further aspect of the present invention, step X1 specifically includes: The image stream data is corrected by the attribute stream data to obtain the first image data. The pose stream data is optimized based on the first image data to generate a grid surface model. The second image data is obtained based on the grid surface model and the optimized pose stream data. Feature matching and filtering are performed based on the second image data and image layer data. First matching point pair set and second matching point pair set are obtained respectively, and spatial mapping model is generated. Third image data is obtained based on spatial mapping model. Confidence data is constructed through third image data, image layer data and second matching point pair set. Based on confidence data, image layer data, and third image data, change probability data, binary change mask data, and semantic change mask data are obtained respectively, and dual-path filtering detection is performed to generate initial task data. Construct a spatial relationship graph, and perform clustering evaluation on the initial task data based on the spatial relationship graph to obtain task cluster data; Among these features, the image layer data is updated in real time.

[0007] As a further aspect of the present invention, change probability data, binary change mask data, and semantic change mask data are obtained based on confidence data, image layer data, and third image data, respectively, and dual-path filtering detection is performed to generate initial task data, including: The image layer data and the third image data are detected by a change detection model to obtain change probability data. Based on the change probability data and confidence data, binarization is performed to obtain binary change mask data. Real-time semantic segmentation data is generated based on third-party image data. The real-time semantic segmentation data is compared and evaluated with image layer data to obtain semantic change mask data. We obtain fused change mask data by combining semantic change mask data and binary change mask data. We then perform connectivity analysis on the fused change mask data to obtain change object data. Finally, we filter the change object data to obtain initial task data.

[0008] As a further aspect of the present invention, the method further includes: The inspection baseline model includes geometric layer data and image layer data; The geometric layer data refers to the spatial structure data of the inspection area; The image layer data refers to the orthophoto data and semantic segmentation data of the inspection area; Drone inspection data includes image stream data, pose stream data, and attribute stream data; The image stream data refers to orthophoto data during UAV inspection; The pose flow data refers to the pose data during UAV inspection. The attribute stream data refers to the metadata during UAV inspections.

[0009] As a further aspect of the present invention, step X2 specifically includes: Obtain robot dog attribute data, and based on the robot dog attribute data and task cluster data, obtain robot dog waiting data; The task-machine dog matching matrix is ​​obtained based on the robot dog waiting data and task cluster data. A scheduling objective function is constructed using the task-machine dog matching matrix and task cluster data. The scheduling objective function is then solved to obtain task assignment data. A robot dog navigation map is generated by using task assignment data and inspection benchmark model. Path search is performed based on the robot dog navigation map to obtain the robot dog's driving route. At the same time, guidance instructions are embedded into the robot dog's driving route through the guidance protocol to generate task instruction data. The task command data is sent to the robot dog and the drone. During the execution, the robot dog continuously outputs motion control commands to control the robot dog in real time.

[0010] As a further aspect of the present invention, a task-machine dog matching matrix is ​​obtained based on machine dog waiting data and task cluster data, and a scheduling objective function is constructed using the task-machine dog matching matrix and task cluster data. The scheduling objective function is then solved to obtain task assignment data, including: The capability matching coefficient is calculated based on the task cluster data and the robot dog waiting data. The path cost coefficient is calculated based on the task cluster data, the robot dog waiting data and the inspection benchmark model. At the same time, the constraint value is obtained. The task-robot dog matching matrix is ​​constructed by the capability matching coefficient, the path cost coefficient and the constraint value. Define decision variables, construct total cost and total capability terms based on the task-robot matching matrix, configure delay penalty and equilibrium terms, construct constraints through task cluster data and robot dog waiting data, and construct scheduling objective function based on total cost, total capability, delay penalty, equilibrium terms and constraints. The scheduling objective function is solved based on the algorithm, the initial task assignment data is obtained, the task conflicts in the initial task assignment data are checked, and the task assignment data is obtained.

[0011] As a further aspect of the present invention, the step of generating a robot dog navigation map through task assignment data and inspection benchmark model, and performing path search based on the robot dog navigation map to obtain the robot dog's driving route includes: Obstacle information data is acquired, and an initial navigation map is generated based on the obstacle information data and the inspection benchmark model; A path cost map is generated from the initial navigation map, and then the path cost map is overlaid on the initial navigation map to generate the robot dog navigation map. Based on the task assignment data and using a global path search algorithm, the robot dog's initial movement route is obtained by searching the navigation map. The initial movement path of the robot dog is smoothed to obtain the robot dog's driving path.

[0012] As a further aspect of the present invention, step X4 specifically includes: The enhanced update data is spatiotemporally located with the inspection baseline model, and a matching relationship between the enhanced update data and the inspection baseline model is established based on the spatial index. The inspection baseline model is updated by matching relationships and enhancing update data, wherein the update operations include creating and updating.

[0013] Furthermore, embodiments of the present invention also provide a joint inspection and control system for drones and robot dogs, comprising: The robot dog collects data based on task instruction data, obtains updated data and enhanced update data, and has built-in optimization and control modules. The optimization module is used to optimize the updated data, and the control module continuously outputs robot dog motion control commands to control the robot dog in real time during the execution process. The drone is used to acquire drone inspection data and guide the drone dog based on task instruction data. The enhancement module is used to acquire initial task data and acquire task cluster data by optimizing the initial task data; The instruction generation module is used to obtain assignment data based on task cluster data and robot dog waiting data, process the assignment data, obtain task instruction data, and send it to the robot dog and drone. An update module is used to update the inspection baseline model; A baseline storage module is used to store the inspection baseline model.

[0014] In this embodiment, the inspection baseline model and UAV inspection data are first acquired, and an initial task data is obtained using a comparison strategy. Based on the initial task, task cluster data is obtained, enabling macroscopic anomalies detected by the UAV to be perceived in real time and converted into tasks required by the robot dog, ensuring task accuracy. Then, robot dog waiting data is obtained through the task cluster data, and a scheduling objective function is constructed and solved to obtain task assignment data. Task instruction data is obtained through the task assignment data, enabling the robot dog and UAV to execute task instruction data. This ensures that the appropriate robot dog arrives at the target area along the optimal route, and also achieves optimal task allocation for multi-machine collaboration in multi-point concurrency, improving the utilization rate of inspection resources. Next, update data is collected by the robot dog and UAV, and enhanced data is obtained simultaneously. Finally, the inspection baseline is updated through the enhanced update data to ensure that a real-time updated inspection baseline model is provided, providing maintenance personnel with a more reliable and comprehensive basis, thereby saving resources used in subsequent maintenance. Attached Figure Description

[0015] Figure 1 This is a flowchart of the steps of a joint inspection and control method of a drone and a robot dog according to the present invention.

[0016] Figure 2 This is a flowchart of step X1 in the joint inspection and control method of a drone and a robot dog of the present invention.

[0017] Figure 3 This is a schematic diagram of the framework of a joint inspection control system for a drone and a robot dog according to the present invention. Detailed Implementation

[0018] The present invention will now be described in detail with reference to the accompanying drawings, providing a comprehensive description of a joint inspection and control method and system for a drone and a robot dog.

[0019] As attached Figure 1 Appendix Figure 2 As shown: Specifically, a joint inspection and control method using a drone and a robotic dog includes the following steps: Step X1: Obtain the inspection baseline model and UAV inspection data. Based on the inspection baseline model and UAV inspection data, and using a comparison strategy, obtain the initial task data. Then, obtain the task cluster data based on the initial task data.

[0020] Understandably, the inspection reference model of the inspection area is retrieved, which is the twin model of the corresponding area. This includes geometric layer data and image layer data. The geometric layer data represents the spatial structure data of the inspection area, namely the 3D mesh file or point cloud file of the inspection area. The image layer data represents the orthophoto data and semantic segmentation data of the inspection area, namely the historical orthophoto of the inspection area with geographic coordinate information and the semantic map aligned with the pixels of the historical orthophoto. In subsequent processes, the orthophoto data in the image layer data is used as the reference image, and the coordinate system of the inspection reference model is unified with the coordinate system of the navigation system of the UAV and the robot dog respectively.

[0021] Furthermore, inspection commands are issued to the drones, which then fly sequentially to the corresponding task areas to perform photographic operations, thereby acquiring drone inspection data. This drone inspection data includes image stream data, pose stream data, and attribute stream data. The image stream data represents the orthophoto data taken by the drone during inspection, i.e., real-time orthophotos of the corresponding task areas taken sequentially by the drone. The pose stream data represents the pose data taken by the drone during inspection, which can be acquired through the drone's built-in sensors. The metadata represents the metadata taken during drone inspection, such as camera intrinsics and distortion coefficients. During drone photography, a fixed shooting time is set for each task area, and the image stream data, pose stream data, and attribute stream data within the same time window are bound together to form synchronized data units. These data units are then sequentially assembled into drone inspection data, each with a unique sequence number and timestamp.

[0022] Furthermore, step X1 specifically includes: Step X1-1: Correct the image stream data using attribute stream data to obtain first image data; optimize the pose stream data based on the first image data and generate a grid surface model; obtain second image data based on the grid surface model and the optimized pose stream data.

[0023] In step X1-1, during drone flight and shooting, attribute stream data and image stream data are acquired in real time. For distortion correction, the camera's intrinsic parameters and distortion coefficients are used to perform geometric transformation on the original captured image to eliminate radial and tangential distortion caused by lens optical characteristics. At the same time, the histogram is analyzed. If the image is overexposed or underexposed overall, this information is recorded, and an attempt is made to adjust the camera exposure parameters in the next frame. In cloudy weather or during early morning and evening inspections, the brightness difference of images captured by the drone at different sun angles may be large. In this case, a physical model-based or deep learning method can be used to normalize the image brightness to a standard lighting condition to reduce the brightness difference, thereby acquiring the first image data. The first image data is represented as the image stream data after the above correction operations, ensuring that the images are comparable in geometry and brightness.

[0024] Next, image feature extraction algorithms such as SIFT and ORB are used to extract features from the continuous first image data, and feature matching is performed between adjacent frames to establish visual associations between images. An incremental SFM algorithm is employed, using the matched feature points and initial pose as the starting point for optimization. Bundle adjustment is used to simultaneously optimize the pose data and the 3D coordinates of the features. During this optimization process, the drift error caused by the initial pose is continuously corrected, and a sparse 3D point cloud is generated. This point cloud represents the terrain features of the corresponding inspection area. The pose flow data and sparse point cloud optimized by the incremental SFM algorithm are registered in the same global coordinate system using coordinate information. Then, the first image data is processed... For image pairs with overlapping areas in the data, a semi-global matching algorithm is used to generate a depth map for each image. The depth maps of all images are then transformed and fused into the global coordinate system based on their optimized pose data to generate a dense 3D point cloud. This point cloud represents the precise terrain of the corresponding inspection area. The dense 3D point cloud is then meshed to generate a raster-style digital surface model, i.e., a raster surface model. Based on the raster surface model and the pose data of each image, each image is orthorectified to eliminate perspective distortion and project it onto a view perpendicular to the ground. Then, color equalization and seam optimization are performed on all corrected images to stitch them together into a complete and seamless real-time digital orthophoto, i.e., the second image data.

[0025] Understandably, discrete and perspective-biased images are transformed into continuous, downward-vertical images with geographic coordinates, thus providing a basis for change comparison and detection in subsequent steps. At this time, the drone collects data, flies back to the default waiting position to wait, and transmits the second image data back to the server.

[0026] Step X1-2: Based on the second image data and the image layer data, feature matching and filtering are performed to obtain the first matching point pair set and the second matching point pair set respectively and generate a spatial mapping model. Based on the spatial mapping model, the third image data is obtained, and confidence data is constructed through the third image data, the image layer data and the second matching point pair set.

[0027] In steps X1-2, Gaussian downsampling is performed on the reference image and the second image data respectively, and an image pyramid is constructed, for example, at the original image, 1 / 2 scale, and 1 / 4 scale, so that it can adapt to different scales of viewpoint changes. First, roughly aligned regions are quickly found at the coarse scale. Then, a matching point pair set is extracted from each layer of the image pyramid, that is, feature points and descriptors with strong invariance to rotation and scaling are extracted. This can be preferentially performed by algorithms such as SIFT (Scale Invariant Feature Transform) or ORB (Oriented FAST and Rotated BRIEF). The descriptor is a high-dimensional vector used to uniquely represent the visual pattern around the feature point.

[0028] Next, between the corresponding image pyramid levels of the reference image and the second image data, the Euclidean distance of the descriptors is calculated to find the most similar candidate matching point in the reference image for each feature point in the second image data. An estimation algorithm such as RANSAC (Random Consistent Sampling) is used, combined with the homography matrix, to find the interior points that conform to the same geometric transformation among all matching points through iterative solution. In the process of solving the interior points, the initial value of the global homography matrix can also be output, and erroneous exterior point matches can be identified, thereby obtaining a set of highly reliable matching point pairs.

[0029] A lightweight semantic segmentation model, such as DeepLabv3+ or SegNet with conventional architecture, is used to generate real-time semantic segmentation data for a second image dataset, which is then temporarily stored in the image layer data. Simultaneously, the semantic segmentation data from the image layer data is combined with the semantic segmentation data to check the label of each pair of matching points in both the semantic segmentation data and the real-time semantic segmentation data. For example, if the label in the semantic segmentation data is "building" while the label in the real-time semantic segmentation data is "vegetation," the point may be a mismatch and can be directly removed. In areas where semantic labels are stable and not easily changed, such as the centerline of a road, even if the feature points are not abundant, matching points with clear semantic correspondences can be manually added based on the edges or corners of the semantic contour to enhance the constraints of these inspection areas, thereby generating a secondary set of matching point pairs.

[0030] Then, a spatial mapping model is generated by matching point pairs twice and using a local weighted transformation model. Specifically, the image is segmented into an irregular triangular network using the matching point pairs as control points. For each triangular region, a local affine transformation is solved using the matching points corresponding to its three vertices, thereby modeling a geometric transformation model, namely the spatial mapping model. Semantic analysis is used to further improve the reliability of matching point pairs, and a suitable mathematical model is used to describe complex geometric deformations, thereby supporting the subsequent geometric registration operation using the spatial mapping model, that is, obtaining third image data based on the spatial mapping model.

[0031] Specifically, for acquiring third image data based on a spatial mapping model, the pixel grid of the reference image is used as a reference to determine the size of the registered output and the geographic coordinates of each output pixel, thereby defining the output grid. Through the spatial mapping model and second image data, and by using reverse mapping and resampling methods, a geometrically registered image, i.e., the third image data, is generated. The reverse mapping and resampling methods are mature technologies and can be found in publicly available literature, so they will not be elaborated further. In the specific implementation process, for pixels that fall outside the effective range after reverse mapping, the method of marking invalid values ​​can be used for processing.

[0032] Finally, the quality of the geometrically registered image can be assessed, and confidence data can be constructed. A difference map or cross-correlation map between the reference image and the third image data is calculated to check whether alignment is achieved in unchanged areas, such as roads, and whether there are ghosting or edge misalignment. Then, based on the density of local matching points, residual band lines, and texture richness, a confidence map with the same size as the image is generated, i.e., confidence data. Areas with low confidence will be given lower weights or temporarily excluded in subsequent transform detection. In practical applications, registration accuracy indices in unchanged areas can also be calculated. Registration accuracy includes, but is not limited to, root mean square error (RMSE) and cross-correlation. RMSE represents the coordinate deviation of the remaining matching point pairs, while cross-correlation measures the statistical dependence between the two images; a higher value indicates higher alignment. A registration accuracy report is then output for registration evaluation and analysis.

[0033] Steps X1-3: Based on confidence data, image layer data, and third image data, change probability data, binary change mask data, and semantic change mask data are obtained respectively, and dual-path filtering detection is performed to generate initial task data.

[0034] Furthermore, steps X1-3 specifically include: Step X1-3-1: The image layer data and the third image data are detected by the change detection model to obtain change probability data. Based on the change probability data and confidence data, binarization is performed to obtain binary change mask data.

[0035] In step X1-3-1, the reference image and third image data are stacked along the channel dimension to form a multi-channel input tensor, or an input tensor processed according to the requirements of subsequent models. The processed reference image and third image data are then input into a pre-trained deep learning model for transform detection. The deep learning model can be a Siamese network with an encoder-decoder structure, such as ChangeDet, BIT, or DSAMNet, to determine whether changes have occurred in the reference image and third image data, and to learn to distinguish between real and pseudo-changes. Real changes can be represented as changes such as new buildings being constructed or objects being removed, while pseudo-changes can be represented as... This model represents changes such as shadow jitter, seasonal vegetation color changes, and temporary moving objects like vehicles. It outputs a change probability map of the same size as the input image, i.e., change probability data. Each pixel in the map has a value of 0-1, representing the confidence level that a real change has occurred at that location. It can also include a change category map, such as new, disappear, or replacement categories. The change probability map is multiplied pixel by pixel with the third image data. In areas with low confidence, the change probability value is reduced accordingly. Then, a set threshold, such as 0.7, is applied to binarize the probability map to obtain a change mask, i.e., binary change mask data.

[0036] Step X1-3-2: Generate real-time semantic segmentation data based on the third image data, compare and evaluate the real-time semantic segmentation data with the image layer data, and obtain semantic change mask data.

[0037] In step X1-3-2, the semantic segmentation model described above is used again to process the third image data to generate real-time semantic segmentation data. The semantic segmentation data temporarily stored in the image layer data is replaced and updated to generate the latest semantic segmentation data. The latest semantic segmentation data is compared with the semantic segmentation data in the image layer data to find all pixels with inconsistent categories and generate an original semantic difference mask, which includes all regions that change from one semantic category to another. However, not all semantic changes represent changes that need attention. At this time, they can be filtered according to a pre-set semantic change rule table. For example, the semantic change rule table can be set to ignore and focus. The change between trees and shrubs can be ignored, and the change from open space to building can be focused. After applying the rule filtering, a semantic change mask is obtained, that is, semantic change mask data.

[0038] Step X1-3-3: Obtain fused change mask data by combining semantic change mask data and binary change mask data; perform connectivity analysis on the fused change mask data to obtain change object data; filter the change object data to obtain initial task data.

[0039] In step X1-3-3, a logical OR operation is performed on the semantic change mask data and the binary change mask data to obtain a fused change mask, i.e., fused change mask data. This ensures that as long as one channel considers a change to have occurred, the changed region is included in the candidate. Then, morphological operations, such as closing operations, are performed on the fused change mask data to fill holes and connect adjacent pixels. Next, connected component labeling is performed, and each independent connected component is regarded as a change object instance. At the same time, attributes are assigned to each change object instance, including area, bounding polygon, dominant change type, confidence, and other data, thereby obtaining change object data. At this point, the change object data can be filtered according to the preset task rule base, for example, filtering out areas with an area of ​​less than 1 square meter. The changes in square meters, or those explicitly marked as imminent or negligible, are then used to calculate a comprehensive priority score for the remaining changed object instances based on their attributes. The calculation method for the comprehensive priority score can be set using a conventional weighted summation formula and the actual process. Based on the priority score, the changed object instances that need to be dispatched to the robot dog for ground verification in the current period are selected, and initial task data is constructed. The initial task data includes a structured initial task list, which includes the task ID and associated instance ID, the target bounding polygon (represented as the target area in the following text), the expected change type and verification requirements, priority, and whether guidance is required. This initial task data completes the accurate conversion from image to task through steps X1-1 to X1-3.

[0040] Steps X1-4: Construct a spatial relationship graph, and perform clustering evaluation on the initial task data based on the spatial relationship graph to obtain task cluster data.

[0041] In steps X1-4, the target area of ​​each task in the initial task data is taken as a node in the graph. The spatial reachability distance between any two target areas is calculated. The reachability distance needs to take into account the actual driving distance along the passable path and the semantic difficulty on the path to determine whether a detour is needed. It can be calculated based on image layer data and third image data. After the spatial relationship graph is constructed, the task nodes can be clustered based on the density clustering algorithm to generate task cluster data.

[0042] Step X2: Obtain robot dog waiting data through task cluster data; construct scheduling objective function and obtain task assignment data through robot dog waiting data and task cluster data; obtain task instruction data through task assignment data; and execute task instruction data between robot dog and drone.

[0043] Furthermore, step X2 specifically includes: Step X2-1: Obtain robot dog attribute data, and obtain robot dog waiting data based on robot dog attribute data and task cluster data.

[0044] In step X2-1, the real-time status information of the robot dog is obtained, namely the robot dog attribute data, including the ID of each robot dog, real-time geographical location, and robot dog capability data (such as obstacle crossing height, sensor type, loading equipment, movement speed, etc.). The robot dog attribute data is compared with the task cluster data, thereby filtering the robot dogs according to the needs of the tasks within the task cluster, filtering out robot dogs that do not have the ability to perform tasks, narrowing the subsequent search range for robot dogs, and thus obtaining the robot dog waiting data.

[0045] Step X2-2: Obtain the task-machine dog matching matrix based on the robot dog waiting data and task cluster data, construct the scheduling objective function using the task-machine dog matching matrix and task cluster data, and solve the scheduling objective function to obtain task assignment data.

[0046] Furthermore, step X2-2 specifically includes: Step X2-2-1: Calculate the capability matching coefficient based on the task cluster data and the robot dog waiting data; calculate the path cost coefficient based on the task cluster data, the robot dog waiting data and the inspection benchmark model; and obtain the constraint value. Construct the task-robot dog matching matrix through the capability matching coefficient, the path cost coefficient and the constraint value.

[0047] In step X2-2-1, for each task cluster in the task cluster data and each robot dog in the robot dog waiting data, a static capability matching score, i.e. capability matching coefficient, is calculated. The score is based on the task requirements in the task cluster, such as the comparison between the sensor and loading equipment required by the task and the capability data corresponding to the robot dog. A complete match can be 1, a partial match can be 0.5, and a mismatch can be 0. For example, a task that requires shooting and data collection has a matching degree of 0 with a robot dog that is not equipped with a high-precision camera.

[0048] Next, the path cost coefficient and constraint value are calculated separately. The path cost coefficient represents the spatial accessibility cost. By calculating the shortest actual driving distance from the robot dog's current position to the edge of the target area of ​​the task cluster, along the latest semantic segmentation data temporarily stored in the inspection benchmark model, this distance is normalized with the robot dog's maximum range and converted into a spatial cost coefficient, with a value range of 0-1. The smaller the value, the better the accessibility. As for the constraint, by combining the state of the task and the robot dog, it is determined whether the robot dog can meet the time window constraint of the task, thus generating a Boolean value, i.e., the constraint value. The above capability matching coefficient, path cost coefficient and constraint value are constructed into a three-dimensional matching degree matrix, i.e., the task-robot dog matching matrix, whose dimension is the number of tasks * robot dogs * 3, where the third dimension stores (capability matching coefficient, path cost coefficient, constraint value).

[0049] Step X2-2-2: Define decision variables, construct total cost and total capability terms based on the task-robot matching matrix, configure delay penalty and equilibrium terms, construct constraints through task cluster data and robot dog waiting data, and construct scheduling objective function based on total cost, total capability, delay penalty, equilibrium terms and constraints.

[0050] In step X2-2-2, a binary decision variable is introduced to represent the assignment of tasks to the robot dogs. Simultaneously, an order variable is introduced for the task sequence of each robot dog, representing the task order. Then, the total cost term, total capability term, delay penalty term, and equilibrium term are constructed, and configured. Specifically, the total cost term is the sum of the binary decision variables multiplied by the path cost coefficient. The total capability term is the sum of the binary decision variables multiplied by (1 - capability matching coefficient) multiplied by the weight. The delay penalty is represented by calculating a penalty term based on the number of times each task is completed. The equilibrium term is represented as... The variance of task load among the robot dogs is penalized, and then constraints can be applied. These constraints include hard constraints and soft constraints. Hard constraints are specific constraint values, such as time windows that must be met. Other hard constraints may include conditions such as each task can only be assigned to one robot dog, and the robot dog's power consumption must meet the energy consumption requirements of all its assigned tasks. Soft constraints can be configured based on actual conditions, such as allowing robot dogs to perform tasks they are good at. The scheduling objective function is constructed using a weighted method with total cost, total capacity, delay penalty, balance, and constraints.

[0051] Step X2-2-3: Solve the scheduling objective function based on the solution algorithm, obtain the initial task assignment data, check the task conflicts in the initial task assignment data, and obtain the task assignment data.

[0052] In step X2-2-3, a metaheuristic algorithm (such as genetic algorithm, particle swarm optimization algorithm or simulated annealing) or a combination of graph theory and operations research methods (such as a vehicle routing problem model with time windows) can be used to solve the problem. Here, we will use genetic algorithm as an example.

[0053] In some possible embodiments, the genetic algorithm is initialized and an initial solution is generated, for example, a batch of task assignment schemes that meet the constraints are randomly generated. Then, within the number of iterations set by the algorithm, evaluation, selection, crossover, mutation, and retention operations are performed cyclically. For the evaluation operation, the objective function value (fitness) of each solution in the current population is calculated, i.e., the chromosome's fitness value. For the selection operation, individuals with high fitness are selected to enter the next generation. For the crossover operation, the selected individuals are crossovered (exchanging some task assignments) and mutated (randomly changing the assignment of a certain task) to generate new solutions, while ensuring that the newly generated solutions meet the constraints. For the retention operation, the best solutions in each generation are retained and directly enter the next generation. When the termination condition is reached, i.e., the set number of iterations is reached, the solution with the minimum fitness value found is output, thereby obtaining task assignment data, which at least includes which robot dog should be assigned to each task, the order in which each robot dog performs its task, and the estimated time of the task.

[0054] Step X2-3: Generate a robot dog navigation map using task assignment data and inspection benchmark model; perform path search based on the robot dog navigation map to obtain the robot dog's driving route; and simultaneously embed guidance instructions into the robot dog's driving route through the guidance protocol to generate task instruction data.

[0055] Furthermore, steps X2-3 specifically include: Step X2-3-1: Obtain obstacle information data and generate an initial navigation map based on the obstacle information data and the inspection benchmark model.

[0056] In step X2-3-1, obstacle information is first acquired, such as temporarily parked vehicles, building materials piled on open ground, large areas of water stains, etc. This obstacle information data can be acquired based on third-party image data and using conventional target detection algorithms, such as the YOLO algorithm. Then, the obstacle data is fused with the latest semantic segmentation data temporarily stored in the image layer data of the information benchmark model. The obstacle information is then labeled on the latest semantic segmentation data to generate an initial navigation map.

[0057] Step X2-3-2: Generate a path cost map from the initial navigation map, and overlay the path cost map onto the initial navigation map to generate the robot dog navigation map.

[0058] In step X2-3-2, a multi-layered cost map is generated based on the initial navigation map for the subsequent path algorithms. The cost of static, impassable areas is infinite, the cost of grassy areas is medium, and the cost of obstacle areas is high. This can be adaptively set based on actual use to guide the subsequent path algorithms to prioritize smoother paths that avoid obstacles. Then, the target area of ​​each task and the preset waiting area for each target area need to be clearly marked on the initial navigation map, such as a safe open area 5-10 meters outside the target area. The path cost map is overlaid on the initial navigation map to generate the robot dog navigation map, and the robot dog navigation map is sent to the robot dog.

[0059] Step X2-3-3: Based on the task assignment data and using the global path search algorithm, search the robot dog's navigation map to obtain the robot dog's initial movement route.

[0060] In step X2-3-3, based on the task execution order, and taking the waiting area of ​​the target region of each task as the center point, as the sequential target point for path planning, a graph search algorithm or a sampling-based fast random tree, such as the A* algorithm or the RRT* algorithm, can be used to search on the path cost map to find a series of global paths from the robot dog's current position to the first target point and then to the subsequent target points. This path geometrically avoids high-cost areas and semantically conforms to the movement rules, thereby obtaining the robot dog's initial movement route.

[0061] Step X2-3-4: Smooth the initial movement path of the robot dog to obtain the robot dog's driving movement path.

[0062] In step X2-3-4, the initial movement route of the robot dog obtained earlier may have issues such as discounting, which need to be smoothed. This can be done by using B-spline curves to optimize the route and generate the robot dog's movement route.

[0063] Furthermore, based on the robot dog's movement route, corresponding path control instructions are generated. If the task requires drone-assisted guidance, a guidance trigger time needs to be set at a specific distance from the target area. At the same time, the system sends a guidance resource reservation request to the drone, specifying the time window and spatial location, thereby reserving a drone to arrive at the target location at the corresponding time to stand by. In the path control instructions, a control instruction is added, that is, instructing the robot dog to wait for drone guidance, and at the same time generating corresponding guidance control instructions for the reserved drone, including the target the drone is flying to, the hovering position, etc., thus generating the final task instruction data.

[0064] Step X2-4: Send the task command data to the robot dog and the drone. During the execution process, the robot dog continuously outputs robot dog motion control commands to control the robot dog in real time.

[0065] In steps X2-4, the robot dog drives based on task command data. It uses its own sensors or onboard equipment to update its local cost map in real time, adding instantaneous obstacles that are not marked globally, such as pedestrians suddenly appearing. Then, based on the guidance of the global path and the newly updated local cost map, a dynamic window algorithm is used to calculate the current optimal speed command, i.e., the robot dog's motion control command. Understandably, the dynamic window algorithm can search for a set of speed pairs in the speed space that can simultaneously approach the global path and avoid local obstacles smoothly and comfortably under the robot dog's dynamic constraints. When the robot dog reaches the path point, the local controller pauses the execution of the robot dog's motion control command and prepares to execute the task command data. For robot dogs with drone guidance commands, the control logic of waiting for guidance is executed until the drone begins guidance. After the drone begins guidance, the robot dog can prepare to execute the task command data.

[0066] X3: Collects and updates data through robot dogs and drones, and evaluates and optimizes the updated data during the collection process to obtain enhanced updated data.

[0067] In step X3, the robot dog calls the corresponding action program template from the local acquisition action template library along with the task instruction data, ensuring the safety of the actions. Then, it can perform acquisition tasks through the corresponding action program template, such as LiDAR, panoramic camera, etc. During the acquisition task, it executes actions in sequence to obtain updated data. For example, in the surround scanning action, it ensures the scanning frequency of the LiDAR, the movement speed of the robot dog, etc. Then, it evaluates the quality of the acquired data. For example, for point cloud data, it can calculate the point cloud density and noise ratio of the current frame in real time. For image data, it can calculate the image clarity and exposure in real time. If the quality of a certain acquisition is not up to standard, such as the image is blurry, it can trigger adaptive resampling or supplementary actions. When the acquired data is from multiple sources, it performs spatiotemporal alignment and fusion to generate fused acquisition data, including image, point cloud, semantic, and other data. After the robot dog completes the acquisition, it sends back the enhanced update data. After sending back, it encapsulates the evaluated and optimized update data, the third image data, and the latest semantic segmentation data temporarily stored in the inspection benchmark model into enhanced update data.

[0068] X4: Updates the inspection baseline model by enhancing and updating data.

[0069] Furthermore, step X4 specifically includes: Step X4-1 involves performing spatiotemporal positioning of the enhanced update data and the inspection baseline model, and establishing a matching relationship between the enhanced update data and the inspection baseline model based on the spatial index.

[0070] In step X4-1, the metadata in the enhanced update data is parsed to obtain data such as the collection timestamp and coordinate system information. Using this information, all data in the enhanced update data, such as point clouds and images, are aligned to the global coordinate system of the inspection baseline model. Using the spatial index structure of the inspection baseline model, all existing entities that intersect with the spatial range (defined by the target region) of the enhanced update data are retrieved and used as candidate entities. Matching of these candidate entities is then performed, including geometric matching, semantic matching, and image matching. The point clouds in the enhanced update data are then compared with the geometric models of the candidate entities. Line matching calculates the geometric fit, matching the images in the enhanced update data with the candidate entities and calculating the image fit. Semantic matching is also performed using the semantics of the enhanced update data and the semantic segmentation data of the candidate entities, calculating the semantic fit. The three types of fit are compared with preset matching thresholds, such as geometric fit > 80%, image fit > 80%, and semantic fit > 80%, to determine whether the data in the enhanced update data belongs to an existing entity or represents a completely new entity. Finally, the association relationship between the enhanced update data and the entities is established, i.e., the matching relationship.

[0071] Step X4-2 involves updating the inspection baseline model using matching relationships and enhanced update data, where the update operation includes creating and updating.

[0072] In step X4-2, based on the matching relationship, a new entity or an updated entity is selected. For a new entity, a triangular mesh model can be generated based on the point cloud in the enhanced update data and a surface reconstruction algorithm to achieve geometric model reconstruction. The attributes of the new entity are automatically initialized based on the semantics in the enhanced update data. The image in the enhanced update data is inserted as a new node, and a topological connection relationship with surrounding entities is automatically established based on its spatial location. For an updated entity, the point cloud in the enhanced update data is non-rigidly registered with the existing entity or locally fused. For example, only the new point cloud is used to replace it, or missing details are supplemented. An updated texture map is generated using the image in the enhanced update data to replace the old texture of the entity. The semantics in the enhanced update data are associated with the attributes of the entity as new attribute fields, and automatic attribute changes are triggered to update the inspection benchmark model.

[0073] This application also provides a joint inspection and control system for drones and robot dogs that can realize the ideas of this application, as shown in the attached document. Figure 3 As shown.

[0074] Specifically, a joint inspection and control system for drones and robotic dogs includes: The robot dog collects data based on task instruction data, obtains updated data and enhanced update data, and has built-in optimization and control modules.

[0075] The optimization module is used to optimize the updated data, and the control module continuously outputs robot dog motion control commands to control the robot dog in real time during the execution process.

[0076] The drone is used to acquire drone inspection data and guide the drone dog based on task instruction data.

[0077] An enhancement module is used to acquire initial task data and to acquire task cluster data by optimizing the initial task data.

[0078] The instruction generation module is used to obtain assignment data based on task cluster data and robot dog waiting data, process the assignment data to obtain task instruction data, and send it to the robot dog and drone.

[0079] An update module is used to update the inspection baseline model.

[0080] A baseline storage module is used to store the inspection baseline model.

[0081] The specific usage and function of this embodiment are explained below: First, the inspection baseline model and UAV inspection data are acquired, and an initial task data is obtained using a comparison strategy. Based on the initial task, task cluster data is obtained, enabling real-time perception of macroscopic anomalies detected by the UAV and converting them into tasks required by the robot dog, ensuring task accuracy. Then, robot dog waiting data is obtained through the task cluster data, and a scheduling objective function is constructed and solved to obtain task assignment data. Task instruction data is obtained through the task assignment data, enabling the robot dog and UAV to execute task instruction data. This ensures that the appropriate robot dog arrives at the target area along the optimal route, and also achieves optimal task allocation for multi-machine collaboration in multi-point concurrency, improving the utilization rate of inspection resources. Next, update data is collected through the robot dog and UAV, and augmented data is obtained simultaneously. Finally, the inspection baseline is updated through the augmented update data to ensure that a real-time updated inspection baseline model is provided, providing maintenance personnel with a more reliable and comprehensive basis, thereby saving resources used in subsequent maintenance.

[0082] The above embodiments can be implemented, in whole or in part, by software, hardware (such as circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via limited means (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0083] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A joint inspection and control method using a drone and a robot dog, characterized in that, Includes the following steps: X1: Obtain the inspection baseline model and UAV inspection data, obtain the initial task data based on the inspection baseline model and UAV inspection data and adopt a comparison strategy, and obtain the task cluster data based on the initial task data; X2: Obtain robot dog waiting data through task cluster data, construct scheduling objective function and obtain task assignment data through robot dog waiting data and task cluster data, obtain task instruction data through task assignment data, and robot dog and drone execute task instruction data; X3: Collects and updates data through robot dogs and drones, and evaluates and optimizes the updated data during the collection process to obtain enhanced updated data; X4: Updates the inspection baseline model by enhancing and updating data.

2. The joint inspection and control method of a drone and a robot dog according to claim 1, characterized in that, Step X1 specifically includes: The image stream data is corrected by the attribute stream data to obtain the first image data. The pose stream data is optimized based on the first image data to generate a grid surface model. The second image data is obtained based on the grid surface model and the optimized pose stream data. Feature matching and filtering are performed based on the second image data and image layer data. First matching point pair set and second matching point pair set are obtained respectively, and spatial mapping model is generated. Third image data is obtained based on spatial mapping model. Confidence data is constructed through third image data, image layer data and second matching point pair set. Based on confidence data, image layer data, and third image data, change probability data, binary change mask data, and semantic change mask data are obtained respectively, and dual-path filtering detection is performed to generate initial task data. Construct a spatial relationship graph, and perform clustering evaluation on the initial task data based on the spatial relationship graph to obtain task cluster data; Among these features, the image layer data is updated in real time.

3. The joint inspection and control method of a drone and a robot dog according to claim 2, characterized in that, Based on confidence data, image layer data, and third-party image data, change probability data, binary change mask data, and semantic change mask data are obtained respectively, and dual-path filtering detection is performed to generate initial task data, including: The image layer data and the third image data are detected by a change detection model to obtain change probability data. Based on the change probability data and confidence data, binarization is performed to obtain binary change mask data. Real-time semantic segmentation data is generated based on third-party image data. The real-time semantic segmentation data is compared and evaluated with image layer data to obtain semantic change mask data. We obtain fused change mask data by combining semantic change mask data and binary change mask data. We then perform connectivity analysis on the fused change mask data to obtain change object data. Finally, we filter the change object data to obtain initial task data.

4. The joint inspection and control method of a drone and a robot dog according to claim 2, characterized in that, The method further includes: The inspection baseline model includes geometric layer data and image layer data; The geometric layer data refers to the spatial structure data of the inspection area; The image layer data refers to the orthophoto data and semantic segmentation data of the inspection area; Drone inspection data includes image stream data, pose stream data, and attribute stream data; The image stream data refers to orthophoto data during UAV inspection; The pose flow data refers to the pose data during UAV inspection. The attribute stream data refers to the metadata during UAV inspections.

5. The joint inspection and control method of a drone and a robot dog according to claim 1, characterized in that, Step X2 specifically includes: Obtain robot dog attribute data, and based on the robot dog attribute data and task cluster data, obtain robot dog waiting data; The task-machine dog matching matrix is ​​obtained based on the robot dog waiting data and task cluster data. A scheduling objective function is constructed using the task-machine dog matching matrix and task cluster data. The scheduling objective function is then solved to obtain task assignment data. A robot dog navigation map is generated by using task assignment data and inspection benchmark model. Path search is performed based on the robot dog navigation map to obtain the robot dog's driving route. At the same time, guidance instructions are embedded into the robot dog's driving route through the guidance protocol to generate task instruction data. The task command data is sent to the robot dog and the drone. During the execution, the robot dog continuously outputs motion control commands to control the robot dog in real time.

6. The joint inspection and control method of a drone and a robot dog according to claim 5, characterized in that, A task-machine dog matching matrix is ​​obtained based on the robot dog waiting data and task cluster data. A scheduling objective function is constructed using the task-machine dog matching matrix and task cluster data. The scheduling objective function is then solved to obtain task assignment data, including: The capability matching coefficient is calculated based on the task cluster data and the robot dog waiting data. The path cost coefficient is calculated based on the task cluster data, the robot dog waiting data and the inspection benchmark model. At the same time, the constraint value is obtained. The task-robot dog matching matrix is ​​constructed by the capability matching coefficient, the path cost coefficient and the constraint value. Define decision variables, construct total cost and total capability terms based on the task-robot matching matrix, configure delay penalty and equilibrium terms, construct constraints through task cluster data and robot dog waiting data, and construct scheduling objective function based on total cost, total capability, delay penalty, equilibrium terms and constraints. The scheduling objective function is solved based on the algorithm, the initial task assignment data is obtained, the task conflicts in the initial task assignment data are checked, and the task assignment data is obtained.

7. The joint inspection and control method of a drone and a robot dog according to claim 5, characterized in that, The process of generating a robot dog navigation map using task assignment data and an inspection baseline model, and then performing path search based on the robot dog navigation map to obtain the robot dog's driving route includes: Obstacle information data is acquired, and an initial navigation map is generated based on the obstacle information data and the inspection benchmark model; A path cost map is generated from the initial navigation map, and then the path cost map is overlaid on the initial navigation map to generate the robot dog navigation map. Based on the task assignment data and using a global path search algorithm, the robot dog's initial movement route is obtained by searching the navigation map. The initial movement path of the robot dog is smoothed to obtain the robot dog's driving path.

8. The joint inspection and control method of a drone and a robot dog according to claim 1, characterized in that, Step X4 specifically includes: The enhanced update data is spatiotemporally located with the inspection baseline model, and a matching relationship between the enhanced update data and the inspection baseline model is established based on the spatial index. The inspection baseline model is updated by matching relationships and enhancing update data, wherein the update operations include creating and updating.

9. A joint inspection and control system for a drone and a robot dog, used to implement the method described in any one of claims 1 to 8, characterized in that, include: The robot dog collects data based on task instruction data, obtains updated data and enhanced update data, and has built-in optimization and control modules. The optimization module is used to optimize the updated data, and the control module continuously outputs robot dog motion control commands to control the robot dog in real time during the execution process. The drone is used to acquire drone inspection data and guide the drone dog based on task instruction data. The enhancement module is used to acquire initial task data and acquire task cluster data by optimizing the initial task data; The instruction generation module is used to obtain assignment data based on task cluster data and robot dog waiting data, process the assignment data, obtain task instruction data, and send it to the robot dog and drone. An update module is used to update the inspection baseline model; A baseline storage module is used to store the inspection baseline model.

Images