Fire-fighting mobile robot air-ground collaborative navigation method, equipment, medium and product

Abstract

The invention discloses a fire-fighting mobile robot air-ground collaborative navigation method and device, a medium and a product, and relates to the field of path planning, the method comprises the following steps: using a plurality of unmanned aerial vehicles to carry out aerial photography on a fire scene, and obtaining a plurality of continuous images in real time; performing global image splicing on a plurality of continuous images according to a set image overlapping index to obtain a real-time fire scene global image; carrying out semantic segmentation and identification on the real-time fire scene global image by adopting an image semantic segmentation network based on UNet; determining a planned route of the mobile robot by adopting a global path planning algorithm according to a departure place and a target place of the mobile robot, the fire scene global image after semantic segmentation and corresponding GPS information; and controlling the mobile robot to move according to the planned route. The fire scene condition can be monitored in real time, and the rescue and fire extinguishing efficiency, stability and intelligence of the fire-fighting mobile robot are improved.

Description

Technical Field

[0001] The present application relates to the field of path planning, and in particular to an air-ground collaborative navigation method, equipment, medium and product for a fire-fighting mobile robot. Background Art

[0002] When dealing with large-scale vicious fire and explosion accidents, firefighters need to fully understand the disaster situation at the disaster site and surrounding areas, and promptly grasp the trend of the disaster spread. They need to send personnel to the scene for reconnaissance. However, for some large-scale, regional disaster sites, sending personnel to conduct reconnaissance using a local, step-by-step approach is very time-consuming and laborious, and the scene is often full of dangers, posing a serious threat to the lives of the personnel entering the reconnaissance.

[0003] At present, mobile robots mostly use the navigation method of "artificial perception / decision-making + remote operation". The operator observes the situation around the mobile robot through a camera, plans the mobile robot's path and controls its movement. This navigation method has obvious disadvantages when the mobile robot is out of the operator's line of sight. After a fire or explosion accident, the topography of the road on the scene often changes greatly, and the original map information is no longer reliable. In addition, the mobile robot has a limited field of view and cannot obtain a global image in real time. If it only relies on the local environmental information around it obtained by the mobile robot, the mobile robot is prone to affecting the efficiency of reconnaissance work due to wrong path selection. In addition, due to factors such as low visibility at the scene, psychological tension, and unstable communication links, operators are prone to operating errors and collisions with surrounding obstacles, reducing the reliability of the fire mobile robot's movement.

[0004] Unmanned aerial vehicles (UAVs) are one of the more commonly used devices. UAVs have the advantages of "standing high and seeing far", and can obtain a wider field image. In order to overcome the limitations of ground mobile robots, air-ground mobile robot collaboration has become a hot topic in the current multi-agent research field. This model provides a wide range of environmental perception capabilities through flying mobile robots and provides navigation information for ground mobile robots. Through the air-ground collaborative system composed of UAVs and unmanned vehicles, the overall picture can be obtained at high altitudes, and the passable roads can be obtained through image segmentation, guiding the unmanned vehicle to quickly reach the target location.

[0005] Compared with the traditional single firefighting mobile robot, the air-ground collaborative multi-agent system can generate a path to the target point more quickly. Combining the properties of drones and unmanned vehicles to conduct feasibility analysis and realize intelligent and efficient rescue and firefighting countermeasures is of great significance to fire fighting and rescue.

[0006] However, the existing multi-agent system for air-ground collaboration has defects in intelligence and autonomy. On the one hand, it is highly dependent on human supervision, with low individual intelligence levels. In the face of complex and changing environments, autonomous decision-making and adaptability are poor, and the collaborative process is mostly based on human preset rules and lacks autonomy. On the other hand, system modeling is difficult and the model is difficult to explain. Deep learning algorithms are difficult to deploy directly due to air-ground terminal resources and real-time requirements. In addition, the current isomorphic strategy is mostly adopted, and the complementary advantages of air-ground mobile robots are not fully explored. Local observation decisions also limit the potential for cooperation, and the collaborative effect is not good.

[0007] The communication link is also full of problems. The air-to-ground mobile robot has a high demand for high bandwidth, but it is difficult to meet the demand in reality. The communication range is easily limited. Obstacles and long distances will cause interruptions or unstable signals. Even if efforts are made to reduce bandwidth, resource consumption is still high. Furthermore, the system has weak fault tolerance. Under the strong coupling characteristics, once some subsystems fail, the failure is easy to spread through the communication network, and the existing control strategy is difficult to suppress, which not only affects collaborative operations, but may even cause system crashes, resulting in economic losses and casualties.

[0008] Therefore, based on the above problems, there is an urgent need to provide an air-ground collaborative navigation method for a fire-fighting mobile robot, so as to be able to monitor the fire scene situation in real time and improve the efficiency, stability and intelligence of the fire-fighting mobile robot in rescue and fire-fighting. Summary of the invention

[0009] The purpose of this application is to provide a fire-fighting mobile robot air-ground collaborative navigation method, equipment, medium and product, which can monitor the fire scene situation in real time and improve the efficiency, stability and intelligence of the fire-fighting mobile robot in rescue and fire-fighting.

[0010] To achieve the above objectives, this application provides the following solutions: In a first aspect, the present application provides an air-ground collaborative navigation method for a firefighting mobile robot, the air-ground collaborative navigation method for a firefighting mobile robot comprising: Use multiple drones to take aerial photos of the fire scene and obtain multiple continuous images in real time; The global image is spliced ​​on multiple continuous images by setting the image overlap index to obtain a real-time global image of the fire scene; For the real-time global image of the fire scene, the UNet-based image semantic segmentation network is used to perform semantic segmentation and identification; According to the mobile robot's starting point, target point, semantically segmented global image of the fire scene and corresponding GPS information, a global path planning algorithm is used to determine the mobile robot's planned route; The mobile robot is controlled to move according to the planned route.

[0011] Optionally, the set image overlap index is an image overlap rate greater than or equal to 80%.

[0012] Optionally, the step of performing global image stitching on multiple continuous images by setting an image overlap index to obtain a real-time global image of the fire scene specifically includes: Use SURF algorithm to extract feature points and descriptors of each image; According to the feature points of each image and the descriptors of the feature points, the k-dtree algorithm and k-nn algorithm are used for feature matching; The RANSAC algorithm is used to screen the matched feature points to obtain the H matrix between the images; The H matrix between images is corrected using the waveform correction method; The real-time global image of the fire scene is obtained by using the H matrix between the corrected images and setting the image overlap index.

[0013] Optionally, the method further includes performing global image splicing on multiple continuous images by setting an image overlap index to obtain a real-time global image of the fire scene, and then further includes: Exposure compensation is performed on the real-time global image of the fire scene using block compensation; The segmentation linear filtering method is used to find and fuse the seam lines of the global image of the fire scene after exposure compensation.

[0014] Optionally, the real-time global image of the fire scene is semantically segmented and labeled using a UNet-based image semantic segmentation network, and the process also includes: Build an image semantic segmentation network based on UNet; Train the UNet-based image semantic segmentation network.

[0015] Optionally, the global path planning algorithm is algorithm.

[0016] In a second aspect, the present application provides an air-ground collaborative navigation device for a fire-fighting mobile robot, the air-ground collaborative navigation device for a fire-fighting mobile robot comprising: An image acquisition module is used to use multiple drones to take aerial photos of the fire scene and obtain multiple continuous images in real time; A global image determination module is used to perform global image splicing on multiple continuous images by setting image overlap indexes to obtain a real-time global image of the fire scene; The semantic segmentation module is used to perform semantic segmentation and identification on the real-time global image of the fire scene using the UNet-based image semantic segmentation network; The route planning module is used to determine the planned route of the mobile robot using a global path planning algorithm based on the mobile robot's starting point, target point, the global image of the fire scene after semantic segmentation, and the corresponding GPS information; The execution module is used to control the mobile robot to move according to the planned route.

[0017] In a third aspect, the present application provides a computer device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the air-ground collaborative navigation method of the fire-fighting mobile robot.

[0018] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the air-ground collaborative navigation method of the fire-fighting mobile robot.

[0019] In a fifth aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the air-ground collaborative navigation method of the fire-fighting mobile robot.

[0020] According to the specific embodiments provided in this application, this application has the following technical effects: The present application provides a method, device, medium and product for air-to-ground collaborative navigation of a fire-fighting mobile robot. The method obtains a real-time global image of the fire scene by stitching a plurality of continuous images together with a set image overlap index, and then updates the global image of the fire scene in real time, thereby realizing disaster monitoring of a large-scale fire scene. The method adopts a global path planning algorithm according to the mobile robot's starting point, target point, semantically segmented global image of the fire scene and corresponding GPS information, and then accurately identifies obstacles, plans multiple paths and sorts them, and shares them with the mobile robot to assist it in quickly reaching the designated location. The present application dynamically adjusts the path, and triggers it in real time with the drone and the mobile robot to ensure the completion of the dynamic path planning adjustment, thereby further improving the efficiency, stability and intelligence of the fire-fighting mobile robot in rescue and firefighting. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0022] Figure 1 This is a flow chart of an air-ground collaborative navigation method for a firefighting mobile robot in one embodiment of the present application; Figure 2 This is a schematic diagram of the image stitching process; Figure 3 This is a schematic diagram of the actual coverage of the image; Figure 4 This is a schematic diagram of k-nn matching; Figure 5 It is a schematic diagram of image relationship; Figure 6 This is a schematic diagram of the H matrix correction process; Figure 7 Schematic diagram of camera coordinates; Figure 8 It is a schematic diagram of the mapping relationship; Fig. 9 This is a schematic diagram of the seam line; Fig.10 This is a training flowchart for the image semantic segmentation network based on UNet; Fig.11 This is a schematic diagram of the eight-way passability of the mobile robot; Fig.12 It is a schematic diagram of the positions of the mobile robot and obstacles; Fig.13 Select a grid diagram for the mobile robot; Fig.14 Schematic diagram of the recalculated mobile robot passing through the grid; Fig.15 The final path diagram is shown below. DETAILED DESCRIPTION

[0023] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0024] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0025] In an exemplary embodiment, Figure 1 As shown, a method for air-ground collaborative navigation of a firefighting mobile robot is provided, and the method includes the following S101 to S105. Among them: S101 uses multiple drones to take aerial photos of the fire scene and obtain multiple continuous images in real time; S102, performing global image splicing on multiple continuous images by setting image overlap indexes to obtain a real-time global image of the fire scene; like Figure 2 As shown in the figure, the actual coverage of the collected images is calculated according to the flight altitude of the drone, and the flight mission is planned according to the 80% overlap area requirement. After the image is collected, image stitching is performed. Among them, image stitching also includes registration and fusion. Registration is to match and superimpose multiple images to obtain the homography matrix H and correct it, determine the reference mapping to obtain the preliminary stitching image; among them, image fusion is to correct color inconsistency and eliminate stitching gaps.

[0026] In order to successfully stitch the drone aerial images completely and smoothly, it is necessary to plan the flight mission according to the actual coverage of the images and the corresponding GPS information, so as to capture images with sufficient overlap. Ideally, the overlapping area of ​​the captured images should be no less than 60% in the heading direction and no less than 40% in the lateral direction. Due to the unstable flight during the drone flight control process, the actual overlapping area of ​​the captured images is not enough, resulting in the final stitching failure or poor effect.

[0027] Therefore, in order to ensure the success of image stitching as much as possible, 80% overlap is used as the index when designing the flight path. The diagonal field of view (DFOV) of the camera used for aerial photography on the drone should not be too large, otherwise the edge distorted image will be taken, which is not conducive to the calculation of the image coverage area. It is recommended to use a camera with a diagonal field of view of no more than 90°.

[0028] According to the actual coverage of the obtained image and Figure 3 The relationship shown can be obtained according to the formula Calculate the line spacing that satisfies the theoretical P (greater than or equal to 80%) image overlap ratio ;in, is the line interval of the image overlap rate, P is the theoretical (greater than or equal to 80%) image overlap rate, and A, A', B, and B' are the left and right endpoint coordinates of the two images respectively.

[0029] When planning and designing UAV flight missions, it is necessary not only to consider the image overlap rate, but also to slow down and switch paths at the turning point according to the UAV flight speed. Since the UAV will produce a certain pitch angle when slowing down and turning, the angle of the image collected in the turning area will change too much, which is not conducive to image stitching. Therefore, based on the above problems, it is necessary to appropriately extend the length of the front and rear ends of the path above the target area.

[0030] S102 specifically includes: S21, extracting feature points and descriptors of feature points of each image using SURF algorithm; The SURF algorithm is an upgraded version of SIFT, which not only improves the calculation speed but also has better robustness. The core idea of ​​the algorithm is to use the Hessian matrix to detect extreme points, use the box filter to obtain the approximate value of Gaussian blur, reduce the dimension of feature description, and improve efficiency.

[0031] The SURF algorithm mainly includes the following steps: (1) Pyramid construction: The spatial pyramid structure of the SURF algorithm keeps the image size unchanged during the creation process and continuously adjusts the box filter scale to achieve the creation of scale space, avoiding alternate point sampling and thus simplifying the calculation.

[0032] (2) Detect feature points: After the pyramid is created, the extreme points of each scale image need to be obtained through the Hessian matrix. The processed pixels are compared, and the extreme points are regarded as coarse feature points. Then they are screened through non-maximum suppression, and the three-dimensional linear interpolation method is used to remove points below the threshold, and the remaining ones are regarded as feature points.

[0033] (3) Determine the direction of the feature point: Use the Haar wavelet response to obtain the sum of the horizontal and vertical features of all points as the horizontal gradient and vertical gradient, and then use the Gaussian function for weighted processing to traverse and generate the vector with the longest vector length as the direction of the feature point, thereby ensuring the rotation invariance of the feature vector.

[0034] (4) Constructing descriptors: Referring to the scale of feature points, select a certain square range around each feature point and divide the range into smaller parts. Use Haar features to construct feature vectors. Finally, use the feature vector set of these subdivided areas as the descriptor of the feature point.

[0035] S22, using k-dtree algorithm and k-nn algorithm to perform feature matching according to the feature points of each image and the descriptors of the feature points.

[0036] The k-dtree algorithm is a generalization of a binary search tree in a K-dimensional space. The kd tree can quickly obtain information about the nearest and next nearest neighbor feature points with similar descriptors between the feature points of the two images. After finding them, the k-nn algorithm can be used to verify that the matching relationship is satisfied. In this application, k in the k-nn algorithm is 2.

[0037] Assume that there is a feature point A in image 1, and find the nearest neighbor feature point C and the next nearest neighbor feature point B in image 2 through k-dtree. The distance between point A and point B is LAB, and the distance between point A and point C is LAC. The matching diagram is as follows: Figure 4 shown.

[0038] If LAB and LAC satisfy the formula in the k-nn algorithm , it means that the matching feature point A in image 1 is C in image 2; otherwise, there is no matching feature point in image 2 for feature point A in image 1.

[0039] in, is the k-nn matching confidence, ranging from (0,1). According to the statistics in David G. Lowe's paper, points with similar distances to the nearest neighbor and the next nearest neighbor are more likely to have incorrect matches, so they are more likely to be "bad points", so increase A higher value of can leave more accurate matching point pairs, but will reduce the number of detected matching point pairs.

[0040] S23, using the RANSAC algorithm to screen the matched feature points to obtain an H matrix between the images.

[0041] The main steps of the RANSAC algorithm in feature point pair screening are: (1) Select four pairs of matching points obtained by feature detection, requiring that any three pairs are not collinear, and solve the H matrix (Homography matrix) with only 8 degrees of freedom.

[0042] (2) Using the obtained H matrix, perform matrix transformation on the feature points in one of the images except the feature points selected in step (1), obtain the estimated positions of these feature points in the other image, and record the distance D between the mapped point and the actual point.

[0043] (3) Compare the distance D with the threshold. The feature points whose distance is less than the threshold are called inliers, and the rest are outliers.

[0044] (4) Repeat the above steps a certain number of times and select the model that generates the most inliers.

[0045] (5) Using the interior point pairs in the model selected in step (4), calculate the H matrix again.

[0046] S24, using a waveform correction method to correct the H matrix between images.

[0047] After feature detection and matching, these selected matching inliers are used to evaluate and calculate the homography matrix H between the images. If there is a projection mapping relationship between the two images, then there must be a 3×3 H matrix between the two images, through which one image can be projected onto the other. Suppose the camera takes two images in different directions, such as Figure 5 As shown, assuming that the homogeneous coordinate point of the mapping of point P in the real world in image A is , the homogeneous coordinate point mapped in image B is ,but and There is a formula The transformation relationship between them.

[0048] By using the RANSAC algorithm to screen feature point pairs, we have obtained the H matrix between each pair of images. Then we can project all images onto a reference image for subsequent stitching. However, there is a certain error in the H matrix obtained by using RANSAC in feature detection and matching, which will lead to poor stitching results. There are two sources of error: (1) The calculation of the H matrix using the inlier pairs obtained by image feature detection and matching is a rough calculation and has certain errors.

[0049] (2) Image benchmark mapping does not directly use the rough H matrix between the benchmark image and all images, but uses the indirect mapping relationship of reliable edges. The mapping relationship between the benchmark image and all images is obtained by superimposing the H matrices of reliable images, and the error will also be accumulated, resulting in low accuracy of the overall mapping relationship.

[0050] In order to reduce the mapping error between the reference image and other images, the homography H matrix needs to be optimized. The optimization idea is to first roughly estimate the camera parameters through the relationship between the homography matrix H and the camera's internal and external parameters, and then refine the camera parameters through the bundle adjustment method. After refinement, the camera rotation parameters need to be waveform corrected, and finally the homography matrix H is recalculated using the camera parameters. The optimization flow chart is as follows: Figure 6 shown.

[0051] Based on the imaging principle, the camera parameters can be roughly estimated by using the relationship between the known homography matrix H and the camera parameters. The light reflected by the three-dimensional object is projected onto the imaging plane inside the camera through the camera pinhole. The object shape mapped by the imaging plane and the three-dimensional object itself are inverted. In order to facilitate coordinate transformation, a virtual imaging plane with a distance equal to the focal length is assumed between the pinhole and the three-dimensional object.

[0052] Construct the camera coordinate system XYZ, and the virtual imaging plane establishes the coordinate system xy, such as Figure 7 As shown. Assume that the coordinates of the point Q in the space in the coordinate system XYZ are , mapped to the virtual imaging plane as point q. The coordinates of point q in the coordinate system xy are , the coordinates in the XYZ coordinate system are .

[0053] According to the following formula, the conversion relationship between the virtual imaging plane coordinate system xy and the camera plane coordinate system XYZ is obtained.

[0054] Translation relationship ; Scaling Relationship ; ; in, and is the conversion of focal length f in horizontal and vertical axes and pixels, and is the coordinate translation, K is the camera internal parameter, and V represents the principal point translation.

[0055] According to the formula The conversion relationship between the virtual imaging plane coordinate system and the world coordinate system is obtained.

[0056] Among them, (X′, Y′, Z′, 1) is the coordinate of the world coordinate system; [R|t] is the camera external parameter.

[0057] Assume that at some point in the real world In image 1, the image coordinates are , in image 2, the image coordinates are , the relationship between the H matrix and the camera parameters between the two images can be obtained by the following formula.

[0058] .

[0059] .

[0060] .

[0061] .

[0062] Among them, H 10 Represents the homography matrix from image 1 to image 2; R 10 Represents the rotation matrix from camera 1 to camera 2.

[0063] The coordinate translation in the internal parameter can be regarded as no, then K=V, the formula Can be converted into a formula The focal length f of image 1 can be obtained by 0 and image 2 focal length f 1 When stitching multiple images, it is necessary to solve the focal lengths of each pair of images to obtain multiple estimated focal lengths. At this time, the median of this set of data can be taken as the focal length of all image cameras.

[0064] After the camera focal length f is obtained, it can be calculated by the formula Calculate the rotation matrix R in the camera extrinsic parameters.

[0065] In the rough estimation of camera parameters, the internal and external parameters of the camera are obtained. In order to reduce the error of the homography matrix H, the camera parameters need to be refined. This application uses the bundle adjustment method to adjust the parameters. The main idea of ​​the bundle adjustment method is that the light beam reflected by the three-dimensional object passes through the optical center of the camera lens to the imaging plane in a straight line. If the light beam from the actual point on the three-dimensional object and the corresponding point on the imaging plane to the optical center of the camera lens is not on the same straight line, the parameters of the camera structure and viewing angle need to be corrected to achieve the optimal ideal collinear state.

[0066] Assume that there are n objects in the real world and m images are taken with a camera. represents the actual coordinates of object i in image j, represents image j, represents object i, represents the predicted coordinates of object i mapped to image j by matrix. Because of the existence of error, the predicted coordinates and the actual coordinates are not equal, which is equivalent to not being collinear. In order to reduce the error and achieve collinearity as much as possible, the camera parameters can be adjusted so that the error function As small as possible, and then select the set of camera parameters with the smallest error as the optimized parameters.

[0067] Among them, when object i has an estimated point in image j, Takes 1; otherwise, takes 0. Represents the Euclidean distance between the estimated point and the actual point.

[0068] The LM algorithm is used in the optimization process of this application, which combines the Gauss-Newton method and the gradient descent method. The remapping error and the ray emission error are used as error indicators, and convergence iterations are performed continuously. The convergence step size is continuously reduced and it is stopped when it converges to a certain degree. The final convergence effect is quite good, but when encountering camera parameters with large errors, it may take a longer convergence operation time.

[0069] After the camera parameters are corrected by the bundle adjustment method, a waveform effect will be generated, causing the final stitched image to present a "snake-shaped" distribution. This is because the images collected by the drone cannot be taken in a completely horizontal state. The camera has a certain tilt and there is a certain angle between it and the gravity axis. Therefore, in order to straighten the "snake-shaped" distribution, it is necessary to introduce a global correction matrix Q for waveform correction. Multiply each camera rotation matrix R by the global correction matrix Q so that the y-axis of the global image is perpendicular to the x-axis of each generation of stitched aerial images. The waveform correction formula is as follows: .

[0070] in, , R k Represents the rotation matrix of the kth camera.

[0071] Through waveform correction, the camera extrinsic parameters can be corrected, and finally the corrected camera extrinsic parameters can be used to recalculate and obtain a more accurate H matrix.

[0072] Through the optimized and more accurate homography H matrix, one of the images to be stitched can be used as the reference image, and the remaining images can be projected onto the reference image plane using the H matrix for stitching. The initial H matrix is ​​calculated from the inliers between the two images. The more inliers there are between the two images, the more accurate the calculated H matrix is. Therefore, in order to improve the accuracy of the H matrix, reliable edges with more inliers are used to connect them during the reference mapping process to convert the H matrix. Figure 8 , assuming that image A is the reference image, the overlapping area between image B and image C is larger than the area between image A and image C. Generally speaking, the number of inliers between image B and image C is greater than the number of inliers between image A and image C. Therefore, in order to improve the accuracy of mapping, image B is directly mapped using the H matrix between AB, while image C needs to be mapped to A through the H matrix of BC and AB. It can be regarded that image C is first mapped to image B and finally mapped to the reference image A.

[0073] For C, A is the reference image and B is the intermediate image. The mapping of an image from the intermediate image to the reference image can be expressed by the following formula: .

[0074] Among them, pA represents the coordinates of point p in image A, pC represents the coordinates of point 𝑝 in image C, HBA represents the homography matrix mapping image B to image A, and HCB represents the homography matrix mapping image C to image B.

[0075] In order to realize the above-mentioned benchmark mapping method, the maximum spanning tree method is used to determine the benchmark image and connection method. The steps are as follows: (1) Take each image as a node of the maximum spanning tree and calculate the number of internal points between each pair of images as the distance between the node edges.

[0076] (2) For closed node states, nodes need to be connected according to the maximum number of internal points. Nodes A, B, and C are connected to each other. There is little overlap between C and A. C needs to be mapped to A using B. Therefore, the node edge between node C and node A needs to be deleted, leaving only the reliable edge with more internal points. This will give the maximum spanning tree.

[0077] (3) Calculate the distance from each node to the other nodes, take the maximum value calculated as the value of the node, and finally select the node with the smallest node value as the central node, which is the reference image.

[0078] S25, using the modified H matrix between the images and setting the image overlap index to obtain a real-time global image of the fire scene.

[0079] After the reference image is determined, reference mapping is performed according to the connection mode between different images in the maximum spanning tree to obtain a preliminary stitching image.

[0080] The preliminary stitched images have been obtained through image registration, but since the drone image acquisition is affected by the sensor exposure and the weather and light of the shooting scene, the images will have inconsistencies in color and brightness. The stitched images may have color differences, and exposure compensation is needed to eliminate the inconsistency, which is beneficial to the final presentation of the visual effect. Therefore, after S102, it also includes: The exposure compensation of the real-time global image of the fire scene is performed using block compensation.

[0081] The segmentation linear filtering method is used to find and fuse the seam lines of the global image of the fire scene after exposure compensation.

[0082] At present, the commonly used methods of exposure compensation for image stitching are gain compensation and block compensation. Gain compensation is to make the illumination of the overlapping images close to the same by assigning a specific gain coefficient to each image. Its error function is as follows: .

[0083] in, represents the overlapping part of image i and image j, m is the number of images used for image stitching, and are the gain coefficients of images i and j respectively, and are the average intensities of images i and j in the overlapping area.

[0084] The core idea of ​​block compensation is to divide the image into different blocks of the same size, assign a corresponding gain function to each block, and perform gain compensation on the block in this way. Because the block segmentation and corresponding gain compensation are implemented, the accuracy of block compensation is better, and the complexity of the gain compensation algorithm can also be reduced. Therefore, in order to obtain a more accurate stitching image, the exposure compensation method will be used for block compensation. In addition, because block compensation is exposure compensation with blocks as the object, after compensation, it is necessary to apply the segmentation linear filtering method for smoothing filtering to eliminate the abrupt connection between different blocks and process the whole into a harmonious visual state.

[0085] The global images of the fire scene after exposure compensation have good visual consistency, but the overlapping areas between the two images cannot be simply superimposed, otherwise the stitched image will have obvious overlapping traces. Therefore, it is necessary to find a seam line with the most similar pixels between the overlapping areas of the two images. The graph cut method is a seam line search method in image stitching. Its core idea is to regard the image as a directed graph, the image pixels as ordinary nodes, and the results of image clustering as terminal nodes. The image can be divided into two types of terminal nodes using the graph cut method, that is, all pixels will be classified into one of the two types of terminal nodes through the graph cut method, which also divides two areas. The pixel boundary between the areas is the seam line of the overlapping area. Assume that there are images A and B with a certain overlapping area. After finding the seam line of the overlapping area, the overlapping area will be covered by the image mask, leaving only the part on one side of the seam line of image A and the part on the other side of the seam line of image B for subsequent image stitching, such as Fig. 9 shown.

[0086] After the seam line is found, a tearing feeling will appear at the seam of the stitched image, which is a poor visual effect. In order to make the transition at the seam not appear abrupt, it is necessary to use fusion technology for certain processing. Common processing methods include feathering and multi-band fusion. If the feathering fusion operation is used, the seam can be smoothed well to achieve a certain gradient effect. However, the disadvantage of this fusion operation is that the seam line is blurred compared to the unprocessed image part, which is not conducive to the preservation of details. The multi-band fusion operation merges the data on multiple bands together, which can well preserve the local splicing details.

[0087] In air-ground collaborative path planning, it is necessary to preserve the details of any area of ​​the spliced ​​map, so that the drivable area obtained by semantic segmentation can meet the detection requirements. Therefore, in the air-ground collaborative path planning project, it is best to use multi-band fusion operation to preserve details as much as possible. The operation process of multi-band fusion operation is as follows: (1) Create a Laplacian pyramid for each image.

[0088] .

[0089] Among them, An is the n-th layer of Laplacian pyramid, Bn is the n-th layer of Gaussian pyramid, and E is the expansion operation.

[0090] (2) For overlapping areas, merge the overlapping layers of the Laplacian pyramid.

[0091] (3) Perform inverse Laplace transform on the merged pyramids and fuse them to obtain the final image; .

[0092] Among them, C represents the synthetic pyramid, D represents the fused pyramid, and E represents the expansion operation.

[0093] S103, using a UNet-based image semantic segmentation network to perform semantic segmentation and labeling on the real-time global image of the fire scene.

[0094] In order to realize the path planning of the mobile robot, it is necessary to segment the feasible area and obstacle area in the mosaic image of the drone aerial photography. The image segmentation of the passable area uses a deep neural network to achieve binary segmentation and output a raster map.

[0095] The training flow chart of the image semantic segmentation network based on UNet is as follows Fig.10 as shown; Fig.10 As shown in the figure, through the detection of drivable areas, the global environment image spliced ​​by drone aerial photography can be divided into three areas: road, open space and background. Then the segmented images are processed to generate an occupancy grid map.

[0096] UNet is a commonly used image semantic segmentation network with a U-shaped structure, including encoding, decoding and skip connection parts, and the layers are connected by maximum pooling or upsampling. UNet performs well in semantic and instance segmentation, has good real-time performance, and can meet project requirements, so it was selected as the road segmentation method. Each pixel in the segmentation map is classified as a semantic information of grass, road surface, flat land and background, and each category is represented by a different color.

[0097] The UNet dataset is created using the labelme image annotation plug-in, which is built on Python. The semantic parts of each image in the dataset are manually selected and annotated, and then the created dataset is input into UNet for training. The scale at which the dataset in UNet can achieve a good segmentation effect is 100-200 images. Adding the enhanced dataset will make the segmentation effect of the UNet model more accurate after training.

[0098] Image segmentation is an important research direction in computer vision. It is currently used in target recognition, medical image segmentation, autonomous driving and other fields. Image segmentation is mainly divided into semantic segmentation and instance segmentation. Semantic segmentation is to classify each pixel in the image into several semantic categories without distinguishing different instances in the same category. Each pixel in the segmentation map with semantic labels is classified as a semantic information of grass, road, flat ground and background, and each category is represented by a different color.

[0099] UNet is a commonly used network for image semantic segmentation. It is characterized by simple structure, accurate segmentation, short time consumption, and the ability to perform semantic segmentation based on small datasets. UNet is an improvement on the FCN network. Its structure is U-shaped. It is an encoder-decoder network with five layers, including the encoding part on the left, the decoding part on the right, and the skip connection part in the middle. Each layer is connected by maximum pooling (encoding part) or upsampling (decoding part).

[0100] In the encoding part, each layer of the image needs to go through two convolutions to extract features before it can be input into the next layer, and then the deep feature information is extracted through maximum pooling. In the decoding part, each layer of the image also goes through two deconvolutions, and the image layers are connected through upsampling to gradually restore their resolution to normal size. At the same time, due to the deepening of convolution and pooling in the encoder network, the segmentation may fall into the details and ignore the basic information of the image. Therefore, the same layer is connected through jumps, and the features that preserve the complete image information in the encoding part of the same layer are transmitted to the decoding part of the corresponding layer, which can more completely retain the information of the original image.

[0101] After image convolution, the width and height formula between the input image and the output image is as follows: .

[0102] .

[0103] in, Represents the image width, Represents the image height, represents the convolution kernel value, represents the convolution value, Represents the filling value, and the subscripts in and out represent the input image and output image respectively.

[0104] The advantage of using UNet to perform RGB semantic segmentation on the traversable part of the scene is that it does not require large-scale data set training, and can quickly train the neural network with a small-scale data set. UNet is also one of the most widely used image semantic segmentation networks in the field of autonomous driving. After years of accumulation and practice, its data set resources and types are very rich, such as the VOC2007 data set that can segment 20 types of semantics and road remote sensing data sets that are mostly used in the field of autonomous driving. At the same time, in the simulation scenario of this application, due to the limitation of the scene scale, the amount of data sets is smaller than in reality, so using UNet can obtain more accurate and efficient segmentation.

[0105] The UNet dataset is created using the labelme image annotation plug-in, which is built on Python. The semantic parts of each image in the dataset are manually selected and annotated, and then the dataset is input into UNet for training. Multiple types of semantic labels can be created in labelme, and after saving, a .json file with labeled images can be obtained. The scale at which the dataset in UNet can achieve a good segmentation effect is 100-200 images. Adding an enhanced dataset will make the segmentation effect of the UNet model more accurate after training.

[0106] Real-world images include various interferences such as changes in lighting and perspective. The greater the difference between the interference of the image and the original image in the dataset, the less accurate the segmentation effect. In addition, due to the limitation of scene scale, the collected dataset is small, so it is necessary to expand the dataset through dataset enhancement methods, increase noise, and generate more samples to obtain better segmentation results.

[0107] Commonly used dataset enhancement methods include image transformation (such as rotation, flipping, scaling), adjusting color contrast (such as logarithmic transformation), adding noise (such as Gaussian noise, Poisson distribution noise), etc. This application mainly uses the addition of Gaussian noise and logarithmic change to complete dataset enhancement.

[0108] (1) Gaussian noise: Gaussian noise refers to the noise whose probability distribution function follows the Gaussian distribution. This means that the pixel values ​​of the image are offset by the noise, and the amplitude of the offset satisfies the Gaussian distribution. In Gaussian noise, the probability density function satisfied by the added noise gray value can be expressed as: .

[0109] in, Represents the gray value noise added to the pixel, represents the standard deviation, Represents the expected value, by adjusting and to change the size of the Gaussian noise.

[0110] (2) Logarithmic transformation: Logarithmic transformation is used to expand the areas with lower grayscale values ​​in the image and compress the areas with higher grayscale values, thereby improving the contrast and showing the details of the low grayscale value parts.

[0111] The relationship between pixel values ​​before and after logarithmic transformation can be expressed by the following formula: ; in, represents the input pixel value, represents the output pixel value, It is a custom adjustment parameter that adjusts the degree of logarithmic transformation.

[0112] After completing the enhancement of the image dataset, a dataset of the original image and the enhanced image is created, which is input into UNet for training until the loss converges, and the spliced ​​image is input into UNet for training to obtain the segmentation effect.

[0113] S104, according to the starting location, target location, semantically segmented global image of the fire scene and corresponding GPS information of the mobile robot, a global path planning algorithm is used to determine the planned route of the mobile robot; the global path planning algorithm is algorithm. The algorithm can effectively simplify the search process and improve the efficiency of path solving, and is particularly suitable for global path planning.

[0114] The core principle of the algorithm is based on a detailed evaluation of node costs. For each node n, three key quantities are defined: the actual cost g(n) from the starting node to the current node, which is calculated by accumulating the weights of the edges on the path from the starting node to the current node, reflecting the real cost that has been paid; The estimated cost h(n) from the current node to the target node is usually estimated based on some heuristic function to predict future costs; The comprehensive cost f(n) = g(n) + h(n) is used to measure the overall quality of the node.

[0115] During the search process, the algorithm continuously maintains an open list and a closed list. The open list stores the nodes to be expanded, and the closed list stores the nodes that have been expanded. Each time, the node with the smallest f(n) value is selected from the open list for expansion.

[0116] Taking a two-dimensional grid map as an example, g(n) can be the number of steps from the starting point to the current point, and h(n) can be estimated using Manhattan distance, Euclidean distance, etc. For example, if the current node coordinates are (x1, y1) and the target node coordinates are (x2, y2), the Manhattan distance h(n) is calculated as |x1 - x2| + |y1 - y2|.

[0117] Through this clever cost evaluation and node selection mechanism, the algorithm greatly improves search efficiency while ensuring that the optimal path is found.

[0118] As an extension of Dijkstra's algorithm, the algorithm is widely used in pathfinding and graph traversal due to its high efficiency. For example, it is widely used in games such as StarCraft. Before understanding the algorithm, you need to know several concepts: The Search Area: The search area in the figure is divided into a simple two-dimensional array. Each element of the array corresponds to a small square. Of course, the area can also be divided into five-pointed stars, rectangles, etc. The center point of a unit is usually called the search area node.

[0119] OpenList: The nodes to be detected in the path planning process are stored in OpenList, and the grids that have been detected are stored in CloseList.

[0120] Parent node: A node used for backtracking in path planning. It can be considered as a parent node pointer in a doubly linked list structure during development.

[0121] Path Sorting: The specific node to move to is determined by the following formula: F(n)=G+H. G represents the moving cost from the initial position A along the generated path to the specified grid to be tested. H specifies the estimated moving cost from the grid to be tested to the target node B.

[0122] The algorithm adopts a heuristic strategy and uses path-related heuristic information to guide the search, that is, it evaluates the cost of each search node and uses this to determine the priority of the node search. Different from the inefficiency of the traditional blind path search strategy, The algorithm can effectively simplify the search process and improve the efficiency of path solving, and is particularly suitable for global path planning.

[0123] Evaluation function of the algorithm The definition of is the minimum cost estimate from the initial node through n nodes to the destination node. Its general form is as follows: ; in, The cost that node n has paid to get from the starting point, The estimated cost for node n to reach the target node.

[0124] The size of can determine the comprehensive priority of n nodes. The smaller the value of is compared to other nodes, the smaller the cost is and the higher the priority of node search is. From the perspective of evaluation function, The algorithm can be seen as a combination of the Dijkstra algorithm and the best greedy algorithm. In a grid map, the Dijkstra algorithm can find the optimal solution to the path problem. The distance between node n and the starting point is used as the evaluation criterion of the Dijkstra algorithm, and the optimal path is searched by continuously expanding outward. The best greedy algorithm uses the distance between node n and the target node as the evaluation criterion, and selects the node with the lowest current cost as the node to be searched each time. In this way, the path can be solved while searching fewer nodes, but it may not be the optimal solution. In global path planning, The algorithm can be viewed as taking Dijkstra's cost as , the cost of the best greedy algorithm is taken as ,Through this combination, the optimal solution can be obtained efficiently.

[0125] exist In the algorithm, there are two sets, one is used to store the node set openlist to be detected, and each time the cost in the openlist is prioritized. The small node n is searched, and the other is used to store the detected set closelist in the operation. The overall code description is as follows: (1) Initialize openlist and closelist.

[0126] (2) Put the set starting point into the openlist and give it the highest priority.

[0127] (3) If there are points in the openlist, the node n with the highest priority is selected for processing. The processing conditions are divided into A and B: A. If node n is the target point, then trace back from node n to the parent node to the starting point, find the final result path, and the search ends.

[0128] B. If node n is not the target point, delete node n from the openlist to the closelist, then traverse the adjacent nodes near node n, skip the adjacent nodes that are already in the closelist, add the nodes that are neither in the closelist nor in the openlist to the openlist, and set their priorities using the evaluation function.

[0129] (4) Repeat step (3) until the search is completed and the final path is obtained.

[0130] In the evaluation function, It is called a heuristic function. The choice of heuristic function affects the efficiency of the algorithm and also determines Can the algorithm find the optimal solution path? If the evaluation in the greedy optimal algorithm (that is, the minimum estimated cost of the path from state n to the target state) is called , then the selection of heuristic function can be roughly divided into the following three cases: (1) If H(n) is always less than or equal to ,but The algorithm will definitely find the optimal path solution. As H(n) continues to decrease, The Dijkstra algorithm property in the algorithm is continuously enhanced, and the computational efficiency is reduced. When H(n) decreases to zero, it degenerates into the Dijkstra algorithm.

[0131] (2) If H(n) is equal to , the efficiency is highest at this time.

[0132] (3) If H(n) is greater than ,but The best greedy algorithm in the algorithm has enhanced properties, fewer search points, and high efficiency, but it may not necessarily obtain the optimal solution path.

[0133] In the distance cost estimation, different distance calculation rules are adopted due to the different movement forms allowed in the map. There are currently three common classifications: (1) If the map only allows movement in four directions: up, down, left, and right, Manhattan distance is used.

[0134] (2) If the map only allows movement in eight directions: up, down, left, right, upper left, lower left, upper right, and lower right, then the diagonal distance is used.

[0135] (3) Cartography allows movement in any direction, so Euclidean distance is used.

[0136] In the simplified model, H uses the traditional Manhattan distance, which is the sum of the horizontal and vertical distances.

[0137] The area to be searched is divided into square grids. This is the first step in pathfinding, simplifying the search area. This special method simplifies the search area into a 2D array. Each item in the array represents a grid, and its state is either walkable or unwalkable. The algorithm finds the shortest path from A to B. The side length of each square is 10, that is, the cost of moving vertically and horizontally is 10. Therefore, the cost of moving diagonally is approximately 14. The specific steps are as follows: Start from the starting point A and add it to an openlist (open list) composed of squares. This openlist is like a shopping list. The squares in the openlist may be passed along the way, or they may not be passed. Therefore, it can be regarded as a list to be checked. Check the 8 squares adjacent to A and add the walkable or reachable squares to the openlist. And set the starting point A as the parent node of these squares. Then remove A from the openlist and add it to the closedlist (closed list). Every square in the closelist does not need to be paid attention to.

[0138] like Fig.11 As shown in the figure, the dark green square is the starting point A, and its outer frame is bright blue, indicating that the square has been added to the closelist. The black squares adjacent to it need to be checked, and their outer frames are bright green. Each black square has a gray pointer pointing to its parent node A.

[0139] Next, you need to select a square from the openlist that is adjacent to the starting point A, and choose the one with the smallest F value. Fig.12 Some squares in the game. In the lettered squares, G=10. This is because there is only one square horizontally from the start point. The squares directly above, below, and to the left of the start point all have a G value of 10, and the diagonal squares all have a G value of 14. The H value is calculated by estimating the Manhattan distance from the start point to the end point (the red square), moving only horizontally and vertically, and ignoring obstacles along the way. Using this method, the square to the right of the start point is 3 squares away from the end point, so H=30. The square above this square is 4 squares away from the end point (note that only the horizontal and vertical distances are calculated), so H=40.

[0140] like Fig.12 As shown in the figure, after comparing the F values ​​of the nodes in the openlist, it is found that the node on the right side of the starting point A has the smallest F value of 40. This node is selected as the current processing node, and this point is deleted from the OpenList and moved to the CloseList.

[0141] Check the 8 grids around this node. If they are impassable (such as walls, water, or other illegal terrain) or already in the CloseList, ignore them. Otherwise, execute the following steps: If the adjacent grid of the current processing node is already in the OpenList, check whether this path is better, that is, calculate whether the grid reached through the current processing node has a smaller G value. If not, do nothing. On the contrary, if the G value is smaller, set the parent node of that grid to the current processing node (the selected grid), and then recalculate the F value and G value of that grid.

[0142] If the adjacent grid of the currently processed node is not in the OpenList, add it and set its parent node to this node.

[0143] Continue searching according to the above rules, and select the square to the right of the starting point as the current processing node. Its outer frame is highlighted with a blue line and is placed in the closelist. Then check the squares adjacent to it. The three squares to its right are walls, so ignore them. The square to its left is the starting point, which is in the closelist and is also ignored. The other four adjacent squares are all in the openlist, and you need to check whether the path to get there via the current node is better. Look at the square above, its current G value is 14. If you get there via the current square, the G value will be 20 (10 of which is the G value from the starting point to the current square, and the G value of 10 for moving vertically from the current square to the square above must be added), so this is not the optimal path. Looking at the picture, you will understand that moving directly from the starting point along the diagonal to that square is better than moving horizontally and then vertically.

[0144] After checking the 4 adjacent squares that are already in the openlist, no better path is found through the current node, so no change is made. Next, we need to select the next node to be processed. So we traverse the openlist again. Now there are only 7 squares in the openlist, and we need to select the one with the smallest F value. This time, there are two squares with F values ​​of 54. Considering the speed, it is faster to select the square that was added to the openlist last. So we select the square to the lower right of the starting point, such as Fig.13 shown.

[0145] Next, take the square with an F value of 54 in the lower right corner of the starting point as the current processing node, and check its adjacent squares. It is found that there is a wall on its right (the square below the wall is also ignored, assuming that the corner of the wall cannot be directly crossed). In this way, there are 5 adjacent squares left. The 2 squares below the current square have not been added to the openlist, so add them and set the current square as their father. Among the remaining 3 squares, 2 are already in the closelist (one is the starting point, and the other is the square above the current square, with the outer frame highlighted), so ignore them. The last square, which is the square to the left of the current square, checks whether there is a smaller G value when reaching there via the current square. If not, prepare to select the next square to be processed from the openlist.

[0146] Repeat this process until the end point is added to the openlist. Fig.14 Notice that the parent of the square 2 squares below the starting point is different now. Previously it had a G value of 28 and pointed to the square above and to the right. Now it has a G value of 20 and points to the square directly above it. This is because somewhere in the pathfinding process the G value is lower when using a new path, so the parent node is reset and the G and F values ​​are recalculated.

[0147] like Fig.15 As shown, starting from the end point, move along the arrow toward the parent node until you return to the starting point, which is the final path.

[0148] The stitched global image is transformed into a global occupancy grid map using image processing methods such as binarization, downsampling, image erosion, and coordinate transformation.

[0149] S105, controlling the mobile robot to move according to the planned route.

[0150] The following is an explanation through a specific embodiment. Before the UAV flies, the image coverage is calculated based on the UAV camera resolution, camera diagonal field of view, UAV flight altitude and other parameters to determine whether the image collected at this altitude can cover the target area. If the theoretical calculation meets the requirements, the UAV flight mission data can be set and image acquisition can be performed; if the theoretical calculation does not meet the requirements, the flight altitude needs to be adjusted. In the test, the UAV flew at an altitude of 100m and the camera resolution was 1920×1080. After image acquisition, a series of aerial images of the target area that meet the 60% overlap rate of the heading are selected for stitching; during image registration, algorithms such as Surf are used to detect, match and filter feature points, the camera parameters are corrected to obtain the H matrix, and the maximum spanning tree is used to determine the benchmark mapping relationship. After registration, fusion is performed, and after compensation and multiple operations, the global map result is obtained through image stitching technology; through the semantic segmentation of the global map, the semantic information of each drivable area, the semantics of flat land, grass and road surface are obtained; after the data set is produced, the improved Unet model is used for training, and the data set is divided into training set and validation set at a ratio of 9:1. 50 epochs are performed to make the loss gradually converge and obtain the training model. The trained improved Unet model is used to detect the drivable area of ​​the rescue area synthesized by image stitching to obtain a semantic segmentation map; since the semantic segmentation is for three categories: roads, open spaces and obstacles, among which roads and open spaces belong to the drivable area, it is necessary to perform an image binarization to convert roads and open spaces into one category. At the same time, a 3×3 square convolution kernel is used to erode the image, and then the generated image digital information is extracted to obtain a grid map. The image coordinates are transformed into an image form with the bottom corner as the origin to generate an occupancy grid map. The drivable area semantic segmentation map is converted into an occupancy grid map after image processing, and then the grid map is generated based on Path planning of the algorithm. Assuming that the mobile robot in the grid map of this embodiment can move in eight adjacent directions, the diagonal distance heuristic function is used. Algorithms can plan paths quickly and efficiently.

[0151] Based on the same inventive concept, the embodiment of the present application also provides a firefighting mobile robot air-ground collaborative navigation device for implementing the firefighting mobile robot air-ground collaborative navigation method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more firefighting mobile robot air-ground collaborative navigation device embodiments provided below can refer to the limitations of the firefighting mobile robot air-ground collaborative navigation method above, and will not be repeated here.

[0152] In an exemplary embodiment, an air-ground collaborative navigation device for a firefighting mobile robot is provided, comprising: An image acquisition module is used to use multiple drones to take aerial photos of the fire scene and obtain multiple continuous images in real time; A global image determination module is used to perform global image splicing on multiple continuous images by setting image overlap indexes to obtain a real-time global image of the fire scene; The semantic segmentation module is used to perform semantic segmentation and identification on the real-time global image of the fire scene using the UNet-based image semantic segmentation network; The route planning module is used to determine the planned route of the mobile robot using a global path planning algorithm based on the mobile robot's starting point, target point, the global image of the fire scene after semantic segmentation, and the corresponding GPS information; An execution module is used to control the mobile robot to move according to the planned route. In an exemplary embodiment, a computer device is provided, which can be a server or a terminal. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for air-ground collaborative navigation of a fire-fighting mobile robot is implemented.

[0153] In an exemplary embodiment, a computer-readable storage medium is provided, storing a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0154] In an exemplary embodiment, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0155] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0156] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0157] The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a distributed database based on blockchain, etc., but is not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but is not limited thereto.

[0158] In this application, all actions to obtain signals, information or data are carried out in compliance with the relevant data protection laws and policies of the country where they are located and with the authorization given by the owner of the corresponding device.

[0159] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0160] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A fire fighting mobile robot air-ground collaborative navigation method, characterized in that: The air-ground collaborative navigation method of the fire-fighting mobile robot comprises: Use multiple drones to take aerial photos of the fire scene and obtain multiple continuous images in real time; The global image is spliced ​​on multiple continuous images by setting the image overlap index to obtain a real-time global image of the fire scene; For the real-time global image of the fire scene, the UNet-based image semantic segmentation network is used to perform semantic segmentation and identification; According to the mobile robot's starting point, target point, semantically segmented global image of the fire scene and corresponding GPS information, a global path planning algorithm is used to determine the mobile robot's planned route; The mobile robot is controlled to move according to the planned route.

2. The air-ground collaborative navigation method of a firefighting mobile robot according to claim 1 is characterized in that: The set image overlap index is greater than or equal to an image overlap rate of 80%.

3. The air-ground collaborative navigation method of a firefighting mobile robot according to claim 1, characterized in that: The method of performing global image stitching on multiple continuous images by setting image overlap index to obtain a real-time global image of the fire scene specifically includes: Use SURF algorithm to extract feature points and descriptors of each image; According to the feature points of each image and the descriptors of the feature points, the k-dtree algorithm and k-nn algorithm are used for feature matching; The RANSAC algorithm is used to screen the matched feature points to obtain the H matrix between the images; The H matrix between images is corrected using the waveform correction method; The real-time global image of the fire scene is obtained by using the H matrix between the corrected images and setting the image overlap index.

4. The air-ground collaborative navigation method of a firefighting mobile robot according to claim 1, characterized in that: The method further comprises: performing global image splicing on multiple continuous images by setting image overlap index to obtain a real-time global image of the fire scene; and then: Exposure compensation is performed on the real-time global image of the fire scene using block compensation; The segmentation linear filtering method is used to find and fuse the seam lines of the global image of the fire scene after exposure compensation.

5. The air-ground collaborative navigation method of a firefighting mobile robot according to claim 1, characterized in that: The real-time global image of the fire scene is semantically segmented and marked using a UNet-based image semantic segmentation network, which also includes: Build an image semantic segmentation network based on UNet; Train the UNet-based image semantic segmentation network.

6. The air-ground collaborative navigation method of a firefighting mobile robot according to claim 1, characterized in that: The global path planning algorithm is algorithm.

7. A fire fighting mobile robot air-ground collaborative navigation device, characterized in that: The air-ground collaborative navigation device of the fire fighting mobile robot comprises: An image acquisition module is used to use multiple drones to take aerial photos of the fire scene and obtain multiple continuous images in real time; A global image determination module is used to perform global image splicing on multiple continuous images by setting image overlap indexes to obtain a real-time global image of the fire scene; The semantic segmentation module is used to perform semantic segmentation and identification on the real-time global image of the fire scene using the UNet-based image semantic segmentation network; The route planning module is used to determine the planned route of the mobile robot using a global path planning algorithm based on the mobile robot's starting point, target point, the global image of the fire scene after semantic segmentation, and the corresponding GPS information; The execution module is used to control the mobile robot to move according to the planned route.

8. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the air-ground collaborative navigation method for a fire-fighting mobile robot according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the air-ground collaborative navigation method for a fire-fighting mobile robot described in any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the air-ground collaborative navigation method for a fire-fighting mobile robot described in any one of claims 1 to 6 is implemented.

Images