A real-time flame positioning method for fire-fighting robots in a fire scene

Abstract

The present application relates to a kind of real-time flame positioning method of fire-fighting robot under fire scene, belong to robot mapping positioning and computer vision image segmentation technical field, the positioning method includes: S1, making data set and training flame segmentation model;S2, mobile fire-fighting robot real-time positioning and mapping, obtain the camera pose of current time;S3, application image segmentation algorithm divides current image frame flame area;S4, extract matching feature points in flame area and use binocular depth estimation algorithm to obtain the coordinate of flame point;S5, using clustering algorithm separates different flame instance and filters false detection flame point;S6, the clustering center obtained in S5 is calculated and output flame center point positioning coordinate.The present application enhances the credibility and robustness of flame area extraction, improves the positioning accuracy of flame center point in real world, can help fire-fighting robot to carry out real-time accurate fire extinguishing operation.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of robot mapping and positioning and computer vision image segmentation, and in particular to a real-time flame positioning method for a fire-fighting robot in a fire scene. BACKGROUND

[0002] Image segmentation algorithms using computer vision technology can achieve pixel-level positioning of the target, which is very important for applications that require accurate positioning. At the same time, with the continuous development of the field of deep learning research, image segmentation methods based on deep learning have been able to achieve high accuracy. However, for accurate positioning of the flame, it is still necessary to combine a method that can obtain accurate real-world coordinates.

[0003] Chinese patent application CN108038867A discloses a fire detection and positioning method based on multi-feature fusion and stereo vision. The method mainly uses an improved Vibe algorithm and an OHTA color segmentation method to extract suspected flame areas and then determines whether they are real flame areas by combining image shape features. A stereo matching algorithm is then used for flame positioning. However, the above method uses multiple traditional image processing methods to determine the flame area, which has the problems of complex calculation and low robustness. In addition, the calculated flame points are not further filtered and fused to enhance the credibility, which can easily cause false detection. SUMMARY

[0004] The technical problem to be solved by the present application is to provide a real-time flame positioning method for a fire-fighting robot in a fire scene, which solves the problems of inaccurate flame positioning and difficulty in achieving real-time for the fire-fighting robot. The method enables the fire-fighting robot to detect fire in real time and quickly locate the flame center world coordinates when a fire occurs, thereby achieving automatic fire extinguishing and fire extinguishing operations.

[0005] To solve the above technical problems, the technical solution adopted by the present application is:

[0006] A real-time flame positioning method for a fire-fighting robot in a fire scene, comprising the following steps:

[0007] S1, making a data set and training a flame segmentation model;

[0008] S2, moving the fire-fighting robot to position and map in real time, and obtaining the camera pose at the current time;

[0009] S3, applying an image segmentation algorithm to segment the flame area in the current image frame;

[0010] S4, extracting matching feature points in the flame area and using a binocular depth estimation algorithm to obtain the coordinates of the flame points;

[0011] S5, using a clustering algorithm to separate different flame instances and filter false flame detection points;

[0012] S6, calculating the cluster center output by S5 to output the flame center point positioning coordinates.

[0013] Further improvement of the technical scheme of the present application is that S1 specifically comprises:

[0014] 1.1, using a web crawler tool to collect images of fire scenes in a search engine gallery, and intercepting part of the video images of the fire scene, cleaning the unqualified image data, and then merging into a total flame image data set, using an image labeling tool to label the outline of the flame in the image to obtain a flame segmentation data set;

[0015] 1.2, training the flame data set using an image segmentation model; during the training process, the image data is preprocessed by data enhancement, including: horizontal flip, multi-scale change, random cropping and random color jittering with a certain probability;

[0016] 1.3, deploying the trained model to a fire-fighting robot operation platform and using tensorrt acceleration.

[0017] Further improvement of the technical scheme of the present application is that S2 specifically comprises:

[0018] 2.1, using a slam algorithm for positioning and mapping, after initialization, for each newly added image frame, the matching relationship with the previous frame is calculated, and the PnP algorithm is used to calculate the relative pose between the two frames, so that the pose of the camera corresponding to the current image frame can be obtained;

[0019] The pose calculation formula used is:

[0020]

[0021] Wherein, p1 and p2 are the coordinates of the matching feature points on the previous frame and the current frame image, and F is the fundamental matrix; is the transpose of the p2 matrix;

[0022] 2.2, at least eight matching point pairs are required to solve the above formula; after solving F, the rotation matrix and translation matrix of the camera, i.e. the pose of the current camera, are obtained by using matrix decomposition method combined with the camera intrinsic parameter.

[0023] Further improvement of the technical scheme of the present application is that S3 specifically comprises: for the image collected by the fire-fighting robot camera, the color image of each key frame in the slam algorithm is input into the image segmentation model trained and deployed by S1 to obtain the flame segmentation result of the image; if the segmentation result does not contain a flame area, no further processing is performed; if the segmentation result contains a flame area, the binary image of the result is passed to the next step.

[0024] Further improvement of the technical scheme of the present application is that S4 specifically comprises:

[0025] 4.1, extract the feature point detection and matching result of the image frame in the slam algorithm, and perform an AND operation on the matched feature points and the segmentation result obtained in step S3 to obtain the matching feature point pair (Pli, Pri) of the flame area corresponding to the left and right images; for the obtained matching point pair, the depth information of the flame point Pi is obtained using a binocular depth estimation calculation method, which is represented by the following formula:

[0026]

[0027] Wherein, b represents the center distance of the binocular camera, and the unit is meter; f is the focal length of the camera, and the unit is pixel; x l , x r is the horizontal coordinate of the matched flame point of the left and right eyes, and the unit is pixel;

[0028] 4.2, combined with the camera pose corresponding to the current image frame calculated by the slam algorithm in S2, the world coordinates Pwi=(xi,yi,zi) of the real flame point are obtained.

[0029] Further improvement of the technical scheme of the present application is that S5 specifically comprises: using a spatial point clustering algorithm to cluster the real world coordinate flame point set obtained in step S4, the number of clusters is not fixed, which is determined by the clustering algorithm according to the actual distribution of data, that is, the number of clusters is equal to the number of fire locations; at the same time, it is set that only when the number of flame points in each cluster after clustering is greater than a certain threshold value, the flame cluster is considered as a real flame, otherwise it is considered as a model false detection point and discarded.

[0030] Further improvement of the technical scheme of the present application is that S6 specifically comprises: according to the clustering information of the points obtained in S5, the centroid of all flame points in each class is calculated, that is, the centroid is taken as the center position Pt(x~i,y~i,z~i) of the flame in space; then the center position of the flame is passed to the fire-fighting robot for fire extinguishing operation.

[0031] Due to the adoption of the above technical scheme, the present application has achieved the following technical progress:

[0032] The application greatly improves the effectiveness and accuracy of flame region extraction by adopting a deep learning-based image segmentation algorithm, and can adapt to various fire environments; based on the extracted flame region, a binocular depth estimation method is adopted, combined with a robot simultaneous localization and mapping algorithm, so that the real-world coordinates of the flame points can be obtained; on this basis, a spatial density-based clustering algorithm is used to filter out false flame points, and the center coordinates of the fire pile are obtained by calculating the cluster center, so that the mobile fire-fighting robot can more accurately locate the flame center position for more effective fire extinguishing operation. Based on this method, the probability of fire occurrence and expansion can be greatly reduced, and the loss of people's life and property caused by the fire can be greatly reduced. BRIEF DESCRIPTION OF DRAWINGS

[0033] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor;

[0034] Figure 1 is the overall flow chart of the real-time flame positioning method of the fire-fighting robot in the fire scene in the embodiment of the present application;

[0035] Figure 2 is the flame segmentation effect diagram in the embodiment of the present application;

[0036] Figure 3 is the flame point coordinate conversion schematic diagram in the embodiment of the present application;

[0037] Figure 4 is the flame point positioning diagram in the embodiment of the present application. DETAILED DESCRIPTION

[0038] It should be noted that the terms "include" and "have" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device containing a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0039] The embodiment of the application provides a real-time flame positioning method of a fire-fighting robot in a fire scene, aiming at the problem of low credibility of flame area extraction, adopts a lightweight image segmentation model based on deep learning to enhance the credibility and robustness of flame area extraction, and simultaneously aims at the problem of inaccurate real position positioning of a flame point, combines a robot simultaneous positioning and mapping method, and adopts a spatial point clustering algorithm to perform flame point clustering and filtering to improve the positioning accuracy of a flame center point in a real world.

[0040] The application will be further described in detail below with reference to the drawings and embodiments:

[0041] As shown in the figure, a real-time flame positioning method of a fire-fighting robot in a fire scene comprises the following steps: Figure 1

[0042] S1, making a data set and training a flame segmentation model;

[0043] Specifically, a network crawler tool is used to collect images of fire scenes in a search engine library, and the keywords are forest fire, indoor fire and factory fire; part of the video images of the flame scene are intercepted, the unqualified image data are cleaned, and then merged into a total flame image data set; an image labeling tool is used to label the contour area of the flame in the image to obtain a flame segmentation data set, and a total of 500 flame instances are obtained.

[0044] The flame data set is trained using an image segmentation model STDC-Seg. During the training process, the image data are preprocessed through data enhancement, including horizontal flip with a probability of 0.5, multi-scale change (320*320, 512*512, 768*768), random cropping with a probability of 0.4, and HSV color space jitter with a probability of 0.4.

[0045] The trained model is deployed on a fire-fighting robot operation platform and the model inference acceleration is performed using a tensorrt tool.

[0046] S2, real-time positioning and mapping of the fire-fighting robot, and obtaining the camera pose at the current time;

[0047] Specifically, the control platform used is Nvidia NX, and an algorithm based on ov2slam is used to map and position the environment around the robot. After initialization, for each newly added image frame, the matching relationship with the previous frame is calculated, and a PnP algorithm is used to calculate the relative pose between the two frames, so that the pose of the camera corresponding to the current image frame can be obtained. The pose calculation formula used is:

[0048]

[0049] ​Wherein, p1, p2 are the coordinates of the matching feature points on the previous frame and the current frame image, F is the fundamental matrix, is the transpose of p2 matrix.

[0050] At least eight matching point pairs are required to solve the above formula; after solving F, the rotation matrix and translation matrix of the camera, i.e. the pose Rt, Tt of the camera at the current time t, can be obtained by using matrix decomposition method.

[0051] S3, applying image segmentation algorithm to segment the flame region of the current image frame;

[0052] Specifically, for the image collected by the robot camera, in the slam algorithm, for each detected key frame, the RGB image corresponding to the key frame is input into the image segmentation algorithm trained and deployed in S1, and the flame segmentation result of the image is output, and the segmentation result is shown in Figure 2 If the segmentation result does not contain the flame region, no further processing is performed. If the segmentation result contains the flame region, the binary image of the result is passed to the next step for processing.

[0053] S4, extracting matching feature points in the flame region and combining the binocular depth estimation algorithm to obtain the coordinates of the flame points;

[0054] Extract the feature point detection and matching result of the image frame in the slam algorithm, and perform the AND operation between the matching feature points and the segmentation result obtained in S3 step, so as to obtain the matching feature point pair (Pli, Pri) of the flame region corresponding to the left and right images. For the obtained matching point pair in the flame region, the depth information of the flame point Pi is obtained by using the binocular depth estimation calculation method, which is expressed by the formula as follows:

[0055]

[0056] Wherein, b represents the center distance of the binocular camera, and the unit is meter; f is the focal length of the camera, and the unit is pixel; x l , x r are the horizontal coordinates of the matching flame points of the left and right eyes, and the unit is pixel.

[0057] After the depth zc of the flame point is recovered, the pixel value (u, v) of the point and the parameters of the camera are combined, and the coordinate transformation can be used to calculate the coordinates of the flame point in the camera coordinate system Pfc=(xc, yc, zc), which is expressed by the formula as follows:

[0058]

[0059] Wherein, f, u0, v0 are the intrinsic parameters of the camera, which can be obtained by camera calibration.

[0060] The camera pose corresponding to the current image frame calculated by the slam algorithm in S2 is combined, and the world coordinates Pfw=(xw, yw, zw) of the real flame point are obtained through coordinate transformation. The conversion of the camera coordinates of the flame point to the world coordinates is shown in Figure 3 The preliminary positioning result of the flame point is shown in Figure 4

[0061] S5, using a clustering algorithm to separate different flame instances and filter false detection flame points;

[0062] A spatial point clustering algorithm is used to cluster the real world coordinate flame point set obtained in step S4. Specifically, the DBSCAN clustering algorithm is used, which clusters according to the density and adjacency of the point set in space. The number of clusters is not fixed, and is determined by the clustering algorithm according to the actual distribution of the data, i.e., the number of clusters is equal to the number of fire locations. At the same time, the number of flame points in each cluster after clustering is set to be greater than a certain threshold, and the flame cluster is considered to be a real flame, otherwise it is considered to be a model false detection point and is discarded.

[0063] S6, calculating the clustering center output flame center point positioning coordinates obtained in S5.

[0064] According to the clustering information of the points obtained in S5, the centroid of all flame points in each class is calculated, i.e., the centroid is taken as the center position Pt(x~i, y~i, z~i) of the flame in space. Then the center position of the flame can be transmitted to the fire-fighting robot, and after combining with the GPS auxiliary positioning to reduce the error, the fire-fighting robot can perform fire-fighting operations. The flame centroid P ci The calculation formula is as follows:

[0065]

[0066] where N ci represents the number of points in the i-th cluster point set, and P cij represents the coordinates of the j-th point in the i-th cluster point set.

[0067] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, and are not limited thereto; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or make equivalent substitutions for part or all of the technical features; and these modifications or substitutions do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.​

Claims

1. A real-time flame positioning method for a firefighting robot in a fire scene, characterized in that: The method comprises the following steps: S1, making a data set and training a flame segmentation model; S2, real-time positioning and mapping of the mobile fire-fighting robot to obtain the camera pose at the current time; S3, applying an image segmentation algorithm to segment the flame region in the current image frame; Specifically, for the image collected by the camera of the fire-fighting robot, the color image of each key frame is input into the image segmentation model trained and deployed in S1 in the slam algorithm to obtain the flame segmentation result of the image; if the segmentation result does not contain a flame region, no further processing is performed; if the segmentation result contains a flame region, the binary image of the result is passed to the next step; S4, extracting matching feature points in the flame region and using a binocular depth estimation algorithm to obtain the coordinates of the flame points; Specifically, 4.1, extracting the feature point detection and matching result of the image frame in the slam algorithm, performing an AND operation on the matched feature points and the segmentation result obtained in S3, so as to obtain the matching feature point pair (Pli, Pri) of the flame region corresponding to the left and right images; for the obtained matching point pair, the depth information of the flame point Pi is obtained by using a binocular depth estimation calculation method, which is represented by the following formula: wherein b represents the center distance of the binocular camera, in meters; f is the focal length of the camera, in pixels; x l , x r is the horizontal coordinate of the matched flame point, in pixels; 4.2, combining the camera pose corresponding to the current image frame calculated by the slam algorithm in S2 to obtain the world coordinates Pwi=(xi, yi, zi) of the real flame point; S5, using a clustering algorithm to separate different flame instances and filter the misdetected flame points; S6, calculating the clustering center obtained in S5 to output the positioning coordinates of the flame center point.

2. The real-time flame positioning method for fire-fighting robots in a fire scene according to claim 1, characterized in that: In S1, specifically comprising: 1.1, using a web crawler tool to collect images of fire scenes in a search engine gallery, and intercepting video images of part of the fire scene, cleaning the unqualified image data, and then merging into a total flame image data set, using an image labeling tool to label the outline of the flame in the image to obtain a flame segmentation data set; 1.2, training the flame data set using an image segmentation model; during the training process, the image data is preprocessed by data enhancement, including: horizontal flip, multi-scale change, random cropping, and random color jittering with a certain probability; 1.3, deploying the trained model to the fire-fighting robot operation platform and using tensorrt for acceleration.

3. The method of claim 1, wherein the method comprises: In S2, specifically comprising: 2.1, using a slam algorithm for positioning and mapping, after initialization, for each newly added image frame, the matching relationship with the previous frame is calculated, and a PnP algorithm is used to calculate the relative pose between the two frames, so as to obtain the pose of the camera corresponding to the current image frame; The pose calculation formula used is: Wherein, p1, p2 are the coordinates of the matching feature points on the previous frame and the current frame image, and F is the fundamental matrix; is the transpose of the p2 matrix; 2.2, at least eight matching point pairs are required to solve the above formula; after solving F, the rotation matrix and translation matrix of the camera are obtained by using matrix decomposition method, that is, the pose of the current camera.

4. The real-time flame positioning method for fire-fighting robots in a fire scene according to claim 1, characterized in that: S5 specifically includes: using a spatial point clustering algorithm to cluster the real-world coordinate flame point set obtained in S4, the number of clusters is not fixed, and is determined by the clustering algorithm according to the actual distribution of data, that is, the number of clusters is equal to the number of ignition positions; at the same time, it is considered that the flame cluster is a real flame only when the number of flame points in each cluster after clustering is greater than a certain threshold, otherwise it is considered to be a model false detection point and is discarded.

5. The real-time flame positioning method for fire-fighting robots in a fire scene according to claim 1, characterized in that: S6 specifically includes: according to the clustering information of the points obtained in S5, calculating the centroid of all flame points of each class, i.e. taking the centroid as the center position of the flame in the space Subsequently, the center position of the flame is transmitted to the fire-fighting robot for fire extinguishing operation.

Images