A robot-assisted flame recognition and positioning method

Abstract

The present invention discloses a method for robot-assisted flame visual recognition and positioning, which relates to the field of computer vision and includes the following steps: using a calibrated camera to obtain a video image; obtaining a suspected flame dynamic target based on an improved background difference method based on a flame color model; calculating and extracting a series of static features of the target; inputting the fused model into a trained support vector machine (SVM); if it is determined to be a flame, then calculating the rough positioning of the suspected flame based on the monocular, binocular, or multi-camera vision that identified the flame, and guiding the robot to a specified position; the robot's pan / tilt platform is aligned with the suspected flame position, and dynamic and temperature features are extracted respectively. After the features are fused, they are input into another SVM; if the result of the second recognition is still a flame, fine positioning is performed. The present invention has high recognition accuracy and strong applicability, and can minimize the hazards of fire by providing early warning and precise spatial positioning of flame targets.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to a robot-assisted flame recognition and positioning method. Background Art

[0002] As humanity develops, its use of fire has reached new heights. However, while fire brings immense convenience to our lives, its frequent occurrence poses a significant threat to life, property, and even society. Major fires are often caused by late detection, missed opportunities for fire control, or ineffective initial containment. Therefore, identifying and locating the fire source as quickly as possible is crucial to reducing or even preventing the damage it causes.

[0003] Currently, common fire monitoring solutions, including various detectors that are often fixedly installed in key areas within a space, convert the physical indicators of a fire, such as temperature, smoke, heat, and radiation, collected by the sensors into voltage or current signals. These signals then use set thresholds to determine if a fire is currently occurring. However, these solutions often fail to accurately provide information on the location of the flame, the size of the fire, or the cause of the fire. Furthermore, traditional detectors are susceptible to the constraints of physical space. The structure and ventilation of the indoor environment can easily affect the detector's detection range. For large and spacious buildings like museums, factories, and warehouses, traditional detectors struggle to meet the needs of timely and effective fire detection and location. With the increasing popularity of video surveillance and the continuous development of image processing technology, research on methods for using video images to monitor fires has gradually become mainstream.

[0004] The current limitations of visual flame detection lie in its high false alarm rate and low detection rate. Highly saturated objects with colors similar to flames, such as sunsets, lights, and balloons, can easily lead visual recognition to mistakenly identify flames as triggering alarms. By raising the relevant thresholds, this in turn makes it impossible to identify true flames. On the other hand, for small targets in the early stages of a flame, limited by the resolution of the camera and the limitations of traditional algorithms, visual recognition struggles to immediately issue early warnings for small flames, let alone determine the specific location of the initial fire. Furthermore, commonly used flame location methods, which determine the depth position of flame targets based on the difference in binocular vision field, also have limitations during use. Binocular or multi-camera methods rely on the simultaneous capture of flame targets, but the camera field of view is often limited, and the randomly generated flame positions are difficult to predict. Traditional methods cannot guarantee the specific location of the flame.

[0005] In summary, the current flame identification and positioning methods still have great limitations, and many problems need to be solved urgently. Summary of the Invention

[0006] The purpose of the present invention is to overcome the technical defects of existing flame identification and positioning methods, and propose a robot-assisted flame identification and positioning method. This method accurately identifies the location of the flame while ensuring a high accuracy rate, and realizes real-time 3D position positioning of the initial fire, greatly improving the time of fire emergency response and is also applicable to more complex and general scenarios.

[0007] To achieve the above objectives, the technical solution of the present invention is as follows: A robot-assisted flame identification and positioning method, specifically:

[0008] First, a camera is used to capture images in real time. The first stable image captured is used as the background frame, which is updated in real time. Pixels are calculated using a background difference method based on a flame color model. If a pixel is larger than a set threshold, it is considered a suspected flame target through morphological operations and verified. Static features of the suspected flame target are extracted and input into an SVM for feature fusion. The resulting output is a preliminary flame identification, and the location of the suspected flame area is calculated.

[0009] Based on the output of the preliminary flame identification and the location of the suspected flame area, the mobile robot uses navigation and positioning algorithms to go to the suspected flame area, rotate the pan-tilt camera, and use the visible light camera and infrared camera to re-identify the flame, extract the corresponding dynamic features and temperature features, fuse the dynamic features and temperature features, and input them into the second SVM for binary classification. If it is judged not to be a flame, the background frame is updated; if it is judged to be a flame, the precise spatial position of the flame is calculated.

[0010] Compared with the prior art, the present invention has the following beneficial effects:

[0011] 1. This invention uses a background subtraction method based on an improved flame color model. This method is computationally simple, and its background value is taken from a static scene, making it more suitable for foreground detection at a fixed angle. It also exhibits strong adaptability to changing scenes, allowing for updating the background image. It also combines RGB and HSI models to eliminate interference from dynamic factors, such as sudden changes in lighting and pedestrian movement.

[0012] 2. The fixed camera recognition method in this invention extracts more robust static features, taking advantage of the long installation distance. This reduces computation time while also training the SVM to perform preliminary identification of suspected flame targets. The robot's secondary recognition process combines visible light with infrared imagery, addressing both the vulnerability of visible light to smoke obstruction and interference, and the problem of infrared's low resolution, which can lead to misidentification of heat-generating or heat-reflecting objects. This combination overcomes the shortcomings of both methods and improves flame recognition accuracy.

[0013] 3. The present invention adopts a method that combines coarse positioning and fine positioning, rather than just installing one or more sets of binocular cameras, which takes into account both economic applicability and the robustness of the system. The location where a fire occurs is often random and it is impossible for it to be within the preset field of view of the binocular camera. If you want to expand the scope of detection, you need to set up more cameras to solve this problem. The fact is that how to arrange the cameras, how to rearrange a set of reasonable camera positions in a new environment to more completely cover the area where a fire may occur, is difficult to deal with the problem of economy and robustness. The present invention does not focus on accurately identifying the location of the flame in the first stage, but adopts a combination of coarse positioning and fine positioning to locate the location of the flame in two steps in a more flexible way, which provides an important basis for the subsequent solution of fire hazards with or without people.

[0014] 4. This invention is applicable to large indoor spaces and complex structures, such as exhibition halls, art galleries, computer rooms, auditoriums, high-rise buildings, and shopping malls; it is also applicable to special spaces such as tunnels, mines, oil rigs, and ancient buildings; and it is also applicable to isolated or hazardous areas, such as nuclear power plants, hazardous material storage areas, chemical plants, and sewage treatment plants. This invention has a wide range of applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 A flowchart of a robot-assisted flame identification and positioning method provided by the present invention;

[0016] Figure 2 Schematic diagram of the local binary method algorithm used in the present invention;

[0017] Figure 3 This is a schematic diagram of the visual coarse positioning of the present invention;

[0018] Figure 4 This is a schematic diagram of the robot-assisted positioning of the present invention. DETAILED DESCRIPTION

[0019] The present invention is a robot-assisted flame identification and positioning method, which requires a camera and a robot for implementation. To more clearly illustrate the technical solutions in the embodiments of the present invention, the following will further describe in detail a specific embodiment of the robot-assisted flame identification and positioning method of the present invention in conjunction with the drawings in the specification of this application. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of this application.

[0020] The present invention provides a robot-assisted flame identification and positioning method, and its specific operation process is as follows: Figure 1 The specific method is as follows:

[0021] (1) Install enough cameras in the space to ensure that every area where a fire may occur is covered. Each camera has been calibrated with a checkerboard grid and has obtained internal and external parameters. Its coordinate position M in the global coordinate system is also known. i , i∈{1,2,3…N}, N is the number of installed cameras.

[0022] (2) Each camera uploads the collected video data to the processing terminal and the cloud. The camera obtains the first stable image as the background frame. If the background does not change suddenly, the background frame is updated every set period of time, such as once an hour. At the same time, the background difference algorithm is used. The generation and dynamic growth of the flame can be responded to by the background difference algorithm. The grayscale value of each pixel in the current frame is subtracted from the grayscale value of the corresponding pixel in the background frame, and the absolute value is taken to obtain the pixel point. The formula is as follows:

[0023] |It(x,y)-Bt(x,y)|

[0024] Among them, It is the grayscale function of the current frame, and Bt is the grayscale function of the background frame.

[0025] (3) If the pixel point calculated in step (2) is greater than the set threshold, after certain morphological operations (such as expansion, corrosion, opening and closing, etc.), it is regarded as a suspected flame target. At the same time, the RGB and HSI channel values of the image pixel point corresponding to the suspected flame target are calculated to verify whether the relationship conforms to the flame color model combined with the RGB model and the HSI model. If it conforms, the suspected flame target is obtained. The RGB model divides visible light into three channels: red, blue, and green, and uses certain mathematical formulas and thresholds to determine whether the pixel point conforms to the color characteristics of the flame. The HSI model separates brightness, saturation, and color, making up for the shortcomings of a single color model. The relationship between the flame color model combined with the RGB model and the HSI model is as follows:

[0026]

[0027] R T is the threshold of the red component, S T is the saturation threshold, R T Range is 115-135, S T The range is 55-65.

[0028] (4) Extracting static features of the suspected flame target obtained in step (3), wherein the static features include texture features, roundness features, sharp angle features, and center of gravity height features. The obtained features are subjected to feature fusion.

[0029] Since the temperature inside the flame is uneven, the flame has a layered structure and certain texture characteristics. The present invention adopts the local binary pattern (LBP) method to analyze the flame texture characteristics. The grayscale values of the adjacent 8 pixels are compared with the center pixel of the window as the threshold. If the surrounding pixel value is greater than the center pixel value, the position of the pixel is marked as 1, otherwise it is 0. Figure 2 shown.

[0030] The circularity feature is the degree of circular similarity, and the formula is as follows:

[0031]

[0032] Among them, C is the circularity; L is the circumference; S is the area.

[0033] The sharp corner feature is often a narrow and long feature of flames. Where f(v) is the number of pixels in a row of sharp corners, and f(v-1) is the number of pixels in the previous row of sharp corners. This ratio is used to calculate whether the target has sharp corners or how many sharp corners there are.

[0034] The center of gravity height characteristic is the ratio of the center of gravity height of the flame area to the total height. The formula is as follows:

[0035]

[0036] Among them, H c is the height of the center of gravity of the flame area, and H is the total height.

[0037] (5) The texture features, circularity features, sharp corner features, and center of gravity height features obtained in step (4) are input into the trained support vector machine (SVM) for feature fusion. The embodiment of the present invention uses the Sigmoid kernel function as the training method for the SVM nonlinear feature mapping. If the output result is not a suspected flame target, steps (2)-(4) are repeated and the background frame is updated. On the other hand, if the output is a suspected flame target, coarse positioning is performed, that is, the suspected flame generation area is calculated.

[0038] (6) Taking the center of gravity of the suspected flame target as its image coordinates (u, v, 1) on the original image, the actual coordinate value of the target flame can be calculated and the area where the suspected flame is generated can be obtained.

[0039] The image coordinates are obtained from the center point of the corresponding target in the original video stream. Given the camera intrinsic parameters, the environment coordinates and posture of the camera installation, and assuming that the flame is burning on the ground, the two-dimensional coordinates of the flame on the ground can be calculated. The line connecting this position and the camera coordinates is the theoretically possible position of the flame. However, considering that this method has a large error, a range with a higher confidence level is output.

[0040] Specifically, the formula is as follows:

[0041]

[0042] Where M is the camera's intrinsic parameter matrix, R is the rotation matrix, t is the translation matrix, and s is the depth of the suspected flame target point in the Z direction of the camera coordinate system (calculated from the camera's intrinsic and extrinsic parameters); the solution set P about the suspected flame target position is obtained, and P is a straight line in space. Project the solution set P on the ground to obtain the projection set P0, {(x,y)|y=kx+b,m≤x≤n}, and then take the average value of the solution is the center of the circle and the diameter is As the diameter, make a circle, such as Figure 3 As shown, the suspected fire area is transmitted to the mobile robot.

[0043] (7) The mobile robot can be a tracked, footed or wheeled robot, and is equipped with sensors such as an inertial meter, an encoder, a laser radar, a pan-tilt camera, and a high-resolution camera. After receiving the instruction to go to the suspected flame target area, the mobile robot can autonomously locate and navigate, and can control its movement through a terminal or a local area network. It uses the navigation and positioning algorithm to autonomously plan the path and take necessary obstacle avoidance movements. After reaching the suspected flame area, the robot stops moving and starts to rotate the pan-tilt to search for the suspected flame target. Since the position of the suspected target is roughly known, the pan-tilt rotates in the suspected direction. The infrared camera it carries is used to capture the high-temperature area exceeding the set threshold. After capturing it, it is aimed at the suspected flame so that it appears in the center of the visible light camera and infrared camera images. The target image is larger than that of the fixed camera, and it is easier to extract features.

[0044] (8) The visible light camera extracts the dynamic characteristics of the target, namely the flame area change characteristics and the flame flicker characteristics. The flame area is judged based on the color model. In each frame, the flame area changes continuously. At the same time, the brightness of the pixels in the flame area also changes over time. The brightness of the pixels inside the flame changes less than that outside the flame, but the brightness of the pixels inside the flame changes more than that of the pixels in the ambient area. This characteristic can be used to describe the flame flicker characteristics. It can be seen that the flame flicker frequency is in the range of 6Hz-9Hz. At the same time, the infrared camera extracts the temperature characteristics of the target, such as the temperature gradient value and the distribution characteristics of the high-temperature area.

[0045] (9) The dynamic features in the visible light image obtained in step (8) and the temperature features in the infrared image are fused and input into the trained second support vector machine SVM for binary classification. If the output is not a flame, return to step (2) and update the background frame. If the output is a flame, the fire is determined and precise spatial positioning is enabled. Given the actual posture of the robot and the pitch angle and rotation angle of the gimbal, as well as the flame position in the image collected by the robot, a calculation method similar to step (6) can be used to obtain the solution set Pr obtained by the camera on board the robot to identify the flame. The solution set P and the solution set Pr of the suspected flame target position are both straight lines in space. The intersection point between the two straight lines or the midpoint of the minimum distance between the two straight lines is obtained, which is the precise position coordinate of the flame obtained, as shown in FIG. Figure 4 As shown in the figure, this method requires less computation than traditional methods, can synchronize images captured by the robot camera and the fixed camera, and output the calculated precise coordinates in real time. Furthermore, since flames have jumping characteristics and the coordinates of their flame cores also change in real time, the method of the present invention can obtain real-time coordinates more accurately.

[0046] (10) The precise location of the flame is calculated according to step (9), and the fire information is transmitted to the automatic alarm controller, and further processing is performed manually.

[0047] Finally, it should be noted that the above content is a further detailed description of the present invention in combination with specific preferred embodiments. It cannot be determined that the specific embodiments of the present invention are limited to these. For ordinary technicians in the technical field to which the present invention belongs, they can make several simple deductions or substitutions without departing from the concept of the present invention, which should be regarded as belonging to the scope of patent protection determined by the submitted claims of the present invention.

Claims

1. A robot-assisted flame identification and positioning method, characterized in that: The main steps include: First, a camera is used to capture images in real time. The first stable image captured is used as the background frame, which is updated in real time. Pixels are calculated using a background difference method based on a flame color model. If a pixel is larger than a set threshold, it is considered a suspected flame target through morphological operations and verified. Extract the static features of the suspected flame target and input them into SVM for feature fusion, output the preliminary identification of the flame, and calculate the location of the suspected flame occurrence area; Based on the initial flame identification and the location of the suspected flame, the mobile robot uses navigation and positioning algorithms to move to the suspected flame area, rotate the pan / tilt, and use the visible light camera and infrared camera to re-identify the flame, extract the corresponding dynamic and temperature features, fuse the dynamic and temperature features, and input them into the second SVM for binary classification. If it is judged not to be a flame, the background frame is updated; if it is judged to be a flame, the precise spatial position of the flame is calculated. In the process of calculating the suspected flame location and the precise flame spatial location, the center of gravity of the flame target is first used as its image coordinates (u, v, 1) on the original image. The two-dimensional coordinates of the target flame on the ground are calculated based on the camera intrinsic parameters, the environment coordinates of the camera installation, and the posture. The line connecting these two-dimensional coordinates and the camera coordinates is the flame generation area. The formula is as follows: Among them, M is the camera internal parameter matrix, R is the rotation matrix, t is the translation matrix, and s is the depth of the suspected flame target point in the Z direction of the camera coordinate system; The above formulas are used to obtain the solution set P for the suspected flame target position and the solution set Pr for the precise flame position identified by the camera carried by the robot; In the calculation of the suspected fire area, the solution set P is projected onto the ground to obtain the projection set P0, {(x, y)|y=kx+b,m≤x≤n}, and then the average value of the solution is used. is the center of the circle and the diameter is Draw a circle with the diameter as the fire suspected area; In the calculation of the precise flame spatial position: the solution set P and the solution set Pr of the suspected flame target position are straight lines in space, and the intersection point between the two straight lines or the midpoint of the minimum distance between the two straight lines is calculated to obtain the precise flame spatial coordinate position.

2. The robot-assisted flame identification and positioning method according to claim 1, characterized in that: The calculation of the pixel point is specifically as follows: subtract the grayscale value of each pixel point in the current frame from the grayscale value of the corresponding pixel in the background frame, and take the absolute value to obtain the pixel point. The formula is as follows: |It(x,y)-Bt(x,y)| Among them, It is the grayscale function of the current frame, and Bt is the grayscale function of the background frame.

3. The robot-assisted flame identification and positioning method according to claim 1, characterized in that: The suspected flame target is obtained through morphological operation and verified as follows: if the calculated pixel point is greater than the set threshold, the suspected flame target is obtained through morphological operations such as expansion, corrosion, opening and closing, and the RGB and HSI channel values of the image pixel points corresponding to the suspected flame target are calculated simultaneously to verify whether it conforms to the flame color model combined with the RGB model and the HSI model. The formula for the verification relationship is: R T is the threshold of the red component, S T is the saturation threshold, R T Range is 115-135, S T The range is 55-65.

4. The robot-assisted flame identification and positioning method according to claim 1, characterized in that: The static features include texture features, roundness features, sharp corner features and center of gravity height features; The circularity feature is the degree of circular similarity, and the formula is as follows: Where C is the circularity; L is the perimeter; S is the area; The sharp corner feature is a narrow and long feature of the flame, through the ratio Control, where f(v) is the number of pixels in a row of sharp corners, and f(v-1) is the number of screams in the previous row of the sharp corner. This ratio is used to calculate whether the suspected flame target has sharp corners or how many sharp corners it has; The center of gravity height characteristic is the ratio of the center of gravity height of the flame area to the total height, and the formula is as follows: Among them, H c is the height of the center of gravity of the flame area, and H is the total height.

5. The robot-assisted flame identification and positioning method according to claim 1, characterized in that: The mobile robot is a crawler-type, foot-type or wheel-type robot equipped with sensors such as an inertial meter, an encoder, a laser radar, a pan-tilt camera and a high-resolution camera.

6. The robot-assisted flame identification and positioning method according to claim 1, characterized in that: The dynamic characteristics include flame area change characteristics and flame flicker frequency characteristics.

7. The robot-assisted flame identification and positioning method according to claim 6, characterized in that: In the flame flicker characteristic, the flame flicker frequency is 6 Hz-9 Hz.

Images