Fire scene trapped personnel posture recognition and positioning method based on computer vision

Abstract

This invention discloses a computer vision-based method for posture recognition and localization of trapped personnel in fire scenes, relating to the fields of computer vision and fire rescue. It solves the problems of difficult human body recognition, inaccurate posture estimation, and low positioning accuracy in existing fire scene environments. This invention simultaneously acquires visible light and thermal infrared images, constructs a human body probability function by combining temperature, gradient, and fire source temperature difference, and extracts candidate regions for the human body. It determines the smoke fluctuation frequency through temporal spectral analysis of continuous images, establishes a temporal buffer window to fusion and compensate for key human body points, overcoming smoke occlusion. The compensated key points are aligned with a semantic label map to construct a crushing index to determine whether a trapped person is crushed. The true distance is iteratively solved through coupled geometric optics and thermal radiation attenuation to achieve precise three-dimensional positioning. This invention significantly improves the accuracy and stability of posture recognition and localization in dense smoke fire scenes, providing reliable support for fire rescue.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and fire rescue, specifically a method for posture recognition and positioning of people trapped in a fire based on computer vision. Background Technology

[0002] With the rapid development of computer vision technology, image analysis-based personnel detection, pose estimation, and localization technologies have been widely applied in fields such as security monitoring and human-computer interaction. In fire rescue scenarios, quickly and accurately locating trapped personnel and assessing their condition is crucial for developing rescue plans and ensuring personnel safety. However, the fire scene environment is characterized by high temperatures, dense smoke, and complex and variable lighting, posing severe challenges to traditional visual recognition methods.

[0003] Currently, human detection and pose estimation technologies based on visible light images are relatively mature in normal environments. Deep learning networks can extract key points of the human body and identify pose categories. However, in fire environments, dense smoke severely obscures visible light images, causing human features to become blurred or even completely invisible, making effective identification difficult to achieve solely based on visible light. Thermal infrared imaging technology can penetrate smoke of a certain concentration to obtain temperature distribution information of objects, providing a new technical approach for personnel detection in fire environments. However, thermal infrared images lack texture details, making it difficult to perform accurate pose estimation on their own.

[0004] Furthermore, dynamic smoke interference, obstacle obstruction, and thermal radiation attenuation in fire scenes make accurate location of trapped personnel difficult. Traditional monocular visual ranging methods rely on clear image feature points, which are prone to significant errors under smoke interference; while distance estimation based on thermal radiation intensity is affected by multiple factors such as smoke concentration and ambient temperature, resulting in uncertainty. Therefore, how to integrate the advantages of visible light and thermal infrared images to achieve accurate attitude recognition and reliable location of trapped personnel in complex fire environments has become an urgent technical problem to be solved in this field.

[0005] The following problems exist in the existing technology:

[0006] Existing methods for detecting human bodies in fire scenes mostly rely on single visible light or thermal infrared images; visible light images are easily obscured by dense smoke, resulting in the loss of human features; although simple thermal infrared images can penetrate smoke, they are difficult to distinguish between human bodies and interference sources such as high-temperature fire sources and heated objects, which can easily lead to false detections. Conventional pose estimation networks are designed for clear images. However, in a fire scene, dynamic smoke intermittently obscures the images, causing a decrease in the confidence of key human figures or even their loss. As a result, pose recognition has poor continuity and low accuracy. Existing technologies can only identify a person's posture and cannot combine environmental obstacle information to determine whether a person is trapped under collapsed objects, furniture, etc., making it difficult to provide a basis for assessing rescue priorities. Traditional monocular vision ranging relies on geometric optical models, but in dense smoke environments, the blurring of image feature points leads to large ranging errors. Although thermal infrared ranging can reflect the relationship between distance and radiation intensity, it is difficult to separate the influence of smoke concentration and distance when used alone, resulting in poor positioning reliability. Summary of the Invention

[0007] The present invention aims to solve at least one of the technical problems existing in the prior art; to this end, the present invention proposes a computer vision-based method for posture recognition and localization of people trapped in a fire, in order to solve the above-mentioned technical problem.

[0008] The first aspect of this invention provides a method for posture recognition and localization of trapped personnel in a fire based on computer vision, comprising the following steps: S1: Simultaneously acquire visible light and thermal infrared images of the fire scene environment, and obtain the pose parameters of the imaging device; S2: Based on the temperature value, gradient magnitude, and the difference between the pixel temperature value and the preset typical temperature of the fire source for each pixel in the thermal infrared image, a human body probability function is established, and the connected regions of pixels whose human body probability function values ​​are greater than a preset threshold are taken as human body candidate regions. S3: Perform temporal spectral analysis on the pixels in the candidate human body region in multiple consecutive frames of images to determine the smoke fluctuation frequency, and establish a temporal buffer window based on the smoke fluctuation frequency; extract human body key points and corresponding confidence scores in the current frame through a pose estimation network, and take human body key points with confidence scores lower than a preset threshold as key points to be compensated; extract human body key points corresponding to the image coordinates of the key points to be compensated from the historical frames in the temporal buffer window and fuse them to obtain compensated human body key points; determine the pose category of the trapped person based on the compensated human body key points. S4: Perform semantic segmentation on the visible light image and the thermal infrared image to generate a semantic label map of environmental obstacles; spatially align the compensated human key points with the semantic label map, calculate the proportion of human key points falling into the obstacle area, and determine whether the trapped person is in a crushed state based on the proportion and the posture category. S5: Based on the compensated human body key points and the pose parameters of the imaging device, perform geometric optical calculations to obtain the preliminary distance; establish a thermal radiation attenuation observation equation based on the thermal infrared grayscale observation value and preset physical parameters; use the preliminary distance and preset smoke concentration coefficient as initial values ​​to perform coupled iterative solution on the thermal radiation attenuation observation equation to obtain the converged true distance; update the three-dimensional coordinates of the trapped person based on the true distance. S6: Output the posture category, the pressure state determination result, and the three-dimensional coordinates.

[0009] Preferably, the human body probability function in S2 Build as:

[0010] Where T(p) is the temperature value of pixel p, and G(p) is the gradient magnitude of pixel p. The preset gradient threshold constant, To preset the typical temperature of the fire source, To preset the transition band width, The standard normal cumulative distribution function is... and These are the mean and standard deviation of the temperature, which are updated in real time based on the non-fire source pixel set.

[0011] Preferably, in step S2, selecting connected pixel regions whose human body probability function values ​​are greater than a preset threshold as human body candidate regions includes the following steps: Temperature value lower than And the gradient magnitude is lower than The pixels constitute the non-fire source pixel set The average temperature is updated in real time using robust statistical methods. with standard deviation : , where mad is the absolute deviation of the median; Based on the updated and Recalculate the human probability function ,Will The value is greater than the preset threshold. Connectivity analysis is performed on the pixels, and the resulting connected components are used as candidate regions for the human body.

[0012] Preferably, in step S3, performing temporal spectral analysis on pixels within the human body candidate region in multiple consecutive frames of images to determine the smoke fluctuation frequency, and establishing a temporal buffer window based on the smoke fluctuation frequency, includes the following steps: Extract pixel temperature values ​​within the human candidate region from multiple consecutive frames of images, calculate the average temperature of each frame to form a time-domain sequence, perform Fourier transform on the time-domain sequence to obtain the spectral amplitude distribution, and take the frequency corresponding to the maximum spectral amplitude as the main frequency of smoke fluctuation. The length of the timing buffer window covering at least three smoke fluctuation cycles is determined based on the ratio of the smoke fluctuation main frequency to the preset image acquisition frame rate.

[0013] Preferably, in step S3, human key points with confidence levels below a preset threshold are selected as key points to be compensated. Human key points corresponding to the image coordinates of the key points to be compensated are extracted from historical frames within the temporal buffer window and fused to obtain compensated human key points. This includes the following steps: Mark human key points in the current frame whose confidence level is lower than a preset threshold as key points to be compensated; For each key point to be compensated, human body key points with the same image coordinate position in historical frames are extracted from historical frames within the temporal cache window, and the fusion weight of each historical frame human body key point is calculated; the fusion weight depends on the time interval between the historical frame and the current frame and the smoke concentration estimate of the historical frame. The coordinates of human body key points in all historical frames are weighted and summed according to the fusion weights to obtain the compensated coordinates of human body key points.

[0014] Preferably, step S4 includes the following steps: The visible light image and the thermal infrared image are semantically segmented respectively, and the segmentation results are adaptively fused according to the smoke concentration to generate a semantic label map containing preset obstacle categories; Spatially align the compensated human key points with the semantic tag map to obtain the obstacle category at the location of each human key point; Based on the confidence level of each human body key point and whether the human body key point falls into a preset obstacle category set, a pressure index is constructed. :

[0015] Where J represents the total number of key points on the human body. Let j be the obstacle category corresponding to the j-th human body key point. For a preset set of obstacle categories, Let J be the confidence level of the j-th human key point; Based on the compression index and the posture category, it is determined whether the trapped person is in a state of compression.

[0016] Preferably, in step S5, a preliminary distance is obtained by performing geometrical optical calculations based on the compensated human body key points and the pose parameters of the imaging device; a thermal radiation attenuation observation equation is established based on the thermal infrared grayscale observation values ​​and preset physical parameters, including the following steps: Based on the image coordinates of the foot key points in the compensated human body key points and the pose parameters of the imaging device, the preliminary distance is calculated by solving the ground projection geometric model; Based on the average grayscale observation value of the human body region in the thermal infrared image Preset reference grayscale Given a preset attenuation constant k, the actual distance d to be determined, and the dust concentration coefficient c to be determined, establish the thermal radiation attenuation observation equation: .

[0017] Preferably, in step S5, the preliminary distance and the preset smoke concentration coefficient are used as initial values ​​to perform coupled iterative solution of the thermal radiation attenuation observation equation to obtain the converged true distance. The three-dimensional coordinates of the trapped personnel are then updated based on the true distance, including the following steps: With the aforementioned preliminary distance and preset smoke and dust concentration coefficient As a distance estimate and estimated smoke and dust concentration The initial value is used to iteratively optimize the objective function derived from the thermal radiation attenuation observation equation using the gradient descent method. In each iteration, the distance estimate is simultaneously corrected according to the following coupled update equation. and estimated smoke and dust concentration :

[0018] Where g is the number of iterations, and k is a preset decay constant. The iteration step size, These are the weighting coefficients for each item; the iteration continues until both the change in distance and the change in smoke and dust concentration are less than the preset threshold. The converged distance estimate is used as the true distance d, and the converged dust concentration estimate is used as the dust concentration coefficient c. The location confidence is obtained based on the error information during the iterative convergence process. The three-dimensional coordinates of the trapped personnel are updated based on the ratio between the actual distance and the preliminary distance.

[0019] Compared with the prior art, the beneficial effects of the present invention are: This invention constructs a probability function for the human body that integrates temperature value, gradient amplitude, and temperature difference with the fire source, and dynamically updates background temperature parameters based on the set of non-fire source pixels, effectively distinguishing the human body from the fire source and high-temperature background, and significantly reducing the false detection rate. This invention determines the smoke fluctuation frequency by performing temporal spectral analysis on continuous images, establishes an adaptive temporal buffer window, and performs historical frame weighted fusion compensation for key points with low confidence, effectively overcoming the interference of intermittent smoke occlusion on pose estimation and ensuring recognition continuity. This invention generates obstacle label maps through multimodal semantic segmentation, aligns the compensated human body key points with the label map space, and constructs a crush index model based on key point confidence to achieve quantitative determination of whether trapped personnel are crushed, providing key information for rescue decisions; This invention couples the preliminary distance calculated by geometric optics with the thermal radiation attenuation observation equation for iterative solution, while optimizing the distance and smoke concentration coefficients. It effectively separates the influence of smoke interference on the distance measurement, outputs the true distance and positioning reliability, and realizes high-precision three-dimensional positioning in a fire environment. Attached Figure Description

[0020] Figure 1 This is a schematic diagram of the method flow of the present invention. Detailed Implementation

[0021] The technical solution of the present invention will be clearly and completely described below with reference to the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0022] Please see Figure 1 This invention provides a computer vision-based method for posture recognition and localization of people trapped in a fire, comprising the following steps: S1: Simultaneously acquire visible light and thermal infrared images of the fire scene environment, and obtain the pose parameters of the imaging device; S2: Based on the temperature value, gradient magnitude, and the difference between the pixel temperature value and the preset typical temperature of the fire source for each pixel in the thermal infrared image, a human body probability function is established, and the connected regions of pixels whose human body probability function values ​​are greater than a preset threshold are taken as human body candidate regions. S3: Perform temporal spectral analysis on the pixels in the candidate human body region in multiple consecutive frames of images to determine the smoke fluctuation frequency, and establish a temporal buffer window based on the smoke fluctuation frequency; extract human body key points and corresponding confidence scores in the current frame through a pose estimation network, and take human body key points with confidence scores lower than a preset threshold as key points to be compensated; extract human body key points corresponding to the image coordinates of the key points to be compensated from the historical frames in the temporal buffer window and fuse them to obtain compensated human body key points; determine the pose category of the trapped person based on the compensated human body key points. S4: Perform semantic segmentation on the visible light image and the thermal infrared image to generate a semantic label map of environmental obstacles; spatially align the compensated human key points with the semantic label map, calculate the proportion of human key points falling into the obstacle area, and determine whether the trapped person is in a crushed state based on the proportion and the posture category. S5: Based on the compensated human body key points and the pose parameters of the imaging device, perform geometric optical calculations to obtain the preliminary distance; establish a thermal radiation attenuation observation equation based on the thermal infrared grayscale observation value and preset physical parameters; use the preliminary distance and preset smoke concentration coefficient as initial values ​​to perform coupled iterative solution on the thermal radiation attenuation observation equation to obtain the converged true distance; update the three-dimensional coordinates of the trapped person based on the true distance. S6: Output the posture category, the pressure state determination result, and the three-dimensional coordinates.

[0023] Specifically, on fire rescue robots or drone platforms, synchronously triggered visible light cameras and thermal infrared imagers are deployed. The visible light camera has a resolution of 1920×1080 pixels and a frame rate of 30 frames per second, used to capture the texture details of the fire environment; the thermal infrared imager has a resolution of 640×512 pixels and a frame rate of 30 frames per second, used to penetrate dense smoke in the fire to obtain the thermal radiation distribution. The two cameras achieve frame-level synchronization through hardware trigger lines, ensuring that the visible light image and thermal infrared image acquired at each moment are strictly aligned in time. Subsequently, the dual-light system is jointly calibrated. A checkerboard calibration board is simultaneously imaged by visible light and thermal infrared, and after extracting corner points, the rotation matrix and translation vector between the two cameras are calculated to achieve pixel-level registration of the image coordinate system. The pose parameters of the imaging equipment include the camera intrinsic parameter matrix, camera height, gimbal angle, and the robot's position in the world coordinate system. = =1200 pixels, principal point =960 pixels =540 pixels and distortion coefficient, obtained through offline checkerboard calibration. Camera height is measured in real time by a laser rangefinder on the robot, with a typical value of 1.5 meters in the undulating ground environment of a fire scene. Gimbal angle is fed back in real time by a gimbal encoder, including horizontal and pitch angles, with an accuracy of 0.1 degrees. Robot world coordinates Positioning accuracy is achieved in centimeters through real-time estimation by a lidar SLAM system.

[0024] A human probability function is established based on thermal infrared images, and candidate regions for human bodies are extracted. Temporal spectral analysis is performed on these candidate regions across multiple consecutive frames to determine smoke fluctuation frequencies, and a temporal buffer window is established. Human keypoints and their corresponding confidence scores are extracted from the current frame using a pose estimation network. Keypoints with confidence scores below a preset threshold are designated as keypoints to be compensated. Keypoints with the same image coordinates from historical frames are extracted and weighted to obtain compensated human keypoints, and their pose categories are determined. Semantic segmentation is performed on both visible light and thermal infrared images. An obstacle semantic label map is generated through adaptive fusion based on smoke concentration. The compensated human keypoints are spatially aligned with the label map, and the proportion of human keypoints falling into the obstacle area is calculated. Combined with the pose category, it is determined whether the trapped person is under pressure. A preliminary distance is obtained through ground projection geometry calculation based on the compensated human keypoints and the imaging device's pose parameters. Simultaneously, a thermal radiation attenuation observation equation is established. Using the preliminary distance and a preset smoke concentration coefficient as initial values, the observation equation is coupled and iteratively solved to obtain the converged true distance. The three-dimensional coordinates of the trapped person are updated based on the ratio between the true distance and the preliminary distance.

[0025] The system ultimately outputs the identified posture category, crush status determination result, and 3D coordinates. Posture categories include standing, sitting, lying down, or curled up; the crush status is determined as either crushed or not crushed. The 3D coordinates are calculated based on the robot's world coordinates and the ratio between the actual distance and the initial distance. The origin of the coordinates is consistent with the world coordinate system used by the robot's SLAM system, and the unit is meters. The output method is determined according to the actual needs of fire rescue. Data packets are sent in real-time to the on-site command center via a wireless communication module (such as Wi-Fi operating at 2.4GHz or a wired relay), or stored in local solid-state storage for post-event analysis, or visualized on the robot's real-time map interface as flashing icons and text labels. The data packet contains a timestamp, posture category, crush status, 3D coordinates, and overall confidence level. An example data packet contains: timestamp "2026-05-20T14:30:25Z", posture category "lying down", crush status "crushed", 3D coordinates (12.5m, 8.3m, 0.0m), and confidence level 0.92. Overall confidence level in data packets The formula used to characterize the overall reliability of the identification and localization results is as follows: , The value ranges from 0 to 1, with typical values ​​between 0.85 and 0.95. The mean of the confidence scores for all human keypoints output by the pose estimation network; For location reliability.

[0026] In one embodiment of the present invention, the human body probability function in S2 Build as:

[0027] Where T(p) is the temperature value of pixel p, and G(p) is the gradient magnitude of pixel p. The preset gradient threshold constant, To preset the typical temperature of the fire source, To preset the transition band width, The standard normal cumulative distribution function is... and These are the mean and standard deviation of the temperature, which are updated in real time based on the non-fire source pixel set.

[0028] Specifically, during thermal infrared imaging, the raw response value of each pixel is converted into a physical temperature value through a sensor calibration curve. For any pixel p in the thermal infrared image, its temperature value T(p) is directly provided by the thermal infrared imager, in Kelvin (K) or degrees Celsius (°C).

[0029] The gradient magnitude G(p) of a pixel reflects the severity of local temperature changes and is obtained by spatial difference calculation of the temperature image. The Sobel operator is used to calculate the gradients in the horizontal and vertical directions:

[0030] Then gradient magnitude The gradient amplitude is dimensionless, and its actual range depends on the severity of temperature changes. The gradient amplitude at the edge of a fire source can reach several hundred, while it is usually below 10 in the smooth areas of the human body.

[0031] Human probability function It consists of the product of three factors, specifically: The first item The standard normal cumulative distribution function is used to map temperature values ​​to probability space; and These represent the mean and standard deviation of the temperature in the non-fire source area, respectively; when the pixel temperature T(p) approaches... At that time, this term is close to 0.5; this term ensures that the probability function focuses on the human physiological temperature range. The second term... Used to suppress high gradient regions such as the edge of a fire source; preset gradient threshold constant. The sensitivity to inhibition is controlled, typically ranging from 5 to 10, for example, a value of 8; this factor monotonically decreases as the gradient increases; for example, when G(p) = When this term is 0.75, and G(p) = 3 When this term is set to 0.4375, this design significantly reduces the probability of high-gradient pixels at the edge of the fire source, while retaining a higher probability in smooth areas inside the human body. (Third term) The high-temperature cutoff term is implemented using a sigmoid function to achieve smooth suppression; the typical temperature of the fire source is preset. Based on the flame temperature range, it is typically set to 500K (approximately 227℃). This value is an approximate lower limit of the flame temperature in common fire scenes, ensuring that areas with temperatures higher than this are effectively suppressed; preset transition zone width. Controlling the transition range from normal temperature to high temperature, typically ranging from 10K to 20K, for example... =15K; when T(p) is much lower than When T(p) is much higher than 0, the Sigmoid term is close to 0, and this term is close to 1; when T(p) is much higher than 0, the Sigmoid term is close to 0, and this term is close to 1. At this time, the Sigmoid term is close to 1, and this term is close to 0, thus completely excluding high-temperature pixels in the core flame region. After multiplying the above three terms, only pixels that simultaneously satisfy the conditions of moderate temperature, gentle gradient, and non-high temperature can obtain a higher probability value for human bodies; in practical applications, the method described in claim 3 can be updated in real time. and Then calculate pixel by pixel. The probability values ​​of all pixels representing the human body constitute a human body probability map.

[0032] In one embodiment of the present invention, step S2, which involves selecting connected pixel regions whose human body probability function values ​​are greater than a preset threshold as human body candidate regions, includes the following steps: Temperature value lower than And the gradient magnitude is lower than The pixels constitute the non-fire source pixel set The average temperature is updated in real time using robust statistical methods. with standard deviation : , where mad is the absolute deviation of the median; Based on the updated and Recalculate the human probability function ,Will The value is greater than the preset threshold. Connectivity analysis is performed on the pixels, and the resulting connected components are used as candidate regions for the human body.

[0033] Specifically, for the current frame's thermal infrared image, reliable non-fire source pixels are selected, with the selection criteria considering both temperature value and gradient magnitude. The temperature requirement is that the pixel temperature value be lower than the preset typical fire source temperature. The gradient condition requires that the pixel gradient magnitude be less than 3. ,in This is a preset gradient threshold constant. The gradient threshold is set to 3. The goal is to ensure the exclusion of high-gradient regions such as the edges of fire sources, while preserving pixels with gentle gradients inside the human body and in the environmental background. Both conditions must be met simultaneously; that is, pixel p must satisfy T(p) < 0. And G(p) < 3 Only then are they classified into the non-fire source pixel set. This dual screening mechanism ensures The pixels in the image primarily come from the environmental background and human body area, excluding the fire source and its edges.

[0034] Obtain the set of non-fire source pixels Then, the temperature distribution parameters of the set are estimated as needed in the probability function of the corresponding human body. and Because residual abnormally high temperature points (such as briefly drifting flame fragments) or sensor noise may exist in the fire scene environment, the arithmetic mean and standard deviation are sensitive to these outliers, which can lead to distorted parameter estimation. Therefore, a robust statistical method is adopted, using the median and median absolute deviation (MAD) instead of the arithmetic mean and standard deviation.

[0035] Average temperature Defined as Median of all pixel temperature values: The median is not sensitive to outliers; even if a small number of high-temperature pixels are mixed into the set, the median will not shift significantly. For example, It contains 100 pixels, of which 99 have a temperature between 300K and 320K, and one abnormally high temperature point has a temperature of 450K. The arithmetic mean is raised to about 321K, while the median remains at around 310K, which more accurately reflects the typical background temperature.

[0036] Temperature standard deviation The median absolute deviation (MAD) is estimated and defined as follows: That is, the temperature of each pixel and the median. The median of the absolute deviation. To convert MAD to a value consistent with the standard deviation, multiply by a coefficient of 1.4826, derived based on the normal distribution assumption, such that 1.4826 × MAD equals the standard deviation when the data follows a normal distribution; therefore If outliers exist, the median is not sensitive to outliers. It can still robustly reflect the dispersion of the main data.

[0037] These two robust statistical parameters and Calculations are performed independently on each frame of the image, enabling dynamic updates and allowing the human probability function to adapt to changes in the fire environment temperature; for example, when the overall environment temperature rises. The temperature will increase accordingly, and the temperature window of the probability function will shift accordingly.

[0038] Get the updated and Then, substitute these values ​​into the human body probability function formula to recalculate the human body probability values ​​for all pixels in the current frame. This ensures that the probability function is always based on the real-time temperature background of the current frame, rather than fixed parameters.

[0039] The entire human body probability map is then segmented using a threshold. A preset probability threshold is used. Used to distinguish human body pixels from non-human body pixels, the typical value range is 0.5 to 0.7, and the preferred value is 0.6; The choice needs to strike a balance between detection sensitivity and false positive rate: Too low a value can cause a large number of background pixels to be misidentified as human bodies. Excessive settings may miss real human body areas; in practical applications, optimization can be achieved through experimental data calibration. For each pixel p, if... If a pixel is found to be a candidate body pixel, it is marked as such; otherwise, it is marked as background. All candidate body pixels constitute a binary image. Connectivity analysis is performed on the binary image to aggregate adjacent candidate body pixels into connected regions. Connectivity analysis can use four-neighbor or eight-neighbor connectivity criteria, with eight-neighbor connectivity being preferred to ensure region integrity. The analysis results in several connected regions, each composed of a group of consecutively located pixels. Since noise or small-area interference can generate a small number of isolated pixels forming tiny connected regions, a minimum human body area threshold can be preset for filtering. The preset minimum human body area threshold can be estimated based on the imaging distance and typical human body size, for example, it can be set to 100 pixels (at a resolution of 640×512). Only connected regions with an area greater than this threshold are retained as valid candidate regions. All connected regions that meet the area condition are considered as the final human body candidate regions.

[0040] In one embodiment of the present invention, step S3 involves performing temporal spectral analysis on pixels within the human body candidate region in multiple consecutive frames of images to determine the smoke fluctuation frequency, and establishing a temporal buffer window based on the smoke fluctuation frequency, including the following steps: Extract pixel temperature values ​​within the human candidate region from multiple consecutive frames of images, calculate the average temperature of each frame to form a time-domain sequence, perform Fourier transform on the time-domain sequence to obtain the spectral amplitude distribution, and take the frequency corresponding to the maximum spectral amplitude as the main frequency of smoke fluctuation. The length of the timing buffer window covering at least three smoke fluctuation cycles is determined based on the ratio of the smoke fluctuation main frequency to the preset image acquisition frame rate.

[0041] Specifically, regarding the current moment To obtain a sequence of candidate human regions from multiple consecutive frames of thermal infrared images; let the number of consecutive frames be N, and the selection of N must ensure that it covers at least two complete smoke fluctuation cycles to obtain reliable spectral information, typically 30 frames. For each frame t∈[ N+1, Extract the human candidate region from this frame. Calculate the average temperature of the region by taking the temperature values ​​of all pixels within the region. ,in This represents the total number of pixels in the candidate region of the frame; finally, the average temperature sequence is obtained. .

[0042] Performing a Fast Fourier Transform (FFT) on the average temperature series yields the spectral amplitude distribution. Where n is the index of the average temperature sequence; i is the imaginary unit, satisfying... = 1; f is the frequency, ranging from 0 to the Nyquist frequency. / 2, This represents the image acquisition frame rate (typically 30fps). Due to the physical characteristics of smoke fluctuations, the effective frequency range is limited to 0.2Hz to 5Hz; components below 0.2Hz typically correspond to slow changes in ambient temperature (such as fire development or overall environmental warming), while components above 5Hz are mostly sensor noise or high-frequency interference. The frequency corresponding to the maximum spectral amplitude within the limited frequency range is the dominant frequency of the smoke fluctuation. .

[0043] Obtain the main frequency of smoke fluctuation Next, a temporal buffer window needs to be established to store image data and pose estimation results from historical frames. The length of the temporal buffer window must meet two conditions: first, it must cover a sufficient number of fluctuation cycles to ensure information redundancy; second, it should not be too long to avoid computational burden and response latency. This method sets the temporal buffer window to cover at least three complete smoke fluctuation cycles. The reason for choosing three cycles is that within one cycle, the smoke occlusion state completes a loop from none to some and back to none. Three cycles ensure that, in most cases, for any occluded human keypoint, there are at least several frames of complete, unoccluded information within the window. The formula for calculating the length of the temporal buffer window is: in, Here, the function is the floor function; ratio This represents the number of frames contained in each fluctuation cycle. Multiplying this by 3 gives the total number of frames required for three cycles. The integer part is then used as the length of the timing buffer window.

[0044] After establishing the temporal buffer window, the data within the window is dynamically maintained using a first-in, first-out (FIFO) approach. When a new frame arrives, its image, human candidate region, pose estimation results, and other data are stored in the window, while the oldest frame in the window is removed, maintaining a constant window length of M. If the dominant frequency of smoke fluctuations in the current frame changes due to variations in the fire environment, the window length needs to be recalculated and adjusted. To avoid instability caused by frequent adjustments, update conditions can be set; for example, if the newly calculated dominant frequency deviates from the current dominant frequency by more than 20%, the temporal buffer window length is updated.

[0045] In one embodiment of the present invention, in step S3, human key points with confidence levels below a preset threshold are selected as key points to be compensated. Human key points corresponding to the image coordinates of the key points to be compensated are extracted from historical frames within a temporal buffer window and fused to obtain compensated human key points. This includes the following steps: Mark human key points in the current frame whose confidence level is lower than a preset threshold as key points to be compensated; For each key point to be compensated, human body key points with the same image coordinate position in historical frames are extracted from historical frames within the temporal cache window, and the fusion weight of each historical frame human body key point is calculated; the fusion weight depends on the time interval between the historical frame and the current frame and the smoke concentration estimate of the historical frame. The coordinates of human body key points in all historical frames are weighted and summed according to the fusion weights to obtain the compensated coordinates of human body key points.

[0046] Specifically, the pose estimation network employs a lightweight convolutional neural network architecture, such as MobileHRNet or LiteHRNet, with the current frame as input. The network generates candidate human body regions (cropped and scaled to a fixed size, such as 256×256 pixels) and outputs J human keypoints as heatmaps or directly regressed coordinate values. If using heatmaps, the pose estimation network generates a probability heatmap for each human keypoint. The value of each pixel in the heatmap represents the probability of the keypoint existing at that location. After Softmax normalization, the location of the maximum value in the heatmap is taken as the coordinate of the human keypoint; this maximum value is the confidence score of the keypoint. The value range is [0,1], where This is the current frame. If coordinate regression is used, the pose estimation network directly outputs the coordinates of human keypoints and their corresponding confidence values. In practical applications, heatmaps are more common and have higher accuracy. The number of human keypoints J is usually 17, covering major joints such as the head, shoulders, elbows, wrists, hips, knees, and ankles.

[0047] Preset reliability threshold Used to distinguish reliable human keypoints from keypoints to be compensated, with a typical value range of 0.4 to 0.6. For each human keypoint j in the current frame, if its confidence level... If the key point of the human body is determined to be unreliable due to smoke obscuring or image blurring, it is marked as a key point to be compensated and added to the compensation set. The remaining key points of the human body are directly retained as the final key point coordinates. For the set to be compensated... For each human keypoint in the image, the human keypoint information of the same image coordinate position in the historical frames needs to be extracted from the historical frames in the temporal buffer window. The temporal buffer window has a length of M and stores the thermal infrared images of the most recent M frames and the corresponding pose estimation results (including the coordinates and confidence of human keypoints in the historical frames).

[0048] Because trapped people in a fire are usually stationary or only move slightly, the same image coordinates The corresponding position in a historical frame is highly likely to correspond to the same part of the human body. Therefore, for each historical frame, its frame index t∈[ M, [1] Extract image coordinates from the pose estimation results of this frame. Historical frame human body key point coordinates and confidence level If the coordinate position is not detected as a human body key point in the historical frame (e.g., the body moves out of the field of view due to human movement), it is considered an invalid point.

[0049] Fusion weights The time interval between the historical frame and the current frame, as well as the smoke density of the historical frame, must be considered simultaneously. Historical frames with more recent time intervals are more similar to the current frame in terms of human pose, and should therefore have a higher weight. Historical frames with higher smoke density have lower image quality and lower reliability of human keypoints, and should therefore have a lower weight. The specific calculation formula is as follows: ;in, The preset time decay coefficient controls the decay rate over time, with a typical value range of 0.1 to 0.5. The preset smoke attenuation coefficient controls the suppression strength of smoke concentration on the weight, with a typical value range of 1.0 to 3.0. The smoke density estimate for frame t is calculated using the dark channel prior algorithm; first, the dark channel of the image is calculated. , where o represents the color channel (red, green, blue). Let q be the brightness value of pixel q on channel o. Define a neighborhood window for pixel p; then calculate the transmittance. ,in, A value of 0.95 is used to retain a small amount of distant fog. The atmospheric light value A is the maximum intensity of the original image corresponding to the brightest 0.1% of pixels in the dark channel. The mean transmittance of the entire image is used as the estimated smoke concentration for that frame. The values ​​are normalized to the interval [0,1], where 0 represents no smoke and 1 represents complete occlusion.

[0050] For each keypoint j to be compensated, the coordinates of the corresponding human body keypoints extracted from all historical frames are weighted and summed. Simultaneously, the confidence level of the human body keypoints in the historical frames themselves is introduced as a second-layer weight to further suppress the influence of low-quality historical frames. The fusion formula is: ,in, The coordinates of human body key points in historical frames t with the same image coordinates. The confidence level is determined by the coordinate position in a historical frame. If no valid historical frame human keypoint is found at this coordinate location (i.e., ...), the confidence level is determined by the following: If the value is 0 or not detected, the contribution is 0. If there are no valid historical frame human keypoints in all historical frames (denominator is zero), then the keypoint cannot be compensated, and interpolation of adjacent human keypoints is used or the coordinates of the current frame are directly retained. After performing the above fusion calculation on all keypoints to be compensated, the compensated coordinates of each keypoint are obtained. The high-confidence human keypoints, which were previously retained directly, are merged to form the complete compensated human keypoint set for the current frame. .

[0051] Compensated human keypoints are used to determine the posture category of the trapped person. Common methods include geometric rule-based classification or lightweight classifiers. Taking geometric rules as an example, body features such as torso tilt angle are first calculated based on the coordinates of the human keypoints. (Angle between the line connecting the midpoint of the shoulder and the midpoint of the hip and the horizontal plane), relative height ratio (The ratio of the height to the width of the smallest bounding rectangle of the human body), knee flexion (Angle between thigh and calf), height of key points on the human body from the ground (distance between foot key points and the ground reference plane). Then, the pose category is determined by a preset threshold rule: if and If, it is determined to be standing; and If it is determined to be sitting; and If it is determined to be lying down; Approximately lying down, but with knee flexion. Furthermore, if the body's projected area is less than 60% of the normal lying position, it is considered curled up.

[0052] In one embodiment of the present invention, step S4 includes the following steps: The visible light image and the thermal infrared image are semantically segmented respectively, and the segmentation results are adaptively fused according to the smoke concentration to generate a semantic label map containing preset obstacle categories; Spatially align the compensated human key points with the semantic tag map to obtain the obstacle category at the location of each human key point; Based on the confidence level of each human body key point and whether the human body key point falls into a preset obstacle category set, a pressure index is constructed. :

[0053] Where J represents the total number of key points on the human body. Let j be the obstacle category corresponding to the j-th human body key point. For a preset set of obstacle categories, Let J be the confidence level of the j-th human key point; Based on the compression index and the posture category, it is determined whether the trapped person is in a state of compression.

[0054] Specifically, semantic segmentation is performed on the synchronously registered visible light and thermal infrared images respectively. The semantic segmentation network uses a lightweight real-time model such as ESNet or Fast-SCNN. Its encoder extracts multi-scale features through convolution and downsampling, and the decoder restores the resolution through upsampling. Finally, each pixel outputs a semantic segmentation function. dimensional vector ( (where the number of categories is preset), this vector is normalized using the Softmax function to obtain the probability that the pixel belongs to each category. For example, let the total number of categories be... =3 (e.g., walls, furniture, others), the original vector output by the semantic segmentation network for a pixel p is [2.0, 1.0, 0.1], where =2.0 (corresponding to the wall score) =1.0 (corresponding furniture score) =0.1 (corresponding to other scores); Softmax calculation formula is: Where 'a' is the target category index. To sum the index values ​​for all categories; the calculation yields... ≈7.39, ≈2.72, ≈1.11, the denominator is calculated to be 11.22; therefore, the probability of the wall type is... / 11.22≈0.659, probability of furniture category / 11.22≈0.242, other class probabilities / 11.22≈0.099. The semantic segmentation network was pre-trained on a dataset containing common obstacles in a fire scene. During training, the cross-entropy loss function was used, the optimizer was Adam, and the learning rate was set to 0.001, iterating until convergence. The categories labeled in the training set included: walls, collapsed furniture, cabinets, tables, chairs, doors, etc. Let the visible light segmentation probability map be... (p,m), the thermal infrared segmentation probability map is (p,m), where p is the pixel coordinate and m is the category index (m=1,2,…, ).

[0055] In actual fire scenes, visible light images become blurred under dense smoke, while thermal infrared images are less affected by smoke but lack detail; therefore, it is necessary to estimate the smoke concentration based on the current frame. (Based on the dark channel prior algorithm and normalized to [0,1]) Adaptive fusion of the two segmentation results is performed. Fusion weights. Defined as: ,in The preset smoke concentration threshold is typically set to 0.6. When... hour, <1, primarily segmented by visible light; when hour, =1, using full thermal infrared segmentation. The fused probability map is as follows: The final semantic label map is obtained by taking the class with the highest probability for each pixel. The value of each pixel in the label image represents the obstacle category at that location.

[0056] The compensated set of human key points (Each key point has image coordinates) and confidence level Mapping to the semantic label graph L, we obtain the obstacle category corresponding to each keypoint. If the keypoint coordinates are not integers, bilinear interpolation is used to determine their category. Define the compression index. To quantify the degree to which a person is pinned down by an obstacle: ,in This is a pre-defined set of obstacle categories, typically including categories such as collapsed furniture, cabinets, and tables that could potentially crush a person; This is an indicator function that takes the value 1 when the condition is true and 0 otherwise. This represents the confidence level of key points on the human body. The index ranges from [0,1], with values ​​closer to 1 indicating a higher probability of being pressed.

[0057] The state of being trapped is determined by combining the trapped person's posture category (from S3) and the compression index. The determination rule is: if... If the value is >0.3 and the posture category is lying down or curled up, then it is determined to be being crushed; if A score >0.5 and a sitting posture are also considered as being trapped; other situations are considered as not being trapped. The thresholds for determining the trapped state (0.3, 0.5) can be adjusted based on actual fire scene data. For example, if a trapped person is lying down, and 8 out of 17 key points on the human body fall into the area of ​​collapsed furniture, and the average confidence level of these key points is 0.8, while the confidence level of the remaining key points is 0.9, then the trapped index is calculated to be 0.44 (assuming an average confidence level of 0.85), which is greater than 0.3 and the posture is lying down, so it is determined as being trapped.

[0058] In one embodiment of the present invention, in step S5, a preliminary distance is obtained by performing geometrical optical calculations based on the compensated human body key points and the pose parameters of the imaging device; a thermal radiation attenuation observation equation is established based on the thermal infrared grayscale observation values ​​and preset physical parameters, including the following steps: Based on the image coordinates of the foot key points in the compensated human body key points and the pose parameters of the imaging device, the preliminary distance is calculated by solving the ground projection geometric model; Based on the average grayscale observation value of the human body region in the thermal infrared image Preset reference grayscale Given a preset attenuation constant k, the actual distance d to be determined, and the dust concentration coefficient c to be determined, establish the thermal radiation attenuation observation equation: .

[0059] Specifically, from the compensated set of key human body points Foot keypoints are selected as the ranging reference. Pose estimation networks typically define keypoints 15 and 16 as the left and right ankles, respectively, and these two keypoints can be used directly. If the confidence level of the foot keypoint is too low due to occlusion (e.g., confidence level below 0.3), then the ordinate of all keypoints is used. The largest point is used as a substitute; this point is the lowest in the image and closest to the ground contact point.

[0060] Let the coordinates of the selected foot keypoint image be... The units are pixels. The coordinates are converted to normalized camera coordinates based on the camera intrinsic parameter matrix K of the imaging device. The camera intrinsic parameter matrix K includes the focal length. , and main point , The value has been obtained through offline chessboard calibration, and its typical value is... = =1200 pixels, =960 pixels, =540 pixels. Normalized coordinates A ray corresponding to the camera coordinate system has the following direction vector: Based on the current horizontal angle fed back by the gimbal. and pitch angle Rotate the direction vector to the world coordinate system: ,in The rotation matrix, which is composed of the angles of the gimbal, can be calculated using the Euler angle formula.

[0061] Let the position of the camera's optical center in the world coordinate system be... ;in Equal to the camera height, measured in real time by a laser rangefinder mounted on the robot; a typical value is 1.5 meters in the undulating terrain of a fire scene; optical center coordinates. With robot world coordinates The relationship depends on the installation position of the gimbal. It is generally assumed that the two overlap or have a fixed offset, which can be obtained through calibration and compensated in advance.

[0062] Starting from the optical center along the direction The ray parameter equation is: Where s is the distance parameter along the ray, with the same units as the coordinate units (meters). Assuming the ground is a horizontal plane with equation Z=0; setting Z=0 in the ray equation, solve for the parameter s: Substituting the obtained s into the X and Y expressions, we can obtain the world coordinates of the foot keypoints on the ground. : .

[0063] The linear distance from the camera's optical center to the key point on the foot is the initial distance. The unit is meters, and the calculation formula is: This distance is calculated based on pure geometric optics principles and has high accuracy when there is no smoke interference. However, in the dense smoke environment of a fire scene, errors will occur due to light refraction and thermal radiation attenuation, requiring subsequent correction.

[0064] In the current frame of the thermal infrared image In the process, the candidate regions of the human body are extracted. The grayscale values ​​of all pixels within the image. The grayscale value of each pixel in a thermal infrared image is typically 14 bits of raw data, exhibiting a linear or non-linear relationship with radiation temperature. Calculate the average grayscale observation value of the candidate human body region. ,in, This represents the total number of pixels contained in the human body candidate region. is the grayscale value of pixel p.

[0065] Thermal infrared radiation propagates through a smoke medium following the Lambert-Beer law and decays exponentially; a reference gray level and a decay constant are preset; an observation equation for thermal radiation attenuation is established: , where d is the actual distance to be determined, and c is the dust concentration coefficient to be determined.

[0066] The standard grayscale value of the human body at a reference distance without attenuation can be obtained through offline calibration by setting a preset reference grayscale. The calibration method is as follows: In a smoke-free environment, place the blackbody or human body at a known distance. Measure its average grayscale value at a distance of 1 meter (for example). Meanwhile, we set the dust concentration coefficient to 0 (i.e., no attenuation) and substitute it into the attenuation equation. achievable However, for more precise calibration, sensor characteristics can be considered, and the values ​​can be directly taken... That's it. A typical 14-bit thermal imager has a grayscale value range of 0 to 16383, and the grayscale value for the human body is usually between 8000 and 12000. It can be taken as around 10,000, or calibrated according to actual measurement.

[0067] The preset attenuation constant k is a constant related to the characteristics of the environmental medium, and its unit is 1. The value was obtained through experimental calibration. The calibration method was as follows: In a uniform smoke environment, the blackbody was placed at two different distances. and Measure the grayscale value respectively and At this point, the dust concentration coefficient c is the same (unknown), and from the attenuation equation, we get: , Dividing the two equations, we get Taking the logarithm, we get Since c is unknown, k cannot be solved independently. In practical applications, we can assume that c=1 (i.e., standard dust concentration) under calibration conditions, thus obtaining... The typical value range is from 0.05 to 0.2, for example, k=0.1.

[0068] In one embodiment of the present invention, in step S5, the preliminary distance and the preset smoke concentration coefficient are used as initial values ​​to perform coupled iterative solution of the thermal radiation attenuation observation equation to obtain the converged true distance. The three-dimensional coordinates of the trapped personnel are then updated based on the true distance, including the following steps: With the aforementioned preliminary distance and preset smoke and dust concentration coefficient As a distance estimate and estimated smoke and dust concentration The initial value is used to iteratively optimize the objective function derived from the thermal radiation attenuation observation equation using the gradient descent method. In each iteration, the distance estimate is simultaneously corrected according to the following coupled update equation. and estimated smoke and dust concentration :

[0069] Where g is the number of iterations, and k is a preset decay constant. The iteration step size, These are the weighting coefficients for each item; the iteration continues until both the change in distance and the change in smoke and dust concentration are less than the preset threshold. The converged distance estimate is used as the true distance d, and the converged dust concentration estimate is used as the dust concentration coefficient c. The location confidence is obtained based on the error information during the iterative convergence process. The three-dimensional coordinates of the trapped personnel are updated based on the ratio between the actual distance and the preliminary distance.

[0070] Specifically, after obtaining the initial distance, grayscale observations, and preset physical parameters, it is necessary to simultaneously solve for the actual distance and the smoke and dust concentration coefficient. This is a nonlinear joint estimation problem, which is iteratively optimized using the gradient descent method. First, the initial value of the distance estimate is set. Take as the initial distance The initial value of the estimated smoke and dust concentration Take as the preset dust concentration coefficient .in This value can be set based on fire scene experience, with a typical range of 0.2 to 0.8, for example, 0.4. The objective function is constructed as follows: ,in, , , These are the weighting coefficients used to balance the contributions of geometric constraints, radiation observation errors, and prior information; typical value ranges are as follows: It ranges from 0.3 to 0.7. It ranges from 0.2 to 0.5. The range is from 0.1 to 0.4; the example value is... =0.5, =0.3, =0.2.

[0071] By minimizing the objective function, which includes geometric constraints, radiation observation errors, and prior constraints, the coupled update equation is obtained: Wherein, the superscript (g) represents the estimated value of the g-th iteration, g = 0, 1, 2, ...; k is the preset decay constant; Preset reference grayscale. Iteration step size. and This controls the magnitude of each update, typically ranging from 0.001 to 0.1. An example value is... =0.01, =0.005. Repeat the iteration until the condition is met. and ;in This is the threshold for the change in distance, typically set to 0.01 meters. The threshold for the change in smoke and dust concentration is typically set to 0.01. A maximum number of iterations (e.g., 50) is also set to prevent infinite loops. Once the convergence condition or the maximum number of iterations is reached, the current estimate is used as the final result. Let the converged distance estimate be... The estimated smoke and dust concentration is Then the actual distance d = Smoke concentration coefficient c= .

[0072] After iterative convergence, the location confidence level is calculated based on the error information during the convergence process. This level characterizes the reliability of the distance estimate, and its value ranges from [0,1]. The distance variance is estimated based on the local curvature of the objective function at the optimal solution. The second-order partial derivative of the objective function with respect to d is approximately: ; Variance of distance estimates Taking the reciprocal of the second-order partial derivative, the location confidence is defined as: ,in Let be the standard deviation, in meters. This mapping converts the standard deviation into a confidence level: the smaller the standard deviation, the closer the confidence level is to 1; the larger the standard deviation, the closer the confidence level is to 0. For example... =0.1 meters ≈0.905.

[0073] The 3D coordinates of the trapped personnel are updated based on the ratio between the actual distance and the initial distance. First, the scaling factor is calculated: Then, the ground projection coordinates of the key foot points are determined. Scale by a scaling factor to obtain real-world coordinates : ;in The coordinates for the robot's world are provided in real time by the SLAM system. If the trapped person is on a non-horizontal surface (such as lying on a step), the Z coordinate can be adjusted accordingly based on the height of the lowest key point of the human body. However, in this method, the ground is assumed to be horizontal, so Z=0.

[0074] The above embodiments are only used to illustrate the technical methods of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical methods of the present invention without departing from the spirit and scope of the technical methods of the present invention.

Claims

1. A method for posture recognition and localization of trapped personnel in a fire based on computer vision, characterized in that, Includes the following steps: S1: Simultaneously acquire visible light and thermal infrared images of the fire scene environment, and obtain the pose parameters of the imaging device; S2: Based on the temperature value, gradient magnitude, and the difference between the pixel temperature value and the preset typical temperature of the fire source for each pixel in the thermal infrared image, a human body probability function is established, and the connected regions of pixels whose human body probability function values ​​are greater than a preset threshold are taken as human body candidate regions. S3: Perform temporal spectral analysis on the pixels within the human candidate region in multiple consecutive frames of images to determine the smoke fluctuation frequency, and establish a temporal buffer window based on the smoke fluctuation frequency; The pose estimation network extracts human key points and corresponding confidence scores of the current frame. Human key points with confidence scores below a preset threshold are taken as key points to be compensated. Human key points corresponding to the image coordinates of the key points to be compensated are extracted from historical frames in the temporal buffer window and fused to obtain compensated human key points. The pose category of the trapped person is determined based on the compensated human key points. S4: Perform semantic segmentation on the visible light image and the thermal infrared image to generate a semantic label map of environmental obstacles; spatially align the compensated human key points with the semantic label map, calculate the proportion of human key points falling into the obstacle area, and determine whether the trapped person is in a crushed state based on the proportion and the posture category. S5: Based on the compensated human body key points and the pose parameters of the imaging device, perform geometric optical calculations to obtain the preliminary distance; establish a thermal radiation attenuation observation equation based on the thermal infrared grayscale observation value and preset physical parameters; use the preliminary distance and preset smoke concentration coefficient as initial values ​​to perform coupled iterative solution on the thermal radiation attenuation observation equation to obtain the converged true distance; update the three-dimensional coordinates of the trapped person based on the true distance. S6: Output the posture category, the pressure state determination result, and the three-dimensional coordinates.

2. The method for posture recognition and localization of trapped personnel in a fire based on computer vision according to claim 1, characterized in that, The human probability function in S2 Build as: Where T(p) is the temperature value of pixel p, and G(p) is the gradient magnitude of pixel p. Here, represents the preset gradient threshold constant, and represents the preset typical temperature of the fire source. To preset the transition band width, The standard normal cumulative distribution function is... and These are the mean and standard deviation of the temperature, which are updated in real time based on the non-fire source pixel set.

3. The method for posture recognition and localization of trapped personnel in a fire based on computer vision according to claim 1, characterized in that, In step S2, the connected pixel regions whose human body probability function values ​​are greater than a preset threshold are selected as human body candidate regions, including the following steps: Temperature value lower than And the gradient magnitude is lower than The pixels constitute the non-fire source pixel set The average temperature is updated in real time using robust statistical methods. with standard deviation : , where mad is the absolute deviation of the median; Based on the updated and Recalculate the human probability function ,Will The value is greater than the preset threshold. Connectivity analysis is performed on the pixels, and the resulting connected components are used as candidate regions for the human body.

4. The method for posture recognition and localization of trapped personnel in a fire based on computer vision according to claim 1, characterized in that, In step S3, temporal spectral analysis is performed on pixels within the human body candidate region in multiple consecutive frames of images to determine the smoke fluctuation frequency, and a temporal buffer window is established based on the smoke fluctuation frequency, including the following steps: Extract pixel temperature values ​​within the human candidate region from multiple consecutive frames of images, calculate the average temperature of each frame to form a time-domain sequence, perform Fourier transform on the time-domain sequence to obtain the spectral amplitude distribution, and take the frequency corresponding to the maximum spectral amplitude as the main frequency of smoke fluctuation. The length of the timing buffer window covering at least three smoke fluctuation cycles is determined based on the ratio of the smoke fluctuation main frequency to the preset image acquisition frame rate.

5. The method for posture recognition and localization of trapped personnel in a fire based on computer vision according to claim 4, characterized in that, In step S3, human key points with confidence levels below a preset threshold are selected as key points to be compensated. Human key points corresponding to the image coordinates of the key points to be compensated are extracted from historical frames within the temporal buffer window and fused to obtain compensated human key points. This includes the following steps: Mark human key points in the current frame whose confidence level is lower than a preset threshold as key points to be compensated; For each key point to be compensated, human body key points with the same image coordinate position in historical frames are extracted from historical frames within the temporal cache window, and the fusion weight of each historical frame human body key point is calculated; the fusion weight depends on the time interval between the historical frame and the current frame and the smoke concentration estimate of the historical frame. The coordinates of human body key points in all historical frames are weighted and summed according to the fusion weights to obtain the compensated coordinates of human body key points.

6. The method for posture recognition and localization of trapped personnel in a fire based on computer vision according to claim 1, characterized in that, S4 includes the following steps: The visible light image and the thermal infrared image are semantically segmented respectively, and the segmentation results are adaptively fused according to the smoke concentration to generate a semantic label map containing preset obstacle categories; Spatially align the compensated human key points with the semantic tag map to obtain the obstacle category at the location of each human key point; Based on the confidence level of each human body key point and whether the human body key point falls into a preset obstacle category set, a pressure index is constructed. : Where J represents the total number of key points on the human body. Let j be the obstacle category corresponding to the j-th human body key point. For a preset set of obstacle categories, Let J be the confidence level of the j-th human key point; Based on the compression index and the posture category, it is determined whether the trapped person is in a state of compression.

7. The method for posture recognition and localization of trapped personnel in a fire based on computer vision according to claim 1, characterized in that, In step S5, a preliminary distance is obtained by performing geometrical optical calculations based on the compensated human body key points and the pose parameters of the imaging device; a thermal radiation attenuation observation equation is established based on the thermal infrared grayscale observation values ​​and preset physical parameters, including the following steps: Based on the image coordinates of the foot key points in the compensated human body key points and the pose parameters of the imaging device, the preliminary distance is calculated by solving the ground projection geometric model; Based on the average grayscale observation value of the human body region in the thermal infrared image Preset reference grayscale Given a preset attenuation constant k, the actual distance d to be determined, and the dust concentration coefficient c to be determined, establish the thermal radiation attenuation observation equation: .

8. The method for posture recognition and localization of trapped personnel in a fire based on computer vision according to claim 7, characterized in that, In step S5, the preliminary distance and the preset smoke concentration coefficient are used as initial values. The thermal radiation attenuation observation equation is solved iteratively in a coupled manner to obtain the converged true distance. The three-dimensional coordinates of the trapped personnel are then updated based on the true distance. This includes the following steps: With the aforementioned preliminary distance and preset smoke and dust concentration coefficient As a distance estimate and estimated smoke and dust concentration The initial value is used to iteratively optimize the objective function derived from the thermal radiation attenuation observation equation using the gradient descent method. In each iteration, the distance estimate is simultaneously corrected according to the following coupled update equation. and estimated smoke and dust concentration : Where g is the number of iterations, and k is a preset decay constant. The iteration step size, These are the weighting coefficients for each item; the iteration continues until both the change in distance and the change in smoke and dust concentration are less than the preset threshold. The converged distance estimate is used as the true distance d, and the converged dust concentration estimate is used as the dust concentration coefficient c. The location confidence is obtained based on the error information during the iterative convergence process. The three-dimensional coordinates of the trapped personnel are updated based on the ratio between the actual distance and the preliminary distance.

Images