Methods and systems for extracting the trajectory of fire monitor jets and locating their impact points.

Abstract

This invention belongs to the field of artificial intelligence technology, specifically relating to a method and system for extracting the trajectory of a fire monitor jet, locating its impact point, and related technologies. Based on the SwinTransformer, this solution introduces a deformable attention module and a spatial convolutional neural network module, ultimately designing an image segmentation network capable of more accurately capturing the trajectory of the fire monitor jet. Building upon this, the invention acquires real-time images of the fire monitor jet's tip using a binocular camera. The image segmentation network extracts the jet trajectory segmentation results from the monocular images within the real-time images. A stereo matching depth estimation scheme based on the IGEV++ model estimates the pixel depth information from the stereo images. Finally, the pixel points of the jet trajectory impact point are identified from the segmented images, and the impact point is located by combining the pixel position and depth information. This invention solves the problems of insufficient prediction accuracy and susceptibility to environmental interference in existing fire monitor trajectories and impact points.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, specifically relating to a method for extracting the trajectory of a fire monitor jet, an image-based method for locating the landing point of a fire monitor jet, a fire monitor jet landing point location module, and an automated fire monitor system. Background Technology

[0002] In some constrained fire rescue scenarios, it is necessary to assess the fire situation and the landing point of fire monitors based on images transmitted from the scene. However, currently, there are few solutions that can predict the landing point of fire monitor trajectories. Most existing solutions use jet trajectory calculation and image processing methods to solve this problem. For example, Gao Guangzhu et al. utilized the difference in gray values ​​between the water column centerline and the water column edge, used Hough transform to extract the water column centerline, and calculated the intersection of the water column centerlines at different angles to obtain the water column landing point. Chen Jing used traditional image processing segmentation methods to segment the water cannon jet image. Based on this, she calculated, analyzed, and fitted the water cannon jet trajectory curve to obtain the trajectory landing point. Wang Dongyue et al. also first extracted the edge gradient features of the water cannon jet trajectory using traditional image processing methods, and then further analyzed and processed them to obtain the trajectory landing point. Zhou Junjie et al. used a lightweight deep learning semantic segmentation network to segment the water cannon jet. Based on prior knowledge such as contour gradient and jet firing direction, they obtained the jet landing point with the highest probability. Tian Hong proposed a MobileNetV2 model with hybrid dilated convolution to segment the jet trajectory, and then used methods such as curve fitting to obtain the landing point of the jet trajectory.

[0003] The water cannon trajectory extraction methods provided above mainly include traditional image processing methods and deep learning-based methods. Traditional image processing methods require a significant investment of time and effort to design manual feature extraction methods and are only applicable to specific scenarios. For complex scenarios or other scenarios (such as fire training scenarios where the movement of firefighters affects recognition; natural factors such as wind and light affecting recognition), feature extraction capabilities are unstable and generalization ability is weak. Deep learning-based methods can largely solve this problem. However, existing methods have insufficient training data for their network models, lack scene diversity, and have not been further optimized for the characteristics of water cannon trajectories, resulting in poor robustness and insufficient accuracy in water cannon trajectory segmentation.

[0004] In summary, existing methods for predicting the trajectory and impact point of fire monitors primarily involve performing various mathematical calculations, analyses, and fittings on the jet profile before finally selecting the correct result. These methods rely on sophisticated mathematical calculations and derivations, resulting in low accuracy in impact point prediction. Some methods also require prior knowledge of the water cannon jet orientation, making them unsuitable for generalization to other scenarios. Furthermore, these methods are weak against interference. Other uncertainties in the environment, such as water pressure and wind force, can easily affect impact point prediction. Summary of the Invention

[0005] To address the problems of insufficient accuracy in predicting the trajectory and landing point of fire monitors and susceptibility to environmental interference in existing solutions, this invention provides a method for extracting the trajectory of fire monitor jets, an image-based method for locating the landing point of fire monitor jets, a fire monitor jet landing point location module, and an automated fire monitor system.

[0006] This invention is achieved using the following technical solution:

[0007] A method for extracting the trajectory of a fire monitor jet, comprising:

[0008] A jet segmentation model is constructed to generate segmented images containing water cannon jet trajectories from an input image. In the jet segmentation model, the input image first undergoes feature extraction through a backbone network consisting of four feature extraction layers. The first feature extraction layer includes one linear embedding layer and two SwinWater blocks; the second feature extraction layer includes one block fusion unit and two SwinWater blocks; the third feature extraction layer includes one block fusion unit and 18 SwinWater blocks; and the fourth feature extraction layer includes one block fusion unit and two SwinWater blocks. The outputs of the four feature extraction layers serve as inputs to four WSCNN modules; the outputs of all WSCNN modules are then used as input to a Mask2Former network, which generates the segmented image based on the input multi-scale feature maps.

[0009] Each SwinWater Block in the backbone network includes one SwinTransformer module and one Deformable Attention (DA) module. The first feature map output by the SwinTransformer module of the previous layer is processed by the SwinTransformer module of the current layer to produce the first feature map. The second feature map output by the DA module of the previous layer and the output of the SwinTransformer module of the current layer are input together into the DA module of the current layer for processing to obtain the second feature map.

[0010] The WSCNN module consists of an inverted bottleneck block, an SCNN network, a layer normalization layer, and a feedforward neural network. The feature maps output from the corresponding feature extraction layer are input into the WSCNN module. They are first processed by the inverted bottleneck block and the SCNN network. The input and output features of the SCNN network are then residually connected, and finally processed by the layer normalization layer and the feedforward neural network to become the final output of the WSCNN module.

[0011] Based on the jet segmentation model constructed above, real water cannon images with jet trajectory label information are used as sample images, and a large number of sample images are used to form a sample dataset. Then, the sample images are divided into training, validation, and test sets according to a preset ratio, and used to train, validate, and test the jet segmentation model. Finally, the parameters of the network model with the best performance after testing are retained, and it is used to segment the water cannon jet trajectory in the input image.

[0012] As a further improvement of the present invention, in the backbone network, the input of the first feature extraction layer is an RGB image, and the input of the remaining feature extraction layers is the output of the last SwinWater Block in the previous layer; the linear embedding layer in the first feature extraction layer cuts the input image into small pieces and embeds them into the embedding; compared with the input features, the height and width of the output features of each feature extraction layer in the backbone network are halved, and the number of channels is doubled.

[0013] As a further improvement of the present invention, the two alternating SwingTransformer modules sequentially include an LN layer, a W-MSA module, an LN layer, an MLP module, an LN layer, a SW-MSA module, an LN layer, and an MLP module; wherein the outputs of the W-MSA module, the SW-MSA module, and the two MLP modules are respectively residually connected to the input of their adjacent preceding LN layer.

[0014] As a further improvement to the present invention, the data processing procedure in the two alternating SwinTransformer modules is as follows:

[0015]

[0016] In the above formula, z l-1 z l and z l+1 These are the feature maps of layers l-1, l, and l+1 extracted by the SwinTransformer module, respectively. and represents the output features of the W-SWA module and the SW-SWA module, respectively; LN is the layer normalization operation; MLP represents the data processing of the multilayer perceptron; W-SWA and SW-SWA represent the data processing of two different types of window self-attention mechanisms.

[0017] As a further improvement of this invention, the Mask2Former network includes a four-layer pixel decoder corresponding to the backbone network and a three-layer Transformer decoder. The first three layers of the pixel decoder are skip-connected to the corresponding layers of the Transformer decoder; the final output of the Transformer decoder serves as the class. The output of the Transformer decoder is multiplied by the output of the last layer of the pixel decoder to generate an initial mask, which is then restored to the original image resolution through bilinear interpolation.

[0018] As a further improvement of the present invention, during the training phase of the jet segmentation model, the quality of the mask generated by Mask2Former is jointly optimized using binary cross-entropy loss (BCE) and Dice loss, and the optimal match is calculated using the Hungarian matching strategy to improve the consistency of target segmentation.

[0019] The present invention also includes an image-based method for locating the impact point of a fire monitor jet, comprising:

[0020] Acquire live images of the water cannon jet tip captured by a pre-calibrated binocular camera, including monocular images from a single-side view and stereo images from both sides.

[0021] The jet trajectory in a monocular image from any side view is segmented using a pre-trained jet segmentation model provided in the aforementioned method for extracting the trajectory of a fire monitor jet. The resulting trajectory segmentation map is then obtained.

[0022] A pre-trained IGEV++-based stereo matching depth estimation network generates a disparity estimation map for each pixel in a stereo image.

[0023] The landing point of the jet is identified from the trajectory segmentation map, and the three-dimensional coordinates of the jet landing point are generated by combining the pixel position of the landing point in the monocular image and the depth information provided by the disparity estimation map.

[0024] As a further improvement of the present invention, after identifying the water jet landing point P0, a predetermined number of pixels are offset in the opposite direction of the jet spray direction to determine a reference point; a reference line perpendicular to the image width direction is drawn at the reference point; N points are uniformly selected in the area where the reference line intersects with the water column, denoted as P1 to P2. N Finally, calculate P0 to P... N The average depth is used as the predicted depth of the landing point.

[0025] The present invention also includes a fire monitor jet impact point positioning module, which includes a memory, a processor, and a computer program stored in the memory and running on the processor. When the processor executes the computer program, it uses the fire monitor jet trajectory extraction method described above to generate the three-dimensional coordinates of the fire monitor jet impact point based on the real-time image containing the fire monitor jet captured by the binocular camera.

[0026] The present invention also includes an automated fire monitor system, comprising: a fire monitor with adjustable spray direction, a binocular camera, and a controller. The binocular camera is used to acquire real-time images including the tip of the fire monitor jet when the fire monitor is activated. The controller includes a fire monitor jet impact point positioning module as described above; the controller is used to generate three-dimensional coordinates of the fire monitor jet impact point based on the real-time images using the fire monitor jet impact point positioning module; and then adaptively adjust the fire monitor's spray direction based on the three-dimensional coordinates of the fire monitor jet impact point and the on-site fire situation.

[0027] The technical solution provided by this invention has the following beneficial effects:

[0028] This invention leverages the powerful feature extraction capabilities of the Swin Transformer, introduces a Deformable Attention module to enhance long-distance dependency modeling, and utilizes a Spatial Convolutional Neural Network (WSCNN) to improve the structural integrity and spatial information transmission capabilities of the water jet trajectory. This results in a jet segmentation network capable of segmenting the fire monitor jet trajectory from an image. Compared to traditional methods, this invention's segmentation network can more accurately capture the water monitor jet trajectory, maintain high segmentation accuracy in complex environments, and possesses stronger generalization ability and robustness.

[0029] By utilizing a designed jet segmentation network, this invention redesigns an image-based method for locating the impact point of a fire monitor jet. This method acquires monocular and stereo images of the water cannon jet from a binocular camera, then extracts the estimated jet trajectory and its corresponding depth information from both images, ultimately calculating the spatial coordinates of the impact point at the end of the jet trajectory. This scheme effectively solves the error problem caused by traditional fitting calculation methods. Furthermore, it does not require prior knowledge of the water cannon jet's orientation and is unaffected by uncontrollable factors such as water pressure and wind force in the environment, enabling real-time depth estimation. This scheme also lays the foundation for designing automated, unmanned fire monitors and has broad practical value. Attached Figure Description

[0030] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0031] Figure 1 This is a flowchart of the steps for extracting the trajectory of a fire monitor jet provided in Embodiment 1 of the present invention.

[0032] Figure 2 This is an overall architecture diagram of the jet segmentation model constructed in Embodiment 1 of the present invention.

[0033] Figure 3 This is a schematic diagram of the SwinWater Block used in the jet segmentation model of Embodiment 1 of the present invention.

[0034] Figure 4 This is a schematic diagram of the SwinTransformer module used in the SwinWater Block of Embodiment 1 of the present invention.

[0035] Figure 5 This is a schematic diagram of the WSCNN module used in the jet segmentation model of Embodiment 1 of the present invention.

[0036] Figure 6 Part (a) in the figure is the architecture of the Mask2Former network used in the jet segmentation model of Embodiment 1 of the present invention. Figure 6 Part (b) is a schematic diagram of the Transformer decoder in the Mask2Former network.

[0037] Figure 7 This is a flowchart of the steps of the image-based fire monitor jet positioning method provided in Embodiment 2 of the present invention.

[0038] Figure 8 This is a schematic diagram of a scenario for the adaptive jet landing point identification method provided in Embodiment 2 of the present invention.

[0039] Figure 9 This is a schematic diagram of a binocular camera with jet trajectory landing point positioning function provided in Embodiment 3 of the present invention.

[0040] Figure 10 To verify the experimental scenario of comparing the positioning accuracy of water columns. Detailed Implementation

[0041] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0042] Example 1

[0043] This embodiment provides a method for extracting the trajectory of a fire monitor jet. Considering the specific characteristics of fire monitor jet trajectory extraction, this embodiment designs a novel jet segmentation model that combines Transformer and CNN modules to capture low-level details while also understanding high-level semantics. The new module introduces a deformable attention (DA) module into the backbone network to enhance the model's ability to capture long-distance dependencies, resulting in more accurate extracted water monitor jet trajectories. Furthermore, addressing the strong prior structure characteristics of water monitor jet trajectories, this invention also introduces a WSCNN module in the feature extraction part. This enhances the jet structure information by transmitting information along different directions in space, making the segmentation results more complete and coherent.

[0044] Specifically, such as Figure 1 As shown, the method for extracting the trajectory of a fire monitor jet provided in this embodiment includes the following steps:

[0045] S1: Construct a jet segmentation model that generates a segmented image of the water cannon jet trajectory contained in the input image.

[0046] like Figure 2 As shown, the jet segmentation model constructed in this embodiment consists of three parts: a backbone network, a WSCNN (Water Spacial Convolutional Neural Net) module, and a Mask2Former network. The backbone network comprises four feature extraction layers: the first layer includes one linear embedding layer and two SwinWater blocks; the second layer includes one patch merging unit and two SwinWater blocks; the third layer includes one patch merging unit and 18 SwinWater blocks; and the fourth layer includes one patch merging unit and two SwinWater blocks.

[0047] In the jet segmentation model, the input image is first processed by a backbone network consisting of four feature extraction layers for feature extraction. The outputs of the four feature extraction layers are used as inputs to four WSCNN modules. Each WSCNN module outputs four feature maps at different scales, which are then fed into the Mask2Former network. The Mask2Former network is used to generate segmented water cannon jet trajectory images based on the input multi-scale feature maps.

[0048] To make the architecture and principle of the jet segmentation model constructed in this embodiment clearer, the details of each part of the network model will be described in detail below.

[0049] (I) SwinWater Block

[0050] In the jet segmentation model designed in this embodiment, the acquired image containing the water basin jet is first processed by the backbone network for feature extraction. In this embodiment, the original input image is an RGB image, denoted as H×W×3, where H represents the image height, W represents the image width, and 3 represents the number of channels, i.e., C=3. Since the original input image size is large, the backbone network in this embodiment first divides the input image into multiple small blocks through a linear embedding layer, generating a corresponding embedding vector for each image block. Then, the generated embedding features are input into the SwinWater Block for feature extraction. The feature extraction layer of the backbone network in this embodiment mainly utilizes the newly designed SwinWater Block to extract features at different scales. The backbone network consists of four layers, each using a different number of SwinWater Blocks, in a ratio of 2:2:18:2. In the second to fourth feature extraction layers, the output of the previous feature extraction layer is first processed by a block fusion unit, and then processed by different numbers of SwinWater Blocks for feature extraction.

[0051] In the backbone network, the input to the first feature extraction layer is an RGB image, and the input to the remaining feature extraction layers is the output of the last SwinWater Block in the previous layer. Compared to the input features, the height and width of the output features of each feature extraction layer in the backbone network are halved, while the number of channels is doubled. Specifically, the sizes of the feature maps output by the four feature extraction layers are as follows: and

[0052] SwinWater Block is a new feature extraction module improved upon the SwinTransformer module in this embodiment, such as... Figure 3 As shown, the SwinWater Block designed in this embodiment includes one SwinTransformer module and one DA (Deformable Attention) module. Each SwinWater Block processes and outputs the two input feature maps in parallel. Specifically, the first feature map output by the previous SwinTransformer module (denoted as: After being processed by the SwinTransformer module of the current layer, the output is the first feature map of the current layer (denoted as: The second feature map output by the previous DA module (denoted as: ) and the output of the current layer's SwinTransformer module (i.e. The two inputs are fed into the DA module of the current layer, and the output of the DA module of the current layer is the second feature map (denoted as: In each SwinWater Block, the output of the DA module serves as the value vector, while the output of the SwinTransformer module serves as the query vector.

[0053] In this embodiment, the SwinTransformer module in the SwinWater Block adopts an existing module in the Swin Transformer deep learning model for visual tasks, such as... Figure 4 As shown, the two alternating SwinTransformer modules consist of 8 layers, sequentially including the first LN (Layer Norma) layer, the W-MSA (in-window multi-head self-attention) module, the second LN layer, the first MLP (Multilayer Perceptron) module, the third LN layer, the SW-MSA (window-offset multi-head self-attention) module, the fourth LN layer, and the second MLP module. The outputs of the W-MSA module, the SW-MSA module, and the two MLP modules are residually concatenated with the inputs of their respective preceding LN layers. The data processing procedure is as follows:

[0054]

[0055] In the above formula, z l-1 z l and z l+1 These are the feature maps of layers l-1, l, and l+1 extracted by the SwinTransformer module, respectively. and represents the output features of the W-SWA module and the SW-SWA module, respectively; LN is the layer normalization operation; MLP represents the data processing of the multilayer perceptron; W-SWA and SW-SWA represent the data processing of two different types of window self-attention mechanisms.

[0056] (II) WSCNN Module

[0057] The WSCNN module used in the jet segmentation model of this embodiment is a network module used to enhance the feature maps extracted by the backbone network. This module is designed under the inspiration of existing Spatial Convolutional Neural Networks (SCNN). Water cannon jets under enormous pressure are generally white and cylindrical, exhibiting significant prior structural characteristics. Therefore, this embodiment designs a WSCNN module to extract features and transmit information along different directions—downward, upward, forward, and backward—enhancing the spatial structural information of the jet and making the jet segmentation results more coherent. Specifically, the WSCNN module in this embodiment consists of one inverted bottleneck block, one SCNN network, one layer normalization layer, and one feedforward neural network. The four WSCNNs in this embodiment are used to process the outputs of corresponding layers in the backbone network, such as... Figure 5 As shown, after the feature map of the corresponding layer in the backbone network enters the WSCNN module, it is first processed by the inverted bottleneck block and the SCNN network. The WSCNN module also performs residual connections on the input and output features of the SCNN network. Then, the obtained feature map is processed by layer normalization and feedforward neural network to obtain the final output of the WSCNN module.

[0058] (III) Mask2Former Network

[0059] This embodiment designs a jet segmentation model that uses the Mask2Former network as the segmentation head. Mask2Former is an existing segmentation network and a general image segmentation framework suitable for tasks such as semantic segmentation, instance segmentation, and panoptic segmentation. Its core idea is to treat the segmentation task as a mask prediction problem and generate high-quality segmentation results through a Transformer decoder. Specifically, after extracting image features from the input image using a pre-trained backbone network, the global and local information in the image features can be enhanced using the multi-scale feature pyramid in WSCNN. Then, the query-guided Transformer decoder in the extracted feature information progressively optimizes the mask prediction. Simultaneously, a masked attention mechanism is used to compute attention only within the target region to improve computational efficiency and enhance detail information. In practical applications, the Mask2Former network exhibits stronger global modeling capabilities, better adaptability to complex scenes, and maintains excellent performance across different types of segmentation tasks.

[0060] like Figure 6As shown, the Mask2Former network includes a four-layer pixel decoder corresponding to the backbone network and a three-layer Transformer decoder. The first three layers of the pixel decoder are skip-connected to the corresponding layers of the Transformer decoder; the final output of the Transformer decoder serves as the class. The output of the Transformer decoder is multiplied by the output of the last layer of the pixel decoder to generate an initial mask, which is then restored to the original image resolution using bilinear interpolation.

[0061] S2: Based on the jet segmentation model constructed above, real water cannon images with jet trajectory label information are used as sample images, and a large number of sample images are used to form a sample dataset. Then, the sample images are divided into training set, validation set and test set according to a preset ratio, and used to train, validate and test the jet segmentation model.

[0062] Besides the design of the network model, its training is also crucial. This embodiment plans to use a dataset of no fewer than 5000 finely labeled water cannon trajectory segmentation images to train the constructed jet segmentation model. During the training phase, the mIOU metric is used to measure the experimental results, and the training parameters are adjusted based on these results. Ultimately, a water cannon jet trajectory segmentation model with high accuracy and strong generalization ability will be trained.

[0063] During the training phase of the jet segmentation model, this embodiment uses binary cross-entropy loss (BCE) and Dice loss to jointly optimize the quality of the mask generated by Mask2Former, and calculates the optimal match through the Hungarian matching strategy to improve the consistency of target segmentation.

[0064] S3: Retain the parameters of the network model that performs best after testing, and use it to segment the water jet trajectory in the input image.

[0065] Based on the above, the solution provided in this embodiment introduces a deformable attention module and a spatial convolutional neural network module on top of the SwinTransformer. The deformable attention module enhances long-range dependency modeling capabilities, while the spatial convolutional neural network improves the structural integrity and spatial information transmission capabilities of the water jet trajectory. Compared to traditional methods, the solution provided by this invention can more accurately capture the water cannon jet trajectory, maintain high segmentation accuracy in complex environments, and possesses stronger generalization ability and robustness.

[0066] Example 2

[0067] Based on the scheme in Example 1, this embodiment further provides an image-based method for locating the impact point of a fire monitor jet, such as... Figure 7As shown, the scheme includes the following process:

[0068] 1. Acquire live images of the water cannon jet tip captured by a pre-calibrated binocular camera, including monocular images from a single-side view and stereo images from both sides.

[0069] The binocular camera in this embodiment includes two visible light cameras. For example, in practical applications, an industrial camera model MV-CE200-10UC can be used to build the required binocular camera. In the deployment, the two visible light cameras are fixed on the same device, spaced 20 centimeters apart to ensure a certain parallax. Simultaneously, the optical axes of the two cameras are set on the same horizontal plane. Next, the intrinsic and extrinsic parameters of the two cameras are calibrated using Zhang's calibration method. Finally, in practical applications, the two cameras are used to simulate human eyes taking two images from left and right perspectives, respectively. This embodiment can correct the two images from the left and right perspectives using epipolar geometry theory. The image obtained from either perspective is a monocular image, consistent with a planar image taken by a conventional camera, providing only color information for each pixel. Combining the monocular images from the left and right perspectives yields depth information for each pixel, which is then fused to obtain a "stereo image" containing depth information.

[0070] It is important to emphasize that the image-based fire monitor jet landing point positioning method provided in this embodiment mainly identifies the trajectory of the fire monitor jet captured in the image, and then combines the pixel information of the jet tip to deduce the corresponding spatial information, thereby achieving the positioning of the fire monitor jet landing point. Therefore, the real-time image containing the water cannon jet acquired by the binocular camera in this scheme must capture the tip of the jet. In addition, this embodiment only needs to capture the tip of the water cannon jet, and does not care whether the captured water cannon jet is complete.

[0071] Second, using a pre-trained jet segmentation model provided in the aforementioned method for extracting fire monitor jet trajectories, image segmentation is performed on the jet trajectory in the monocular image from any side viewpoint to obtain a trajectory segmentation map. The principle of the jet segmentation model has been detailed above and will not be repeated here. Simultaneously, a pre-trained IGEV++-based stereo matching depth estimation network is used to generate a disparity estimation map for each pixel based on the stereo image.

[0072] In this embodiment, the IGEV++ model is a stereo matching network based on deep learning. The stereo matching process of the model includes four stages: matching cost calculation, cost aggregation, disparity calculation, and disparity correction. Traditional stereo matching algorithms require manual feature design, which makes it difficult to solve problems such as missing object textures, blurred boundaries, and occlusion. IGEV++ is an efficient stereo matching depth estimation network. Its core advantage lies in combining iterative geometry encoding volume (IGEV) for feature enhancement, thereby improving the accuracy and stability of disparity estimation. Compared with traditional methods, IGEV++ has the following advantages: (1) Stronger geometric modeling capability: It processes coarse-grained geometric information in large disparity scenes through multi-range geometry encoding volume (MGEV), while extracting fine structures in small disparity regions, thereby improving matching accuracy. (2) More efficient feature aggregation method: It uses a recursive geometric interaction mechanism to gradually optimize disparity prediction, making the matching results more accurate after each iteration. (3) Better robustness: In complex scenes such as low texture, occlusion, and dynamic blur, IGEV++ can effectively reduce false matches and improve the stability of disparity estimation. (4) Optimized computational efficiency: Compared with IGEV, IGEV++ further optimizes the cost volume calculation and information transmission mechanism, reducing computational costs while ensuring high accuracy, making it more suitable for practical application scenarios.

[0073] By applying a stereo matching depth estimation network based on IGEV++, this embodiment can accurately and efficiently generate a disparity estimation map for each pixel using the corrected left and right view images acquired by a binocular camera.

[0074] 3. Identify the jet's landing point from the trajectory segmentation map, and generate the three-dimensional coordinates of the jet's landing point by combining the pixel position of the landing point in the monocular image and the depth information provided by the disparity estimation map.

[0075] In practical applications, the center position of the water cannon trajectory's end point is the jet's impact point. Based on the segmented image of the jet trajectory, this embodiment can easily find the pixel corresponding to the impact point from the image. Next, by combining the pixel position of the water cannon jet trajectory's impact point in the monocular image and the intrinsic and extrinsic parameters of the binocular camera, partial location information of the impact point can be obtained. Furthermore, by combining the depth information of each pixel obtained from the stereo image by the stereo matching depth estimation network, the three-dimensional coordinates of the jet trajectory's impact point in the world coordinate system can be obtained.

[0076] Considering that a bubble jet typically appears as a relatively thick column of water in an image, to avoid discrepancies between the image-identified landing point and the actual landing point center, this embodiment further optimizes the solution by providing an adaptive jet landing point identification method. Specifically, as shown... Figure 8 As shown, the content of this plan is as follows:

[0077] After identifying the water jet impact point P0 (yellow dot in the image), a predetermined number of pixels (let's say α) are offset in the opposite direction of the jet direction to determine a reference point. A reference line perpendicular to the image width is drawn at the reference point; N points, denoted as P1 to P2, are uniformly selected in the area where the reference line intersects the water jet. N (Red dots in the diagram); Finally, calculate P0 to P... N The average depth is used as the predicted depth of the landing point. In a typical scheme of this embodiment, α = 20 and N = 5.

[0078] Example 3

[0079] Based on the solution in Embodiment 1, this embodiment further provides a computer program product or storage medium, which includes a computer program. When the computer program is executed by a processor, it uses the fire monitor jet trajectory extraction method as in Embodiment 1 to generate the three-dimensional coordinates of the fire monitor jet landing point based on the real-time image containing the fire monitor jet captured by the binocular camera.

[0080] Based on the solution in Embodiment 1, this embodiment further provides a fire monitor jet landing point positioning module, which includes a memory, a processor, and a computer program stored in the memory and running on the processor. When the processor executes the computer program, it uses the fire monitor jet trajectory extraction method as in Embodiment 1 to generate the three-dimensional coordinates of the fire monitor jet landing point based on the real-time image containing the water monitor jet acquired by the binocular camera.

[0081] The fire monitor jet landing point positioning module provided in this embodiment is essentially a computer device. In practical applications, this computer device can be an embedded device integrated into the back end of a binocular camera. For example... Figure 9 As shown, the device employs two visible light cameras, model MV-CE200-10UC industrial cameras. The two cameras are packaged in a single box. Each binocular camera integrates a data processing chip, which is used for water jet trajectory extraction, water jet trajectory depth estimation, and water jet trajectory impact point prediction. This chip can be a Rockchip RV1126. In addition, the device can be equipped with a small display screen for real-time display of the binocular camera images, water jet trajectory segmentation results, and water jet trajectory impact point prediction results. The display screen can be an ATK 7-inch V2 TFTLCD.

[0082] In practical applications, the binocular cameras are fixed at a certain distance (about 60 meters) from the water cannon jet trajectory using a bracket. When the water cannon jet trajectory appears, the two visible light cameras simultaneously capture images of the water cannon jet trajectory, obtaining left and right view images. The cameras continuously acquire left and right view images at a frame rate of 30fps and a resolution of 20 megapixels.

[0083] Taking the acquisition of a single frame of an image as an example, the fire monitor jet impact point positioning module first preprocesses the left and right view images during operation, correcting the left and right images using calibrated camera intrinsic and extrinsic parameters. Next, the image size is resized to 1920x1080. The left image serves as input for the water cannon jet trajectory extraction task, while the left and right images are used as a stereo camera image pair as input for the water cannon jet trajectory depth estimation task. Then, combining the segmented image obtained from the water cannon jet trajectory extraction task and the depth information of each pixel obtained from the water cannon jet trajectory depth estimation task, the world coordinates of the water cannon jet trajectory impact point are obtained. Finally, the water cannon jet trajectory image captured by the camera, the segmented water cannon jet trajectory, and the landing point of the water cannon jet trajectory are all displayed in real time on the device's screen.

[0084] In other embodiments, the fire monitor jet landing point positioning module can also be a separate device independent of the imaging equipment. Specifically, in this embodiment, the computer equipment can be a smart terminal capable of executing programs, such as a tablet computer, laptop computer, desktop computer, rack server, blade server, tower server, or cabinet server (including a standalone server or a server cluster composed of multiple servers), etc.

[0085] The computer device described in this embodiment includes, but is not limited to, a memory and a processor that can be interconnected via a system bus. The memory (i.e., readable storage medium) includes flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory can be an internal storage unit of the computer device, such as the hard disk or RAM of the computer device. In other embodiments, the memory can also be an external storage device of the computer device, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc. Of course, the memory can also include both internal storage units and external storage devices of the computer device. In this embodiment, the memory is typically used to store the operating system and various application software installed on the computer device. Furthermore, the memory can also be used to temporarily store various types of data that have been output or will be output.

[0086] In some embodiments, a processor may be a central processing unit (CPU), a graphics processing unit (GPU), a controller, a microcontroller, a microprocessor, or other data processing chip. The processor is typically used to control the overall operation of a computer device. In this embodiment, the processor is used to run program code stored in memory or process data.

[0087] Example 4

[0088] Building upon embodiments 1 and 3, this embodiment further provides an automated fire monitor system, comprising: a fire monitor with adjustable spray direction, a binocular camera, and a controller. The binocular camera is used to acquire real-time images including the tip of the fire monitor jet when activated. The controller includes an embedded fire monitor jet impact point positioning module, as described in embodiment 3. In this embodiment, the controller generates three-dimensional coordinates of the fire monitor jet impact point based on the real-time images using the fire monitor jet impact point positioning module; then, it adaptively adjusts the fire monitor's spray direction by combining the three-dimensional coordinates of the fire monitor jet impact point with the on-site fire situation.

[0089] The automated fire monitor system provided in this embodiment can acquire the landing point of the sprayed water jet using a binocular camera, which is then used to guide the spray angle of the fire monitor. In practical applications, the automated fire monitor system of this embodiment can be mounted on unmanned vehicles, robots, or other equipment. Through this adaptive adjustment method, the automated fire monitor system provided in this embodiment can achieve automated, unmanned control of the fire scene in restricted environments, thereby enabling fire rescue operations in environments inaccessible to firefighters.

[0090] Verification Experiment

[0091] To verify the effectiveness of the method for extracting the trajectory of fire monitor jets and the method for locating the landing point of fire monitor jets based on images provided by this invention, technicians conducted relevant verification experiments.

[0092] I. Image Segmentation

[0093] This experiment was conducted as follows: Figure 2 The proposed scheme constructs a jet segmentation network and serves as the experimental group. Then, PSPNet, DeeplabV3Plus, DeeplabV3Plus, HRNet, ConvNeXt-S, ConvNeXt-B, ViT, Swin-S, and SegFormer schemes are selected as control groups. The performance of the experimental and control group schemes is then compared. In the comparative experiments, IoU, mIoU, PA, and mPA are selected as performance evaluation metrics. The performance evaluation results of the control and experimental group schemes in the comparative experiments are shown in Table 1 below.

[0094] Table 1: Performance comparison of different treatments in the experimental and control groups

[0095]

[0096] Analysis of the data in the table above reveals that the solution provided by this invention outperforms existing solutions in all indicators, demonstrating superior water jet trajectory segmentation performance.

[0097] Furthermore, this invention includes ablation experiments to verify the contributions of the DA module and the WSCNN module to the performance of the jet segmentation network designed in this invention. In the ablation experiments, the Swin-S scheme, lacking both the DA and WSCNN modules, serves as the control group. The experimental group lacking only the DA module is designated Swin-WSCNN, the experimental group lacking only the WSCNN module is designated Swin-DA, and the experimental group containing both the DA and WSCNN modules is designated Swin-Water. The IoU was used as the performance evaluation metric in the ablation experiments. The performance of each scheme is shown in Table 2 below:

[0098] Table 2: Performance of different schemes in ablation experiments

[0099] Swin-S Swin-DA Swin-WSCNN Swin-Water DA × √ × √ WSCNN × × √ √ IoU 77.76 79.13 78.12 79.91

[0100] Analysis of the data in the table above reveals that the complete solution of this invention achieves the best performance. The WSCNN module contributes more to the performance than the DA module. Compared with the Swin-S solution, the performance of this invention is enhanced by 2.15%.

[0101] II. Landing Point Positioning Effect

[0102] This experiment is as follows: Figure 10 In the scenario shown, the positioning test of the water jet impact point of a fire monitor was conducted using the solution of this invention. In the test experiment, the positioning accuracy of the IGEV++ scheme of this invention as the stereo matching network was compared with that of the control group schemes using PCW-Net, GwcNet, RAFT-Stereo, ACVNet, and IGEV at jet depths of 2.022, 5.066, and 8.004. Figure 10 (a), (b), and (c) in the table correspond to jet scenarios at depths of 2 meters, 5 meters, and 8 meters, respectively. Furthermore, the performance differences of different schemes before and after adopting the Adaptive Jet Landing Point Recognition Method (AWCLPL) of this invention were compared during the testing process. The test results are shown in the table below:

[0103] Table 3: Water column landing point positioning results for different schemes

[0104]

[0105] Analysis of the data in the table above reveals that the scheme using the three-dimensional matching network in this invention achieves higher positioning accuracy compared to other schemes. Furthermore, the positioning accuracy can be further improved by using the adaptive jet landing point identification method provided by this invention.

[0106] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for extracting the trajectory of a fire monitor jet, characterized in that, It includes: A jet segmentation model is constructed to generate segmented images containing water cannon jet trajectories from an input image. The input image is first processed by a backbone network consisting of four feature extraction layers: the first layer includes one linear embedding layer and two SwinWater blocks; the second layer includes one block fusion unit and two SwinWater blocks; the third layer includes one block fusion unit and 18 SwinWater blocks; and the fourth layer includes one block fusion unit and two SwinWater blocks. The outputs of the four feature extraction layers serve as inputs to four WSCNN modules. The outputs of all WSCNN modules are then used as inputs to a Mask2Former network, which generates the segmented image based on the input multi-scale feature map. The SwinWater Block includes a SwinTransformer module and a deformable attention module (DA). The first feature map output by the upper-layer SwinTransformer module is processed by the current-layer SwinTransformer module and output as the first feature map. The second feature map output by the upper-layer DA module and the output of the current-layer SwinTransformer module are input into the current-layer DA module for processing to obtain the second feature map. In the WSCNN module, the input features are processed sequentially through an inverted bottleneck block and an SCNN network. The input and output features of the SCNN network are then processed by residual connections, followed by layer normalization and a feedforward neural network. We used real water cannon images containing a large number of jet trajectory label information as sample datasets, and divided them into training, validation and test sets for training, validation and testing of the jet segmentation model; we retained the parameters of the network model with the best test performance and used it to segment the water cannon jet trajectory of the input image.

2. The method for extracting the trajectory of a fire monitor jet as described in claim 1, characterized in that: In the backbone network, the input to the first feature extraction layer is an RGB image, and the input to the remaining feature extraction layers is the output of the last SwinWater Block in the previous layer. The linear embedding layer in the first feature extraction layer segments the input image into small pieces and embeds them into the embedding; compared with the input features, the height and width of the output features of each feature extraction layer in the backbone network are halved, and the number of channels is doubled.

3. The method for extracting the trajectory of a fire monitor jet as described in claim 1, characterized in that: The two alternating SwinTransformer modules sequentially include an LN layer, a W-MSA module, an LN layer, an MLP module, an LN layer, a SW-MSA module, an LN layer, and an MLP module connected in sequence; wherein the outputs of the W-MSA module, the SW-MSA module, and the two MLP modules are residually connected to the inputs of their adjacent preceding LN layers.

4. The method for extracting the trajectory of a fire monitor jet as described in claim 3, characterized in that: The data processing procedure of the two alternating SwinTransformer modules is as follows: In the above formula, z l-1 z l and z l+1 These are the feature maps of layers l-1, l, and l+1 extracted by the SwinTransformer module, respectively. and These represent the output characteristics of the W-SWA module and the SW-SWA module, respectively. LN stands for layer normalization; MLP represents data processing in a multilayer perceptron; W-SWA and SW-SWA represent data processing in two different types of window self-attention mechanisms.

5. The method for extracting the trajectory of a fire monitor jet as described in claim 1, characterized in that: The Mask2Former network includes a four-layer Pixel decoder corresponding to the backbone network and a three-layer Transformer decoder; the first three layers of the Pixel decoder are skip-connected to the corresponding layers of the Transformer decoder; the final output of the Transformer decoder is used as the class; the output of the Transformer decoder and the output of the last layer of the Pixel decoder are multiplied by a matrix to generate an initial mask, which is then restored to the original image resolution through bilinear interpolation.

6. The method for extracting the trajectory of a fire monitor jet as described in claim 5, characterized in that: During the training phase of the jet segmentation model, the quality of the mask generated by Mask2Former is jointly optimized using binary cross-entropy loss (BCE) and Dice loss, and the optimal match is calculated using the Hungarian matching strategy to improve the consistency of target segmentation.

7. A method for locating the impact point of a fire monitor jet based on an image, characterized in that, It includes: Acquire live images of the water cannon jet tip captured by a pre-calibrated binocular camera; the live images include monocular images from a single-side view and stereo images from both sides. The jet trajectory in the monocular image from any one side view is segmented using a pre-trained jet segmentation model provided in the method for extracting the trajectory of a fire monitor jet as described in any one of claims 1-6, to obtain a trajectory segmentation map. A disparity estimation map for each pixel is generated from the stereo image using a pre-trained IGEV++-based stereo matching depth estimation network. The landing point of the jet is identified from the trajectory segmentation map, and the three-dimensional coordinates of the landing point are generated by combining the pixel position of the landing point in the monocular image and the depth information provided by the disparity estimation map.

8. The image-based method for locating the impact point of a fire monitor jet as described in claim 7, characterized in that: After identifying the water jet impact point P0, a predetermined number of pixels are offset in the opposite direction of the jet spray direction to determine a reference point; a reference line perpendicular to the image width direction is drawn at the reference point; N points are uniformly selected in the area where the reference line intersects with the water column, denoted as P1 to P2. N Finally, calculate P0 to P... N The average depth is used as the predicted depth of the landing point.

9. A fire monitor jet impact point positioning module, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, it uses the method for extracting the trajectory of the fire monitor jet as described in claim 7 or 8 to generate the three-dimensional coordinates of the landing point of the fire monitor jet based on the real-time image containing the water monitor jet captured by the binocular camera.

10. An automated fire monitor system, comprising: A fire monitor with adjustable spray direction; A binocular camera is used to capture live images, including the tip of the water cannon jet, when the fire monitor is activated. The controller includes a fire monitor jet landing point positioning module as described in claim 9; the controller is used to generate three-dimensional coordinates of the fire monitor jet landing point based on the real-time image through the fire monitor jet landing point positioning module; and then adaptively adjust the spray direction of the fire monitor by combining the three-dimensional coordinates of the fire monitor jet landing point and the fire situation on site.

Images