Method and device for predicting track of dangerous rock mass based on fusion of visible light and infrared thermal imaging

Abstract

The invention provides a dangerous rock body track prediction method and device based on fusion of visible light and infrared thermal imaging, and belongs to the technical field of computers, and the method comprises the steps: obtaining an infrared thermal imaging video and a visible light video of a dangerous rock region in a first time period, and obtaining a fusion video based on the infrared thermal imaging video and the visible light video; inputting the fused video into a trained dangerous rock mass collapse and sliding identification model, determining rockfall position data corresponding to a current image frame based on the current image frame and a previous image frame in the fused video, obtaining a first feature vector based on the current image frame, carrying out an embedding operation on the first feature vector to obtain an embedded vector, and carrying out the recognition of the collapse and sliding of the dangerous rock mass. And inputting the embedded vector into a down-sampling module to obtain an image feature vector corresponding to the current image frame, and inputting the image feature vector corresponding to the current image frame and the rockfall position data into an up-sampling module to predict rockfall position data corresponding to the next image frame. The method can accurately identify the rockfall and can improve the prediction accuracy of the rockfall trajectory.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method and device for predicting dangerous rock mass trajectories based on the fusion of visible light and infrared thermal imaging. Background Art

[0002] The occurrence of dangerous rock collapse on highway slopes is often characterized by strong suddenness and great destructive power, which can block traffic in a short period of time, causing huge economic losses and hidden dangers to personnel safety. In addition, geological disasters have also increased the difficulty and cost of maintaining transportation routes in mountainous areas. Frequent repair work not only consumes a lot of manpower and material resources, but also increases the uncertainty of transportation operations. Therefore, frequent geological disasters have become one of the key factors restricting the safe and efficient operation of mountainous transportation. Dangerous rocks on the slopes of traffic corridors are characterized by many points, dispersion, and small individual scale. If traditional methods are used for monitoring, it will consume a lot of manpower, material and financial resources, and there are also many problems in actual operation. For dangerous rock bodies with hidden dangers, using simple, effective and intelligent methods for monitoring will be the key to effectively prevent and avoid dangerous rock body collapse disasters.

[0003] In the existing technology, most of the visible light monitoring equipment is used to monitor the dangerous rock area. The visible light sensor provides high-resolution scene information through light reflection imaging, but it is difficult for the visible light sensor to distinguish obvious targets from the background environment at night or in bad weather. For example, when using visible light monitoring equipment to monitor rockfall on the rock slope of a highway, it is impossible to accurately distinguish between the rockfall and the background throughout the day, and the sensitivity of identifying the trajectory of the rockfall is low, resulting in technical problems such as failure to identify the rockfall and inaccurate prediction of the rockfall trajectory. Therefore, how to accurately identify the rockfall and improve the prediction accuracy of the rockfall trajectory is a technical problem that needs to be solved urgently. Summary of the invention

[0004] In view of this, an embodiment of the present invention provides a dangerous rock trajectory prediction method and device based on the fusion of visible light and infrared thermal imaging to eliminate or improve one or more defects existing in the prior art.

[0005] One aspect of the present invention provides a dangerous rock trajectory prediction method based on the fusion of visible light and infrared thermal imaging, the method comprising: Acquire an infrared thermal imaging video and a visible light video of the dangerous rock area in a first time period, and obtain a fused video based on the infrared thermal imaging video and the visible light video; The fused video is input into a trained dangerous rock mass landslide recognition model, the dangerous rock mass landslide recognition model determines the rockfall position data corresponding to the current image frame based on the current image frame and the previous image frame in the fused video, and obtains a first feature vector based on the current image frame, embeds the first feature vector to obtain an embedded vector, inputs the embedded vector into a downsampling module of the dangerous rock mass landslide recognition model to obtain the image feature vector corresponding to the current image frame, inputs the image feature vector corresponding to the current image frame and the rockfall position data into an upsampling module to predict the rockfall position data corresponding to the next image frame, and the rockfall position data includes the vertex coordinates of the detection box corresponding to the rockfall area.

[0006] In some embodiments of the present invention, Obtaining a fused video based on the infrared thermal imaging video and the visible light video, including: The infrared thermal imaging video and the visible light video are input into a trained first video fusion model, and the first video fusion model splices the image frames of the infrared thermal imaging video and the image frames of the visible light video to obtain a spliced ​​image, and sequentially passes through a convolution layer, a downsampling layer, a first hole convolution layer, an upsampling layer, a transposed convolution layer, and a second hole convolution layer to obtain a fused image corresponding to the spliced ​​image; The fused images corresponding to the stitched images are stitched together to obtain a fused video.

[0007] In some embodiments of the present invention, Obtaining a fused video based on the infrared thermal imaging video and the visible light video, including: Determine the brightness value of each image frame in the visible light video, calculate a first weight value of the infrared thermal imaging video and a second weight value of the visible light video based on each brightness value, calculate a normalized weight coefficient based on the first weight value and the second weight value, and calculate a fused video based on the normalized weight coefficient, the infrared thermal imaging video and the visible light video.

[0008] In some embodiments of the present invention, the calculation formula of the brightness value is: ; Wherein, l(t) represents the brightness value of the t-th image frame in the visible light video, r(t) represents the R channel data of the t-th image frame in the visible light video, g(t) represents the G channel data of the t-th image frame in the visible light video, and b(t) represents the B channel data of the t-th image frame in the visible light video; The calculation formula of the first weight value is: , where l min Indicates the minimum brightness value, l max Indicates the maximum brightness value; The calculation formula for the second weight value is: ,in, is the second weight value, is the first weight value; The calculation formula of the normalized weight coefficient is: , , , where α represents the normalized weight coefficient, Represents the calculation result of the gradient amplitude corresponding to the image frame in the visible light video, Represents the normalized result of the image matrix corresponding to the image frame in the infrared thermal imaging video.

[0009] In some embodiments of the present invention, the fused video is calculated based on the normalized weight coefficient, the infrared thermal imaging video and the visible light video, including: based on Calculate and obtain the R channel data of the fused image corresponding to the image frame; based on Calculate and obtain the G channel data of the fused image corresponding to the image frame; based on Calculate and obtain the B channel data of the fused image corresponding to the image frame; Among them, A r , A g and A b They represent the R channel data, G channel data and B channel data of the image frame in the visible light video, respectively. r , B g and B b They respectively represent the R channel data, G channel data and B channel data of the image frame in the infrared thermal imaging video, and α represents the normalized weight coefficient.

[0010] In some embodiments of the present invention, the method further comprises: Constructing a first sample data set, a first loss function and an initial first video fusion model, and pre-training the initial first video fusion model based on the first sample data set and the first loss function to obtain a trained first video fusion model, wherein the sample data in the first sample data set includes infrared thermal imaging video sample data, visible light video sample data and fusion video sample data; The first loss function is: , where N represents the total number of sample data in the first sample data set, L i represents the predicted value of the fused image, Represents the fused image label value.

[0011] In some embodiments of the present invention, obtaining a fused video based on the infrared thermal imaging video and the visible light video includes: Performing image registration on the image frames in the infrared thermal imaging video and the image frames in the visible light video; and / or, Obtain infrared thermal imaging video and visible light video of the dangerous rock area in the first time period, including: Infrared thermal imaging equipment and visible light equipment at a preset distance from the dangerous rock area respectively shoot infrared thermal imaging videos and visible light videos of the dangerous rock area, and the infrared thermal imaging equipment and visible light equipment are both fixed on equipment support rods.

[0012] In some embodiments of the invention, the method comprises: Constructing a second sample data set, a second loss function and an initial dangerous rock mass collapse and slide recognition model, pre-training the initial dangerous rock mass collapse and slide recognition model based on the second sample data set and the second loss function to obtain a trained dangerous rock mass collapse and slide recognition model, wherein the sample data in the second sample data set includes fused video sample data and rockfall position data sample data corresponding to the next image frame; The second loss function is: ,in, , ρ(p, g) represents the distance between the predicted value of the rockfall center location data and the label value of the rockfall center location data, Q represents the predicted value of the rockfall location data, R represents the label value of the rockfall location data, p represents the predicted value of the rockfall center location data, and g represents the label value of the rockfall center location data. represents the intersection-over-union loss, and Loss represents the second loss function.

[0013] According to another aspect of the present invention, a dangerous rock trajectory prediction device based on the fusion of visible light and infrared thermal imaging is also disclosed. The device includes a processor, a memory, and a computer program stored in the memory. The processor is used to execute the computer program. When the computer program is executed, the device implements the steps of the method described in any of the above embodiments.

[0014] According to yet another aspect of the present invention, a computer-readable storage medium is disclosed, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method described in any of the above embodiments are implemented.

[0015] The dangerous rock mass trajectory prediction method disclosed in the above embodiment of the present invention combines infrared thermal imaging and visible light imaging technology, breaking through the limitations of single visible light monitoring at night or in severe weather conditions, and can achieve all-weather and all-time monitoring. In addition, the fusion video of the present application further achieves a more comprehensive and detailed prediction of the location of falling rocks through the dangerous rock mass collapse and slide recognition model, and can efficiently monitor the start of dangerous rock mass collapse and slide and the movement trajectory of falling rocks; therefore, the dangerous rock mass trajectory prediction method can not only accurately identify falling rocks, but also improve the prediction accuracy of falling rock trajectories.

[0016] In addition, the dangerous rock mass trajectory prediction method of this application adopts intelligent monitoring means, which not only improves the recognition ability of the model, but also greatly reduces the consumption of manpower, material and financial resources, while reducing the maintenance cost and operational uncertainty of mountain transportation routes.

[0017] Additional advantages, purposes, and features of the present invention will be described in part in the following description, and will become apparent to those skilled in the art after studying the following, or may be learned from the practice of the present invention. The purposes and other advantages of the present invention may be achieved and obtained by the structures specifically indicated in the specification and the accompanying drawings.

[0018] Those skilled in the art will appreciate that the objectives and advantages that can be achieved with the present invention are not limited to the above specific description, and the above and other objectives that can be achieved by the present invention will be more clearly understood from the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings described herein are used to provide a further understanding of the present invention, constitute a part of this application, and do not constitute a limitation of the present invention. The components in the drawings are not drawn to scale, but are only for illustrating the principles of the present invention. In order to facilitate the illustration and description of some parts of the present invention, the corresponding parts in the drawings may be enlarged, that is, they may become larger relative to other components in the exemplary device actually manufactured according to the present invention. In the drawings: Figure 1 The present invention is a flowchart of a dangerous rock trajectory prediction method based on the fusion of visible light and infrared thermal imaging according to an embodiment of the present application.

[0020] Figure 2 This is a flow chart of a dangerous rock trajectory prediction method based on the fusion of visible light and infrared thermal imaging according to another embodiment of the present application.

[0021] Figure 3 This is a schematic diagram of the layout of a dangerous rock mass collapse monitoring device according to an embodiment of the present application.

[0022] Figure 4 Schematic diagram of the architecture of a first video fusion model according to an embodiment of the present application.

[0023] Figure 5 Schematic diagram of the architecture of a dangerous rock mass collapse and landslide identification model according to an embodiment of the present application.

[0024] Figure 6 A schematic diagram of the composition of an electronic device according to an embodiment of the present application.

[0025] Reference numerals: Sound and light alarm 1 solar energy 2 infrared thermal imaging device 3 visible light device 4 battery 5 video preliminary processing device 6 electronic device 300 processor 310 memory 320. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments and the accompanying drawings. Here, the illustrative embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0027] It should also be noted that, in order to avoid obscuring the present invention due to unnecessary details, only structures and / or processing steps closely related to the solutions according to the present invention are shown in the accompanying drawings, while other details that are not closely related to the present invention are omitted.

[0028] It should be emphasized that the term “include / comprises” when used herein refers to the presence of features, elements, steps or components, but does not exclude the presence or addition of one or more other features, elements, steps or components.

[0029] It should also be noted here that, unless otherwise specified, the term "connection" in this article can refer not only to a direct connection, but also to an indirect connection with an intermediary, not only to a wired connection, but also to a wireless connection, and the specific connection can be changed based on the actual application scenario.

[0030] In order to better understand the present invention, the terms related to the technical solution are explained as follows: Infrared thermal imaging monitoring equipment passively receives the infrared thermal radiation of the target itself, which is independent of climatic conditions and can work normally regardless of day or night. The core of infrared and visible light image fusion is to integrate the complementary information of the two source images to generate a fused image with better visual effects; in the field of intelligent identification of falling rock trajectories, the fusion technology of visible light and infrared thermal imaging provides more comprehensive information to the machine learning model, improves the accuracy of falling rock identification, and shows good adaptability.

[0031] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In the accompanying drawings, the same reference numerals represent the same or similar components, or the same or similar steps.

[0032] Figure 1 FIG. 1 is a flow chart of a dangerous rock trajectory prediction method based on the fusion of visible light and infrared thermal imaging according to an embodiment of the present application. Figure 1 As shown, the dangerous rock mass trajectory prediction method at least includes steps S10 to S20.

[0033] Step S10: Obtain an infrared thermal imaging video and a visible light video of the dangerous rock area in a first time period, and obtain a fused video based on the infrared thermal imaging video and the visible light video.

[0034] In this step, infrared thermal imaging equipment and visible light equipment at a preset distance from the dangerous rock area can be used to respectively shoot infrared thermal imaging video and visible light video of the dangerous rock area, wherein the infrared thermal imaging equipment 3 and the visible light equipment 4 are both fixed on the equipment support rod. Specifically, firstly, a dangerous rock body collapse monitoring equipment (such as Figure 3 As shown), ensure that the entire movement path of the dangerous rock collapse is within the field of view of the infrared thermal imaging device and the visible light device; illustratively, a support rod can be fixed at a certain distance from the slope, and the support rod can be an iron vertical rod, and the infrared thermal imaging device and the visible light device are both fixed in the middle and upper part of the iron vertical rod; in addition, the support rod can be located in an area 30m-50m away from the slope, and in order to ensure reliable monitoring of the infrared thermal imaging device and the visible light device, the infrared thermal imaging device and the visible light device can be powered by solar energy 2 (360w / 60A). Furthermore, a waterproof box can be installed in the middle of the support rod, and a battery is placed in the waterproof box. The battery stores excess electricity and provides electricity for the entire system when solar energy 2 is not generating electricity.

[0035] In one embodiment, a fused video is obtained based on the infrared thermal imaging video and the visible light video, including: inputting the infrared thermal imaging video and the visible light video into a trained first video fusion model, the first video fusion model splicing the image frames of the infrared thermal imaging video and the image frames of the visible light video to obtain a spliced ​​image, and sequentially passing through a convolution layer, a downsampling layer, a first hole convolution layer, an upsampling layer, a transposed convolution layer, and a second hole convolution layer to obtain a fused image corresponding to the spliced ​​image; splicing the fused images corresponding to each of the spliced ​​images to obtain a fused video. In this embodiment, the fusion of the infrared thermal imaging video and the visible light video is completed based on the trained first video fusion model. In order to obtain a trained first video fusion model, in some embodiments, the dangerous rock trajectory prediction method further includes the following steps: constructing a first sample data set, a first loss function, and an initial first video fusion model, pre-training the initial first video fusion model based on the first sample data set and the first loss function to obtain a trained first video fusion model, the sample data in the first sample data set includes infrared thermal imaging video sample data, visible light video sample data, and fused video sample data; the first loss function is: , where N represents the total number of sample data in the first sample data set, L i represents the predicted value of the fused image, Represents the fused image label value.

[0036] In order to obtain the sample data in the first sample data set, the rockfall can be simulated from the slope with the risk of collapse and landslide to obtain the dangerous rock video sample data. Exemplarily, the collapse and landslide phenomenon of the dangerous rock body can be simulated on the slope, and the infrared thermal imaging device and the visible light device can be used to shoot the video in real time. The camera can be an automatic zoom camera; the captured video data is further transmitted to the video recorder, and uploaded to the remote computer through the 4G router and the switch, and the dangerous rock body collapse and landslide video database is established in the computer. When the infrared thermal imaging device and the visible light device are used to shoot the dangerous rock video, the infrared thermal imaging device and the visible light device are kept at the same horizontal plane, and the distance between the infrared thermal imaging device and the visible light device can be set to 20cm, and the distance between the detection device and the object to be measured is further adjusted (such as 20m, 40m, 60m, etc.) to obtain the infrared thermal imaging video sample data and the visible light video sample data.

[0037] In addition, when a fused video is obtained based on the infrared thermal imaging video and the visible light video, the image frames in the infrared thermal imaging video and the image frames in the visible light video may be registered. Specifically, images are extracted frame by frame from the obtained visible light video and infrared thermal imaging video, and the thermal infrared image and the visible light image are preliminarily registered. The registration algorithm may be a feature-based method (such as SIFT, RIFT, etc.) or a region-based method (such as CFOG, HOPC, etc.). After the preliminary registration is completed, precise registration is performed. After the registration, the shooting areas of the image frames in the infrared thermal imaging video and the image frames in the visible light video are made consistent. In addition, the image frames are fine-tuned to reduce the deformation of the images, and the thermal infrared image frames and the visible light image frames of the same area and resolution are obtained. The thermal infrared image frames provide temperature distribution, and the visible light image frames provide texture and details. The registered thermal infrared image frames and the visible light image frames are fused to obtain a fused image.

[0038] In another embodiment, the fused video can also be obtained based on the following steps: determining the brightness value of each image frame in the visible light video, calculating a first weight value of the infrared thermal imaging video and a second weight value of the visible light video based on each brightness value, calculating a normalized weight coefficient based on the first weight value and the second weight value, and calculating the fused video based on the normalized weight coefficient, the infrared thermal imaging video and the visible light video. In one embodiment, the above steps can be used to determine the fused video sample data based on the infrared thermal imaging video sample data and the visible light video sample data.

[0039] Furthermore, the calculation formulas for the brightness value, the first weight value, the second weight value, and the normalized weight coefficient are as follows: ; ; ; ; ; ; Wherein, l(t) represents the brightness value of the t-th image frame in the visible light video, r(t) represents the R channel data of the t-th image frame in the visible light video, g(t) represents the G channel data of the t-th image frame in the visible light video, and b(t) represents the B channel data of the t-th image frame in the visible light video; l min Indicates the minimum brightness value (when there are multiple sample data, l min represents the average value of the minimum visible light intensity of multiple sample data in one day), l max Indicates the maximum brightness value (when there are multiple sample data, l max Represents the average value of the maximum visible light intensity of multiple sample data in one day); is the second weight value (the weight value corresponding to the infrared thermal imaging video), is the first weight value (the weight value corresponding to the visible light video); represents the importance of features corresponding to visible light video, represents the importance of the features corresponding to the infrared thermal imaging video; α represents the normalized weight coefficient, Represents the gradient amplitude calculation result corresponding to the image frame in the visible light video (that is, the gradient result of [0, 1] obtained by edge algorithms such as Canny operator and Sobel operator in the image frame of the visible light video), It represents the normalized result of the image matrix corresponding to the image frame in the infrared thermal imaging video (that is, the normalized result obtained by normalizing the matrix after the temperature matrix is ​​derived from the image frame in the infrared thermal imaging video). In addition, when determining the importance of the features corresponding to the infrared thermal imaging video, the rockfall position detection frame can be marked in the visible light image as the target area, and the temperature difference matrix between the local temperature of the target area and the surrounding background temperature can be calculated and normalized.

[0040] In some embodiments, the fused video is calculated based on the normalized weight coefficient, the infrared thermal imaging video and the visible light video, which may specifically include: Calculate the R channel data of the fused image corresponding to the image frame; based on Calculate the G channel data of the fused image corresponding to the image frame; based on The B channel data of the fused image corresponding to the image frame is calculated. r , A g and A b They represent the R channel data, G channel data and B channel data of the image frame in the visible light video, respectively. r , Bg and B b In the above embodiment, the weight value corresponding to the image frame of the visible light video is determined based on the brightness value of the image frame in the visible light video, so that the weight of the image frame of the visible light video during the daytime is higher, while the weight of the image frame of the infrared thermal imaging video during the nighttime is higher.

[0041] Step S20: input the fused video into a trained dangerous rock mass landslide recognition model, the dangerous rock mass landslide recognition model determines the rockfall position data corresponding to the current image frame based on the current image frame and the previous image frame in the fused video, and obtains a first feature vector based on the current image frame, embeds the first feature vector to obtain an embedded vector, inputs the embedded vector into a downsampling module of the dangerous rock mass landslide recognition model to obtain the image feature vector corresponding to the current image frame, inputs the image feature vector corresponding to the current image frame and the rockfall position data into an upsampling module to predict the rockfall position data corresponding to the next image frame, and the rockfall position data includes the vertex coordinates of the detection box corresponding to the rockfall area.

[0042] In this step, the rockfall position data corresponding to the next image frame is predicted based on the trained dangerous rock collapse and landslide recognition model. The input information of the dangerous rock collapse and landslide recognition model is the fused video obtained in step S10, and the output information is the rockfall position data corresponding to the next image frame; in specific predictions, the dangerous rock collapse and landslide recognition model inputs the video at the current moment and before the current moment, and the output is the rockfall position data at the next moment.

[0043] Exemplarily, in order to obtain a trained dangerous rock mass collapse and landslide identification model prediction, the dangerous rock mass trajectory prediction method may also include the following steps: constructing a second sample data set, a second loss function and an initial dangerous rock mass collapse and landslide identification model, and pre-training the initial dangerous rock mass collapse and landslide identification model based on the second sample data set and the second loss function to obtain a trained dangerous rock mass collapse and landslide identification model, wherein the sample data in the second sample data set includes fused video sample data and rockfall position data sample data corresponding to the next image frame.

[0044] Furthermore, the fused video sample data may be the fused video sample data in the first sample data set. When constructing the second sample data set, a rockfall detection frame is annotated for each image frame in each fused video sample data, that is, each image frame is derived from the fused video, a rockfall position detection frame is annotated in each image frame as a target area, and the four vertex coordinates of the detection frame are used as labels, which are the rockfall position data in the corresponding image frame, and the dangerous rock mass collapse identification data set data is determined based on the rockfall position data in each image frame. , , Indicates fusion video The corresponding dangerous rock mass collapse and slide identification dataset, A fusion video showing a simulated rock collapse The i image frame exported in the image is merged into a 4*2 matrix with the four vertex coordinates of the rockfall detection frame. , the order of the four vertex coordinates in the matrix is ​​lower left, upper left, upper right, lower right, , n is the total number of image frames; derive the dataset from all fused videos , forming a data set for identifying dangerous rock mass collapse and slide, , m is the total number of fused videos. In this embodiment, the initial dangerous rock mass collapse and slide identification model prediction is pre-trained based on the dangerous rock mass collapse and slide identification data set data.

[0045] Figure 5 This is a schematic diagram of the architecture of the dangerous rock mass collapse identification model of an embodiment of the present application. The model is improved by transformer and combines the encoding and decoding forms. The model starts reasoning with the collapse x, y, w, h detected by the start token and the optical flow algorithm. The generation of each token will refer to the token generated in the previous order, and finally generate [x, y, w, h, end] to inform the user that the prediction is complete. Figure 5 As shown, when pre-training the model, the fusion video can be first Image analysis can be performed, that is, the existing dynamic target detection algorithm can be used for dynamic target detection; for example, by analyzing the optical flow field to predict the start of dangerous rock collapse, the rough range of dangerous rock collapse is delineated with the initial detection frame, and the four vertex coordinates of the initial detection frame are combined to form a 4*2 matrix , . Further extract the fused video from the dataset data Corresponding As a sequence set, the image frame where the initial detection box is located is , The next image frame is In addition, , , As model input, , Output feature vector by slicing the image and importing it into the linear projection layer and the eigenvector , and , Respectively with the position embedding vector and Combined, the image features are extracted through the downsampling module, and the output image features are further imported into the upsampling module. Add the start term (such as start) and import it into the upsampling module through the text encoder. The upsampling module output vector is then output through the text decoder. Predicted rockfall locations .

[0046] During the training process, the loss function Loss is used to calculate the prediction box With label The difference ; ; Among them, ρ(p, g) represents the distance between the predicted value of the rockfall center location data and the label value of the rockfall center location data, Q represents the predicted value of the rockfall location data, R represents the label value of the rockfall location data, p represents the predicted value of the rockfall center location data, and g represents the label value of the rockfall center location data. represents the intersection-over-union loss, and Loss represents the second loss function.

[0047] For the dangerous rock mass collapse and landslide identification model, a machine learning optimizer of existing technology, such as the Adam optimizer, can be selected to train the dangerous rock mass collapse and landslide identification model through the loss function and the optimizer until it converges.

[0048] in addition, Figure 4This is a schematic diagram of the architecture of the first video fusion model of an embodiment of the present application. For example, based on the first video fusion model, a visible light video and an infrared thermal imaging video are used to generate a fused video, that is, a visible light image D1 and a thermal infrared image D2 are used to generate a fused image, and finally a plurality of fused images are spliced ​​together to obtain a fused video. In this embodiment, the infrared thermal imaging video and the visible light video are fused based on an improved UNet network to obtain a more comprehensive and detailed fused video. When generating a fused image, the visible light image D1 and the thermal infrared image D2 are spliced ​​into 6-channel data as the model input. After the convolution operation with a convolution kernel of 7×7, a step size of 2, and a channel number of 64, the output is a 256×256×64 tensor. Then, the 256×256×64 tensor is further subjected to a downsampling module to extract image features. The downsampling module is a four-time pooling layer and a resnet network, and the size of the output tensor is 16×16×512. The output of the downsampling module is then subjected to four dilated convolutions, and the output of the dilated convolutions of five stages is added, and the output size is a tensor of 16×16×512. The output of the dilated convolution is then upsampled four times to obtain a 256×256×64 tensor. At the same time, the downsampled feature output of each layer is added to the upsampled input of each layer through the self-attention module. Finally, the output of the model, i.e., a three-channel fused image, is obtained through a 4×4 transposed convolution operation and a convolution operation with a dilation rate of 1.

[0049] In addition, when training the first video fusion model, the adam optimizer can be selected to train the deep network through the loss function and the optimizer until convergence. In the first video fusion model, each frame of the infrared thermal imaging video and the visible light video is used as input information, and each frame of the fused image is output and reassembled into a video, ultimately generating a more comprehensive and detailed fused video.

[0050] Figure 3 FIG. 1 is a schematic diagram of the arrangement of a dangerous rock mass collapse monitoring device according to an embodiment of the present application. Figure 3As shown, a waterproof box can be further provided in the middle and lower part of the equipment support rod of the dangerous rock body collapse monitoring device, and a video recorder, a hard disk, a 4G router, a switch, etc. are located in the waterproof box, and the video recorder, the hard disk, the 4G router, the switch, etc. are combined into a video preliminary processing device 6 to provide a local area network and cloud support for the entire system, so as to facilitate users to view online. In addition, an audible and visual alarm 1 can be further provided on the top of the support rod to remind pedestrians and vehicles to pay attention to falling rocks when there is a risk of collapse of the dangerous rock body on the slope. That is, in some embodiments, the video of the collapse of the dangerous rock body on the slope is monitored based on the dangerous rock body trajectory prediction method, and alarm measures are taken when the collapse of the dangerous rock body is detected. The alarm measures include sound and light alarms at points 1000m before and after the highway line, sound and light alarms on the monitoring equipment, and SMS notifications to users. In addition, when the motion trajectory of the dangerous rock body is identified in the fused video, if the collapse of the dangerous rock body is detected, the collapse detection frame of the dangerous rock body identified by each image frame is exported, and the midpoints of the detection frames are connected in sequence, and the connection line of the midpoints is the trajectory of the collapse of the dangerous rock body.

[0051] Correspondingly, the present invention also provides a dangerous rock trajectory prediction device based on the fusion of visible light and infrared thermal imaging, the device comprising a processor, a memory and a computer program stored in the memory, the processor is used to execute the computer program, and when the computer program is executed, the device implements the steps of the method described in any of the above embodiments.

[0052] Figure 2 FIG. 1 is a flow chart of a dangerous rock trajectory prediction method based on the fusion of visible light and infrared thermal imaging according to another embodiment of the present application. Figure 2 As shown, the dangerous rock mass trajectory prediction method specifically includes the following steps: Step S110: Setting up dangerous rock mass collapse monitoring equipment to ensure that the entire movement path of the dangerous rock mass collapse is within the field of view of the infrared thermal imaging equipment and the visible light equipment.

[0053] Step S120, simulating the collapse of dangerous rock mass and shooting a video. Rockfall is simulated on a field slope, and a video of a slope with potential collapse and slide danger is shot in real time.

[0054] Step S130: fuse the thermal imaging video and the visible light video using a machine learning method to fuse the infrared thermal imaging video and the visible light video to obtain a more comprehensive and detailed fused video.

[0055] Step S140: Construct a video dataset of dangerous rock mass collapse and landslide. Mark the rockfall detection frame and construct a video dataset of dangerous rock mass collapse and landslide.

[0056] Step S150, training a dangerous rock mass collapse and slide recognition model. Specifically, the dangerous rock mass collapse and slide recognition model is trained using the labeled dangerous rock mass collapse and slide video dataset.

[0057] Step S160, identification detection and alarm are performed through the dangerous rock mass collapse and slide identification model. The video of the dangerous rock mass collapse and slide of the slope is monitored, and alarm measures are taken when the dangerous rock mass collapse and slide phenomenon is detected, and the collapse and slide path is identified.

[0058] According to the above embodiments, it can be found that the dangerous rock mass trajectory prediction method based on the fusion of visible light and infrared thermal imaging of the present application can achieve seamless monitoring during the day and at night by simulating rockfall on the slope and shooting videos of visible light and infrared thermal imaging after fusion, thus overcoming the limitations of a single imaging technology. The position of the dangerous rock collapse in each image frame in the simulated video is marked, and these marks are learned using machine learning to obtain a trajectory recognition model for the collapse of dangerous rock masses, thereby realizing the automatic recognition of the collapse trajectory of dangerous rock masses on slopes in the field, which is fast and efficient. The method comprehensively applies infrared thermal imaging, visible light imaging and machine learning technology, laying the foundation for the study of the movement characteristics of the collapse of dangerous rock masses on slopes, and providing reliable technical support for the protection of transportation structures and early warning of geological disasters.

[0059] The embodiment of the present invention further provides an electronic device 300, Figure 6 FIG. 1 is a schematic diagram of the composition of an electronic device according to an embodiment of the present application. Figure 6 As shown, the electronic device includes a processor 310 and a memory 320 .

[0060] The embodiment of the present invention also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the method described in any of the above embodiments are implemented. The computer-readable storage medium can be a tangible storage medium, such as a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a floppy disk, a hard disk, a removable storage disk, a CD-ROM, or any other form of storage medium known in the technical field.

[0061] It should be understood by those skilled in the art that the exemplary components, systems and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software or a combination of the two. Whether it is performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention. When implemented in hardware, it can be, for example, an electronic circuit, an application-specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present invention are programs or code segments used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link via a data signal carried in a carrier.

[0062] It should be clear that the present invention is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present invention.

[0063] In the present invention, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with features of other embodiments or replace features of other embodiments.

[0064] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the embodiments of the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for predicting dangerous rock mass trajectory based on the fusion of visible light and infrared thermal imaging, characterized in that: The method comprises: Acquire an infrared thermal imaging video and a visible light video of the dangerous rock area in a first time period, and obtain a fused video based on the infrared thermal imaging video and the visible light video; The fused video is input into a trained dangerous rock mass landslide recognition model, the dangerous rock mass landslide recognition model determines the rockfall position data corresponding to the current image frame based on the current image frame and the previous image frame in the fused video, and obtains a first feature vector based on the current image frame, embeds the first feature vector to obtain an embedded vector, inputs the embedded vector into a downsampling module of the dangerous rock mass landslide recognition model to obtain the image feature vector corresponding to the current image frame, inputs the image feature vector corresponding to the current image frame and the rockfall position data into an upsampling module to predict the rockfall position data corresponding to the next image frame, and the rockfall position data includes the vertex coordinates of the detection box corresponding to the rockfall area.

2. The method for predicting dangerous rock mass trajectory based on the fusion of visible light and infrared thermal imaging according to claim 1 is characterized in that: Obtaining a fused video based on the infrared thermal imaging video and the visible light video, including: The infrared thermal imaging video and the visible light video are input into a trained first video fusion model, and the first video fusion model splices the image frames of the infrared thermal imaging video and the image frames of the visible light video to obtain a spliced ​​image, and sequentially passes through a convolution layer, a downsampling layer, a first hole convolution layer, an upsampling layer, a transposed convolution layer, and a second hole convolution layer to obtain a fused image corresponding to the spliced ​​image; The fused images corresponding to the stitched images are stitched together to obtain a fused video.

3. The method for predicting dangerous rock mass trajectory based on the fusion of visible light and infrared thermal imaging according to claim 1 is characterized in that: Obtaining a fused video based on the infrared thermal imaging video and the visible light video, including: Determine the brightness value of each image frame in the visible light video, calculate a first weight value of the infrared thermal imaging video and a second weight value of the visible light video based on each brightness value, calculate a normalized weight coefficient based on the first weight value and the second weight value, and calculate a fused video based on the normalized weight coefficient, the infrared thermal imaging video and the visible light video.

4. The method for predicting dangerous rock mass trajectory based on the fusion of visible light and infrared thermal imaging according to claim 3 is characterized in that: The calculation formula of brightness value is: ； Wherein, l(t) represents the brightness value of the t-th image frame in the visible light video, r(t) represents the R channel data of the t-th image frame in the visible light video, g(t) represents the G channel data of the t-th image frame in the visible light video, and b(t) represents the B channel data of the t-th image frame in the visible light video; The calculation formula of the first weight value is: , where l min Indicates the minimum brightness value, l max Indicates the maximum brightness value; The calculation formula for the second weight value is: ,in, is the second weight value, is the first weight value; The calculation formula of the normalized weight coefficient is: , , , where α represents the normalized weight coefficient, Represents the calculation result of the gradient amplitude corresponding to the image frame in the visible light video, Represents the normalized result of the image matrix corresponding to the image frame in the infrared thermal imaging video.

5. The method for predicting dangerous rock mass trajectory based on the fusion of visible light and infrared thermal imaging according to claim 4 is characterized in that: The fused video is obtained by calculating based on the normalized weight coefficient, the infrared thermal imaging video and the visible light video, including: based on Calculate and obtain the R channel data of the fused image corresponding to the image frame; based on Calculate and obtain the G channel data of the fused image corresponding to the image frame; based on Calculate and obtain the B channel data of the fused image corresponding to the image frame; Among them, A r , A g and A b They represent the R channel data, G channel data and B channel data of the image frame in the visible light video, respectively. r , B g and B b They respectively represent the R channel data, G channel data and B channel data of the image frame in the infrared thermal imaging video, and α represents the normalized weight coefficient.

6. The method for predicting dangerous rock mass trajectory based on the fusion of visible light and infrared thermal imaging according to claim 2 is characterized in that: The method further comprises: Constructing a first sample data set, a first loss function and an initial first video fusion model, and pre-training the initial first video fusion model based on the first sample data set and the first loss function to obtain a trained first video fusion model, wherein the sample data in the first sample data set includes infrared thermal imaging video sample data, visible light video sample data and fusion video sample data; The first loss function is: , where N represents the total number of sample data in the first sample data set, L i represents the predicted value of the fused image, Represents the fused image label value.

7. The method for predicting dangerous rock mass trajectory based on the fusion of visible light and infrared thermal imaging according to claim 1 is characterized in that: Obtaining a fused video based on the infrared thermal imaging video and the visible light video, including: Performing image registration on the image frames in the infrared thermal imaging video and the image frames in the visible light video; and / or, Obtain infrared thermal imaging video and visible light video of the dangerous rock area in the first time period, including: Infrared thermal imaging equipment and visible light equipment at a preset distance from the dangerous rock area respectively shoot infrared thermal imaging videos and visible light videos of the dangerous rock area, and the infrared thermal imaging equipment and visible light equipment are both fixed on equipment support rods.

8. The method for predicting dangerous rock mass trajectory based on the fusion of visible light and infrared thermal imaging according to claim 1 is characterized in that: The method comprises: Constructing a second sample data set, a second loss function and an initial dangerous rock mass collapse and slide recognition model, pre-training the initial dangerous rock mass collapse and slide recognition model based on the second sample data set and the second loss function to obtain a trained dangerous rock mass collapse and slide recognition model, wherein the sample data in the second sample data set includes fused video sample data and rockfall position data sample data corresponding to the next image frame; The second loss function is: ,in, , ρ(p, g) represents the distance between the predicted value of the rockfall center location data and the label value of the rockfall center location data, Q represents the predicted value of the rockfall location data, R represents the label value of the rockfall location data, p represents the predicted value of the rockfall center location data, and g represents the label value of the rockfall center location data. represents the intersection-over-union loss, and Loss represents the second loss function.

9. A device for predicting dangerous rock mass trajectory based on the fusion of visible light and infrared thermal imaging, the device comprising a processor, a memory and a computer program stored in the memory, characterized in that: The processor is used to execute the computer program. When the computer program is executed, the device implements the steps of the method according to any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images