Multispectral fusion display method and system for smart media high-definition video

Abstract

The invention provides a multispectral fusion display method and system for a smart media high-definition video. The method comprises the following steps: collecting infrared light and visible light video data for feature point identification, processing according to feature point coordinate data to obtain spatial transformation model parameters, performing feature extraction on the infrared light and visible light video data, processing according to feature extraction data to obtain spatial distance data, the infrared light feature data and the visible light feature data are combined for processing to obtain a feature fusion evaluation index, and the infrared light feature data and the visible light feature data are subjected to spatial registration according to spatial transformation model parameters and then are subjected to feature fusion; and processing according to the feature fusion vector to obtain a texture feature anomaly evaluation index and an edge feature anomaly evaluation index, further processing to obtain a fire risk evaluation index, and carrying out fire risk grade labeling display at a video end. Therefore, the technology of evaluating the fire risk through the multispectral fusion technology and carrying out video end annotation display is realized.

Description

Technical Field

[0001] The present application relates to the field of big data and multi-spectral fusion technology, and more specifically, to a multi-spectral fusion display method and system for smart media high-definition video. Background Art

[0002] As an important part of smart construction sites, video surveillance can intuitively present the real-time status of the construction site and provide strong support for remote management. In recent years, multispectral imaging technology has been widely used. Multispectral imaging can simultaneously obtain image information of multiple different spectral bands, covering multiple spectral ranges such as visible light and infrared light. Images of different spectral bands have unique sensitivity to different object features and environmental conditions. By fusing multispectral images, the advantages of each spectral band image can be fully integrated to make up for the shortcomings of single spectral imaging, thereby generating fused images containing richer and more accurate information, providing stronger technical support for video surveillance and management of smart construction sites. However, there is currently a lack of a multispectral fusion display technology for smart media high-definition video designed for fire risk scenarios. Even in the early stage of a fire when the flame is small, fire risk assessment can be achieved based on the color texture changes of visible light and the thermal sensitivity of infrared light, providing stronger support for fire prevention and control in smart construction sites.

[0003] In view of the above problems, effective technical solutions are urgently needed. Summary of the invention

[0004] The purpose of this application is to provide a multi-spectral fusion display method and system for smart media high-definition video, which is a multi-spectral fusion display technology for smart media high-definition video designed for fire risk scenarios. It obtains the fire risk assessment level by fusing multi-spectral data containing both infrared light and visible light, and performs feature anomaly assessment on the feature-fused data, and then annotates and displays the fire risk assessment level on the video end.

[0005] The present application also provides a multi-spectral fusion display method for smart media high-definition video, comprising the following steps: Collect infrared light video data and visible light video data and perform feature point recognition to obtain infrared light feature point coordinate data and visible light feature point coordinate data, and process them to obtain space transformation model parameters; Extract features from the infrared light video data and the visible light video data respectively to obtain infrared light feature data and visible light feature data, and process them to obtain spatial distance data; Obtaining a feature fusion evaluation index according to the spatial distance data and the infrared light feature data and the visible light feature data, and performing spatial registration on the infrared light feature data and the visible light feature data according to the spatial transformation model parameters; Perform feature fusion on the registered infrared light feature data and visible light feature data to obtain a feature fusion vector, and obtain a texture feature anomaly assessment index based on the feature fusion vector processing; Obtain edge feature abnormality assessment index based on feature fusion vector processing; The fire risk assessment index is obtained according to the texture feature anomaly assessment index and the edge feature anomaly assessment index, and the fire risk level is marked and displayed on the video end.

[0006] Optionally, in the multi-spectral fusion display method of smart media high-definition video described in the present application, the collecting of infrared light video data and visible light video data and the identification of feature points, obtaining infrared light feature point coordinate data and visible light feature point coordinate data, and processing to obtain spatial transformation model parameters include: Collect infrared light video data and visible light video data, perform feature point recognition on the infrared light video data and the visible light video data, and obtain infrared light feature point coordinate data and visible light feature point coordinate data; The infrared light feature point coordinate data includes infrared light corner point coordinate data, infrared light contour point coordinate data, infrared light texture intersection point coordinate data, infrared light local extreme point coordinate data and infrared light texture mutation point coordinate data; The visible light feature point coordinate data includes visible light corner point coordinate data, visible light contour point coordinate data, visible light texture intersection point coordinate data, visible light local extreme point coordinate data and visible light texture mutation point coordinate data; The infrared light feature point coordinate data and the visible light feature point coordinate data are input into a preset space transformation model for processing to obtain space transformation model parameters.

[0007] Optionally, in the multi-spectral fusion display method of smart media high-definition video described in the present application, the feature extraction of infrared light video data and visible light video data is performed respectively to obtain infrared light feature data and visible light feature data, and the processing is performed to obtain spatial distance data, including: Extracting features from the infrared light video data and the visible light video data respectively to obtain infrared light feature data and visible light feature data; The infrared light feature data includes thermal radiation pixel value, infrared light texture feature data, infrared light edge feature data and infrared light contour feature data; The infrared light texture feature data includes boundary thermal contrast and boundary thermal correlation data; the infrared light edge feature data includes edge thermal pixel gradient value, edge thermal pixel gradient direction change frequency and edge thermal pixel gradient direction change angle; the infrared light contour feature data includes thermal radiation area centroid coordinate data and thermal radiation area area; The visible light feature data includes visible light texture feature data, visible light edge feature data and visible light contour feature data, and the visible light texture feature data includes boundary color contrast and boundary color correlation data; The visible light edge feature data includes edge pixel gradient value, edge pixel gradient direction change frequency and edge pixel gradient direction change angle, and the visible light contour feature data includes contour centroid coordinate data and contour area; A spatial distance calculation is performed based on the heat radiation area centroid coordinate data and the contour centroid coordinate data to obtain spatial distance data.

[0008] Optionally, in the multi-spectral fusion display method of smart media high-definition video described in the present application, the processing of the spatial distance data and the infrared light feature data and the visible light feature data to obtain a feature fusion evaluation index, and spatially registering the infrared light feature data and the visible light feature data according to the spatial transformation model parameters includes: Processing is performed according to the spatial distance data and the contour area and the thermal radiation area to obtain a feature fusion evaluation index; The feature fusion evaluation index is compared with a preset feature fusion evaluation index threshold. If the threshold comparison result meets the preset threshold comparison result requirement, the infrared light feature data and the visible light feature data are spatially aligned according to the spatial transformation model parameters.

[0009] Optionally, in the multi-spectral fusion display method of smart media high-definition video described in the present application, the feature fusion of the registered infrared light feature data and the visible light feature data to obtain a feature fusion vector, and the texture feature anomaly assessment index is obtained according to the feature fusion vector processing, including: The registered thermal radiation feature data, infrared light texture feature data, infrared light edge feature data, infrared light contour feature data and the visible light texture feature data, visible light edge feature data and visible light contour feature data are combined to generate a feature fusion vector; The texture feature anomaly assessment index is obtained according to the boundary thermal contrast, boundary thermal correlation data and the boundary color contrast, boundary color correlation data.

[0010] Optionally, in the multi-spectral fusion display method of smart media high-definition video described in the present application, the step of obtaining an edge feature anomaly evaluation index according to feature fusion vector processing includes: The edge feature anomaly assessment index is obtained according to the edge thermal pixel gradient value, the edge thermal pixel gradient direction change frequency, the edge thermal pixel gradient direction change angle and the edge pixel gradient value, the edge pixel gradient direction change frequency, and the edge pixel gradient direction change angle.

[0011] Optionally, in the multi-spectral fusion display method of the smart media high-definition video described in the present application, the fire risk assessment index is obtained according to the texture feature anomaly assessment index and the edge feature anomaly assessment index, and the fire risk level is marked and displayed on the video end, including: Obtaining a fire risk assessment index according to the thermal radiation pixel value, the texture feature abnormality assessment index and the edge feature abnormality assessment index; The fire risk assessment index is compared with a preset fire risk assessment index threshold, the fire risk level is determined according to the risk level to which the threshold comparison result belongs, and the fire risk level is marked and displayed on the video end.

[0012] In a second aspect, the present application provides a multi-spectral fusion display system for smart media high-definition video, the system comprising: a memory and a processor, wherein the memory stores a program of a multi-spectral fusion display method for smart media high-definition video, and when the program of the multi-spectral fusion display method for smart media high-definition video is executed by the processor, the following steps are implemented: Collect infrared light video data and visible light video data and perform feature point recognition to obtain infrared light feature point coordinate data and visible light feature point coordinate data, and process them to obtain space transformation model parameters; Extract features from the infrared light video data and the visible light video data respectively to obtain infrared light feature data and visible light feature data, and process them to obtain spatial distance data; Obtaining a feature fusion evaluation index according to the spatial distance data and the infrared light feature data and the visible light feature data, and performing spatial registration on the infrared light feature data and the visible light feature data according to the spatial transformation model parameters; Perform feature fusion on the registered infrared light feature data and visible light feature data to obtain a feature fusion vector, and obtain a texture feature anomaly assessment index based on the feature fusion vector processing; Obtain edge feature abnormality assessment index based on feature fusion vector processing; The fire risk assessment index is obtained according to the texture feature anomaly assessment index and the edge feature anomaly assessment index, and the fire risk level is marked and displayed on the video end.

[0013] Optionally, in the multi-spectral fusion display system of the smart media high-definition video described in the present application, the collecting of infrared light video data and visible light video data and the identification of feature points, obtaining infrared light feature point coordinate data and visible light feature point coordinate data, and processing to obtain spatial transformation model parameters include: Collect infrared light video data and visible light video data, perform feature point recognition on the infrared light video data and the visible light video data, and obtain infrared light feature point coordinate data and visible light feature point coordinate data; The infrared light feature point coordinate data includes infrared light corner point coordinate data, infrared light contour point coordinate data, infrared light texture intersection point coordinate data, infrared light local extreme point coordinate data and infrared light texture mutation point coordinate data; The visible light feature point coordinate data includes visible light corner point coordinate data, visible light contour point coordinate data, visible light texture intersection point coordinate data, visible light local extreme point coordinate data and visible light texture mutation point coordinate data; The infrared light feature point coordinate data and the visible light feature point coordinate data are input into a preset space transformation model for processing to obtain space transformation model parameters.

[0014] In a third aspect, the present application also provides a computer-readable storage medium, which stores a multi-spectral fusion display method program for smart media high-definition video. When the multi-spectral fusion display method program for smart media high-definition video is executed by a processor, the steps of the multi-spectral fusion display method for smart media high-definition video as described in any one of the above items are implemented.

[0015] From the above, it can be seen that the multi-spectral fusion display method and system for smart media high-definition video provided by the present application is a multi-spectral fusion display technology for smart media high-definition video designed for fire risk scenarios. It obtains the fire risk assessment level by fusing multi-spectral data that contains both infrared light and visible light, and performs feature anomaly assessment on the feature-fused data, and then annotates and displays the fire risk assessment level on the video end.

[0016] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or understood by practicing the embodiments of the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the written description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 A flowchart of a multi-spectral fusion display method for smart media high-definition video provided in an embodiment of the present application; Figure 2 A flowchart of obtaining spatial transformation model parameters for a multi-spectral fusion display method of a smart media high-definition video provided in an embodiment of the present application; Figure 3 A flowchart of obtaining spatial distance data of a multi-spectral fusion display method for smart media high-definition video provided in an embodiment of the present application; Figure 4 A flowchart of spatial registration of a multi-spectral fusion display method for smart media high-definition video provided in an embodiment of the present application. DETAILED DESCRIPTION

[0019] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0020] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0021] Please refer to Figure 1 , Figure 1 The multi-spectral fusion display method of the smart media high-definition video in some embodiments of the present application is a flow chart. The multi-spectral fusion display method of the smart media high-definition video is used in a terminal device, such as a computer, a mobile phone terminal, etc. The multi-spectral fusion display method of the smart media high-definition video includes the following steps: S11, collecting infrared light video data and visible light video data and performing feature point recognition, obtaining infrared light feature point coordinate data and visible light feature point coordinate data, and processing to obtain space transformation model parameters; S12, extracting features from the infrared light video data and the visible light video data respectively to obtain infrared light feature data and visible light feature data, and processing them to obtain spatial distance data; S13, obtaining a feature fusion evaluation index according to the spatial distance data and the infrared light feature data and the visible light feature data, and performing spatial registration on the infrared light feature data and the visible light feature data according to the spatial transformation model parameters; S14, performing feature fusion on the registered infrared light feature data and the visible light feature data to obtain a feature fusion vector, and obtaining a texture feature anomaly assessment index according to the feature fusion vector processing; S15, obtaining an edge feature abnormality assessment index according to feature fusion vector processing; S16. Obtain a fire risk assessment index according to the texture feature anomaly assessment index and the edge feature anomaly assessment index, and annotate and display the fire risk level on the video end.

[0022] It should be noted that this application is a multi-spectral fusion technology designed for construction site fire scenes, which realizes intelligent detection and monitoring of construction site fire risks. In order to correct the geometric deformation differences between images caused by factors such as shooting angle, distance, and equipment differences, so that the shape and position relationship of the same object in the two images can be better mathematically corresponding, it is necessary to spatially align different spectral data. Therefore, it is first necessary to identify the feature points of the collected infrared video data and visible light video data, obtain the spatial transformation model parameters based on the feature point coordinate data processing, and map each pixel in the infrared image to the position corresponding to the visible light image according to this transformation rule based on the spatial transformation model parameters. Then, feature extraction is performed on the infrared light video data and the visible light video data, and spatial distance data is obtained based on feature extraction data processing. The feature fusion evaluation index is then obtained by combining the infrared light feature data and the visible light feature data processing. Since the goal of spatial registration is to accurately align the same objects or features in the visible light and infrared light images in spatial position, the spatial distance between the corresponding feature points and the matching degree of the contour area can be calculated to determine whether the conditions for spatial registration are met. If the requirements are met, the infrared light feature data and the visible light feature data are spatially registered according to the spatial transformation model parameters, and then feature fusion is performed. The texture feature anomaly evaluation index and the edge feature anomaly evaluation index are obtained based on the feature fusion vector processing, and further processed to obtain the fire risk assessment index, and the fire risk level is annotated and displayed on the video end. Thus, the technology of evaluating the fire risk through multi-spectral fusion technology and annotating and displaying it on the video end is realized.

[0023] Please refer to Figure 2 , Figure 2The present invention is a flowchart of obtaining spatial transformation model parameters of a multi-spectral fusion display method of smart media high-definition video in some embodiments of the present application. According to an embodiment of the present invention, the infrared light video data and the visible light video data are collected and feature point recognition is performed to obtain infrared light feature point coordinate data and visible light feature point coordinate data, and the spatial transformation model parameters are obtained by processing, including: S21, collecting infrared light video data and visible light video data, and performing feature point recognition on the infrared light video data and the visible light video data respectively to obtain infrared light feature point coordinate data and visible light feature point coordinate data; S22, the infrared light feature point coordinate data includes infrared light corner point coordinate data, infrared light contour point coordinate data, infrared light texture intersection point coordinate data, infrared light local extreme point coordinate data and infrared light texture mutation point coordinate data; S23, the visible light feature point coordinate data includes visible light corner point coordinate data, visible light contour point coordinate data, visible light texture intersection point coordinate data, visible light local extreme point coordinate data and visible light texture mutation point coordinate data; S24, inputting the infrared light feature point coordinate data and the visible light feature point coordinate data into a preset space transformation model for processing to obtain space transformation model parameters.

[0024] It should be noted that, in this embodiment, spatial registration refers to the use of a spatial transformation model to transform the coordinate system of the infrared light image into a coordinate system that matches the visible light image, in order to correct the geometric deformation differences between the images due to factors such as shooting angles, distances, and equipment differences, so that the shapes and positional relationships of the same object in the two images can correspond better mathematically. In this embodiment, corner points, contour points, texture intersection points, local extreme points, and texture mutation points that are relatively fixed in position under different spectra, have good repeatability and stability, and are representative are selected as feature points to determine the parameters of the spatial transformation model. Among them, corner points are points in the image where the grayscale changes dramatically in multiple directions, contour points refer to pixel points at the edge of objects or scenes in the image, texture intersection points refer to the intersection points of lines in the texture, local extreme points refer to local grayscale maximum points and local grayscale minimum points, and points where the contrast of the grayscale co-occurrence matrix (GLCM) in the texture points is greater than the set threshold can be used as texture mutation points.

[0025] The calculation formula of the spatial transformation model parameters is: ; in, is the infrared light feature point coordinate data, is the coordinate data of the visible light feature points, , , , , and are model parameters; In this embodiment, multiple sets of infrared light corner point coordinate data, infrared light contour point coordinate data, infrared light texture intersection point coordinate data, infrared light local extreme point coordinate data, infrared light texture mutation point coordinate data and their corresponding visible light corner point coordinate data, visible light contour point coordinate data, visible light texture intersection point coordinate data, visible light local extreme point coordinate data, and visible light texture mutation point coordinate data can be substituted into the calculation formula of the spatial transformation model parameters to obtain a group of linear equations, and the model parameters can be obtained after solving them using the least squares method.

[0026] Please refer to Figure 3 , Figure 3 The present invention is a flowchart of obtaining spatial distance data of a multi-spectral fusion display method of a smart media high-definition video in some embodiments of the present application. According to an embodiment of the present invention, the infrared light video data and the visible light video data are respectively extracted to obtain infrared light feature data and visible light feature data, and the spatial distance data is obtained by processing, including: S31, extracting features from the infrared light video data and the visible light video data respectively to obtain infrared light feature data and visible light feature data; S32, the infrared light feature data includes thermal radiation pixel value, infrared light texture feature data, infrared light edge feature data and infrared light contour feature data; S33, the infrared light texture feature data includes boundary thermal contrast and boundary thermal correlation data, the infrared light edge feature data includes edge thermal pixel gradient value, edge thermal pixel gradient direction change frequency and edge thermal pixel gradient direction change angle, and the infrared light contour feature data includes thermal radiation area centroid coordinate data and thermal radiation area area; S34, the visible light feature data includes visible light texture feature data, visible light edge feature data and visible light contour feature data, and the visible light texture feature data includes boundary color contrast and boundary color correlation data; S35, the visible light edge feature data includes edge pixel gradient value, edge pixel gradient direction change frequency and edge pixel gradient direction change angle, and the visible light contour feature data includes contour centroid coordinate data and contour area; S36. Perform spatial distance calculation based on the heat radiation area centroid coordinate data and the contour centroid coordinate data to obtain spatial distance data.

[0027] It should be noted that the calculation of spatial distance can accurately quantify the position difference of objects in visible light and infrared light images. For the same target, such as flame, the thermal radiation mainly comes from the inside of the object, and its centroid position should be close to the centroid position of the object contour. If the spatial distance is too large, it means that the thermal radiation area and the contour may not correspond to the same object, and spatial registration and feature fusion cannot be performed. Therefore, the spatial distance calculation can be performed based on the centroid coordinate data of the thermal radiation area and the centroid coordinate data of the contour to obtain the spatial distance data; The calculation formula of the spatial distance data is: ; in, is the spatial distance data, is the contour centroid coordinate data, It is the centroid coordinate data of the thermal radiation area.

[0028] Please refer to Figure 4 , Figure 4 The present invention is a flowchart of performing spatial registration of a multi-spectral fusion display method of a smart media high-definition video in some embodiments of the present application. According to an embodiment of the present invention, the processing of the spatial distance data and the infrared light feature data and the visible light feature data to obtain a feature fusion evaluation index, and performing spatial registration of the infrared light feature data and the visible light feature data according to the spatial transformation model parameters, includes: S41, performing processing according to the spatial distance data and the contour area and the thermal radiation area to obtain a feature fusion evaluation index; S42. Compare the feature fusion evaluation index with a preset feature fusion evaluation index threshold. If the threshold comparison result meets the preset threshold comparison result requirement, spatially align the infrared light feature data and the visible light feature data according to the spatial transformation model parameters.

[0029] It should be noted that, under normal circumstances, if the same target (such as a flame) is detected, they should have a good correspondence. For example, the area surrounded by the outline of the flame under visible light should be roughly consistent with the area of ​​the thermal radiation area under infrared light. Therefore, the spatial distance data and the outline area and thermal radiation area are processed to obtain the feature fusion evaluation index. If the feature fusion evaluation index is less than or equal to the preset threshold, it means that the same target is detected, and spatial registration and feature fusion can be performed.

[0030] The calculation formula of the feature fusion evaluation index is: ; in, is the feature fusion evaluation index, is the spatial distance data, is the contour area, is the area of ​​heat radiation, is the preset characteristic coefficient (which can be obtained by querying the preset multispectral image database).

[0031] According to an embodiment of the present invention, the step of performing feature fusion on the registered infrared light feature data and the visible light feature data to obtain a feature fusion vector, and obtaining a texture feature anomaly assessment index according to the feature fusion vector processing includes: The registered thermal radiation feature data, infrared light texture feature data, infrared light edge feature data, infrared light contour feature data and the visible light texture feature data, visible light edge feature data and visible light contour feature data are combined to generate a feature fusion vector; The texture feature anomaly assessment index is obtained according to the boundary thermal contrast, boundary thermal correlation data and the boundary color contrast, boundary color correlation data.

[0032] It should be noted that the boundary contrast of the flame is very high under both infrared light and visible light. Because the temperature and color differences between the flame and the surrounding environment are large, the pixel values ​​at the boundary change significantly. And because the flame shape is irregular and changes dynamically, the correlation of the flame boundary is low. Among them, the contrast and correlation data can be obtained based on the gray level co-occurrence matrix (GLCM). Therefore, by comprehensively considering the boundary thermal contrast, boundary thermal correlation, boundary color contrast, and boundary color correlation data, an assessment of the abnormality of texture features can be obtained, thereby achieving the purpose of assessing the risk of fire. The calculation formula of the texture feature abnormality evaluation index is: ; in, is the texture feature abnormality assessment index, is the boundary thermal correlation data, is the boundary color correlation data, is the boundary thermal contrast, is the boundary color contrast, , , and is the preset characteristic coefficient (which can be obtained by querying the preset multispectral image database).

[0033] According to an embodiment of the present invention, the step of obtaining an edge feature anomaly assessment index according to feature fusion vector processing includes: The edge feature anomaly assessment index is obtained according to the edge thermal pixel gradient value, the edge thermal pixel gradient direction change frequency, the edge thermal pixel gradient direction change angle and the edge pixel gradient value, the edge pixel gradient direction change frequency, and the edge pixel gradient direction change angle.

[0034] It should be noted that at the edge of the flame, the thermal pixels (infrared imaging) or color pixels (visible light imaging) change dramatically, and the pixel values ​​differ significantly from the high temperature area inside the flame to the low temperature area outside. At the edge of the flame, the thermal pixel gradient value and the pixel gradient value will increase significantly, and the flame will drift with the wind, and the frequency and angle of change of the edge thermal pixel gradient direction and the frequency and angle of change of the edge pixel gradient direction will increase accordingly. Therefore, the edge feature anomaly assessment index is obtained based on the edge thermal pixel gradient value, the frequency of change of the edge thermal pixel gradient direction, the angle of change of the edge thermal pixel gradient direction, the edge pixel gradient value, the frequency of change of the edge pixel gradient direction, and the angle of change of the edge pixel gradient direction. Among them, the pixel gradient value is used to measure the rate of change of the thermal radiation intensity (thermal signal corresponding to temperature) of the pixel in spatial position, the edge thermal pixel gradient direction change frequency refers to the frequency of thermal pixel gradient direction changes on the edge of the hot area, the edge thermal pixel gradient direction change angle refers to the angle change of the thermal pixel gradient direction from one position to another on the edge, the edge pixel gradient value is used to measure the rate of change of the pixel grayscale (or color intensity, etc.) in the image in spatial position, the edge pixel gradient direction change frequency refers to the frequency of pixel gradient direction changes on the edge of the object in the image, and the edge pixel gradient direction change angle refers to the angle change of the pixel gradient direction from one position to another on the edge.

[0035] The calculation formula of the edge feature abnormality evaluation index is: ; in, is the edge feature abnormality assessment index, , and They are the edge hot pixel gradient value, the edge hot pixel gradient direction change frequency, and the edge hot pixel gradient direction change angle. , and are the edge pixel gradient value, the edge pixel gradient direction change frequency, and the edge pixel gradient direction change angle, respectively. , , , , and is the preset characteristic coefficient (which can be obtained by querying the preset multispectral image database).

[0036] According to an embodiment of the present invention, the processing of obtaining a fire risk assessment index according to the texture feature anomaly assessment index and the edge feature anomaly assessment index, and marking and displaying the fire risk level on the video end, includes: Obtaining a fire risk assessment index according to the thermal radiation pixel value, the texture feature abnormality assessment index and the edge feature abnormality assessment index; The fire risk assessment index is compared with a preset fire risk assessment index threshold, the fire risk level is determined according to the risk level to which the threshold comparison result belongs, and the fire risk level is marked and displayed on the video end.

[0037] It should be noted that when a fire occurs, the thermal radiation pixel value increases significantly, and the fire risk assessment index is obtained by processing the thermal radiation pixel value, the texture feature abnormality assessment index and the edge feature abnormality assessment index; The calculation formula of the fire risk assessment index is: ; in, is the fire risk assessment index, is the thermal radiation pixel value, is the texture feature abnormality assessment index, is the edge feature abnormality assessment index, and is the preset weight coefficient (which can be obtained by querying the preset multispectral image database).

[0038] The present invention also discloses a multi-spectral fusion display system for smart media high-definition video, including a memory and a processor. The memory stores a multi-spectral fusion display method program for smart media high-definition video. When the multi-spectral fusion display method program for smart media high-definition video is executed by the processor, the following steps are implemented: Collect infrared light video data and visible light video data and perform feature point recognition to obtain infrared light feature point coordinate data and visible light feature point coordinate data, and process them to obtain space transformation model parameters; Extract features from the infrared light video data and the visible light video data respectively to obtain infrared light feature data and visible light feature data, and process them to obtain spatial distance data; Obtaining a feature fusion evaluation index according to the spatial distance data and the infrared light feature data and the visible light feature data, and performing spatial registration on the infrared light feature data and the visible light feature data according to the spatial transformation model parameters; Perform feature fusion on the registered infrared light feature data and visible light feature data to obtain a feature fusion vector, and obtain a texture feature anomaly assessment index based on the feature fusion vector processing; Obtain edge feature abnormality assessment index based on feature fusion vector processing; The fire risk assessment index is obtained according to the texture feature anomaly assessment index and the edge feature anomaly assessment index, and the fire risk level is marked and displayed on the video end.

[0039] It should be noted that this application is a multi-spectral fusion technology designed for construction site fire scenes, which realizes intelligent detection and monitoring of construction site fire risks. In order to correct the geometric deformation differences between images caused by factors such as shooting angle, distance, and equipment differences, so that the shape and position relationship of the same object in the two images can be better mathematically corresponding, it is necessary to spatially align different spectral data. Therefore, it is first necessary to identify the feature points of the collected infrared video data and visible light video data, obtain the spatial transformation model parameters based on the feature point coordinate data processing, and map each pixel in the infrared image to the position corresponding to the visible light image according to this transformation rule based on the spatial transformation model parameters. Then, feature extraction is performed on the infrared light video data and the visible light video data, and spatial distance data is obtained based on feature extraction data processing. The feature fusion evaluation index is then obtained by combining the infrared light feature data and the visible light feature data processing. Since the goal of spatial registration is to accurately align the same objects or features in the visible light and infrared light images in spatial position, the spatial distance between the corresponding feature points and the matching degree of the contour area can be calculated to determine whether the conditions for spatial registration are met. If the requirements are met, the infrared light feature data and the visible light feature data are spatially registered according to the spatial transformation model parameters, and then feature fusion is performed. The texture feature anomaly evaluation index and the edge feature anomaly evaluation index are obtained based on the feature fusion vector processing, and further processed to obtain the fire risk assessment index, and the fire risk level is annotated and displayed on the video end. Thus, the technology of evaluating the fire risk through multi-spectral fusion technology and annotating and displaying it on the video end is realized.

[0040] According to an embodiment of the present invention, the collecting of infrared light video data and visible light video data and performing feature point recognition to obtain infrared light feature point coordinate data and visible light feature point coordinate data, and processing to obtain spatial transformation model parameters, includes: Collect infrared light video data and visible light video data, perform feature point recognition on the infrared light video data and the visible light video data, and obtain infrared light feature point coordinate data and visible light feature point coordinate data; The infrared light feature point coordinate data includes infrared light corner point coordinate data, infrared light contour point coordinate data, infrared light texture intersection point coordinate data, infrared light local extreme point coordinate data and infrared light texture mutation point coordinate data; The visible light feature point coordinate data includes visible light corner point coordinate data, visible light contour point coordinate data, visible light texture intersection point coordinate data, visible light local extreme point coordinate data and visible light texture mutation point coordinate data; The infrared light feature point coordinate data and the visible light feature point coordinate data are input into a preset space transformation model for processing to obtain space transformation model parameters.

[0041] It should be noted that, in this embodiment, spatial registration refers to the use of a spatial transformation model to transform the coordinate system of the infrared light image into a coordinate system that matches the visible light image, in order to correct the geometric deformation differences between the images due to factors such as shooting angles, distances, and equipment differences, so that the shapes and positional relationships of the same object in the two images can correspond better mathematically. In this embodiment, corner points, contour points, texture intersection points, local extreme points, and texture mutation points that are relatively fixed in position under different spectra, have good repeatability and stability, and are representative are selected as feature points to determine the parameters of the spatial transformation model. Among them, corner points are points in the image where the grayscale changes dramatically in multiple directions, contour points refer to pixel points at the edge of objects or scenes in the image, texture intersection points refer to the intersection points of lines in the texture, local extreme points refer to local grayscale maximum points and local grayscale minimum points, and points where the contrast of the grayscale co-occurrence matrix (GLCM) in the texture points is greater than the set threshold can be used as texture mutation points.

[0042] The calculation formula of the spatial transformation model parameters is: ; in, is the infrared light feature point coordinate data, is the coordinate data of the visible light feature points, , , , , and are model parameters; In this embodiment, multiple sets of infrared light corner point coordinate data, infrared light contour point coordinate data, infrared light texture intersection point coordinate data, infrared light local extreme point coordinate data, infrared light texture mutation point coordinate data and their corresponding visible light corner point coordinate data, visible light contour point coordinate data, visible light texture intersection point coordinate data, visible light local extreme point coordinate data, and visible light texture mutation point coordinate data can be substituted into the calculation formula of the spatial transformation model parameters to obtain a group of linear equations, and the model parameters can be obtained after solving them using the least squares method.

[0043] According to an embodiment of the present invention, the feature extraction of the infrared light video data and the visible light video data is performed to obtain infrared light feature data and visible light feature data, and the processing is performed to obtain spatial distance data, including: Extracting features from the infrared light video data and the visible light video data respectively to obtain infrared light feature data and visible light feature data; The infrared light feature data includes thermal radiation pixel value, infrared light texture feature data, infrared light edge feature data and infrared light contour feature data; The infrared light texture feature data includes boundary thermal contrast and boundary thermal correlation data; the infrared light edge feature data includes edge thermal pixel gradient value, edge thermal pixel gradient direction change frequency and edge thermal pixel gradient direction change angle; the infrared light contour feature data includes thermal radiation area centroid coordinate data and thermal radiation area area; The visible light feature data includes visible light texture feature data, visible light edge feature data and visible light contour feature data, and the visible light texture feature data includes boundary color contrast and boundary color correlation data; The visible light edge feature data includes edge pixel gradient value, edge pixel gradient direction change frequency and edge pixel gradient direction change angle, and the visible light contour feature data includes contour centroid coordinate data and contour area; A spatial distance calculation is performed based on the heat radiation area centroid coordinate data and the contour centroid coordinate data to obtain spatial distance data.

[0044] It should be noted that the calculation of spatial distance can accurately quantify the position difference of objects in visible light and infrared light images. For the same target, such as flame, the thermal radiation mainly comes from the inside of the object, and its centroid position should be close to the centroid position of the object contour. If the spatial distance is too large, it means that the thermal radiation area and the contour may not correspond to the same object, and spatial registration and feature fusion cannot be performed. Therefore, the spatial distance calculation can be performed based on the centroid coordinate data of the thermal radiation area and the centroid coordinate data of the contour to obtain the spatial distance data; The calculation formula of the spatial distance data is: ; in, is the spatial distance data, is the contour centroid coordinate data, It is the centroid coordinate data of the thermal radiation area.

[0045] According to an embodiment of the present invention, the processing of obtaining a feature fusion evaluation index according to the spatial distance data and the infrared light feature data and the visible light feature data, and spatially registering the infrared light feature data and the visible light feature data according to the spatial transformation model parameters includes: Processing is performed according to the spatial distance data and the contour area and the thermal radiation area to obtain a feature fusion evaluation index; The feature fusion evaluation index is compared with a preset feature fusion evaluation index threshold. If the threshold comparison result meets the preset threshold comparison result requirement, the infrared light feature data and the visible light feature data are spatially aligned according to the spatial transformation model parameters.

[0046] It should be noted that, under normal circumstances, if the same target (such as a flame) is detected, they should have a good correspondence. For example, the area surrounded by the outline of the flame under visible light should be roughly consistent with the area of ​​the thermal radiation area under infrared light. Therefore, the spatial distance data and the outline area and thermal radiation area are processed to obtain the feature fusion evaluation index. If the feature fusion evaluation index is less than or equal to the preset threshold, it means that the same target is detected, and spatial registration and feature fusion can be performed.

[0047] The calculation formula of the feature fusion evaluation index is: ; in, is the feature fusion evaluation index, is the spatial distance data, is the contour area, is the area of ​​heat radiation, is the preset characteristic coefficient (which can be obtained by querying the preset multispectral image database).

[0048] According to an embodiment of the present invention, the step of performing feature fusion on the registered infrared light feature data and the visible light feature data to obtain a feature fusion vector, and obtaining a texture feature anomaly assessment index according to the feature fusion vector processing includes: The registered thermal radiation feature data, infrared light texture feature data, infrared light edge feature data, infrared light contour feature data and the visible light texture feature data, visible light edge feature data and visible light contour feature data are combined to generate a feature fusion vector; The texture feature anomaly assessment index is obtained according to the boundary thermal contrast, boundary thermal correlation data and the boundary color contrast, boundary color correlation data.

[0049] It should be noted that the boundary contrast of the flame is very high under both infrared light and visible light. Because the temperature and color differences between the flame and the surrounding environment are large, the pixel values ​​at the boundary change significantly. And because the flame shape is irregular and changes dynamically, the correlation of the flame boundary is low. Among them, the contrast and correlation data can be obtained based on the gray level co-occurrence matrix (GLCM). Therefore, by comprehensively considering the boundary thermal contrast, boundary thermal correlation, boundary color contrast, and boundary color correlation data, an assessment of the abnormality of texture features can be obtained, thereby achieving the purpose of assessing the risk of fire. The calculation formula of the texture feature abnormality evaluation index is: ; in, is the texture feature abnormality assessment index, is the boundary thermal correlation data, is the boundary color correlation data, is the boundary thermal contrast, is the boundary color contrast, , , and is the preset characteristic coefficient (which can be obtained by querying the preset multispectral image database).

[0050] According to an embodiment of the present invention, the step of obtaining an edge feature anomaly assessment index according to feature fusion vector processing includes: The edge feature anomaly assessment index is obtained according to the edge thermal pixel gradient value, the edge thermal pixel gradient direction change frequency, the edge thermal pixel gradient direction change angle and the edge pixel gradient value, the edge pixel gradient direction change frequency, and the edge pixel gradient direction change angle.

[0051] It should be noted that at the edge of the flame, the thermal pixels (infrared imaging) or color pixels (visible light imaging) change dramatically, and the pixel values ​​differ significantly from the high temperature area inside the flame to the low temperature area outside. At the edge of the flame, the thermal pixel gradient value and the pixel gradient value will increase significantly, and the flame will drift with the wind, and the frequency and angle of change of the edge thermal pixel gradient direction and the frequency and angle of change of the edge pixel gradient direction will increase accordingly. Therefore, the edge feature anomaly assessment index is obtained based on the edge thermal pixel gradient value, the frequency of change of the edge thermal pixel gradient direction, the angle of change of the edge thermal pixel gradient direction, the edge pixel gradient value, the frequency of change of the edge pixel gradient direction, and the angle of change of the edge pixel gradient direction. Among them, the pixel gradient value is used to measure the rate of change of the thermal radiation intensity (thermal signal corresponding to temperature) of the pixel in spatial position, the edge thermal pixel gradient direction change frequency refers to the frequency of thermal pixel gradient direction changes on the edge of the hot area, the edge thermal pixel gradient direction change angle refers to the angle change of the thermal pixel gradient direction from one position to another on the edge, the edge pixel gradient value is used to measure the rate of change of the pixel grayscale (or color intensity, etc.) in the image in spatial position, the edge pixel gradient direction change frequency refers to the frequency of pixel gradient direction changes on the edge of the object in the image, and the edge pixel gradient direction change angle refers to the angle change of the pixel gradient direction from one position to another on the edge.

[0052] The calculation formula of the edge feature abnormality evaluation index is: ; in, is the edge feature abnormality assessment index, , and They are the edge hot pixel gradient value, the edge hot pixel gradient direction change frequency, and the edge hot pixel gradient direction change angle. , and are the edge pixel gradient value, the edge pixel gradient direction change frequency, and the edge pixel gradient direction change angle, respectively. , , , , and is the preset characteristic coefficient (which can be obtained by querying the preset multispectral image database).

[0053] According to an embodiment of the present invention, the processing of obtaining a fire risk assessment index according to the texture feature anomaly assessment index and the edge feature anomaly assessment index, and marking and displaying the fire risk level on the video end, includes: Obtaining a fire risk assessment index according to the thermal radiation pixel value, the texture feature abnormality assessment index and the edge feature abnormality assessment index; The fire risk assessment index is compared with a preset fire risk assessment index threshold, the fire risk level is determined according to the risk level to which the threshold comparison result belongs, and the fire risk level is marked and displayed on the video end.

[0054] It should be noted that when a fire occurs, the thermal radiation pixel value increases significantly, and the fire risk assessment index is obtained by processing the thermal radiation pixel value, the texture feature abnormality assessment index and the edge feature abnormality assessment index; The calculation formula of the fire risk assessment index is: ; in, is the fire risk assessment index, is the thermal radiation pixel value, is the texture feature abnormality assessment index, is the edge feature abnormality assessment index, and is the preset weight coefficient (which can be obtained by querying the preset multispectral image database).

[0055] The present invention discloses a multi-spectral fusion display method and system for smart media high-definition video, which is a multi-spectral fusion display technology for smart media high-definition video designed for fire risk scenarios. The technology fuses multi-spectral data containing both infrared light and visible light, and performs feature anomaly assessment on the feature-fused data to obtain a fire risk assessment level, and then annotates and displays the fire risk assessment level on the video end.

[0056] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0057] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0058] In addition, all functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may be separately used as a unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

Claims

1. A multi-spectral fusion display method for smart media high-definition video, characterized in that: The following steps are involved: Collect infrared light video data and visible light video data and perform feature point recognition to obtain infrared light feature point coordinate data and visible light feature point coordinate data, and process them to obtain space transformation model parameters; Extract features from the infrared light video data and the visible light video data respectively to obtain infrared light feature data and visible light feature data, and process them to obtain spatial distance data; Obtaining a feature fusion evaluation index according to the spatial distance data and the infrared light feature data and the visible light feature data, and performing spatial registration on the infrared light feature data and the visible light feature data according to the spatial transformation model parameters; Perform feature fusion on the registered infrared light feature data and visible light feature data to obtain a feature fusion vector, and obtain a texture feature anomaly assessment index based on the feature fusion vector processing; Obtain edge feature abnormality assessment index based on feature fusion vector processing; The fire risk assessment index is obtained according to the texture feature anomaly assessment index and the edge feature anomaly assessment index, and the fire risk level is marked and displayed on the video end.

2. The multi-spectral fusion display method of smart media high-definition video according to claim 1 is characterized in that: The collecting of infrared light video data and visible light video data and performing feature point recognition to obtain infrared light feature point coordinate data and visible light feature point coordinate data, and processing to obtain space transformation model parameters, includes: Collect infrared light video data and visible light video data, perform feature point recognition on the infrared light video data and the visible light video data, and obtain infrared light feature point coordinate data and visible light feature point coordinate data; The infrared light feature point coordinate data includes infrared light corner point coordinate data, infrared light contour point coordinate data, infrared light texture intersection point coordinate data, infrared light local extreme point coordinate data and infrared light texture mutation point coordinate data; The visible light feature point coordinate data includes visible light corner point coordinate data, visible light contour point coordinate data, visible light texture intersection point coordinate data, visible light local extreme point coordinate data and visible light texture mutation point coordinate data; The infrared light feature point coordinate data and the visible light feature point coordinate data are input into a preset space transformation model for processing to obtain space transformation model parameters.

3. The multi-spectral fusion display method of smart media high-definition video according to claim 2 is characterized in that: The extracting features of the infrared light video data and the visible light video data respectively to obtain infrared light feature data and visible light feature data, and processing to obtain spatial distance data, includes: Extracting features from the infrared light video data and the visible light video data respectively to obtain infrared light feature data and visible light feature data; The infrared light feature data includes thermal radiation pixel value, infrared light texture feature data, infrared light edge feature data and infrared light contour feature data; The infrared light texture feature data includes boundary thermal contrast and boundary thermal correlation data; the infrared light edge feature data includes edge thermal pixel gradient value, edge thermal pixel gradient direction change frequency and edge thermal pixel gradient direction change angle; the infrared light contour feature data includes thermal radiation area centroid coordinate data and thermal radiation area area; The visible light feature data includes visible light texture feature data, visible light edge feature data and visible light contour feature data, and the visible light texture feature data includes boundary color contrast and boundary color correlation data; The visible light edge feature data includes edge pixel gradient value, edge pixel gradient direction change frequency and edge pixel gradient direction change angle, and the visible light contour feature data includes contour centroid coordinate data and contour area; A spatial distance calculation is performed based on the heat radiation area centroid coordinate data and the contour centroid coordinate data to obtain spatial distance data.

4. The multi-spectral fusion display method of smart media high-definition video according to claim 3 is characterized in that: The step of obtaining a feature fusion evaluation index according to the spatial distance data and the infrared light feature data and the visible light feature data, and performing spatial registration on the infrared light feature data and the visible light feature data according to the spatial transformation model parameters includes: Processing is performed according to the spatial distance data and the contour area and the thermal radiation area to obtain a feature fusion evaluation index; The feature fusion evaluation index is compared with a preset feature fusion evaluation index threshold. If the threshold comparison result meets the preset threshold comparison result requirement, the infrared light feature data and the visible light feature data are spatially aligned according to the spatial transformation model parameters.

5. The multi-spectral fusion display method of smart media high-definition video according to claim 4 is characterized in that: The step of performing feature fusion on the registered infrared light feature data and the visible light feature data to obtain a feature fusion vector, and obtaining a texture feature anomaly assessment index according to the feature fusion vector processing includes: The registered thermal radiation feature data, infrared light texture feature data, infrared light edge feature data, infrared light contour feature data and the visible light texture feature data, visible light edge feature data and visible light contour feature data are combined to generate a feature fusion vector; The texture feature anomaly assessment index is obtained according to the boundary thermal contrast, boundary thermal correlation data and the boundary color contrast, boundary color correlation data.

6. The multi-spectral fusion display method of smart media high-definition video according to claim 5 is characterized in that: The step of obtaining the edge feature abnormality assessment index according to the feature fusion vector processing includes: The edge feature anomaly assessment index is obtained according to the edge thermal pixel gradient value, the edge thermal pixel gradient direction change frequency, the edge thermal pixel gradient direction change angle and the edge pixel gradient value, the edge pixel gradient direction change frequency, and the edge pixel gradient direction change angle.

7. The multi-spectral fusion display method of smart media high-definition video according to claim 6 is characterized in that: The obtaining of the fire risk assessment index according to the texture feature anomaly assessment index and the edge feature anomaly assessment index, and marking and displaying the fire risk level on the video end, includes: Obtaining a fire risk assessment index according to the thermal radiation pixel value, the texture feature abnormality assessment index and the edge feature abnormality assessment index; The fire risk assessment index is compared with a preset fire risk assessment index threshold, the fire risk level is determined according to the risk level to which the threshold comparison result belongs, and the fire risk level is marked and displayed on the video end.

8. A multi-spectral fusion display system for smart media high-definition video, characterized in that: The invention comprises a memory and a processor, wherein the memory stores a program of a method for multi-spectral fusion display of a smart media high-definition video, and when the program of the method for multi-spectral fusion display of a smart media high-definition video is executed by the processor, the following steps are implemented: Collect infrared light video data and visible light video data and perform feature point recognition to obtain infrared light feature point coordinate data and visible light feature point coordinate data, and process them to obtain space transformation model parameters; Extract features from the infrared light video data and the visible light video data respectively to obtain infrared light feature data and visible light feature data, and process them to obtain spatial distance data; Obtaining a feature fusion evaluation index according to the spatial distance data and the infrared light feature data and the visible light feature data, and performing spatial registration on the infrared light feature data and the visible light feature data according to the spatial transformation model parameters; Perform feature fusion on the registered infrared light feature data and visible light feature data to obtain a feature fusion vector, and obtain a texture feature anomaly assessment index based on the feature fusion vector processing; Obtain edge feature abnormality assessment index based on feature fusion vector processing; The fire risk assessment index is obtained according to the texture feature anomaly assessment index and the edge feature anomaly assessment index, and the fire risk level is marked and displayed on the video end.

9. The multi-spectral fusion display system of smart media high-definition video according to claim 8, characterized in that: The collecting of infrared light video data and visible light video data and performing feature point recognition to obtain infrared light feature point coordinate data and visible light feature point coordinate data, and processing to obtain space transformation model parameters, includes: Collect infrared light video data and visible light video data, perform feature point recognition on the infrared light video data and the visible light video data, and obtain infrared light feature point coordinate data and visible light feature point coordinate data; The infrared light feature point coordinate data includes infrared light corner point coordinate data, infrared light contour point coordinate data, infrared light texture intersection point coordinate data, infrared light local extreme point coordinate data and infrared light texture mutation point coordinate data; The visible light feature point coordinate data includes visible light corner point coordinate data, visible light contour point coordinate data, visible light texture intersection point coordinate data, visible light local extreme point coordinate data and visible light texture mutation point coordinate data; The infrared light feature point coordinate data and the visible light feature point coordinate data are input into a preset space transformation model for processing to obtain space transformation model parameters.

Images