Feature extraction method, device and storage medium based on thermal imaging video

Abstract

The application discloses a feature extraction method and device based on a thermal imaging video and a storage medium. The feature extraction method based on the thermal imaging video comprises the following steps: extracting a temperature feature of a thermal imaging image in the thermal imaging video; determining an image feature of the thermal imaging image based on the temperature feature; fusing the temperature feature and the image feature to obtain a comprehensive feature of the thermal imaging image; and performing time sequence correlation on the comprehensive features of the thermal imaging images according to time sequence features of the thermal imaging video to obtain a video feature of the thermal imaging video. The above scheme can improve the feature extraction efficiency of the thermal imaging video.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a feature extraction method based on thermal imaging video, a device and a storage medium. BACKGROUND

[0002] Thermal imaging technology is a technology for detecting the surface temperature of an object by infrared rays, which can convert the temperature information of the object into an image for display. In the application scenario of thermal imaging technology, thermal imaging feature extraction is a very key step, which can provide strong support for subsequent analysis and processing.

[0003] However, there are great differences between thermal imaging and optical imaging in imaging principles and feature representations, resulting in that the performance of traditional feature extraction methods on thermal imaging videos is not ideal. Therefore, there is an urgent need for a method capable of extracting features from thermal imaging to improve the efficiency of target recognition based on thermal imaging. SUMMARY

[0004] The present application provides at least a feature extraction method based on thermal imaging video, a device, an apparatus and a computer readable storage medium.

[0005] The first aspect of the present application provides a feature extraction method based on thermal imaging video, comprising: extracting temperature features of thermal imaging images in the thermal imaging video; determining image features of the thermal imaging images based on the temperature features; fusing the temperature features and the image features to obtain comprehensive features of the thermal imaging images; and time-series correlating the comprehensive features of each thermal imaging image according to time-series features of the thermal imaging video to obtain video features of the thermal imaging video.

[0006] In an embodiment, the image features include spatial features and semantic features, and the step of determining the image features of the thermal imaging images based on the temperature features comprises: determining spatial information in the thermal imaging images according to the temperature features; performing feature extraction on the spatial information to obtain the spatial features; and performing semantic feature extraction on the thermal imaging images to obtain the semantic features.

[0007] In an embodiment, the step of determining the spatial information in the thermal imaging images according to the temperature features and performing feature extraction on the spatial information to obtain the spatial features comprises: detecting the spatial information in the thermal imaging images according to the temperature features, wherein the spatial information includes edges and corner points; and performing feature extraction based on the edges and corner points of the thermal imaging images respectively to obtain the spatial features, wherein the spatial features include edge features and corner point features.

[0008] In an embodiment, after the step of detecting the spatial information in the thermal imaging image according to the temperature feature, the method further comprises: performing binary labeling on pixels in the thermal imaging image based on the edge pixels corresponding to the edges and the corner pixels corresponding to the corner points; performing background elimination on the binary labeled thermal imaging image to obtain a foreground of the binary labeled thermal imaging image, the foreground including a target object; and performing semantic feature extraction on the target object to obtain the semantic feature.

[0009] In an embodiment, the step of extracting the temperature feature of the thermal imaging image in the thermal imaging video comprises: performing temperature identification on the thermal imaging image to obtain a temperature value corresponding to each pixel point in the thermal imaging image; and performing temperature coding based on the temperature value to obtain a temperature feature vector of the temperature feature of the thermal imaging image.

[0010] In an embodiment, the step of determining the image feature of the thermal imaging image based on the temperature feature, and fusing the temperature feature and the image feature to obtain a comprehensive feature of the thermal imaging image comprises: extracting an image feature vector of the image feature of the thermal imaging image based on the temperature feature vector of the temperature feature; performing fusion calculation on the temperature feature vector and the image feature vector to obtain a feature fusion result; and fitting the feature fusion result to obtain the comprehensive feature.

[0011] In an embodiment, the step of associating the comprehensive features of the thermal imaging images in time sequence according to the time sequence feature of the thermal imaging video to obtain a video feature of the thermal imaging video comprises: determining a time sequence of each frame of thermal imaging image in the thermal imaging video according to the time sequence feature of the thermal imaging video; and associating the comprehensive features of the thermal imaging images based on the time sequence to obtain the video feature of the thermal imaging video.

[0012] In an embodiment, the step of associating the comprehensive features of the thermal imaging images based on the time sequence to obtain the video feature of the thermal imaging video comprises: extracting a plurality of frames of thermal imaging images in the thermal imaging video; and analyzing the comprehensive features corresponding to the plurality of frames of thermal imaging images based on the time sequence of the plurality of frames of thermal imaging images to obtain the video feature of the thermal imaging video.

[0013] The second aspect of the present application provides a feature extraction device based on thermal imaging video, comprising: a temperature feature extraction module, configured to extract temperature features of thermal imaging images in the thermal imaging video; an image feature determination module, configured to determine image features of the thermal imaging images based on the temperature features; a fusion module, configured to fuse the temperature features and the image features to obtain comprehensive features of the thermal imaging images; and a time sequence correlation module, configured to time sequence correlate the comprehensive features of the thermal imaging images according to time sequence features of the thermal imaging video to obtain video features of the thermal imaging video.

[0014] The third aspect of the present application provides an electronic device, comprising a memory and a processor, wherein the processor is configured to execute program instructions stored in the memory to implement the feature extraction method based on thermal imaging video.

[0015] The fourth aspect of the present application provides a computer readable storage medium, which stores program instructions, and the program instructions are executed by a processor to implement the feature extraction method based on thermal imaging video.

[0016] The above scheme extracts temperature features of thermal imaging images in the thermal imaging video to determine image features of the thermal imaging images, fuses the temperature features and the image features to obtain comprehensive features of the thermal imaging images, and time sequence correlates the comprehensive features of the thermal imaging images based on time sequence features of the thermal imaging video, thereby obtaining video features of the thermal imaging video, and improving the feature extraction efficiency of the thermal imaging video.

[0017] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, but not limiting the present application. BRIEF DESCRIPTION OF DRAWINGS

[0018] The accompanying drawings are incorporated into and form part of the specification, which illustrate embodiments consistent with the present application, and together with the specification, serve to explain the technical solutions of the present application.

[0019] Figure 1 is a flowchart of an exemplary embodiment of the feature extraction method based on thermal imaging video of the present application;

[0020] Figure 2 is a flowchart of determining image features of thermal imaging images based on temperature features in the present application;

[0021] Figure 3 is an effect diagram of feature extraction of thermal imaging images in the present application;

[0022] Figure 4 is a block diagram of the feature extraction device based on thermal imaging video shown by an exemplary embodiment of the present application;

[0023] Figure 5 is a structural schematic diagram of an embodiment of an electronic device of the present application.

[0024] Figure 6 is a structural schematic diagram of an embodiment of a computer readable storage medium of the present application. DETAILED DESCRIPTION

[0025] The scheme of the embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0026] In the following description, specific details are set forth in order to provide a thorough understanding of the present application. However, persons having ordinary skill in the art will appreciate that embodiments of the present application can be practiced without incorporating these specific details.

[0027] The term "and / or" herein is merely an association relationship between associated objects, which means that there can be three relationships, for example, A and / or B can mean that A exists alone, A and B exist together, and B exists alone. In addition, the character " / " herein generally represents an "or" relationship between the front and rear associated objects. In addition, "multiple" herein means two or more than two. In addition, the term "at least one" herein means any one of multiple or any combination of at least two of multiple, for example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0028] Please refer to Figure 1 , Figure 1 is a flowchart of an exemplary embodiment of the feature extraction method based on thermal imaging video of the present application. Specifically, it can include the following steps:

[0029] Step S110, extracting temperature features of thermal imaging images in the thermal imaging video.

[0030] The temperature features include temperature values, which can be represented in the form of feature maps or feature vectors. It can be understood that any object in nature above absolute zero (-273℃) will continuously radiate infrared rays. Thermal imaging converts the infrared radiation of the object surface into a visible image through an optical system, an infrared detector chip, and an electronic processing system. In simple terms, the principle of thermal imaging is to use temperature imaging to convert the invisible infrared energy emitted by the object into a visible thermal imaging image. The pixels in the thermal imaging image can represent the temperature value of the target object being measured.

[0031] Exemplarily, the temperature features of each thermal imaging image in the thermal imaging video are extracted respectively, that is, the temperature features of each thermal imaging image are usually different, each thermal imaging image has its corresponding temperature feature, and the process of extracting the temperature features of the thermal imaging image can be extracting the temperature features of all pixels in each thermal imaging image, or can be image processing on each thermal imaging image to extract the temperature features of the thermal imaging image after image processing; further, the image processing methods include but are not limited to scaling and normalization operations on the thermal imaging image to eliminate the scale difference in the image data and make the extracted features more stable; it can also be to reduce noise of the thermal imaging image by using bilateral filtering, non-local mean filtering and the like, thereby improving the accuracy of feature extraction.

[0032] In step S120, the image features of the thermal imaging image are determined based on the temperature features.

[0033] The image features refer to the visible features represented by the thermal imaging image, and the target object in the thermal imaging image can be analyzed through the image features, for example, the size, shape and type of the target object can be determined through the image features of the thermal imaging image.

[0034] It should be noted that the temperatures of the objects in the thermal imaging image are usually different, so the boundaries or edges between the objects can be determined through the temperature features of the thermal imaging image representing the temperature values of the objects.

[0035] In step S130, the temperature features and the image features are fused to obtain the comprehensive features of the thermal imaging image.

[0036] The comprehensive features represent the features after integrating the multi-scale features of the thermal imaging image.

[0037] Exemplarily, the method of fusing the temperature features and the image features includes but is not limited to feature splicing of the temperature features and the image features, or weighted summation based on the temperature features and the image features, and the feature fusion of the temperature features and the image features of the thermal imaging image can effectively integrate the features of different scales and forms in the thermal imaging video, thereby improving the analysis efficiency of the thermal imaging video and the model performance of extracting the features of the thermal imaging video.

[0038] In step S140, the comprehensive features of each thermal imaging image are time-sequentially associated according to the time sequence features of the thermal imaging video to obtain the video features of the thermal imaging video.

[0039] The time sequence features represent the time sequence of each thermal imaging image in the thermal imaging video, and each thermal imaging image has a corresponding time point on the time axis of the thermal imaging video, that is, each thermal imaging image is arranged in a certain time sequence or frame sequence to form the thermal imaging video.

[0040] The video feature represents the feature information embodied in the thermal imaging video. By processing the video feature using deep learning technology, the content or object in the thermal imaging video can be analyzed.

[0041] It can be understood that the thermal imaging image changes along the time axis of the thermal imaging video, which is equivalent to the video feature of the thermal imaging video also changing along the time axis. Therefore, by correlating the comprehensive features of each thermal imaging image in time sequence, the change process of the video feature of the thermal imaging video can be obtained, so as to realize dynamic analysis of the thermal imaging video, thereby improving the accuracy and robustness of the thermal imaging video feature extraction model in the detection and recognition process.

[0042] It can be seen that the present application determines the image feature of the thermal imaging image by the temperature feature extracted from the thermal imaging image in the thermal imaging video, fuses the temperature feature and the image feature, obtains the comprehensive feature of the thermal imaging image, and time-correlates the comprehensive features of each thermal imaging image based on the time sequence feature of the thermal imaging video. Thus, the video feature of the thermal imaging video can be obtained, and the efficiency of feature extraction of the thermal imaging video is improved.

[0043] Based on the above embodiments, the embodiments of the present application adopt Figure 2 The flowchart shown in the above embodiment details the steps of determining the image feature of the thermal imaging image based on the temperature feature. The image feature includes spatial feature and semantic feature. Specifically, the method of the present embodiment includes the following steps:

[0044] In step S210, the spatial information in the thermal imaging image is determined according to the temperature feature.

[0045] The spatial information refers to the information represented by the thermal imaging image in the spatial dimension. For example, through the spatial information, it can be known that the target object occupies a space or region in the thermal imaging image, which is equivalent to determining the edge, corner point and other spatial information of the target object in the thermal imaging image through the temperature feature.

[0046] As can be known from the foregoing embodiments, the thermal imaging image reflects the content in the image through temperature. It can be understood that, since the temperature values of different objects are different, they are represented as different light and dark regions or different pixel values in the thermal imaging image. Therefore, the spatial information such as edge information and corner point information in the thermal imaging image can be determined according to the temperature feature.

[0047] In step S220, the spatial information is subjected to feature extraction to obtain the spatial feature.

[0048] The spatial feature refers to the feature obtained according to the spatial information. For example, edge detection and feature extraction are performed on the thermal imaging image to obtain the edge feature of the edge; corner point detection and feature extraction are performed on the thermal imaging image to obtain the corner point feature of the corner point.

[0049] Exemplarily, reference can be made to Figure 3 , Figure 3 For the effect of feature extraction of the thermal imaging image in the present application, the thermal imaging image can be input into a pre-trained feature extraction neural network to obtain edge features and corner features output by the feature extraction neural network. It can be seen that through the extraction of spatial features such as edge features and corner features, more rich image features are obtained, which provides more rich information for the subsequent processing process of the thermal imaging image.

[0050] In step S230, semantic features of the thermal imaging image are extracted to obtain semantic features.

[0051] The semantic features refer to a series of high-level features such as categories, textures or colors represented by each object in the thermal imaging image. Through analysis of the semantic features, the attribute information of each object in the thermal imaging image can be recognized. The semantic features are common parameter variables in deep learning technology, which will not be described here.

[0052] Specifically, the method of extracting semantic features of the thermal imaging image can be feature extraction from the thermal imaging image through convolution layers such as standard convolution layers, deformable convolution layers and adaptive local mode convolution layers. It should be noted that in the process of extracting semantic features of the thermal imaging image, the semantic features of the thermal imaging image can also be extracted in combination with the thermal imaging image and the spatial features of the thermal imaging image.

[0053] On the basis of the above embodiments, the spatial information in the thermal imaging image can be determined according to the temperature features, and the spatial features can be obtained by feature extraction of the spatial information.

[0054] The specific steps include:

[0055] The spatial information in the thermal imaging image is detected according to the temperature features, and the spatial information includes edges and corners. The spatial features are obtained by feature extraction based on the edges and corners of the thermal imaging image, and the spatial features include edge features and corner features.

[0056] Exemplarily, taking edges as an example, the regions with large light-dark changes in the thermal imaging image are identified through the temperature features of the thermal imaging image, which is equivalent to identifying the edges or boundaries of the objects in the thermal imaging image. Specifically, the edges are edge-detected and feature-extracted based on the temperature features, and the process can include gradient calculation based on the temperature feature vectors of the thermal imaging image, such as using Sobel, Prewitt, etc. gradient operator to calculate the horizontal and vertical gradient components of the image, and combining these components into gradient magnitude and direction. Along the gradient direction, the neighborhood of each pixel is checked. If the gradient magnitude of the current pixel is greater than that of other pixels in its neighborhood, the pixel is retained. Otherwise, the pixel value of the current pixel is set to zero, thereby achieving the effect of non-maximum suppression. In order to determine the final edge pixels, double-threshold processing is also needed for the image after non-maximum suppression. Specifically, two preset thresholds (high threshold and low threshold) are selected, and the pixels in the image after non-maximum suppression are divided into three categories: strong edge pixels (greater than the high threshold), weak edge pixels (less than or equal to the high threshold and greater than or equal to the low threshold), and non-edge pixels (less than the low threshold). Among them, strong edge pixels can be determined as clear edge points, while weak edge pixels need to be further identified and detected. By detecting the neighborhood of the weak edge pixel, if there is a strong edge pixel in its neighborhood, the weak edge pixel is retained as an edge pixel. Otherwise, it is set as a non-edge pixel. Thus, the edge features in the thermal imaging image are determined through the temperature features of the thermal imaging image, and these edge features are equivalent to the edge features of the objects in the thermal imaging image, and the edge features of the target object can be further extracted therefrom, such as selecting the edge features corresponding to the temperature features matching the human body temperature.

[0057] Taking a corner point as an example, the feature extraction of the corner point and the above edge feature extraction process are similar, and gradient calculation is also needed. Then, a second-order autocorrelation matrix corresponding to each pixel is calculated for each pixel gradient component, which is used to describe the local structure information of the thermal imaging image near the pixel point. Then, the corner response value of each pixel is calculated based on the second-order autocorrelation matrix corresponding to each pixel, which represents the corner characteristic intensity near each pixel point in the thermal imaging image, and then it is judged whether the point is likely to be a corner point. The corner response calculation method can adopt Harris response and Shi-Tomasi response. Non-maximum suppression is performed based on the corner response value of each pixel. Threshold screening is performed on the pixel points after non-maximum suppression to determine the corner points. The threshold screening can be to compare the corner response value of the pixel points after non-maximum suppression with a preset corner threshold, and the pixel points after non-maximum suppression with a corner response value greater than the corner threshold are obtained as corner points.

[0058] Further, after the edge features and the corner features in the thermal imaging image are detected, an edge image and a corner image can be respectively output. The two types of images can be independently used in subsequent processing steps, or they can be combined to form a richer feature representation. For example, the edge image and the corner image can be superimposed to obtain a binary image containing edge and corner information. The binary image can be used as input data for the background elimination layer to eliminate the background and further extract information of the target object.

[0059] Based on the above embodiments, the step of detecting spatial information in the thermal imaging image according to the temperature features further includes: based on the edge pixels corresponding to the edges and the corner pixels corresponding to the corners, binarizing and marking the pixels in the thermal imaging image; performing background elimination on the binarized and marked thermal imaging image to obtain a foreground of the binarized and marked thermal imaging image, and the foreground includes the target object; and performing semantic feature extraction on the target object to obtain semantic features.

[0060] Specifically, after the edges and the corners in the thermal imaging image are detected, the edge pixels and the corner pixels in the thermal imaging image are marked as 1, and the non-edge pixels and the non-corner pixels are marked as 0 to obtain a binarized and marked thermal imaging image; the binarized and marked thermal imaging image is input into a preset background elimination layer to perform background elimination to obtain a target object in a foreground of the thermal imaging image; and semantic features of the target object are extracted.

[0061] Further, in the process of inputting the binarized and marked thermal imaging image into the preset background elimination layer to perform background elimination, a mixture Gaussian model is initialized for each pixel point of the binarized and marked thermal imaging image, and the model is composed of multiple Gaussian distributions, and each Gaussian distribution represents a possible pixel value of the pixel point. The initial parameters (such as mean, variance, and weight) of the Gaussian distribution can be randomly set or preset according to the information of the input data. As the time progresses along the time axis of the thermal imaging video, a new frame of thermal imaging image is obtained in real time, and the mixture Gaussian model at the position of each pixel point is updated according to the new frame of thermal imaging image. The update process can be that for each new pixel value of the pixel point, the closest Gaussian distribution is found, and the parameters of the distribution are updated to gradually adapt to the new pixel value. At the same time, the weights of the Gaussian distributions are adjusted according to the dynamics of the input data. The difference between the pixel value of the pixel point and each Gaussian distribution is calculated, and then the pixel corresponding to the Gaussian distribution with the smallest difference and smaller than a preset threshold is selected to be determined as a background pixel. Otherwise, the pixel is regarded as a foreground pixel, thereby realizing the distinction between the background and the foreground of each pixel point based on the mixture Gaussian model. The pixel value of the pixel point belonging to the background is set to 0 to obtain a binary image containing only the foreground, i.e., a thermal imaging image after background elimination, which is equivalent to obtaining a target object in a foreground of the thermal imaging image.

[0062] Further, in order to reduce the complexity of the data processing process, a pooling layer can also be added to the neural network to pool the thermal image after background removal, realizing the scale invariance of the image. For example, a spatial pyramid pooling layer is used. Specifically, the spatial pyramid pooling layer receives the binary image output from the background removal layer. It can be understood that the background pixels of the binary image have been largely removed at this time. The spatial pyramid pooling layer extracts the feature representation of the image at multiple scales, so that the target object can be better focused in the subsequent processing stage. The level of the spatial pyramid pooling layer determines the granularity of the pooling operation, which affects the feature extraction effect. When selecting the appropriate number of pyramid levels, the computational complexity and feature representation ability need to be considered comprehensively. Generally, the pyramid level can be increased gradually from the first pyramid, and the optimal target level can be obtained through continuous training and experiments of the neural network. For example, a common choice is to use a 3-level pyramid, corresponding to 1x1, 2x2 and 4x4 regions respectively. When selecting the pyramid level, the experimental results and experience guidance in existing literature can also be referred to, which will not be described here. The spatial pyramid pooling layer divides the received image into regions according to the pyramid level. The region division needs to ensure that the number of pixels in each region is as evenly distributed as possible to avoid introducing additional bias in the pooling operation. For each level, the feature map can be divided into equal sub-regions, and the number of sub-regions is equal to the square of the level. For example, for a 2x2 level, the feature map is divided into 4 equal sub-regions. In practical applications, various region division strategies can be used, such as uniform division, division based on image segmentation, etc., to adapt to different application scenarios and feature requirements. Pooling operations are performed in each sub-region to extract features. Specifically, the pooling operation can be max pooling or average pooling. Max pooling extracts the maximum value in the region, which has stronger robustness and anti-noise ability. Average pooling calculates the average value in the region, which can preserve more context information. When selecting the pooling operation, the characteristics and needs of the actual task can be considered. For example, for target detection tasks, max pooling may be more suitable for capturing the salient features of the target. For scene classification tasks, average pooling may be more capable of reflecting the overall feature distribution of the image. It is worth noting that a hybrid pooling strategy can be used to combine the advantages of max pooling and average pooling to achieve better feature extraction effect. The feature vectors of different levels are concatenated to form a unified feature representation. During the merging process, the order and alignment of the feature vectors need to be considered to ensure that the spatial structure and semantic relationship of the features are preserved. For example, the feature vectors can be concatenated in the order of pyramid levels. For a 3-level pyramid, the feature vectors of the 1x1 level and the 2x2 level can be concatenated first, and then the result and the feature vectors of the 4x4 level can be concatenated. This merging strategy can preserve the spatial relationship between features of different levels and realize multi-scale fusion of features.Finally, the merged feature vector is output to the subsequent convolution layer, so that the subsequent convolution layer can extract higher-level feature representations, i.e., semantic features, through the feature vector.

[0063] It should be further pointed out that the convolution layer for extracting semantic features from the thermal imaging image can include, but is not limited to, one or more of a standard convolution layer, a deformable convolution layer, a non-local module convolution layer, and an adaptive local pattern convolution layer. Taking the non-local module convolution layer (ML-Block) as an example, the non-local module convolution layer is a module for capturing long-distance dependencies in deep learning, and is therefore suitable for video recognition tasks. It can be understood that similar image regions between images with similar time sequences in a video should have similar features, and the features of a pixel point can depend on similar pixel points. The non-local operation establishes this dependency relationship by weighted summation of all pixel points on the feature map. The implementation process for applying the non-local operation to the thermal imaging video includes converting the feature map of the thermal imaging image into a form that is easy to calculate similarity based on the target pixel point and the similar pixel point through embedding functions such as linear mapping, convolution, and full connection, to obtain at least two embedded feature maps; calculating the similarity between the embedded feature maps through methods such as dot product, L2 norm, and cosine similarity to obtain a similarity matrix; performing a normalization operation on the similarity matrix to make the result of the subsequent weighted summation have a reasonable range; performing weighted summation on the input feature map according to the similarity calculation result to obtain an output feature map; and the output feature map can also be passed to the adaptive local pattern convolution layer for further extraction of image features.

[0064] In combination with the foregoing, taking the adaptive local mode convolution layer as an example, after receiving the feature map provided by the previous neural network layer, the adaptive local mode convolution layer performs local convolution operation on the feature map to capture finer local structure, which can be achieved by using a smaller convolution kernel (such as 3x3 or 5x5) to effectively extract high-frequency details in the image; convert the extracted local structure information into dynamic convolution kernel parameters, which can be achieved by using a generation network, which can be a fully connected layer, a convolution layer or other learnable mapping functions, the generation network receives the local structure information as input and outputs the corresponding convolution kernel parameters; after having the dynamically generated convolution kernel parameters, apply them to the input feature map for convolution operation, this step is similar to the traditional convolution operation, but the convolution kernel parameters are dynamically generated according to the local structure of the input feature map, thereby realizing adaptive extraction of different features; after completing the convolution operation, the result is passed to the next layer for further processing, which can include activation function, batch normalization and pooling operation, etc. to improve the nonlinear expression ability and robustness of the model; it can be seen that the adaptive local mode convolution realizes adaptive extraction of different features in the input feature map by dynamically generating convolution kernel parameters, this method has stronger flexibility and adaptability in the feature extraction process, and can improve the performance of the model in processing complex image tasks.

[0065] On the basis of the above-mentioned embodiments, the step of extracting the temperature feature of the thermal imaging image in the thermal imaging video is exemplarily described. Specifically, it includes: performing temperature identification on the thermal imaging image to obtain the temperature value corresponding to each pixel point in the thermal imaging image; performing temperature coding based on the temperature value to obtain the temperature feature vector of the temperature feature of the thermal imaging image.

[0066] Specifically, a temperature perception layer can be set in the neural network, which extracts the temperature value corresponding to each pixel point in the image based on the received thermal imaging image; each temperature value is converted into a feature vector with only one element being 1 by using one-hot encoding, or the temperature value is divided into several intervals according to the temperature range by using temperature partition coding, each interval corresponding to a feature vector; the temperature feature in the thermal imaging data is learned by using a convolutional neural network (CNN) or other deep learning models; and then the temperature feature vector is output.

[0067] It should be further explained that before the thermal imaging image is input into the temperature perception layer, the thermal imaging image can also be pre-processed, and the video data is decoded into numerical matrices in memory, which usually have the same spatial resolution and channel number as the thermal imaging data; the numerical matrices are scaled and normalized, wherein the scaling operation can adjust the matrix to a specified size by an interpolation method (such as bilinear interpolation, bicubic interpolation, etc.), thereby eliminating the scale difference in the data, and the normalization operation adjusts the numerical range in the matrix to a standard interval (such as [0, 1] or [-1, 1]) to facilitate the processing of subsequent feature extraction algorithms; denoising methods such as bilateral filtering, non-local mean filtering, total variation denoising, etc. are used for processing, which removes noise while preserving image details, thereby improving the accuracy of feature extraction; the thermal imaging data can also be color space converted (such as from RGB space to Lab space), histogram equalization to improve image contrast, etc., so that the thermal imaging data is converted into a form suitable for feature extraction and classification tasks; finally, the pre-processed data is transmitted to the neural network structure of the temperature perception layer as a two-dimensional thermal imaging data tensor.

[0068] On the basis of the above-mentioned embodiments, the present embodiment exemplarily illustrates the steps of determining the image features of the thermal imaging image based on the temperature features, fusing the temperature features and the image features, and obtaining the comprehensive features of the thermal imaging image. Specifically, it includes: based on the temperature feature vector, extracting the image feature vector of the image features of the thermal imaging image; fusing and calculating the temperature feature vector and the image feature vector to obtain a feature fusion result; fitting the feature fusion result to obtain the comprehensive features.

[0069] As explained in the foregoing embodiments, based on the temperature feature vector, the image feature vector of the thermal imaging image is extracted, which is equivalent to assisting the neural network to determine the features of edges and corners based on the temperature information represented by the thermal imaging image. These features belong to the local features of the thermal imaging image, which can enhance the understanding of the details and structure of the thermal imaging image by the neural network.

[0070] It should be further explained that the feature fusion process includes flattening the feature map into a feature vector, or applying global average pooling or global maximum pooling operations to the feature map to generate a feature vector; then the feature vectors are concatenated or fused together by other means (such as weighted summation, etc.) to obtain the feature fusion result; the feature fusion result is input into the fully connected layer for further fitting processing to obtain the comprehensive features output by the fully connected layer.

[0071] On the basis of the above-mentioned embodiments, the step of obtaining the video feature of the thermal imaging video by associating the integrated features of each thermal imaging image with the time sequence of the thermal imaging video is exemplarily described. Specifically, it includes:

[0072] determining the time sequence of each frame of thermal imaging image in the thermal imaging video according to the time sequence feature of the thermal imaging video; and associating the integrated features of each frame of thermal imaging image based on the time sequence to obtain the video feature of the thermal imaging video.

[0073] It should be noted that in the process of extracting the feature of the thermal imaging video, in addition to considering the content of each frame of thermal imaging image, the time sequence relationship between frames also needs to be considered, therefore, usually after the temperature feature, spatial feature and semantic feature are extracted, a recurrent neural network (RNN) or other tools are used to extract the time sequence feature, which can help us understand the dynamic changes of the video, i.e. each frame of thermal imaging image changes with the time sequence feature of the thermal imaging video.

[0074] Exemplarily, the method of extracting the time sequence feature can be using recurrent neural networks (RNN) and its variants, such as long short-term memory (LSTM) or gated recurrent unit (GRU), etc. These neural networks can process sequence data, remember historical information, and consider these historical information when processing current input; but the traditional RNN has the problems of gradient disappearance and gradient explosion in practical application, which makes it difficult for the network to capture long-distance time sequence dependence relationship. In order to solve this problem, LSTM and GRU introduce a gating mechanism, so that the network can selectively remember or forget historical information, thereby more effectively capturing long-distance time sequence dependence relationship. Specifically, the LSTM network includes one or more LSTM units, each unit is composed of an input gate, a forget gate and an output gate, the input gate controls the inflow of new input information, the forget gate controls the forgetting of historical information, and the output gate controls the output of information. Through this gating mechanism, LSTM can dynamically adjust the transmission and forgetting of information when processing sequence data according to the actual situation of the data; it can be seen that by comprehensively analyzing the features of the thermal imaging video at different time scales and spatial scales, the whole thermal imaging video feature extraction model can capture various complex temperature distributions and dynamic changes in the thermal imaging video, thereby improving the accuracy and robustness of feature extraction.

[0075] Similarly, based on the above embodiments, the step of correlating the comprehensive features of each frame of thermal imaging image based on the time sequence to obtain the video feature of the thermal imaging video is exemplarily described. Specifically, it includes: extracting multiple frames of thermal imaging images in the thermal imaging video; analyzing the comprehensive features corresponding to the multiple frames of thermal imaging images based on the time sequence of the multiple frames of thermal imaging images to obtain the video feature of the thermal imaging video.

[0076] In combination with the foregoing embodiments, the time sequence information is particularly important when processing video data, because it can capture dynamic changes and time dependencies in image sequences, such as in action recognition, behavior analysis, object tracking, and other tasks. The features extracted from each frame of image alone often fail to capture complete dynamic information, at which time the time sequence of multiple frames of image is considered to enhance the recognition ability of the model by capturing time sequence information. The extraction of multiple frames of thermal imaging images in the thermal imaging video can be frame-by-frame acquisition of all frames of thermal imaging images in the thermal imaging video, or extraction of partial frames of thermal imaging images in the thermal imaging video, which is not limited herein. For thermal imaging images with a confidence level of the comprehensive feature lower than a preset confidence threshold, these thermal imaging images can be screened out to avoid deviation and interference when analyzing the video feature of the entire thermal imaging video.

[0077] Further, for the thermal imaging video, the dynamic temperature change of the object can be reflected based on the time sequence information and the multiple frames of thermal imaging images in the thermal imaging video processing process. For example, the video feature of the thermal imaging video obtained by the method of the present application can identify the activity or behavior of the target object in the thermal imaging video, as well as the heat distribution pattern and dynamic temperature change pattern of the object.

[0078] As can be seen, the present application determines the image feature of the thermal imaging image by the temperature feature extracted from the thermal imaging image in the thermal imaging video, fuses the temperature feature and the image feature to obtain the comprehensive feature of the thermal imaging image, and correlates the comprehensive features of each thermal imaging image based on the time sequence feature of the thermal imaging video, thereby obtaining the video feature of the thermal imaging video, and improving the feature extraction efficiency of the thermal imaging video.

[0079] It should be further explained that the execution subject of the feature extraction method based on the thermal imaging video can be a feature extraction device based on the thermal imaging video. For example, the feature extraction method based on the thermal imaging video can be executed by a terminal device or a server or other processing device. The terminal device can be a user equipment (UE), a computer, a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. In some possible implementation manners, the feature extraction method based on the thermal imaging video can be realized by a processor invoking computer readable instructions stored in a memory.

[0080] Figure 4 is a block diagram of a feature extraction device based on a thermal imaging video according to an example embodiment of the present application. As shown in the figure, the example feature extraction device based on the thermal imaging video 400 includes a temperature feature extraction module 410, an image feature determination module 420, a fusion module 430, and a time sequence correlation module 440. Specifically, Figure 4

[0081] The temperature feature extraction module 410 is configured to extract temperature features of thermal imaging images in the thermal imaging video.

[0082] The image feature determination module 420 is configured to determine image features of the thermal imaging images based on the temperature features.

[0083] The fusion module 430 is configured to fuse the temperature features and the image features to obtain comprehensive features of the thermal imaging images.

[0084] The time sequence correlation module 440 is configured to perform time sequence correlation on the comprehensive features of the thermal imaging images according to time sequence features of the thermal imaging video to obtain video features of the thermal imaging video.

[0085] In the example feature extraction device based on the thermal imaging video, the temperature features extracted from the thermal imaging images in the thermal imaging video are used to determine the image features of the thermal imaging images, the temperature features and the image features are fused to obtain the comprehensive features of the thermal imaging images, and the comprehensive features of the thermal imaging images are time sequence correlated based on the time sequence features of the thermal imaging video, thereby obtaining the video features of the thermal imaging video, and improving the feature extraction efficiency of the thermal imaging video.

[0086] The functions of each module can be referred to the embodiments of the feature extraction method based on the thermal imaging video, which will not be described herein.

[0087] Please refer to Figure 5 , Figure 5 ​is a structural schematic diagram of an embodiment of an electronic device of the present application. The electronic device 500 includes a memory 501 and a processor 502, and the processor 502 is configured to execute program instructions stored in the memory 501 to implement the steps in any of the above embodiments of the feature extraction method based on thermal imaging video. In a specific implementation scenario, the electronic device 500 can include but is not limited to a microcomputer, a server, and in addition, the electronic device 500 can also include a notebook computer, a tablet computer, and other mobile devices, which are not limited herein.

[0088] Specifically, the processor 502 is configured to control itself and the memory 501 to implement the steps in any of the above embodiments of the feature extraction method based on thermal imaging video. The processor 502 can also be referred to as a CPU (Central Processing Unit). The processor 502 can be an integrated circuit chip with processing capability. The processor 502 can also be a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. In addition, the processor 502 can be implemented by an integrated circuit chip together.

[0089] The above scheme determines the image features of the thermal imaging images by the temperature features extracted from the thermal imaging images in the thermal imaging video, fuses the temperature features and the image features to obtain the comprehensive features of the thermal imaging images, and time-series correlates the comprehensive features of the thermal imaging images based on the time-series features of the thermal imaging video, so as to obtain the video features of the thermal imaging video, thereby improving the feature extraction efficiency of the thermal imaging video.

[0090] Please refer to Figure 6 , Figure 6 is a structural schematic diagram of an embodiment of a computer-readable storage medium of the present application. The computer-readable storage medium 610 stores program instructions 611 capable of being executed by a processor, and the program instructions 611 are used to implement the steps in any of the above embodiments of the feature extraction method based on thermal imaging video.

[0091] The scheme extracts the temperature feature of the thermal imaging image in the thermal imaging video, determines the image feature of the thermal imaging image, fuses the temperature feature and the image feature, obtains the comprehensive feature of the thermal imaging image, and performs time sequence correlation on the comprehensive feature of each thermal imaging image based on the time sequence feature of the thermal imaging video, so that the video feature of the thermal imaging video can be obtained, and the feature extraction efficiency of the thermal imaging video is improved.

[0092] In some embodiments, the apparatus provided by the embodiments of the present disclosure has functions or includes modules that can be used to perform the methods described in the above method embodiments, and the specific implementation can refer to the description of the above method embodiments. For brevity, it will not be repeated here.

[0093] The above description of each embodiment tends to emphasize the differences between each embodiment, and the same or similar parts can be mutually referred to. For brevity, it will not be repeated here.

[0094] In several embodiments provided in the present application, it should be understood that the disclosed method and device can be implemented in other ways. For example, the above-described device implementation is only schematic; for example, the division of the modules or units is only a logical function division, and there can be another division manner in actual implementation; for example, a unit or component can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual coupling or direct coupling or communication connection can be indirect coupling or communication connection through some interface, device or unit, and can be electrical, mechanical or other forms.

[0095] In addition, each of the function units in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software function unit. When the integrated unit is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application, essentially or in the form of a contribution to the prior art, or all or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes several instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor execute all or part of the steps of the methods in each embodiment of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

Claims

1. A feature extraction method based on thermal imaging video, characterized in that, The method includes: Extract the temperature features of the thermal imaging image from the thermal imaging video; The image features of the thermal imaging image are determined based on the temperature characteristics; the image features include spatial features and semantic features. The step of determining the image features of the thermal imaging image based on the temperature features includes: determining spatial information in the thermal imaging image based on the temperature features; the spatial information includes edges and corners; performing feature extraction on the spatial information to obtain the spatial features; performing semantic feature extraction on the thermal imaging image based on the spatial features to obtain the semantic features; the semantic features are used to characterize the attribute information of each object in the thermal imaging image. The step of extracting semantic features from the thermal imaging image based on the spatial features to obtain the semantic features includes: inputting the spatial features into a pre-constructed mapping function to obtain the corresponding dynamic convolution kernel parameters; and extracting semantic features from the thermal imaging image based on the dynamic convolution kernel parameters to obtain the semantic features. By fusing the temperature features and the image features, a comprehensive feature of the thermal imaging image is obtained; Based on the temporal characteristics of the thermal imaging video, the comprehensive features of each thermal imaging image are temporally correlated to obtain the video features of the thermal imaging video.

2. The method according to claim 1, characterized in that, The step of extracting features from the spatial information to obtain the spatial features includes: Based on the edges and corners of the thermal imaging image, the spatial features are obtained, including edge features and corner features.

3. The method according to claim 2, characterized in that, After the step of detecting the spatial information in the thermal imaging image based on the temperature characteristics, the method further includes: The pixels in the thermal imaging image are binarized and labeled based on the edge pixels corresponding to the edges and the corner pixels corresponding to the corners. Background removal is performed on the binarized thermal imaging image to obtain the foreground of the binarized thermal imaging image, wherein the foreground includes the target object; Semantic features are extracted from the target object to obtain the semantic features.

4. The method according to claim 1, characterized in that, The step of extracting temperature features from the thermal imaging video includes: Temperature identification is performed on the thermal imaging image to obtain the temperature value corresponding to each pixel in the thermal imaging image; Temperature encoding is performed based on the temperature value to obtain the temperature feature vector of the thermal imaging image.

5. The method according to claim 4, characterized in that, The step of determining the image features of the thermal imaging image based on the temperature features, and fusing the temperature features and the image features to obtain the comprehensive features of the thermal imaging image, includes: Based on the temperature feature vector of the temperature features, extract the image feature vector of the thermal imaging image; The temperature feature vector and the image feature vector are fused to obtain the feature fusion result. The feature fusion result is fitted to obtain the comprehensive feature.

6. The method according to claim 1, characterized in that, The step of temporally associating the comprehensive features of each thermal imaging image based on the temporal characteristics of the thermal imaging video to obtain the video features of the thermal imaging video includes: The temporal sequence of each frame of thermal imaging image in the thermal imaging video is determined based on the temporal characteristics of the thermal imaging video. The combined features of each frame of thermal imaging image are correlated based on the time sequence to obtain the video features of the thermal imaging video.

7. The method according to claim 6, characterized in that, The step of associating the comprehensive features of each frame of thermal imaging image based on the temporal sequence to obtain the video features of the thermal imaging video includes: Extract multiple frames of thermal imaging images from the thermal imaging video; Based on the temporal sequence of the multiple thermal imaging images, the comprehensive features corresponding to the multiple thermal imaging images are analyzed to obtain the video features of the thermal imaging video.

8. An electronic device, characterized in that, The method includes a memory and a processor, the processor being configured to execute program instructions stored in the memory to implement the method according to any one of claims 1 to 7.

9. A computer-readable storage medium having program instructions stored thereon, characterized in that, When the program instructions are executed by the processor, they implement the method described in any one of claims 1 to 7.

Images