A multi-modal sensor data fusion method

Abstract

The application discloses a kind of multimodal sensor data fusion methods, belong to multimodal sensor data fusion technical field.The method acquires the first image sequence of first visible light sensor and the second image sequence of first infrared sensor, after time axis alignment, each frame image is respectively implemented multi-scale direction filtering, generates the phase consistency tensor sequence of structure information representation;Through tensor point product operation to two groups of tensor sequence, generate the fusion response graph sequence reflecting multimodal structure matching degree;According to fusion response graph sequence, first frame sequence and second frame sequence complete pixel level weighted fusion, obtain multimodal fusion image sequence.Phase consistency tensor fuses local energy amplitude and direction entropy, can robustly extract cross-modal structure consistency features under complex scene, so that fusion result retains visible light texture details and infrared target saliency simultaneously.

Description

Technical Field

[0001] This invention relates to the field of multimodal sensor data fusion technology, specifically to a multimodal sensor data fusion method. Background Technology

[0002] Multimodal sensor data fusion methods need to address the information complementarity problem between visible light and infrared images under conditions of temporal synchronization and spatial registration. Visible light sensors can capture rich texture and color details in well-lit environments, while infrared sensors are sensitive to thermal radiation targets and are unaffected by changes in illumination. Fusing these two modalities can generate images that combine high detail and target prominence, making them widely applicable in scenarios such as nighttime surveillance, autonomous driving, and industrial inspection. Existing fusion methods often employ multi-scale decomposition, sparse representation, or deep learning-based prediction weights, which have two shortcomings. Firstly, conventional multi-scale decomposition fusion relies on pyramid decomposition of the image followed by fusion based on local contrast or energy. When faced with sudden changes in illumination, low texture, or local occlusion, fusion criteria based on grayscale gradients or energy are easily affected by noise, leading to blurred edges, halos, or broken target contours in the fused image. On the other hand, mainstream pixel-level fusion strategies typically employ fixed weighting coefficients or a single saliency measure, failing to effectively capture the similarity of structural information between visible and infrared images. This prevents the fusion process from adaptively distinguishing the respective advantageous regions of the two modalities, often weakening the integrity of infrared targets while preserving visible light details, or introducing excessive visible light background noise when enhancing infrared targets. Phase consistency, as a local structural measure insensitive to changes in illumination and contrast, has been used for feature detection in single-modal images. However, when extended to multimodal fusion, the challenge of quantifying the degree of structural matching between different modalities remains. Therefore, a fusion method is needed that can jointly characterize local energy intensity and orientation distribution features within the phase consistency framework, and thereby construct a cross-modal structural similarity measure to address the problem of reliable fusion of structural information between visible and infrared images in complex scenes. Summary of the Invention

[0003] This invention provides a multimodal sensor data fusion method based on phase consistency tensors and tensor dot products, addressing the problems of existing fusion methods easily losing structural information in areas with varying illumination and low texture, and the lack of adaptive evaluation of the degree of multimodal structural matching in fusion weights. By converting visible light and infrared images into phase consistency tensors containing local energy amplitude and directional entropy, respectively, and using tensor dot products to measure the degree of matching across modal structures, a fusion response map capable of distinguishing modal dominance regions is generated, thereby achieving pixel-level fusion that is adaptive to scene content.

[0004] To achieve the above objectives, the present invention provides the following technical solution: The present invention provides a multimodal sensor data fusion method, including acquiring a multimodal sensor data set, which includes a first image sequence acquired by a first visible light sensor and a second image sequence acquired by a first infrared sensor. The first image sequence and the second image sequence are time-axis aligned to generate an aligned first frame sequence and a second frame sequence. Preferably, the first acquisition timestamp of each frame in the first image sequence and the second acquisition timestamp of each frame in the second image sequence are extracted. Based on the time difference between the first and second acquisition timestamps, cubic spline interpolation is used to perform inter-frame interpolation on the second image sequence to generate a second frame sequence synchronized with the first image sequence on the time axis. The original first image sequence is used as the aligned first frame sequence, and the interpolated second frame sequence is used as the aligned second frame sequence. This eliminates temporal mismatch caused by differences in sampling frequency or trigger delays between different modal sensors, ensuring the temporal consistency of the corresponding frames in subsequent fusion.

[0005] Multi-scale directional filtering is performed on each first frame image in the first frame sequence to generate a first phase-consistency tensor sequence; multi-scale directional filtering is performed on each second frame image in the second frame sequence to generate a second phase-consistency tensor sequence. As a technical solution of this invention, a Log-Gabor two-dimensional filter bank containing 3 scale parameters and 8 direction parameters is constructed. Each frame image is convolved with all filters in the filter bank to obtain the filtering response at each scale and direction. For each pixel location, the local energy amplitude and local direction entropy are calculated based on the filtering response at all scale directions. The local energy amplitude is used as the amplitude component, and the local direction entropy is used as the direction component to construct a phase-consistency tensor for each pixel location. This phase-consistency tensor is a second-order tensor containing amplitude and direction components. Preferably, the filtered responses are first normalized, and the sum of the normalized filtered responses in the directional dimension is calculated as the sum of the directional responses at that scale. Then, the sums of the directional responses at each scale are accumulated according to pixel position to obtain the total directional response, and the local energy amplitude is calculated based on this. At the same time, the local directional entropy is calculated based on the distribution of the directional responses, thus forming a second-order tensor. Multi-scale directional filtering can effectively extract the structural features and edge information of images at different scales and directions. Moreover, the phase consistency tensor is invariant to changes in illumination and contrast differences, and descriptions reflecting the geometric stability of the image can be obtained from visible light images and infrared images respectively, providing robust feature representations for cross-modal fusion.

[0006] A fused response map sequence is generated based on the tensor dot product between the first phase consistency tensor sequence and the second phase consistency tensor sequence. Tensor dot product is performed on the first and second phase consistency tensors at the same pixel location in each frame image to obtain the dot product scalar for that pixel location. The dot product scalars for all pixel locations are combined into a dot product matrix with the same size as the frame image. Preferably, the dot product matrix is ​​linearly stretched to linearly map it to the 0-1 interval to generate the fused response map sequence. Further, before linear stretching, the overall mean and standard deviation of the dot product matrix are calculated. A lower threshold is set as the mean minus twice the standard deviation, and an upper threshold is the mean plus twice the standard deviation. Elements in the dot product matrix below the lower threshold are truncated to the lower threshold, and elements above the upper threshold are truncated to the upper threshold. Then, linear normalization is performed using the truncated upper and lower thresholds to ensure that the element values ​​of the normalized dot product matrix are in the 0-1 interval. By using tensor dot product operations, higher fusion response values ​​can be assigned to locations with consistent structures and similar directional characteristics in visible light and infrared images, while moderate or lower response values ​​can be assigned to regions rich in complementary information or with significant structural differences. This allows for the automatic differentiation of homogeneous and complementary regions of multimodal information in the fusion response map.

[0007] The first frame sequence and the second frame sequence are fused at the pixel level according to the fusion response map sequence to generate a multimodal fused image sequence. Specifically, the fusion response value at each pixel position in the fusion response map sequence is used as a first weighting coefficient, and the difference between 1 and the fusion response value is used as a second weighting coefficient. For each pixel position, the pixel value of the first frame image is multiplied by the first weighting coefficient to obtain a first weighted component, and the pixel value of the second frame image is multiplied by the second weighting coefficient to obtain a second weighted component. The first weighted component and the second weighted component are then added to obtain the pixel value of the fused image at that pixel position. After traversing all pixel positions, a multimodal fused image sequence is generated. Preferably, each frame of the fused image sequence is subjected to bilateral filtering. The spatial domain standard deviation of the bilateral filter is taken as 5% of the width of the fused image, and the gray domain standard deviation is taken as 10% of the maximum gray value of the fused image. The image after bilateral filtering is used as the final multimodal fused image sequence. By using adaptive weighting based on the fusion response map, the high spatial resolution information of the visible light image can be fully utilized in textured areas, while the radiation characteristics of the infrared image can be enhanced in low-light or occluded areas, avoiding the introduction of artifacts or noise during the fusion process. Bilateral filtering post-processing can further smooth flat areas and preserve edge details, improving the visual quality and spatial consistency of the fused image.

[0008] As a further improvement of the present invention, before acquiring the multimodal sensor data set, the spatial positions of the first visible light sensor and the first infrared sensor are calibrated to obtain rotation matrices and translation vectors. Based on these rotation matrices and translation vectors, the first image sequence and the second image sequence are spatially registered, aligning the pixel coordinates of corresponding frames in the two sequences through homography transformation. The spatially registered first image sequence and the transformed second image sequence are then used as the multimodal sensor data set. Preferably, the first image sequence from the first visible light sensor is used as the reference sequence, and the second image sequence from the first infrared sensor is used as the sequence to be transformed. The homography matrix of each frame in the sequence to be transformed relative to the reference sequence is calculated based on the rotation matrix and translation vectors. Bilinear interpolation is used to remap the pixel coordinates of each frame in the sequence to be transformed, generating a transformed second image sequence spatially aligned with the reference sequence. This eliminates the parallax effect in the spatial dimension, ensuring that the pixel positions of the same scene point accurately coincide in different modal images. This provides a geometrically consistent prerequisite for subsequent temporal alignment and pixel-level fusion, significantly improving the accuracy and reliability of the fusion results.

[0009] The technical effects and advantages provided by the present invention in the above technical solution are as follows: Multi-scale directional filtering based on a Log-Gabor filter is applied to each frame in the first and second frame sequences to extract phase consistency features at multiple scales and directions. Local energy amplitude and local orientation entropy are calculated for each pixel location, and a second-order phase consistency tensor containing amplitude and orientation components is constructed. The local energy amplitude reflects the intensity of the structured information at that location, while the local orientation entropy characterizes the concentration of the orientation distribution. The phase consistency tensor, jointly constructed by these two components, encodes both structural strength and orientation features, stably responding to structural elements such as edges, corners, and textures in the image. Compared to representations that rely solely on gradient amplitude or energy spectrum, this tensor is invariant to changes in illumination intensity and contrast. Therefore, it can still extract consistent representations reflecting the same physical structure under two significantly different modes, visible light and infrared light, providing a robust feature foundation for subsequent cross-modal structure matching. Based on this, tensor dot product operations are performed on the first and second phase consistency tensors at the same pixel location to obtain a dot product scalar reflecting the degree of structural matching between the two modes at that location, forming a fused response map sequence. Tensor dot product integrates the matching relationship between amplitude and orientation components into a single scalar value. A larger dot product value indicates that the location is suitable for preserving visible light details when the two tensors are highly consistent in both amplitude and orientation features; a smaller dot product value indicates that the region should highlight the thermal radiation target of the infrared sensor. An adaptive weight allocation based on this fusion response map is used to perform pixel-by-pixel weighted fusion of the first and second frame images. This ensures that regions with rich visible light texture and salient infrared targets obtain fusion coefficients that match their information credibility, avoiding the loss of details or target weakening that occurs in traditional fixed-weight fusion. Attached Figure Description

[0010] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0011] Figure 1 This is a flowchart of a multimodal sensor data fusion method; Figure 2 This is a flowchart of the image sequence time axis alignment process; Figure 3 This is a schematic diagram of the acquisition timestamp of the second image sequence and the cubic spline interpolation curve; Figure 4 It is a graph showing the local energy amplitude and local orientation entropy of a pixel location under multi-scale directional filtering; Figure 5This is a schematic diagram of the dot product of the first phase consistency tensor and the second phase consistency tensor, and the fused response value. Figure 6 It is a graph comparing pixel values ​​of a multimodal fused image; Figure 7 It is a graph comparing pixel deviations before and after spatial registration. Detailed Implementation

[0012] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0013] See Figure 1 This invention provides a multimodal sensor data fusion method. The method involves acquiring a multimodal sensor data set, including a first image sequence acquired by a first visible light sensor and a second image sequence acquired by a first infrared sensor. The first and second image sequences are time-axis aligned to generate aligned first and second frame sequences. Each first frame in the first frame sequence is subjected to multi-scale directional filtering to generate a first phase-consistency tensor sequence. Each second frame in the second frame sequence is also subjected to multi-scale directional filtering to generate a second phase-consistency tensor sequence. A fused response map sequence is generated based on the tensor dot product between the first and second phase-consistency tensor sequences. Finally, the first and second frame sequences are pixel-level fused using the fused response map sequence to generate a multimodal fused image sequence.

[0014] Example 1: In specific implementation, please refer to Figure 2The process involves aligning the first and second image sequences along their timelines to generate aligned first and second frame sequences. This includes extracting the first acquisition timestamp of each frame in the first image sequence and the second acquisition timestamp of each frame in the second image sequence. During acquisition, the first visible light sensor and the first infrared sensor generate image frames according to their respective acquisition frequencies. Each frame is accompanied by an acquisition timestamp, which can be parsed from the image file's metadata or recorded synchronously by the acquisition software as an independent timestamp file corresponding to the frame number when generating the image sequence. The first acquisition timestamp indicates the time when the first visible light sensor acquired the corresponding frame image, and the second acquisition timestamp indicates the time when the first infrared sensor acquired the corresponding frame image. The extraction operation organizes all timestamps into a continuous time series, with the first acquisition timestamp sequence corresponding one-to-one with the frames of the first image sequence, and the second acquisition timestamp sequence corresponding one-to-one with the frames of the second image sequence.

[0015] Based on the time difference between the first and second acquisition timestamps, cubic spline interpolation is used to perform inter-frame interpolation on the second image sequence, generating a second frame sequence synchronized with the first image sequence on the time axis. In specific implementation, the first acquisition timestamp sequence is used as a reference time axis, and the second image sequence needs to generate corresponding image frames at each time point marked by the first acquisition timestamp sequence. For any time point in the first acquisition timestamp sequence, the relative position of the time point in the second acquisition timestamp sequence is determined, and multiple timestamps in the second acquisition timestamp sequence that are temporally adjacent to the time point and their corresponding image frames are selected. The number of selected image frames is at least 4 frames to construct a cubic spline interpolation function that satisfies the third-order continuity condition. For each pixel channel in the image, a cubic spline interpolation polynomial is constructed with the pixel grayscale value of the selected frame as the function value and the corresponding timestamp as the independent variable. The cubic spline interpolation polynomial has continuous first and second derivatives in adjacent time intervals, and the overall interpolation curve exhibits smooth characteristics. Natural boundary conditions are used, i.e., the second derivative is set to zero at both ends, to determine all coefficients of the interpolation function. The target time point is substituted into the solved cubic spline interpolation polynomial to calculate the interpolated pixel value. After performing the above operation on all pixel positions and all pixel channels of the image, a complete interpolated image frame is generated. The above process is repeated for all time points in the first acquisition timestamp sequence to form a complete interpolation sequence, i.e., the second frame sequence synchronized with the first image sequence on the time axis. When the start of the second acquisition timestamp sequence is later than the start of the first acquisition timestamp sequence, or the end is earlier than the end of the first acquisition timestamp sequence, the interpolation range is limited to the time domain covered by the second acquisition timestamp sequence; the portion exceeding the range can be extrapolated according to the boundary frames or directly discarded.

[0016] The first image sequence is used as the first aligned frame sequence, and the interpolated second frame sequence is used as the second aligned frame sequence. The first aligned frame sequence and the second aligned frame sequence have a completely consistent timeline, that is, the frame number corresponds one-to-one with the timestamp, and subsequent processing can obtain the corresponding frame under the same time index.

[0017] See Figure 3 In the graph, the horizontal axis represents the time axis in seconds, and the vertical axis represents the pixel grayscale value. The orange scatter points represent the grayscale value distribution of the original sampling points in the second image sequence, distributed unevenly across the time axis, exhibiting a discrete sampling state. The blue continuous curve is the grayscale value change curve generated after interpolating the orange sampling points using the cubic spline interpolation method. This interpolation curve shows a smooth connection between any two adjacent sampling points and has continuous first and second derivative properties, ensuring a natural transition of grayscale value changes. The curve shows multiple fluctuations within the time interval of 0 to 10 seconds, with grayscale values ​​varying between approximately 60 and 170, reflecting the dynamic change trend of the pixel grayscale value of the interpolated image over time.

[0018] Example 2: In a specific implementation, multi-scale directional filtering is performed on each first frame image in the first frame sequence to generate a first phase-consistent tensor sequence, including: A two-dimensional filter bank for constructing a Log-Gabor filter is shown, comprising 3 scale parameters and 8 orientation parameters. In a specific implementation, the frequency response of the Log-Gabor filter is defined in the frequency domain, and the transfer function of the two-dimensional Log-Gabor filter in polar coordinates is expressed as the product of the radial and angular components. The number of scale parameters is set to 3, denoted as [missing information - likely a typology or similar designation]. , The values ​​1, 2, and 3 correspond to three filtering scales with progressively decreasing center frequencies. The center frequency is selected as follows: the center frequency of the smallest scale is set to a value equal to a fraction of the Nyquist frequency. The ratio of center frequencies between adjacent scales is a fixed value of 2.1, allowing the filter bank to cover frequency bandwidths of varying widths. The number of directional parameters is set to 8, denoted as... , Integers ranging from 1 to 8, corresponding to values ​​from 0 radians to... radians, adjacent directional intervals are The eight directions of the radian. This leads to the construction of 24 two-dimensional Log-Gabor filters, each determined by a specific set of scale and direction parameters.

[0019] Each first frame image is convolved with all filters in the filter bank to obtain the filtered response at each scale and direction. In a specific implementation, a Fourier transform is performed on the first frame image to obtain its spectrum. The transfer function of each filter in the filter bank is transformed to the frequency domain and multiplied element-wise with the spectrum of the first frame image. An inverse Fourier transform is performed on the multiplication result to obtain the filtered response corresponding to one scale parameter and one direction parameter. This process is repeated, iterating through the first frame image for three scale parameters and eight direction parameters, resulting in 24 filtered response images. Each filtered response image has the same size as the first frame image and records the complex numerical response at each pixel position at that scale and direction. The filtered response consists of a real part and an imaginary part.

[0020] For each pixel location, the local energy magnitude and local orientation entropy are calculated based on the filtered response at all scale directions for that pixel location. Optionally, the local energy magnitude is obtained by aggregating the filtered response magnitudes at each scale direction. The local orientation entropy is calculated based on the proportion of each direction's response in the total response.

[0021] A phase consistency tensor is constructed for each pixel location. This phase consistency tensor is a second-order tensor containing an amplitude component and a direction component. In a specific implementation, the amplitude component is set to the local energy amplitude, and the direction component is set to the local direction entropy. This operation is performed on all pixel locations in the first frame image to obtain the first phase consistency tensor corresponding to the first frame image. Each first frame image in the first frame sequence is processed sequentially to generate a sequence of first phase consistency tensors.

[0022] In specific implementation, the process of constructing the phase consistency tensor for each pixel location further includes: normalizing the filtered response at each scale and in each direction to generate a normalized filtered response. Optionally, the normalization process is performed as follows: for the filtered response image at each scale and in each direction, the root mean square value of the amplitude at all pixel locations in the filtered response image is calculated; the amplitude at each pixel location in the filtered response image is divided by the root mean square value to complete the normalization and generate a normalized filtered response. This normalization process can eliminate gain differences between different filtering channels.

[0023] The sum of all normalized filtered responses along the orientation dimension is calculated as the total orientation response at the stated scale. In a specific implementation, for a fixed scale parameter and a fixed pixel position, the amplitudes of the normalized filtered responses for the eight orientation parameters at that scale are summed to obtain a scalar value, called the total orientation response at that pixel position at that scale. This process is performed for each of the three scales, resulting in a total orientation response for each pixel position at each scale.

[0024] The total directional response at each scale is obtained by summing the directional responses at each pixel location. In a specific implementation, for the same pixel location, the sums of the directional responses calculated at the three scales are added together to obtain the total directional response at that pixel location, denoted as . .

[0025] The local energy amplitude is calculated based on the total directional response, and the local directional entropy is calculated based on the distribution of the responses in each direction. In a specific implementation, the local energy amplitude is directly taken as the square root of the total directional response, i.e., the local energy amplitude. Local directional entropy is obtained by analyzing the relative distribution of the filtered responses in different directions at the pixel location. For the pixel location, the amplitudes of the normalized filtered responses in eight directions at all scales are collected. The amplitudes at different scales in the same direction are summed to obtain the overall directional response intensity for that direction, and the direction is denoted as... The overall directional response intensity is Calculation direction Response probability If all appear If both are zero, then the local directional entropy is directly set to zero. Local directional entropy The calculation formula is: in, Represents local directional entropy. The index represents the direction parameter, and its value is an integer ranging from 1 to 8. Indicates direction The response probability, This represents a logarithmic operation with base 2. When At that time, it was agreed .

[0026] The local energy amplitude and local orientation entropy are used as two tensor elements of the phase consistency tensor to form a second-order tensor. In a specific implementation, at each pixel location, a second-order tensor containing two elements is obtained, the first element being the local energy amplitude. The second element is the local directional entropy. The second-order tensors are arranged according to pixel positions to form the phase consistency tensor of the first frame image. The first frame sequence is processed frame by frame to obtain the first phase consistency tensor sequence. For each second frame image in the second frame sequence, the same multi-scale directional filtering and phase consistency tensor construction process as described above is used to generate the second phase consistency tensor sequence.

[0027] See Figure 4In the graph, the horizontal axis represents the pixel column position, ranging from 0 to approximately 510, reflecting the pixel distribution in the horizontal direction of the first frame image. The left vertical axis, corresponding to the blue solid curve, represents the local energy amplitude, with a value ranging from approximately 0.5 to 11.0, reflecting the overall energy intensity of the multi-scale directional filtering response at the pixel location. The right vertical axis, corresponding to the red dashed curve, represents the local directional entropy, with a value ranging from approximately 0.0 to 3.6 bits, reflecting the uniformity of the filtering response distribution in multiple directions at the pixel location.

[0028] The blue solid line curve exhibits a multi-peak fluctuation characteristic, with obvious peaks appearing at approximately pixel positions 50, 120, 200, 290, 350, and 450, where the local energy amplitude reaches its maximum. This indicates a strong filtering response energy in the corresponding areas, suggesting that the image texture or structural features at these locations are more significant. The curve shows a relatively low energy level between the peaks, exhibiting an overall periodic fluctuation trend.

[0029] The local directional entropy trend of the red dashed curve is relatively independent of the energy amplitude variation. In the pixel column ranges of approximately 140-170, 220-270, and 380-420, the directional entropy reaches relatively high peaks, exceeding 3.0 bits, indicating that the filter response in these regions is relatively dispersed in multiple directions, with a relatively uniform directional distribution. Conversely, near the local energy amplitude peaks, the directional entropy tends to decrease to some extent, showing a relative concentration in direction.

[0030] Example 3: In specific implementation, a fused response map sequence is generated based on the tensor dot product between the first phase consistency tensor sequence and the second phase consistency tensor sequence, including: Tensor dot product operations are performed on the first and second phase-consistency tensors at the same pixel location in each frame image to obtain the dot product scalar for that pixel location. Both the first and second phase-consistency tensors are second-order tensors. The first phase-consistency tensor contains a first magnitude component and a first direction component, and the second phase-consistency tensor contains a second magnitude component and a second direction component. At each pixel location, the tensor dot product operation is performed by multiplying the first and second magnitude components to obtain an magnitude product, multiplying the first and second direction components to obtain a direction product, and then summing the magnitude product and the direction product to obtain the scalar value of the dot product scalar for that pixel location. For each pair of corresponding frame images in the first and second frame sequences, the above tensor dot product operation is performed at all pixel locations to obtain a set of dot product scalars with the same number of pixels as the frame images.

[0031] The dot product scalars of all pixel positions are combined into a dot product matrix, the size of which is the same as the frame image size. In a specific implementation, the dot product scalars are arranged into a two-dimensional matrix according to the row and column order of the pixel positions. The number of rows in the matrix is ​​the same as the height of the frame image, and the number of columns is the same as the width of the frame image, forming the dot product matrix. A dot product matrix is ​​generated for each pair of corresponding frame images, and the sequence of dot product matrices corresponds one-to-one with the sequence of frame images.

[0032] In specific implementation, before linearly stretching the dot product matrix, the following steps are included: calculating the overall mean μ and standard deviation σ of the dot product matrix. The overall mean μ is obtained by summing all elements in the dot product matrix and then dividing by the total number of elements in the dot product matrix. The standard deviation σ is obtained by calculating the square root of the squared deviations of each element from the overall mean μ. A lower threshold is set to μ-2σ, and an upper threshold is set to μ+2σ. Elements in the dot product matrix below the lower threshold are set to the lower threshold, and elements above the upper threshold are set to the upper threshold. The truncation process constrains the extreme values ​​in the dot product matrix to the range determined by the overall mean μ and standard deviation σ. The truncated dot product matrix is ​​linearly normalized using the upper and lower thresholds, so that the elements of the normalized dot product matrix are in the interval between 0 and 1. The linear normalization calculation formula is: in, This represents the element values ​​of the normalized dot product matrix. This represents the value of the corresponding element in the truncated dot product matrix. Indicates the lower threshold. Indicates the upper limit threshold. The value is μ-2σ. The value is μ+2σ. When and When they are equal, that is, when all elements in the dot product matrix have the same value, simply use... Set it to 0.5.

[0033] The normalized dot product matrix is ​​the fused response map corresponding to the frame image, and the fused response maps corresponding to all frame images form a fused response map sequence.

[0034] See Figure 5In the graph, the horizontal axis represents the pixel position index, ranging from 0 to 500, and the vertical axis represents the response value. The gray solid line is the distribution curve of the original dot product scalar, showing the trend of the dot product scalar change corresponding to each pixel position. The values ​​fluctuate significantly, and the dot product scalar at some pixel positions exceeds the normal range, exhibiting obvious extreme high and low values. The green dashed line is the truncated dot product scalar curve. After truncating by using the overall mean μ minus twice the standard deviation σ as the lower threshold and the mean plus twice the standard deviation as the upper threshold, the extreme dot product scalar values ​​exceeding the threshold range are limited within the threshold boundaries. This significantly suppresses abnormal peaks and valleys in the gray solid line, making the curve fluctuation amplitude more stable, and the overall trend basically consistent. The two thin horizontal dashed lines in the graph correspond to the lower and upper thresholds, located at approximately 1 and 9, respectively. The orange solid line represents the fusion response value curve, which is the result of linearly normalizing the truncated dot product matrix to the 0-1 range. The response value is stable between 0.2 and 0.8, showing a relatively smooth fluctuation trend with no abrupt changes or outliers, indicating that the normalization process effectively maps the dot product scalar to the fusion response value. This fusion response value curve is the pixel response distribution of the fusion response map sequence after tensor dot product operation and truncated normalization processing described in Example 3 in a single frame image, reflecting the spatial variation characteristics of the fusion response based on the phase-consistent tensor sequence.

[0035] Example 4: In a specific implementation, the first frame sequence and the second frame sequence are fused at the pixel level according to the fused response map sequence to generate a multimodal fused image sequence, including: The fusion response value at each pixel position in the fused response map sequence is used as the first weighting coefficient, and the difference between 1 and the fused response value is used as the second weighting coefficient. Each frame of the fused response map sequence corresponds one-to-one with the corresponding frame in the aligned first and second frame sequences in terms of time index. For a given frame index, the fused response map of the corresponding frame is read. Each pixel position in the fused response map records a fused response value, with the value ranging from 0 to 1. The coordinate positions in the fused response map... The fusion response value at that point is directly used as the first weighting coefficient. Calculate the second weighting coefficient. The method is Thus, the first weight coefficient matrix and the second weight coefficient matrix with the same size as the fused response map are obtained.

[0036] For each pixel location, the pixel value of the first frame image is multiplied by the first weighting coefficient to obtain the first weighted component, and the pixel value of the second frame image is multiplied by the second weighting coefficient to obtain the second weighted component. In a specific implementation, for the same frame index, the corresponding first frame image in the aligned first frame sequence and the corresponding second frame image in the aligned second frame sequence are obtained. The first frame image is a visible light image, and the second image is an infrared image; the two have achieved pixel-level correspondence through temporal alignment and spatial registration. Regarding coordinate positions... Read the pixel values ​​of the first frame image from the first frame image. Read the pixel values ​​of the second frame image from the second frame image. When the first or second frame image is a multi-channel image, the pixel value refers to the grayscale value of a single channel, and subsequent operations need to be performed separately for each channel. First weighted component Calculated as Second weighted component Calculated as .

[0037] The first weighted component and the second weighted component are added together to obtain the pixel value of the fused image at the pixel location. In a specific implementation, the pixel value of the fused image... Obtained through the following formula: in, Indicates coordinate position The pixel values ​​of the fused image are calculated at the location; This represents the first weighting coefficient, and its value is the coordinate position. The fusion response value at the location; Indicates the coordinate position of the first frame image Pixel value at; This represents the second weighting coefficient, with a value of [value missing]. ; Indicates the coordinate position of the second frame image The pixel value at that location.

[0038] Traverse all pixel positions to obtain a complete single-frame fused image. Perform the above process on all frames in the frame sequence to generate a multimodal fused image sequence.

[0039] After adding the first weighted component and the second weighted component, the method further includes: performing bilateral filtering on each frame of the multimodal fused image sequence, wherein the spatial domain standard deviation of the bilateral filter is 5% of the width of the fused image, and the gray-level domain standard deviation is 10% of the maximum gray value of the fused image; and using the bilaterally filtered image as the final multimodal fused image sequence. In a specific implementation, the kernel function of the bilateral filter includes a spatial domain Gaussian kernel and a gray-level Gaussian kernel, and the standard deviation of the spatial domain Gaussian kernel is... Set to 5% of the width of the merged image, that is ,in The number of pixels for the image width to be merged. The standard deviation of the Gaussian kernel in the grayscale region. The maximum grayscale value of the merged image is set to 10% of the maximum grayscale value. This maximum grayscale value is determined by the image pixel format. If the pixel format is an 8-bit unsigned integer, the maximum grayscale value is 255. For other bit depths, the same principle applies. The neighborhood window size of the bilateral filter is selected as follows. ,in, This indicates rounding up. The bilateral filtering operation is performed on each frame of the fused image using a full-image traversal approach. The filtered image replaces the original fused image, and all images processed by bilateral filtering constitute the final multimodal fused image sequence.

[0040] See Figure 6 In the graph, the horizontal axis represents the pixel column position of the image, ranging from 0 to 500, and the vertical axis represents the pixel grayscale value at the corresponding pixel column position, with a value range of approximately 40 to 160. The curves include three lines: the pixel value curve of the first frame image (blue), the pixel value curve of the second frame image (red), and the pixel value curve of the merged image (yellow).

[0041] The blue curve represents the grayscale value of a visible light image at each pixel column position in the first frame sequence after alignment. The curve fluctuates significantly, exhibiting obvious high-frequency changes and amplitude fluctuations, reflecting the rich details and dramatic grayscale variations in visible light images. The red curve represents the grayscale value of the infrared image in the second frame sequence after alignment of the corresponding frame. The overall grayscale value is higher than that of the blue curve and the fluctuation is more gradual, indicating a more uniform grayscale distribution in the infrared image and relatively stable brightness in multiple areas.

[0042] The yellow curve represents the pixel grayscale values ​​of the fused image obtained according to the multimodal fusion method described in Example 4. This curve generally falls between the blue and red curves, smoothly fusing the grayscale information from both sensor data. In the pixel positions approximately 40-80, 180-220, and 400-440, the fusion curve clearly balances the sharp fluctuations of the blue curve and the flatness of the red curve, exhibiting a smooth transition in grayscale values. Simultaneously, in the pixel positions approximately 100-160 and 300-350, the fusion curve effectively preserves the higher grayscale of the infrared image while moderately enhancing the details of the visible light image.

[0043] Example 5: In practical implementation, before collecting the multimodal sensor data set, the following is also included: The spatial relationship between the first visible light sensor and the first infrared sensor is calibrated to obtain the rotation matrix and translation vector. In a specific implementation, a checkerboard calibration board is used. The surface of the checkerboard calibration board contains a black and white checkerboard pattern, and the geometric positions of the corner points of the checkerboard are precisely known. The checkerboard calibration board is placed within the common field of view of the first visible light sensor and the first infrared sensor, and the orientation of the checkerboard calibration board is adjusted so that both the first visible light sensor and the first infrared sensor can completely capture the entire pattern of the checkerboard calibration board. The first visible light sensor captures a set of visible light calibration images, and the first infrared sensor simultaneously captures a set of infrared calibration images. The two sets of calibration images record the spatial orientation of the checkerboard calibration board from different perspectives. For each visible light calibration image, a corner detection algorithm is used to extract the corner pixel coordinates of the checkerboard calibration board, and a perspective projection equation is established between the corner pixel coordinates and the physical spatial coordinates of the checkerboard calibration board. For each infrared calibration image, the corner pixel coordinates of the checkerboard calibration board are also extracted. The intrinsic parameter matrices of the first visible light sensor and the first infrared sensor are obtained in advance through separate calibration. These intrinsic parameter matrices include the focal length and principal point coordinates. Using the correspondence between corner pixel coordinates in multiple sets of visible and infrared calibration images, a set of constraint equations is constructed, encompassing the rigid body transformation relationship between the coordinate systems of the first visible light sensor and the first infrared sensor. Solving these constraint equations yields the rotation matrix and translation vector. The rotation matrix is ​​a 3×3 orthogonal matrix describing the rotation transformation of the first infrared sensor coordinate system relative to the first visible light sensor coordinate system; the translation vector is a 3×1 vector describing the translation transformation of the origin of the first infrared sensor coordinate system relative to the origin of the first visible light sensor coordinate system.

[0044] Spatial registration is performed on the first image sequence and the second image sequence based on the rotation matrix and translation vector, so that the pixel coordinates of corresponding frames in the two sequences are aligned through homography transformation. It can be understood that the purpose of spatial registration is to eliminate parallax caused by the different installation positions of the first visible light sensor and the first infrared sensor, ensuring that the same scene point is mapped to the same pixel position in the frames of the first image sequence and the second image sequence.

[0045] The spatially registered first image sequence and the first infrared image sequence are used as the acquired multimodal sensor data set.

[0046] In a specific implementation, spatial registration of the first image sequence and the second image sequence is performed based on the rotation matrix and translation vector, including: using the first image sequence from the first visible light sensor as a reference sequence and the second image sequence from the first infrared sensor as the sequence to be transformed. It can be understood that the imaging plane of the first visible light sensor is used as the reference plane, and all subsequent spatial transformations are based on the pixel coordinate system of the reference sequence.

[0047] The homography matrix of each frame in the sequence to be transformed, relative to the reference sequence, is calculated based on the rotation matrix and translation vector. In a specific implementation, the calculation of the homography matrix relies on the assumption of a planar region in the scene, and a calibration plane approximately perpendicular to the optical axis of the first visible light sensor is selected as the reference plane. Let the unit normal vector of the reference plane in the coordinate system of the first visible light sensor be... The distance from the reference plane to the optical center of the first visible light sensor is A rotation matrix is ​​denoted as . The translation vector is denoted as Homography matrix Calculated using the following formula: in, This represents the homography matrix that maps pixel coordinates in the second image sequence to the corresponding pixel coordinates in the first image sequence; it is a 3×3 matrix. The intrinsic parameter matrix of the first infrared sensor is a 3×3 matrix; This represents the rotation matrix from the first infrared sensor coordinate system to the first visible light sensor coordinate system; This represents the translation vector from the first infrared sensor coordinate system to the first visible light sensor coordinate system, and is a 3×1 column vector; This represents the unit normal vector of the reference plane in the coordinate system of the first visible light sensor. This represents the transpose of the unit normal vector, which is a 1×3 row vector; This represents the perpendicular distance from the reference plane to the optical center of the first visible light sensor, with units consistent with those of the translation vector. This represents the inverse matrix of the intrinsic parameter matrix of the first visible light sensor. The value is obtained by measuring the distance from the calibration plate plane to the first visible light sensor during the calibration process. If the actual scene depth variation range is limited, it can be determined based on the average depth of the main targets in the scene. The value.

[0048] Bilinear interpolation is used to remap the pixel coordinates of each frame in the sequence to be transformed, generating a transformed second image sequence spatially aligned with the reference sequence. In a specific implementation, for each reference frame in the first image sequence, the pixel coordinate grid of the reference frame is used as the target grid, and the position of each target pixel in the target grid... Through homography matrix inverse matrix Map back to the source coordinate position in the frame to be transformed The mapping relationship is to multiply the target pixel vector in homogeneous coordinates by... After normalization, the source coordinate floating-point value is obtained. Since the source coordinate floating-point value usually does not fall on integer pixel grid points, bilinear interpolation is used to take the weighted average of the gray values ​​of the four nearest integer pixels around the source coordinate position. The coordinates of the four nearest integer pixels are as follows: , , , ,in, This indicates rounding down. The weights are determined by the horizontal and vertical distances between the source coordinate floating-point values ​​and the four integer pixel coordinates. The bilinear interpolation result is assigned to the target pixel position. The above remapping operation is performed on each frame in the sequence to be transformed, and all remapped frames constitute the transformed second image sequence.

[0049] The transformed second image sequence and the first image sequence are used together as a spatially registered multimodal sensor data set. In the spatially registered multimodal sensor data set, each frame of the first image sequence and the corresponding frame of the transformed second image sequence are spatially aligned in pixel coordinates, and the same scene point is located at the same pixel coordinate position in the two images.

[0050] See Figure 7 In the graph, the horizontal axis represents the pixel row coordinates, ranging from 0 to approximately 500; the vertical axis represents the pixel deviation value, in pixels. Two curves are plotted in the graph: "Spatial Registration Deviation Before" (red dashed line) and "Spatial Registration Deviation After" (green solid line).

[0051] The "spatiotemporal registration deviation" curve fluctuates significantly within the pixel row coordinate range, with the deviation value varying between -3 pixels and +2.5 pixels, exhibiting a clear periodic fluctuation trend. This indicates that when the images acquired by the two sensors are not spatially registered, there is a large parallax error at the corresponding pixel positions, and the error shows a strong non-uniform distribution as the pixel row coordinate changes.

[0052] The "spatial registration deviation" curve is close to zero deviation, and the deviation value mainly fluctuates within the range of ±0.2 pixels. The fluctuation amplitude is significantly reduced and there is no obvious periodic change. This indicates that after calculating the homography matrix based on the rotation matrix and translation vector in Example 5 and performing spatial registration on the second image sequence, the images in the two sequences achieve high-precision pixel-level alignment, which significantly reduces the parallax error between sensors.

[0053] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A method for multimodal sensor data fusion, characterized in that, include: A multimodal sensor data set is collected, which includes a first image sequence collected by a first visible light sensor and a second image sequence collected by a first infrared sensor; The first image sequence and the second image sequence are time-axis aligned to generate an aligned first frame sequence and second frame sequence. Perform multi-scale directional filtering on each first frame image in the first frame sequence to generate a first phase-consistent tensor sequence; Multi-scale directional filtering is performed on each second frame image in the second frame sequence to generate a second phase-consistent tensor sequence. A fused response map sequence is generated based on the tensor dot product between the first phase consistency tensor sequence and the second phase consistency tensor sequence; The first frame sequence and the second frame sequence are fused at the pixel level according to the fusion response map sequence to generate a multimodal fused image sequence.

2. The multimodal sensor data fusion method according to claim 1, characterized in that, The step of performing timeline alignment processing on the first image sequence and the second image sequence to generate aligned first frame sequence and second frame sequence includes: Extract the first acquisition timestamp of each frame in the first image sequence and the second acquisition timestamp of each frame in the second image sequence; Based on the time difference between the first acquisition timestamp and the second acquisition timestamp, cubic spline interpolation is used to perform inter-frame interpolation on the second image sequence to generate a second frame sequence that is synchronized with the first image sequence on the time axis; The first image sequence is used as the first aligned frame sequence, and the interpolated second frame sequence is used as the second aligned frame sequence.

3. The multimodal sensor data fusion method according to claim 1, characterized in that, The step of performing multi-scale directional filtering on each first frame image in the first frame sequence to generate a first phase-consistent tensor sequence includes: Construct a two-dimensional filter bank for a Log-Gabor filter, the filter bank containing 3 scale parameters and 8 orientation parameters; Each first frame image is convolved with all filters in the filter bank to obtain the filter response at each scale and in each direction. For each pixel location, calculate the local energy magnitude and local orientation entropy based on the filtered response of that pixel location across all scale directions; Using the local energy amplitude as the amplitude component and the local orientation entropy as the orientation component, a phase consistency tensor is constructed for each pixel location. The phase consistency tensor is a second-order tensor that includes both amplitude and orientation components.

4. The multimodal sensor data fusion method according to claim 3, characterized in that, The construction of the phase consistency tensor for each pixel location includes: The filter response at each scale and in each direction is normalized to generate a normalized filter response; Calculate the sum of all normalized filtered responses along the directional dimension, and use this as the sum of directional responses at that scale; The total directional response at each scale is summed up according to the pixel position to obtain the total directional response at that pixel position. The local energy amplitude is calculated based on the total directional response, and the local directional entropy is calculated based on the distribution of the responses in each direction. The local energy amplitude and local direction entropy are used as two tensor elements of the phase consistency tensor to form a second-order tensor.

5. The multimodal sensor data fusion method according to claim 1, characterized in that, The step of generating a fused response map sequence based on the tensor dot product between the first phase consistency tensor sequence and the second phase consistency tensor sequence includes: Perform tensor dot product operation on the first phase coherence tensor and the second phase coherence tensor at the same pixel position in each frame image to obtain the dot product scalar at that pixel position; Combine the dot product scalars of all pixel locations into a dot product matrix, the size of which is the same as the frame image size; The dot product matrix is ​​linearly stretched to linearly map the dot product scalar to the interval between 0 and 1, generating a fusion response map sequence.

6. The multimodal sensor data fusion method according to claim 5, characterized in that, Before linearly stretching the dot product matrix, the method further includes: Calculate the overall mean μ and standard deviation σ of the dot product matrix; Set the lower threshold to μ-2σ and the upper threshold to μ+2σ. Set the elements in the dot product matrix that are below the lower threshold to the lower threshold and the elements that are above the upper threshold to the upper threshold. The truncated dot product matrix is ​​linearly normalized using the upper and lower thresholds, so that the elements of the normalized dot product matrix are in the interval between 0 and 1.

7. The multimodal sensor data fusion method according to claim 1, characterized in that, The step of performing pixel-level fusion of the first frame sequence and the second frame sequence based on the fusion response map sequence to generate a multimodal fused image sequence includes: The fusion response value at each pixel position in the fusion response map sequence is used as the first weighting coefficient, and the difference between 1 and the fusion response value is used as the second weighting coefficient. For each pixel position, the pixel value of the first frame image is multiplied by the first weight coefficient to obtain the first weighted component, and the pixel value of the second frame image is multiplied by the second weight coefficient to obtain the second weighted component. The first weighted component and the second weighted component are added together to obtain the pixel value of the fused image at that pixel location; Iterate through all pixel locations to generate a multimodal fused image sequence.

8. The multimodal sensor data fusion method according to claim 7, characterized in that, After adding the first weighted component and the second weighted component, the method further includes: Bilateral filtering is applied to each frame of the fused image sequence, wherein the spatial domain standard deviation of the bilateral filter is 5% of the width of the fused image, and the gray-level domain standard deviation is 10% of the maximum gray value of the fused image. The bilaterally filtered image is used as the final multimodal fused image sequence.

9. The multimodal sensor data fusion method according to claim 1, characterized in that, Before acquiring the multimodal sensor data set, the following is also included: The spatial positional relationship between the first visible light sensor and the first infrared sensor is determined to obtain the rotation matrix and translation vector; Spatial registration is performed on the first image sequence and the second image sequence according to the rotation matrix and translation vector, so that the pixel coordinates of corresponding frame images in the two sequences are aligned by homography transformation; The spatially registered first image sequence and the first infrared image sequence are used as the acquired multimodal sensor data set.

10. A multimodal sensor data fusion method according to claim 9, characterized in that, The step of spatially registering the first image sequence and the second image sequence according to the rotation matrix and translation vector includes: The first image sequence of the first visible light sensor is used as the reference sequence, and the second image sequence of the first infrared sensor is used as the sequence to be transformed. Calculate the homography matrix of each frame in the sequence to be transformed relative to the reference sequence based on the rotation matrix and translation vector; Bilinear interpolation is used to remap the pixel coordinates of each frame in the sequence to be transformed, generating a transformed second image sequence that is spatially aligned with the reference sequence. The transformed second image sequence and the first image sequence are used together as the spatially registered multimodal sensor data set.

Images