A multi-modal visual data fusion processing method and system

Abstract

This invention relates to the field of image recognition technology, specifically to a multimodal visual data fusion processing method and system, comprising the following steps: extracting pixel gradients and temperature matrices, calculating covariance values ​​to generate associated weight vectors, performing dot product and channel splicing to construct a nonlinear reconstruction tensor, extracting classification boundary values ​​to construct a confidence vector, calculating relative entropy to generate penalty term coefficients, performing bias reduction calculations to output a classification decision matrix. In this invention, by weighting heterogeneous data based on the weight vector and reconstructing global confidence distribution features, extracting spatial relative entropy values ​​as correction coefficients and dynamically adjusting the bias of the output probability matrix, this invention suppresses visual frequency band interference caused by light sources, eliminates residual defects caused by feature overlap due to hardware splicing modes, reduces mapping errors while avoiding pixel feature dimension loss and target recognition misjudgment risks, and greatly improves recognition accuracy.

Description

Technical Field

[0001] This invention relates to the field of image recognition technology, and in particular to a multimodal visual data fusion processing method and system. Background Technology

[0002] Image recognition technology involves a system of techniques for extracting information from digital images or videos and performing automatic analysis and processing, mainly covering core processes such as feature extraction, object detection, and semantic segmentation. Traditional multimodal visual data fusion processing methods involve using visible light cameras and infrared sensors to acquire images separately, then aligning pixel coordinates using image registration algorithms, and finally concatenating pixel matrices from different modalities by channel before feeding them into the input layer of a convolutional neural network for feature extraction and classification.

[0003] Traditional multimodal visual data fusion processing methods use a pixel-level channel stitching mode for static feature combination in actual operation. This mode directly compresses heterogeneous data from different sensors into the same receptive field for computation, which forces the nonlinear differences in the spatial feature distribution of visible light and infrared modes to be linearly aligned. This results in severe data feature overlap and feature annihilation phenomena in feature extraction channels under sudden changes in illumination or complex heat source interference, reducing the accuracy of feature representation and increasing the feature space mapping error and recognition bias of subsequent classification layers. Summary of the Invention

[0004] The purpose of this invention is to address the shortcomings of existing technologies by proposing a multimodal visual data fusion processing method and system.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: a multimodal visual data fusion processing method, comprising the following steps: Spatial information of the target scene is collected by cameras and infrared detection equipment, and pixel gradient matrix and temperature mapping matrix are extracted. S2: Perform feature domain alignment calculation on the pixel gradient matrix and the temperature mapping matrix, extract the cross covariance value of the pixel gradient matrix and the temperature mapping matrix in the local receptive field, perform normalization calculation based on the cross covariance value, and generate a cross-modal association weight vector. S3: Perform a dot product operation on the pixel gradient matrix and the temperature mapping matrix based on the cross-modal correlation weight vector, and perform tensor channel splicing and spatial dimension mapping combination calculation based on the dot product result to construct a nonlinear feature reconstruction tensor; S4: Input the nonlinear feature reconstruction tensor into the visible light classifier and the infrared classifier respectively to extract the probability boundary values, aggregate the probability boundary values, construct the confidence distribution vector, calculate the relative entropy mapping value of the confidence distribution vector, and generate the dynamic penalty term coefficient. S5: Perform bias reduction calculation on the initial probability matrix of the decision layer according to the dynamic penalty term coefficient, obtain the matrix calculation difference, adjust the classification decision probability according to the matrix calculation difference, and output the classification decision matrix.

[0006] As a further aspect of the present invention, step S1 specifically comprises: S11: A continuous frame visible light image sequence of the observation environment is acquired in real time by a multi-view image acquisition device deployed in a specific location. The brightness change intensity features and corresponding edge direction distribution features of each pixel in the continuous frame visible light image sequence in the horizontal and vertical coordinate axes are extracted frame by frame. These features are then converted into a set of discrete structural features with spatial coordinate attributes according to the original image resolution size to generate a pixel gradient matrix. S12: Drive the infrared thermal imaging sensor, which is at the same physical observation angle as the multi-view image acquisition device, to collect wavelength thermal radiation energy distribution data of the corresponding environmental area, record the absolute physical temperature scalar value corresponding to each discrete detection coordinate point in the wavelength thermal radiation energy distribution data, aggregate the absolute physical temperature scalar value and combine it with the scene spatial depth information to perform bilinear interpolation smoothing processing based on spatial two-dimensional grid division, and obtain the temperature mapping matrix.

[0007] As a further aspect of the present invention, step S2 specifically comprises: S21: Obtain the pixel gradient matrix and the temperature mapping matrix, set a sliding region traversal window with the same row and column span size as the analysis area, extract the center anchor point position features and neighborhood boundary contour features of each data element in the analysis area during the movement of the sliding region traversal window, and establish a local receptive field. S22: Calculate the mean variance of the internal elements of the pixel gradient matrix and the mean variance of the internal elements of the temperature mapping matrix within the area covered by the local receptive field. Combine the mean variance of the internal elements to obtain the cumulative amount of cooperative change deviation of the corresponding elements of the two at the same spatial position, and extract the cross covariance value. S23: Calculate the maximum and minimum deviation extreme values ​​of the entire set extracted during the sliding process of the local receptive field in the whole image region, and use the maximum and minimum deviation extreme values ​​to perform a linear scaling operation on the cross covariance value to constrain it to the standard positive floating point range, thereby generating a cross-modal association weight vector.

[0008] As a further aspect of the present invention, step S3 specifically comprises: S31: Obtain the cross-modal correlation weight vector, project each component in the cross-modal correlation weight vector onto the corresponding coordinate nodes of the pixel gradient matrix and the temperature mapping matrix according to the spatial position correspondence, perform element-level corresponding multiplication operation, and obtain the weighted gradient feature matrix and the weighted temperature feature matrix. S32: Read the depth level attribute parameters of the weighted gradient feature matrix and the weighted temperature feature matrix, perform stacking alignment and data stream merging of the weighted gradient feature matrix and the weighted temperature feature matrix along the depth level direction, and perform tensor channel splicing operation and spatial dimension mapping combination calculation operation; S33: Using a nonlinear activation function, perform an element-wise nonlinear response mapping operation on the fused high-dimensional data block generated by the tensor channel splicing operation and the spatial dimension mapping combination calculation operation, filter out negative redundant noise features in the fused high-dimensional data block and retain positive saliency features, and establish a nonlinear feature reconstruction tensor.

[0009] As a further aspect of the present invention, step S4 specifically comprises: S41: Obtain the nonlinear feature reconstruction tensor, input the nonlinear feature reconstruction tensor into the visible light feature extraction network and the infrared feature extraction network based on the multilayer perceptron architecture respectively, calculate the logistic regression exponential ratio of the output feature response value in each category label dimension, and extract the probability boundary value; S42: Align, stack, and sum the probability boundary values ​​output by the visible light feature extraction network and the infrared feature extraction network according to the preset category label arrangement order, integrate the distribution of decision opinions under different modalities, and establish a confidence distribution vector; S43: Read the ideal probability reference vector corresponding to the benchmark uniform distribution reference model, calculate the cumulative difference in information content between the confidence distribution vector and the ideal probability reference vector during the information transmission process, output the relative entropy mapping value, extract the scaling penalty multiplier according to the exponential decay law of the relative entropy mapping value, and generate the dynamic penalty term coefficient.

[0010] As a further aspect of the present invention, step S5 specifically comprises: S51: Obtain the initial probability matrix pre-configured by the decision layer and the coefficient of the dynamic penalty term, apply the coefficient of the dynamic penalty term as a bias adjustment multiplication factor to the row and column elements corresponding to the category with lower confidence in the initial probability matrix to perform a numerical reduction operation, and obtain the matrix calculation difference; S52: Extract the state results before and after performing the numerical reduction operation, compare the fluctuation range of the element values ​​corresponding to the same category position one by one, obtain the combination of absolute change deviation value sequence, and update the matrix to calculate the difference; S53: Based on the matrix, calculate the difference and perform probability correction compensation calculation on the target candidate category that is in the decision edge state. Select the category label conclusion corresponding to the element index with the largest value after probability correction compensation calculation, and establish a classification decision matrix.

[0011] As a further aspect of the present invention, the process of extracting the cross covariance value specifically includes: Obtain the mean variance of the internal elements of the pixel gradient matrix and the mean variance of the internal elements of the temperature mapping matrix, and perform a centering elimination operation on the expected value of the product of the corresponding elements of the pixel gradient matrix and the temperature mapping matrix in combination with the mean variance of the internal elements. Read the deviation multiplication cumulative sum parameter obtained after the centering elimination operation, and use the spatial normalization factor containing the constraint of the total number of pixels in the region to perform an arithmetic mean segmentation operation on the deviation multiplication cumulative sum parameter to calculate the mean deviation parameter of the region that eliminates the influence of layout size. The mean deviation parameter of the region is input into a preset hyperbolic tangent nonlinear mapping operator for extreme value smoothing transformation to suppress outlier noise peak response caused by sudden environmental changes and output cross covariance value. The spatial normalization factor is specifically a reciprocal scaling multiplier, preset based on the physical pixel size of the local receptive field, used to balance the differences in the number of pixels within receptive fields of different sizes.

[0012] As a further aspect of the present invention, the process of constructing the nonlinear feature reconstruction tensor specifically includes: Obtain the fused high-dimensional data block generated by performing the tensor channel splicing operation and the spatial dimension mapping combined calculation operation, set the global dynamic pooling receptive field parameter, and use the global dynamic pooling receptive field parameter to perform extreme value filtering operation and mean aggregation operation on the fused high-dimensional data block on each independent channel plane to obtain the channel descriptor information vector; Based on the channel descriptor information vector, a differentiated weight assignment mapping operation is performed on the slice matrices of different depths in the fused high-dimensional data block to suppress the response intensity of shallow texture slice matrices containing redundant background interference information and enhance the response intensity of deep contour slice matrices containing core target semantic information. A sinusoidal wave function parameter containing absolute position encoding information is added along the spatial coordinate dimension of the fused high-dimensional data block. This parameter is then fully connected and fused with the intermediate feature data volume after the differential weight assignment mapping operation to construct a nonlinear feature reconstruction tensor. The global dynamic pooling receptive field parameters are specifically a set of sliding window step size and size dynamically calculated based on the spatial resolution of the fused high-dimensional data block to limit the coverage of the extreme value filtering operation.

[0013] As a further aspect of the present invention, the process of generating the dynamic penalty term coefficient specifically includes: Obtain the expected base probability values ​​of each category in the ideal probability reference vector and the actual judgment probability values ​​of the corresponding categories in the confidence distribution vector. Perform a quotient logarithmic transformation operation on the actual judgment probability values ​​and the expected base probability values ​​to obtain the category bias information parameters. The category bias information parameter and the actual judgment probability value are multiplied together, and the results of the multiplication and combination operations corresponding to all categories are summed globally to output the relative entropy mapping value. Based on the comparison between the relative entropy mapping value and the preset system tolerance upper limit threshold, a nonlinear inverse proportional reduction function operation is applied to the part exceeding the system tolerance upper limit threshold to perform mapping compression transformation, generating a dynamic penalty term coefficient.

[0014] A multimodal visual data fusion processing system is provided, the system being used to implement the above-described multimodal visual data fusion processing method, the system comprising: The target scene acquisition module collects spatial information of the target scene through a camera and an infrared detection device, and extracts the pixel gradient matrix and temperature mapping matrix. The feature domain association module performs feature domain alignment calculation on the pixel gradient matrix and the temperature mapping matrix, extracts the cross-covariance value of the pixel gradient matrix and the temperature mapping matrix in the local receptive field, performs normalization calculation based on the cross-covariance value, and generates a cross-modal association weight vector. The tensor reconstruction module performs a dot product operation on the pixel gradient matrix and the temperature mapping matrix based on the cross-modal correlation weight vector, and performs tensor channel splicing and spatial dimension mapping combination calculation based on the dot product result to construct a nonlinear feature reconstruction tensor. The penalty term generation module inputs the nonlinear feature reconstruction tensor into the visible light classifier and the infrared classifier respectively to extract probability boundary values, aggregates the probability boundary values, constructs a confidence distribution vector, calculates the relative entropy mapping value of the confidence distribution vector, and generates dynamic penalty term coefficients. The decision output module performs bias reduction calculation on the initial probability matrix of the decision layer according to the dynamic penalty term coefficient, obtains the matrix calculation difference, adjusts the classification decision probability according to the matrix calculation difference, and outputs the classification decision matrix.

[0015] Compared with the prior art, the advantages and positive effects of the present invention are as follows: In this invention, the pixel gradient matrix of a visible light image and the temperature mapping matrix of an infrared signal are obtained. The cross-covariance value within the local receptive field is calculated and a cross-modal correlation weight vector is generated. Based on the cross-modal correlation weight vector, channel weighting and feature reconstruction are performed on heterogeneous data to construct a single-modal confidence distribution vector. The relative entropy value between corresponding vectors is extracted as a dynamic penalty term coefficient. The output probability matrix of the decision layer is biased and adjusted according to the dynamic penalty term coefficient to suppress modal interference noise caused by environmental abrupt changes. This solves the feature overlap defect caused by forced linear alignment in the traditional pixel-level stitching mode and reduces spatial mapping error. Attached Figure Description

[0016] Figure 1 This is a flowchart illustrating the overall process of multimodal visual data fusion processing in this invention. Figure 2 This is a flowchart of the multimodal data acquisition and feature matrix generation process of this invention; Figure 3 This is a flowchart of the feature domain alignment and cross-modal weight generation process of the present invention; Figure 4 This is a flowchart of the nonlinear feature reconstruction tensor construction process of the present invention; Figure 5 This is a flowchart of the probability aggregation and dynamic penalty coefficient generation process of the present invention; Figure 6 This is a flowchart of the matrix decision bias adjustment and correction output of the present invention. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this invention clearer, the software-based technical solution is described in detail below with reference to system architecture diagrams and embodiments. It should be understood that the specific embodiments described herein are only for explaining the technical solutions of this invention and do not constitute a limitation on the scope of protection.

[0018] In the description of this invention, the system architecture relationships or data processing flows indicated by terms such as "layer," "module," "interface," "data flow," "client," and "server" are all defined based on the architecture diagram or flowchart corresponding to the embodiments. This way of describing is only used to clearly illustrate the logical relationships between the elements in the technical solution, and not to limit the physical deployment form. The term "multiple" includes two or more technical units, including but not limited to multiple data nodes, processing threads, service instances, or functional components and other scalable elements. The specific number is determined according to the actual business scenario and needs to be specifically specified.

[0019] Please see Figure 1 and Figure 2 This invention provides a technical solution: a multimodal visual data fusion processing method, comprising the following steps: S1: Collect spatial information of the target scene through cameras and infrared detection equipment, and extract pixel gradient matrix and temperature mapping matrix.

[0020] The specific steps of S1 are as follows: S11: Real-time acquisition of continuous frame visible light image sequences of the observation environment through multi-view image acquisition equipment deployed in a specific orientation; extraction of brightness change intensity features and corresponding edge direction distribution features of each pixel in the continuous frame visible light image sequence in the horizontal and vertical coordinate axes; conversion of these features into a set of discrete structural features with spatial coordinate attributes according to the original image resolution size; generation of pixel gradient matrix. S12: Drive the infrared thermal imaging sensor, which is at the same physical observation angle as the multi-view image acquisition device, to collect wavelength thermal radiation energy distribution data of the corresponding environmental area, record the absolute physical temperature scalar value corresponding to each discrete detection coordinate point in the wavelength thermal radiation energy distribution data, aggregate the absolute physical temperature scalar value and combine it with the scene spatial depth information to perform bilinear interpolation smoothing processing based on spatial two-dimensional grid division, and obtain the temperature mapping matrix.

[0021] A high-definition visible light camera, deployed 3.5 meters above the monitoring area, is activated. This camera uses a 1 / 2.8-inch complementary metal-oxide-semiconductor (CMOS) sensor and is set to a sampling frequency of 30 frames per second to acquire a continuous sequence of visible light images with a resolution of 1920 x 1080 pixels, stored in 24-bit true color format. Simultaneously, a long-wave infrared thermal imaging sensor, mounted on the same high-precision 3D pan-tilt base as the high-definition visible light camera, is activated. This sensor uses an uncooled focal plane array detector, with a detection wavelength range locked between 8 and 14 micrometers and a native physical resolution of 640 x 480 pixels. During the data acquisition phase, a synchronization pulse is sent to both acquisition devices via a hardware clock synchronization trigger to ensure that the time axis deviation between the visible light and infrared image frames remains within 2 milliseconds.

[0022] For each frame of the acquired visible light image, a luminance component extraction operation is performed. The red, green, and blue channel pixel values ​​are weighted and fused. The calculation logic is the sum of the red channel value multiplied by 0.299, the green channel value multiplied by 0.587, and the blue channel value multiplied by 0.114, thus generating a single-channel grayscale image. Subsequently, a convolutional scanning process is performed on the grayscale image using horizontal and vertical probe operators with a size of 3 x 3 pixels. The weight matrix of the horizontal probe operator is set with the left column being -1, the middle column being 0, and the right column being 1. The weight matrix of the vertical probe operator is set with the top row being -1, the middle row being 0, and the bottom row being 1. This process obtains the luminance abrupt change intensity features along the horizontal and vertical axes by weighted summation of the luminance values ​​in the neighborhood of each pixel. Taking coordinate point 100,100 as an example, if its horizontal brightness abrupt change intensity feature is 45 and its vertical brightness abrupt change intensity feature is 30, then the edge intensity of this point is calculated using the square root of the sum of the squares of the two features, yielding a value of 54.08. Simultaneously, the arctangent function is calculated using the ratio of the vertical to the horizontal feature values, resulting in an edge direction distribution feature of 33.69 degrees. The brightness abrupt change intensity features of all pixels are arranged according to the original resolution of 1920 multiplied by 1080 to form a pixel gradient matrix.

[0023] In the infrared data processing branch, the infrared thermal imaging sensor captures the wavelength thermal radiation energy distribution data of objects in the target scene in real time. Its internal analog-to-digital conversion circuit converts the radiation energy into a 14-bit digital signal value. Using a pre-calibrated blackbody radiation reference curve, the digital signal of each discrete detection coordinate point is converted into an absolute physical temperature scalar value representing the true temperature of the object's surface. For example, when the ambient temperature is 25 degrees Celsius, the absolute physical temperature scalar value of the detected human target's central region is 36.8 degrees Celsius. Since the physical resolution of the infrared detector is lower than that of the visible light camera, this process requires a bilinear interpolation smoothing operation based on spatial two-dimensional grid division. First, based on the difference in field of view and relative position parameters between the visible light and infrared sensors, a virtual mapping grid is established in a 1920 x 1080 spatial coordinate system, mapping the original temperature points of 640 x 480 to their corresponding positions. For blank coordinate points between the grids, four adjacent physical detection points in the original infrared image are selected, and two linear interpolation operations are performed based on the horizontal and vertical distance weights between the target point and these four points. For example, if the target point is located in the center surrounded by four known temperature points, the infrared temperature data is upsampled to a scale of 1920 multiplied by 1080 by calculating the arithmetic mean of the temperature values ​​of these four points, and finally aggregated to form a temperature mapping matrix aligned with the pixel gradient matrix space.

[0024] Table 1 lists the core sensor parameters and initial data characteristics involved in step S1.

[0025] Table 1. Sensor Acquisition Parameters and Initial Data Characteristics; As shown in Table 1, hardware synchronization and spatial resampling ensured the consistency of the two modal data in the spatiotemporal dimensions.

[0026] The aforementioned visible light high-definition camera refers to an imaging device that uses photoelectric technology to convert visible light image signals into electrical signals.

[0027] The aforementioned long-wave infrared thermal imaging sensor refers to a detection device capable of detecting thermal radiation in the wavelength range of 8 micrometers to 14 micrometers and converting it into a temperature distribution image.

[0028] The bilinear interpolation smoothing operation mentioned above refers to a mathematical operation method that achieves image magnification by weighted averaging of the values ​​of four adjacent points in a two-dimensional space.

[0029] Please see Figure 1 and Figure 3 S2: Perform feature domain alignment calculation on the obtained pixel gradient matrix and temperature mapping matrix, extract the cross covariance values ​​of the pixel gradient matrix and temperature mapping matrix in the local receptive field, perform normalization calculation based on the cross covariance values, and generate cross-modal association weight vector.

[0030] The specific steps of S2 are as follows: S21: Obtain the pixel gradient matrix and temperature mapping matrix, set a sliding region traversal window with the same row and column span size as the analysis area, extract the center anchor point position features and neighborhood boundary contour features of each data element in the analysis area during the movement of the sliding region traversal window, and establish a local receptive field. S22: Calculate the mean variance of the internal elements of the pixel gradient matrix and the mean variance of the internal elements of the temperature mapping matrix within the area covered by the local receptive field. Combine the mean variance of the internal elements to obtain the cumulative amount of cooperative change deviation of the corresponding elements of the two at the same spatial position, and extract the cross covariance value. The process of extracting the cross covariance values ​​specifically includes: Obtain the mean variance of the internal elements of the pixel gradient matrix and the mean variance of the internal elements of the temperature mapping matrix, and perform a centering elimination operation on the expected value of the product of the corresponding elements of the pixel gradient matrix and the temperature mapping matrix by combining the mean variance of the internal elements. Read the accumulated sum of deviations obtained after the centering elimination operation, and use the spatial normalization factor containing the total number of pixels in the region to perform an arithmetic mean segmentation operation on the accumulated sum of deviations to calculate the mean deviation parameter of the region that eliminates the influence of layout size. The mean deviation parameter of the region is input into the preset hyperbolic tangent nonlinear mapping operator for extreme value smoothing transformation to suppress outlier noise peak response caused by sudden environmental changes and output cross covariance value. The spatial normalization factor is specifically a reciprocal scaling multiplier, preset based on the physical pixel size of the local receptive field, used to balance the differences in the number of pixels within receptive fields of different sizes. S23: Statistically extract the maximum and minimum deviation extreme values ​​of the entire set of local receptive fields during the sliding process of the whole image region, and use the maximum and minimum deviation extreme values ​​to perform linear scaling operations on the cross covariance values ​​to constrain them to the standard positive floating-point range, thereby generating a cross-modal association weight vector.

[0031] A pixel gradient matrix and a temperature mapping matrix of size 1920 x 1080 are extracted from the memory buffer. A sliding region traversal window with a span of 15 x 15 pixels is defined, and this window moves across the matrix plane in 1-pixel increments, covering the entire area. When the sliding region traversal window stops at a specific coordinate position, the data elements of 225 pixels within the window are read. The coordinates of the geometric center of this region are extracted as the center anchor point position feature, and the coordinates of the four vertices of the window edge are recorded as the neighborhood boundary contour feature, thereby establishing the local receptive field for the current position. Within the 225 sampling points covered by the local receptive field, the elements in the pixel gradient matrix are summed and divided by 225 to obtain the mean variance of the internal elements of the pixel gradient matrix. Similarly, the mean variance of the internal elements of the temperature mapping matrix within this region is calculated.

[0032] In the process of extracting cross-covariance values, a centering elimination operation is first performed on each corresponding element within the local receptive field. Taking an element in the pixel gradient matrix as an example, its original value is subtracted from the mean variance of the internal elements in that region to obtain the gradient deviation value; simultaneously, the temperature scalar value at the corresponding location is subtracted from the mean temperature region to obtain the temperature deviation value. Subsequently, these two deviation values ​​are multiplied, and all 225 deviation product results within the receptive field are accumulated to obtain the deviation product accumulation sum parameter. To eliminate the influence of the physical pixel size of the local receptive field on the numerical magnitude, a spatial normalization factor is introduced, which is set to 1 divided by 225, i.e., 0.00444. The deviation product accumulation sum parameter is multiplied by this spatial normalization factor to calculate the region mean deviation parameter that eliminates the influence of layout size. If there is a pedestrian edge in the current analysis region, due to the synchronous increase of visible light gradient and infrared temperature, its region mean deviation parameter will show significant positive fluctuations, for example, a value reaching 85.6.

[0033] To mitigate outlier noise peaks caused by sudden environmental changes, such as strong reflections from metallic objects or interference from localized heat sources, a region-meaning deviation parameter is input into a preset hyperbolic tangent nonlinear mapping operator. This operator maps the input deviation parameter to a convergence interval between -1 and +1, outputting the final cross-covariance value. When the region-meaning deviation parameter is 85.6, the cross-covariance value after processing by the operator stabilizes at approximately 0.992. After traversing all 2,073,600 pixels of the image in the sliding region traversal window, the maximum deviation extreme value (e.g., 0.998) and the minimum deviation extreme value (e.g., -0.452) are calculated for the entire set. These two extreme values ​​are then used to perform a linear scaling operation on each cross-covariance value. The logic is to subtract the minimum deviation extreme value from the current value and then divide by the difference between the maximum and minimum deviation extreme values. This operation constrains the originally highly volatile covariance data to a standard positive floating-point range of 0.0 to 1.0, ultimately generating a cross-modal association weight vector with the same size as the original image.

[0034] The advantage of this computational logic lies in its ability to accurately measure the synergy between visible light and infrared features in spatial distribution by calculating the cross-covariance within the local receptive field. This allows for the automatic enhancement of common features and suppression of single-mode noise during subsequent fusion. Experimental results show that, in complex scenes with uneven illumination, the mutual information of feature alignment is improved by 14.2% compared to traditional methods.

[0035] The pixel gradient matrix mentioned above refers to a two-dimensional data matrix composed of the intensity values ​​of brightness abrupt changes of each pixel in the image arranged according to their spatial positions.

[0036] The aforementioned local receptive field refers to the local area of ​​the original image that can be perceived by the current layer neurons or analysis window during image processing.

[0037] The hyperbolic tangent nonlinear mapping operator mentioned above refers to a nonlinear transformation function that maps input data to the interval from negative 1 to positive 1 using the hyperbolic tangent function.

[0038] Please see Figure 1 and Figure 4 S3: Perform dot product operation on the pixel gradient matrix and temperature mapping matrix based on the extracted cross-modal correlation weight vector, and perform tensor channel splicing and spatial dimension mapping combination calculation based on the dot product result to construct a nonlinear feature reconstruction tensor.

[0039] The specific steps for S3 are as follows: S31: Obtain the cross-modal correlation weight vector, project each component in the cross-modal correlation weight vector to the corresponding coordinate nodes of the pixel gradient matrix and the temperature mapping matrix according to the spatial position correspondence, perform element-level corresponding multiplication operation, and obtain the weighted gradient feature matrix and the weighted temperature feature matrix. S32: Read the depth-level attribute parameters of the weighted gradient feature matrix and the weighted temperature feature matrix, perform stacking alignment and data stream merging of the weighted gradient feature matrix and the weighted temperature feature matrix along the depth-level direction, and perform tensor channel splicing operation and spatial dimension mapping combination calculation operation; S33: Using a nonlinear activation function, perform element-wise nonlinear response mapping processing on the fused high-dimensional data block generated by the combined calculation operation of tensor channel splicing and spatial dimension mapping, filter out negative redundant noise features in the fused high-dimensional data block and retain positive saliency features, and establish a nonlinear feature reconstruction tensor. The process of constructing a nonlinear feature reconstruction tensor specifically includes: Obtain the fused high-dimensional data block generated by the combined calculation operation of tensor channel splicing and spatial dimension mapping, set the global dynamic pooling receptive field parameter, and use the global dynamic pooling receptive field parameter to perform extreme value filtering and mean aggregation operations on each independent channel plane of the fused high-dimensional data block to obtain the channel descriptor information vector. Based on the channel descriptor information vector, a differentiated weight assignment mapping operation is performed on the slice matrices of different depths in the fused high-dimensional data block to suppress the response intensity of shallow texture slice matrices containing redundant background interference information and enhance the response intensity of deep contour slice matrices containing core target semantic information. A sinusoidal wave function parameter containing absolute position encoding information is added along the spatial coordinate dimension of the fused high-dimensional data block. This parameter is then fully connected and fused with the intermediate feature data volume after the differential weight assignment mapping operation to construct a nonlinear feature reconstruction tensor. The global dynamic pooling receptive field parameters are specifically a set of sliding window step sizes and dimensions that are dynamically calculated based on the spatial resolution of the fused high-dimensional data blocks to limit the coverage of the extreme value filtering operation.

[0040] Read the cross-modal association weight vector generated in the previous stage. The weight component of each coordinate point in this vector represents the confidence level of the feature at that location in multimodal fusion. Obtain each component in the cross-modal association weight vector and project them onto the corresponding coordinate nodes of the pixel gradient matrix and temperature mapping matrix according to their spatial correspondence, performing element-wise multiplication. Taking coordinate point 200,200 as an example, if the weight component of this point is 0.92, the value of the pixel gradient matrix at this point is 15, and the value of the temperature mapping matrix at this point is 37.0, then the value of the weighted gradient feature matrix at this point is 13.8, and the value of the weighted temperature feature matrix at this point is 34.04. After traversing the entire image, obtain the complete weighted gradient feature matrix and weighted temperature feature matrix.

[0041] The depth-level attribute parameters of the weighted gradient feature matrix and the weighted temperature feature matrix are read. Currently, each matrix has a depth of one channel. The weighted gradient feature matrix and the weighted temperature feature matrix are stacked, aligned, and merged along the depth-level direction. The merged tensor depth is expanded to two channels, with a size of 1920 x 1080 x 2. After performing tensor channel concatenation and spatial dimension mapping operations, a nonlinear activation function is used, with a modified linear unit function selected, to perform element-wise nonlinear response mapping on the fused high-dimensional data block. The logic of this operation is: if the input value is greater than 0, the original value is retained; if the input value is less than or equal to 0, it is forcibly set to 0. This mapping process can filter out negative redundant noise features in the fused high-dimensional data block and retain positive saliency features.

[0042] The process of constructing a nonlinear feature reconstruction tensor specifically includes: acquiring a fused high-dimensional data block generated by performing a combination of tensor channel concatenation and spatial dimension mapping operations; setting a global dynamic pooling receptive field parameter, which is set to a sliding window step size of 32 pixels and a size of 32 x 32 pixels; using the global dynamic pooling receptive field parameter to perform extreme value filtering and mean aggregation operations on each independent channel plane of the fused high-dimensional data block; for example, within the 32 x 32 block of the gradient feature channel, selecting the maximum edge intensity of 125 and calculating the average brightness of 82, thereby obtaining the channel descriptor information vector; and performing differentiated weight assignment mapping operations on the slice matrices of different depths in the fused high-dimensional data block based on the channel descriptor information vector. For the shallow texture slice matrix containing redundant background interference information, i.e., the first channel, a weight coefficient of 0.35 is assigned; for the deep contour slice matrix containing core target semantic information, i.e., the second channel, a weight coefficient of 0.65 is assigned, and the shallow response is suppressed and the deep features are enhanced through product operations.

[0043] Along the spatial coordinate dimensions of the fused high-dimensional data block, namely the horizontal and vertical directions, a sinusoidal wave function parameter containing absolute position encoding information is added. The calculation logic for this parameter involves substituting the horizontal and vertical coordinate values ​​of the coordinate points into sine and cosine functions of different frequencies, with the position encoding fundamental frequency set to 0.0001. The encoded position information is then fused with the intermediate feature data volume after differential weight assignment and mapping operations using a fully connected layer consisting of 512 neurons, ultimately constructing a nonlinear feature reconstruction tensor with rich spatial semantic information.

[0044] Table 2 shows the data distribution of the feature reconstruction process in step S3 in detail.

[0045] Table 2. Data distribution and weight assignment during the feature reconstruction stage; As shown in Table 2, by dynamically allocating weights and using nonlinear activation, the reconstructed tensor significantly improves the saliency of the target features. The advantage of this operational logic is that it introduces absolute position encoding, solving the localization loss problem caused by spatial invariance in traditional depthwise convolution. Experimental results show that in moving target recognition experiments, the feature recall rate is improved by 11.5% compared to the single-channel linear fusion method.

[0046] The aforementioned modified linear unit function refers to a nonlinear activation function whose calculation logic is to take the maximum value between the input value and the original value.

[0047] The aforementioned channel descriptor information vector refers to a vector used to characterize the feature strength of each channel and the global statistical properties in a multi-channel tensor.

[0048] The aforementioned fully connected mapping fusion computation refers to a neural network computation method that fully connects all nodes in the input layer to all nodes in the output layer and assigns different weights for weighted summation.

[0049] Please see Figure 1 and Figure 5 S4: Input the nonlinear feature reconstruction tensor into the visible light classifier and the infrared classifier respectively to extract the probability boundary values, aggregate the probability boundary values, construct the confidence distribution vector, calculate the relative entropy mapping value of the confidence distribution vector, and generate the dynamic penalty term coefficient.

[0050] The specific steps for S4 are as follows: S41: Obtain the nonlinear feature reconstruction tensor, input the nonlinear feature reconstruction tensor into the visible light feature extraction network and the infrared feature extraction network based on the multilayer perceptron architecture respectively, calculate the logistic regression exponential ratio of the output feature response values ​​in each category label dimension, and extract the probability boundary values. S42: Align, stack, and sum the probability boundary values ​​output by the visible light feature extraction network and the infrared feature extraction network according to the preset category label arrangement order, integrate the distribution of decision opinions under different modalities, and establish a confidence distribution vector; S43: Read the ideal probability reference vector corresponding to the benchmark uniform distribution reference model, calculate the cumulative amount of information difference between the confidence distribution vector and the ideal probability reference vector in the information transmission process, output the relative entropy mapping value, extract the scaling penalty multiplier according to the exponential decay law of the relative entropy mapping value, and generate the dynamic penalty term coefficient. The process of generating the dynamic penalty term coefficients specifically includes: Obtain the expected base probability values ​​of each category in the ideal probability reference vector and the actual judgment probability values ​​of the corresponding category in the confidence distribution vector. Perform a quotient logarithmic transformation operation on the actual judgment probability values ​​and the expected base probability values ​​to obtain the category bias information parameters. The category bias information parameter is multiplied and combined with the actual judgment probability value. The results of the multiplication and combination operation corresponding to all categories are summed globally, and the relative entropy mapping value is output. The system compares the relative entropy mapping value with the preset system tolerance upper limit threshold. For the part that exceeds the system tolerance upper limit threshold, a nonlinear inverse proportional reciprocal reduction function is applied to perform mapping compression transformation, generating a dynamic penalty term coefficient.

[0051] A nonlinear feature reconstruction tensor is obtained and input into a visible light feature extraction network and an infrared feature extraction network based on a multilayer perceptron architecture, respectively. The visible light feature extraction network consists of one input layer, three hidden layers, and one output layer. The input layer receives the feature vector of the reconstruction tensor. The first hidden layer has 512 neurons using a linear rectified activation function; the second hidden layer has 256 neurons; and the third hidden layer has 128 neurons. The output layer is set to 10 classification nodes, corresponding to different target category labels, such as vehicles, pedestrians, and cyclists. The network calculates the logistic regression exponential ratio of the output feature response values ​​in each category label dimension through forward propagation. That is, it performs an exponential operation with the natural logarithm as the base and performs summation and normalization across all categories to extract the probability boundary value. For example, in the visible light branch, the probability boundary value for determining the current target as a pedestrian is 0.88.

[0052] The infrared feature extraction network performs the same logical architecture operations, but its weight parameters are pre-optimized using an infrared feature sample set. Its output probability boundary values ​​may be affected by thermal radiation intensity; for example, the pedestrian determination probability boundary value for the same target in the infrared branch is 0.72. Following a preset category label arrangement, the probability boundary values ​​output by the visible light feature extraction network and the infrared feature extraction network are aligned, stacked, and averaged to integrate the decision opinion distributions under different modalities, establishing a confidence distribution vector containing 10 probability components.

[0053] To quantify the determinism of multimodal decision-making, the ideal probability reference vector corresponding to the baseline uniform distribution reference model is read. This vector represents a completely information-free random distribution state, with a base probability expectation of 0.1 for each category. The cumulative difference in information content between the confidence distribution vector and the ideal probability reference vector during information transmission is calculated. Specifically, the base probability expectation values ​​for each category in the ideal probability reference vector and the actual decision probability values ​​for the corresponding categories in the confidence distribution vector are obtained. The actual decision probability values ​​and the base probability expectation values ​​are then converted to logarithmic values ​​to obtain the category bias information content parameter. Subsequently, the category bias information content parameter is multiplied and combined with the actual decision probability values. The results for all 10 categories are then globally summed to output the relative entropy mapping value. The formula for calculating relative entropy is shown in Equation 1. ; in, Represents the numerical value of relative entropy mapping. The first element in the confidence distribution vector represents the... The actual probability values ​​for each category. The first element in the ideal probability reference vector The expected base probability values ​​for each category, with the summation sign associated with a letter identifier. The index number representing the category label, with values ​​ranging from 1 to 10.

[0054] The decision is made by comparing the relative entropy mapping value with a preset system tolerance upper limit threshold of 2.5. If the relative entropy mapping value is 1.8, which is lower than the system tolerance upper limit threshold, a scaling penalty multiplier is directly extracted based on its exponential decay law to generate a dynamic penalty term coefficient. The calculation logic is the natural constant e raised to the power of -1.8, resulting in a value of 0.165. For the portion exceeding the system tolerance upper limit threshold, a nonlinear inverse proportional reduction function operation is applied for mapping compression transformation to cope with extreme decision conflict scenarios and ensure that the generated dynamic penalty term coefficient has physical feedback significance.

[0055] The advantage of this process is that it uses relative entropy to quantify information inconsistencies between modes. When two modes severely conflict, the system can automatically identify low-reliability decisions. Experimental results show that in multi-target overlapping occlusion scenarios, the stability variance of classification decisions is reduced by 21.4% compared to the simple fixed-weight fusion method.

[0056] The aforementioned multilayer perceptron architecture refers to a fully connected feedforward neural network consisting of an input layer, multiple hidden layers, and an output layer.

[0057] The aforementioned relative entropy mapping value refers to a mathematical index that measures the degree of difference between two probability distributions, also known as the Kleiber-Leibler divergence in information theory.

[0058] The aforementioned exponential decay law refers to the law that a physical quantity decreases rapidly in an exponential proportion as the independent variable increases.

[0059] Please see Figure 1 and Figure 6 S5: Perform bias reduction calculation on the initial probability matrix of the decision layer based on the calculated dynamic penalty term coefficient, obtain the matrix calculation difference, adjust the classification decision probability based on the matrix calculation difference, and output the classification decision matrix.

[0060] The specific steps of S5 are as follows: S51: Obtain the initial probability matrix and dynamic penalty term coefficients pre-configured by the decision layer, apply the dynamic penalty term coefficients as bias adjustment multiplication factors to the row and column elements corresponding to the categories with lower confidence in the initial probability matrix, perform numerical reduction operations, and obtain the matrix calculation difference. S52: Extract the state results before and after performing the numerical reduction operation, compare the fluctuation range of the element values ​​corresponding to the same category position one by one, obtain the combination of absolute change deviation value sequence, and update the matrix to calculate the difference; S53: Based on the matrix, calculate the difference and perform probability correction and compensation calculation on the target candidate category that is in the decision edge state. Select the category label conclusion corresponding to the element index with the largest value after probability correction and compensation calculation, and establish a classification decision matrix.

[0061] Obtain the pre-configured 10x1 initial probability matrix of the decision layer and the coefficient of the dynamic penalty term generated in the previous stage, assuming the dynamic penalty term coefficient is 0.165. Use the dynamic penalty term coefficient as a bias adjustment multiplication factor, applying it to the corresponding row and column elements of the category with lower confidence in the initial probability matrix to perform a numerical reduction operation. Set the decision threshold to 0.2. If the original probability value of a category in the initial probability matrix is ​​lower than 0.2, for example, the original probability value of the background category is 0.08, then multiply 0.08 by 0.165 to obtain the reduced value of 0.0132. Obtain the preliminary matrix difference by subtracting the reduced value from the original values ​​of all low-confidence categories.

[0062] Extract the state results before and after the numerical reduction operation, and compare the fluctuation range of the element values ​​corresponding to the same category position one by one. Taking the background category as an example, its fluctuation range is 0.08 minus 0.0132, which equals 0.0668. Calculate the combination of absolute change deviation values ​​for all categories, accumulate the fluctuation range of all categories, and update the matrix to calculate the difference to reflect the overall probability shift. Subsequently, based on the difference calculated by the matrix, perform probability correction compensation calculations for target candidate categories in the decision-edge state. Set the decision-edge interval to 0.4 to 0.6. If the original probability value of the pedestrian's class is 0.52, and it is in the decision-edge state, then half of the previously calculated accumulated fluctuation range value for all categories, for example 0.12, is compensated to that category, and the corrected probability value is updated to 0.64.

[0063] The system selects the category label corresponding to the element index with the largest value after probability correction and compensation calculation. In this example, the corrected pedestrian category probability is 0.64, higher than all other categories. Therefore, the system selects pedestrian as the final judgment and establishes a classification decision matrix. Through this dynamic adjustment mechanism, the system can strengthen the dominant features in marginal states by penalizing low-confidence components when there is noise in intermodal information.

[0064] Table 3 shows a comparison of the probability distributions of each target category before and after the decision correction in step S5.

[0065] Table 3 Comparison of classification probability distributions before and after decision correction; As shown in Table 3, by penalizing low-probability terms and allocating energy to dominant terms, the robustness of decision-making is greatly enhanced. The advantage of this operational logic lies in its simulation of the human visual process of focusing and correcting for blurred targets. Through self-calibration against uncertainty, it significantly improves the system's discriminative power under harsh conditions. Experimental results show that in a test set during heavy rain and fog, the system's overall classification accuracy increased from 82.5% to 93.8%, demonstrating superior technological advancement.

[0066] The aforementioned decision-making layer refers to the level in a multimodal fusion system that performs the final classification or judgment logic processing based on all feature information extracted from the front end.

[0067] The aforementioned initial probability matrix refers to the vector matrix composed of the original class scores output by the classifier without post-processing correction.

[0068] The aforementioned classification decision matrix refers to the output matrix that, after multiple rounds of logical correction, includes the final classification result index and the corresponding confidence component.

[0069] A multimodal visual data fusion processing system is provided, which is used to execute the above-described multimodal visual data fusion processing method. The system includes: The target scene acquisition module collects spatial information of the target scene through a camera and infrared detection equipment, and extracts the pixel gradient matrix and temperature mapping matrix. The feature domain association module performs feature domain alignment calculation on the acquired pixel gradient matrix and temperature mapping matrix, extracts the cross-covariance values ​​of the pixel gradient matrix and temperature mapping matrix in the local receptive field, performs normalization calculation based on the cross-covariance values, and generates a cross-modal association weight vector. The tensor reconstruction module performs a dot product operation on the pixel gradient matrix and temperature mapping matrix based on the extracted cross-modal correlation weight vector, and performs tensor channel splicing and spatial dimension mapping combination calculation based on the dot product result to construct a nonlinear feature reconstruction tensor. The penalty term generation module inputs the nonlinear feature reconstruction tensor into the visible light classifier and the infrared classifier respectively to extract probability boundary values, aggregates the probability boundary values, constructs a confidence distribution vector, calculates the relative entropy mapping value of the confidence distribution vector, and generates dynamic penalty term coefficients. The decision output module performs bias reduction calculation on the initial probability matrix of the decision layer based on the calculated dynamic penalty term coefficient, obtains the matrix calculation difference, adjusts the classification decision probability based on the matrix calculation difference, and outputs the classification decision matrix.

[0070] The above embodiments illustrate preferred embodiments of the present invention. Any equivalent adjustments to the technical solution based on software engineering methods are within the scope of protection, including but not limited to: implementing algorithm logic using different programming languages, refactoring functional modules into services, adjusting data interaction protocols, and optimizing resource scheduling strategies. Any implementation scheme derived from reasonable modifications to the data processing flow, service call chain, or system architecture layer without departing from the core technology of the present invention should be considered within the protection scope defined by the technical solution of the present invention.

Claims

1. A method for multimodal visual data fusion processing, characterized in that, Includes the following steps: S1: Collect spatial information of the target scene through cameras and infrared detection equipment, and extract pixel gradient matrix and temperature mapping matrix; S2: Perform feature domain alignment calculation on the pixel gradient matrix and the temperature mapping matrix, extract the cross covariance value of the pixel gradient matrix and the temperature mapping matrix in the local receptive field, perform normalization calculation based on the cross covariance value, and generate a cross-modal association weight vector. S3: Perform a dot product operation on the pixel gradient matrix and the temperature mapping matrix based on the cross-modal correlation weight vector, and perform tensor channel splicing and spatial dimension mapping combination calculation based on the dot product result to construct a nonlinear feature reconstruction tensor; S4: Input the nonlinear feature reconstruction tensor into the visible light classifier and the infrared classifier respectively to extract the probability boundary values, aggregate the probability boundary values, construct the confidence distribution vector, calculate the relative entropy mapping value of the confidence distribution vector, and generate the dynamic penalty term coefficient. S5: Perform bias reduction calculation on the initial probability matrix of the decision layer according to the dynamic penalty term coefficient, obtain the matrix calculation difference, adjust the classification decision probability according to the matrix calculation difference, and output the classification decision matrix.

2. The multimodal visual data fusion processing method according to claim 1, characterized in that, The specific steps of S1 are as follows: S11: A continuous frame visible light image sequence of the observation environment is acquired in real time by a multi-view image acquisition device deployed in a specific location. The brightness change intensity features and corresponding edge direction distribution features of each pixel in the continuous frame visible light image sequence in the horizontal and vertical coordinate axes are extracted frame by frame. These features are then converted into a set of discrete structural features with spatial coordinate attributes according to the original image resolution size to generate a pixel gradient matrix. S12: Drive the infrared thermal imaging sensor, which is at the same physical observation angle as the multi-view image acquisition device, to collect wavelength thermal radiation energy distribution data of the corresponding environmental area, record the absolute physical temperature scalar value corresponding to each discrete detection coordinate point in the wavelength thermal radiation energy distribution data, aggregate the absolute physical temperature scalar value and combine it with the scene spatial depth information to perform bilinear interpolation smoothing processing based on spatial two-dimensional grid division, and obtain the temperature mapping matrix.

3. The multimodal visual data fusion processing method according to claim 1, characterized in that, The specific steps of S2 are as follows: S21: Obtain the pixel gradient matrix and the temperature mapping matrix, set a sliding region traversal window with the same row and column span size as the analysis area, extract the center anchor point position features and neighborhood boundary contour features of each data element in the analysis area during the movement of the sliding region traversal window, and establish a local receptive field. S22: Calculate the mean variance of the internal elements of the pixel gradient matrix and the mean variance of the internal elements of the temperature mapping matrix within the area covered by the local receptive field. Combine the mean variance of the internal elements to obtain the cumulative amount of cooperative change deviation of the corresponding elements of the two at the same spatial position, and extract the cross covariance value. S23: Calculate the maximum and minimum deviation extreme values ​​of the entire set extracted during the sliding process of the local receptive field in the whole image region, and use the maximum and minimum deviation extreme values ​​to perform a linear scaling operation on the cross covariance value to constrain it to the standard positive floating point range, thereby generating a cross-modal association weight vector.

4. The multimodal visual data fusion processing method according to claim 1, characterized in that, The specific steps in S3 are as follows: S31: Obtain the cross-modal correlation weight vector, project each component in the cross-modal correlation weight vector onto the corresponding coordinate nodes of the pixel gradient matrix and the temperature mapping matrix according to the spatial position correspondence, perform element-level corresponding multiplication operation, and obtain the weighted gradient feature matrix and the weighted temperature feature matrix. S32: Read the depth level attribute parameters of the weighted gradient feature matrix and the weighted temperature feature matrix, perform stacking alignment and data stream merging of the weighted gradient feature matrix and the weighted temperature feature matrix along the depth level direction, and perform tensor channel splicing operation and spatial dimension mapping combination calculation operation; S33: Using a nonlinear activation function, perform an element-wise nonlinear response mapping operation on the fused high-dimensional data block generated by the tensor channel splicing operation and the spatial dimension mapping combination calculation operation, filter out negative redundant noise features in the fused high-dimensional data block and retain positive saliency features, and establish a nonlinear feature reconstruction tensor.

5. The multimodal visual data fusion processing method according to claim 1, characterized in that, The specific steps of S4 are as follows: S41: Obtain the nonlinear feature reconstruction tensor, input the nonlinear feature reconstruction tensor into the visible light feature extraction network and the infrared feature extraction network based on the multilayer perceptron architecture respectively, calculate the logistic regression exponential ratio of the output feature response value in each category label dimension, and extract the probability boundary value; S42: Align, stack, and sum the probability boundary values ​​output by the visible light feature extraction network and the infrared feature extraction network according to the preset category label arrangement order, integrate the distribution of decision opinions under different modalities, and establish a confidence distribution vector; S43: Read the ideal probability reference vector corresponding to the benchmark uniform distribution reference model, calculate the cumulative difference in information content between the confidence distribution vector and the ideal probability reference vector during the information transmission process, output the relative entropy mapping value, extract the scaling penalty multiplier according to the exponential decay law of the relative entropy mapping value, and generate the dynamic penalty term coefficient.

6. The multimodal visual data fusion processing method according to claim 1, characterized in that, The specific steps of S5 are as follows: S51: Obtain the initial probability matrix pre-configured by the decision layer and the coefficient of the dynamic penalty term, apply the coefficient of the dynamic penalty term as a bias adjustment multiplication factor to the row and column elements corresponding to the category with lower confidence in the initial probability matrix to perform a numerical reduction operation, and obtain the matrix calculation difference; S52: Extract the state results before and after performing the numerical reduction operation, compare the fluctuation range of the element values ​​corresponding to the same category position one by one, obtain the combination of absolute change deviation value sequence, and update the matrix to calculate the difference; S53: Based on the matrix, calculate the difference and perform probability correction compensation calculation on the target candidate category that is in the decision edge state. Select the category label conclusion corresponding to the element index with the largest value after probability correction compensation calculation, and establish a classification decision matrix.

7. The multimodal visual data fusion processing method according to claim 3, characterized in that, The process of extracting the cross covariance values ​​specifically includes: Obtain the mean variance of the internal elements of the pixel gradient matrix and the mean variance of the internal elements of the temperature mapping matrix, and perform a centering elimination operation on the expected value of the product of the corresponding elements of the pixel gradient matrix and the temperature mapping matrix in combination with the mean variance of the internal elements. Read the deviation multiplication cumulative sum parameter obtained after the centering elimination operation, and use the spatial normalization factor containing the constraint of the total number of pixels in the region to perform an arithmetic mean segmentation operation on the deviation multiplication cumulative sum parameter to calculate the mean deviation parameter of the region that eliminates the influence of layout size. The mean deviation parameter of the region is input into a preset hyperbolic tangent nonlinear mapping operator for extreme value smoothing transformation to suppress outlier noise peak response caused by sudden environmental changes and output cross covariance value. The spatial normalization factor is specifically a reciprocal scaling multiplier, preset based on the physical pixel size of the local receptive field, used to balance the differences in the number of pixels within receptive fields of different sizes.

8. The multimodal visual data fusion processing method according to claim 4, characterized in that, The process of constructing the nonlinear feature reconstruction tensor specifically includes: Obtain the fused high-dimensional data block generated by performing the tensor channel splicing operation and the spatial dimension mapping combined calculation operation, set the global dynamic pooling receptive field parameter, and use the global dynamic pooling receptive field parameter to perform extreme value filtering operation and mean aggregation operation on the fused high-dimensional data block on each independent channel plane to obtain the channel descriptor information vector; Based on the channel descriptor information vector, a differentiated weight assignment mapping operation is performed on the slice matrices of different depths in the fused high-dimensional data block to suppress the response intensity of shallow texture slice matrices containing redundant background interference information and enhance the response intensity of deep contour slice matrices containing core target semantic information. A sinusoidal wave function parameter containing absolute position encoding information is added along the spatial coordinate dimension of the fused high-dimensional data block. This parameter is then fully connected and fused with the intermediate feature data volume after the differential weight assignment mapping operation to construct a nonlinear feature reconstruction tensor. The global dynamic pooling receptive field parameters are specifically a set of sliding window step size and size dynamically calculated based on the spatial resolution of the fused high-dimensional data block to limit the coverage of the extreme value filtering operation.

9. The multimodal visual data fusion processing method according to claim 5, characterized in that, The process of generating the dynamic penalty term coefficient specifically includes: Obtain the expected base probability values ​​of each category in the ideal probability reference vector and the actual judgment probability values ​​of the corresponding categories in the confidence distribution vector. Perform a quotient logarithmic transformation operation on the actual judgment probability values ​​and the expected base probability values ​​to obtain the category bias information parameters. The category bias information parameter and the actual judgment probability value are multiplied together, and the results of the multiplication and combination operations corresponding to all categories are summed globally to output the relative entropy mapping value. Based on the comparison between the relative entropy mapping value and the preset system tolerance upper limit threshold, a nonlinear inverse proportional reduction function operation is applied to the part exceeding the system tolerance upper limit threshold to perform mapping compression transformation, generating a dynamic penalty term coefficient.

10. A multimodal visual data fusion processing system, characterized in that, The system is used to implement the multimodal visual data fusion processing method according to any one of claims 1-9, and the system includes: The target scene acquisition module collects spatial information of the target scene through a camera and infrared detection equipment, and extracts the pixel gradient matrix and temperature mapping matrix. The feature domain association module performs feature domain alignment calculation on the pixel gradient matrix and the temperature mapping matrix, extracts the cross-covariance value of the pixel gradient matrix and the temperature mapping matrix in the local receptive field, performs normalization calculation based on the cross-covariance value, and generates a cross-modal association weight vector. The tensor reconstruction module performs a dot product operation on the pixel gradient matrix and the temperature mapping matrix based on the cross-modal correlation weight vector, and performs tensor channel splicing and spatial dimension mapping combination calculation based on the dot product result to construct a nonlinear feature reconstruction tensor. The penalty term generation module inputs the nonlinear feature reconstruction tensor into the visible light classifier and the infrared classifier respectively to extract probability boundary values, aggregates the probability boundary values, constructs a confidence distribution vector, calculates the relative entropy mapping value of the confidence distribution vector, and generates dynamic penalty term coefficients. The decision output module performs bias reduction calculation on the initial probability matrix of the decision layer according to the dynamic penalty term coefficient, obtains the matrix calculation difference, adjusts the classification decision probability according to the matrix calculation difference, and outputs the classification decision matrix.

Images