Terahertz imaging and visible light image fusion detection method

Abstract

The present application relates to the technical field of image fusion, in particular to a terahertz imaging and visible light image fusion detection method, which obtains terahertz and visible light images and divides the images into blocks, reads the gray scale sum value and marks the source, extracts the block path and filters out non-continuous items, extracts the direction consistent path to identify continuous moving blocks, analyzes the source distribution and compares the spatial offset, selects the offset slow area mode to extract features for fusion processing, and forms a dominant image fusion content set. The present application marks the image source through the block gray scale sum value, establishes the structure correspondence relationship between images, enhances the content traceability, combs the block path based on the time sequence and excludes non-continuous items, strengthens the continuation expression of the area change, extracts the path direction consistency, improves the recognition ability of the block dynamic direction, combines the source classification to compare the spatial offset, selects the offset slow area as the fusion reference, and enhances the matching degree of the block structure coordination and information fusion.

Description

Technical Field

[0001] This invention relates to the field of image fusion technology, and in particular to a method for detecting the fusion of terahertz imaging and visible light images. Background Technology

[0002] Image fusion technology involves the collaborative processing of multi-source image information, with core processes including image preprocessing, registration, fusion, and reconstruction. By integrating information from different imaging modalities, it improves the expressive power and recognition efficiency of images, and is commonly used in scenarios such as security monitoring, medical diagnosis, and remote sensing analysis. Fusion methods cover infrared and visible light, multispectral and hyperspectral, as well as deep learning-based image fusion methods.

[0003] The traditional terahertz imaging and visible light image fusion detection method utilizes the complementary information between terahertz and visible light images to achieve target recognition through steps such as image registration, fusion, and detection. Specifically, it includes first performing image distortion correction and feature extraction, then achieving image registration through corner matching and contour alignment; in the fusion stage, key features are extracted using wavelet transform and principal component analysis to complete image fusion; and finally, target detection and recognition are achieved through image segmentation and edge detection.

[0004] Traditional image fusion processes the entire image as a unit, lacking a response mechanism to dynamic differences between regions and failing to characterize temporal changes. The registration process heavily relies on feature point matching, exhibiting weak adaptability to inter-modal deformation and resolution differences, easily leading to positioning misalignment. Furthermore, the fusion process ignores inter-modal stability differences and fails to establish effective fusion criteria, resulting in unclear regional overlap and blurred structural contours. In scenes with significant image deformation or drastic dynamic changes, this negatively impacts the accuracy and interpretability of image fusion. Summary of the Invention

[0005] To address the technical problems existing in the prior art, embodiments of the present invention provide a method for fusion detection of terahertz imaging and visible light images. The technical solution is as follows: The terahertz imaging and visible light image fusion detection method includes the following steps: S1: Acquire images from the terahertz imaging mode and visible light imaging mode during the flight mission, divide the images into equal-area patches, read the pixel grayscale values ​​within each patch sequentially and sum them, use the patch positions as association points, complete the annotation of image source information, and obtain the set of grayscale sum values ​​and patch positions corresponding to each other. S2: Call the patch data in the continuous image frames corresponding to the gray sum value and patch position, extract the spatial position of each frame patch according to time, sort out the continuous patch path, remove interrupted patches, and connect the remaining patch positions as the change trajectory to obtain the patch gray centroid path sequence. S3: Based on the path direction in the gray-scale centroid path sequence of the blocks, extract the movement trend between blocks along the path direction line, filter out paths that maintain consistency in direction changes, identify continuous movement of blocks, extract block data with a clear movement direction, and obtain a continuous block path data group. S4: Call the image source of the tile in the continuous tile path data group, divide the tile into two groups: terahertz and visible light, check the position distribution of the two groups in the image, compare the spatial offset performance between the sources, and obtain the regional offset tendency information between image modes. S5: Based on the modality with a relatively gentle shift in the inter-modal region shift tendency information of the image, the modality image is imported into the fusion processing stage. Corresponding block features are introduced according to the block structure. The block features are centrally processed and organized into the content on which the image fusion operation is based, thus obtaining the dominant image fusion content set.

[0006] As a further embodiment of the present invention, the gray-level summation value and the tile position correspondence set includes tile spatial positioning parameters, tile gray-level total index, and image modality category label; the tile gray-level centroid path sequence includes path continuity identifier, tile gray-level center distribution, and tile time series information; the continuous tile path data set includes stable movement direction, path duration, and tile sequence number; the image modality inter-regional offset tendency information includes modality source distribution range, offset direction attribute, and inter-modality relative displacement value; and the dominant image fusion content set includes tile feature extraction results, fusion reference structure, and dominant modality image benchmark.

[0007] As a further aspect of the present invention, the step of obtaining the set of corresponding grayscale sum values ​​and tile positions is as follows: S101: Based on the terahertz imaging images and visible light imaging images acquired during the flight mission, each image is divided into multiple patches according to a unified area standard. Boundary positioning information is extracted in the horizontal and vertical directions of the image, and pixel regions are divided according to the spacing to ensure that the pixel range covered by each patch is consistent, thus obtaining the patch pixel region distribution set. S102: Based on the pixel range of each region in the pixel region distribution set of the image blocks, extract the grayscale content of the corresponding region from the image item by item, read all pixel grayscale values ​​in sequence and perform accumulation processing, complete the grayscale calculation in the region block by block, and obtain the cumulative combination of image block grayscale values; S103: Call the image coordinate information of the corresponding region and the accumulated gray level combination of the image patch, extract the modal content of the original image, add the image source category on the basis of the coordinates, complete the pairing operation between the region position, image category and gray level calculation, and obtain the image patch gray level value and modal position information set.

[0008] As a further aspect of the present invention, the step of obtaining the grayscale centroid path sequence of the image patch is as follows: S201: Call the data of the corresponding block region of each image frame in the block gray value and modal position information set, extract the region position in each frame in chronological order, connect the coordinate information of the same block index in each frame along the time axis to form a linear path, and obtain the block temporal movement path line segment set; S202: Based on whether there are missing tile positions in each path of the tile time-series moving path segment set within consecutive frames, extract the tile path index of the frame sequence interruption, filter out broken paths according to the continuity requirement, and retain the remaining uninterrupted paths to the next stage to obtain a continuous area moving path set. S203: Based on the position index of the corresponding map block in each frame of the continuous area movement path set, the coordinates of the same map block in each frame are sequentially connected in time order to extract the position change trend, and the map block is arranged in the order of the frame marked path structure to obtain the region gray-scale centroid change sequence.

[0009] As a further aspect of the present invention, the step of obtaining the continuous tile path data group is as follows: S301: Based on the gray-scale centroid path sequence of the image blocks, extract the gray-scale centroid coordinates of the image blocks in the path, calculate the direction change of the corresponding direction lines, filter out paths with discontinuous direction changes, and obtain a path angle sequence with consistent direction. S302: Based on the tile number in the direction-consistent path angle sequence and the relative position of the tile in the image, calculate the movement trend along the path direction, extract continuous movement path segments, and obtain the continuous path displacement change trend. S303: Based on the directional information in the displacement change trend of the continuous path, extract the block sequence with consistent direction, and import the corresponding path number and coordinate content to obtain the continuous block path data group.

[0010] As a further aspect of the present invention, the step of obtaining the image inter-modal region offset tendency information is as follows: S401: Call the image source information of the tiles involved in the continuous tile path data group, determine the type of the tile according to the naming identifier of the original image corresponding to the tile, divide the tile into two categories: terahertz source and visible light source, and obtain the distribution information of tile source type; S402: Based on the classification results in the distribution information of the source type of the image patch, extract the coordinate position of each type of image patch in the original image, and count the coverage area and relative position of each type of image patch in the image space to obtain the spatial location distribution characteristics of the image patch. S403: Based on the relative position data in the spatial distribution characteristics of the image blocks, calculate the coordinate offset relationship between the terahertz image blocks and the visible light image blocks, classify the offset directions in each region, and obtain the regional offset tendency information between image modes.

[0011] As a further aspect of the present invention, the step of obtaining the dominant image fusion content set is as follows: S501: Based on the slow offset behavior in the inter-modal region offset tendency information of the image, extract the complete image content of the corresponding modal image, and combine it with the original tile path range to determine the tile index and corresponding region in the image that should participate in subsequent processing, and obtain the imported image tile index information; S502: Call the tile position in the imported image tile index information, extract the edge shape, gray scale distribution and texture structure information of the corresponding tile in the original image, and organize the various features in the tile according to the position order to obtain the tile feature combination structure; S503: Based on the various block feature contents in the block feature combination structure, allocate them to the input channels in the image fusion processing flow, arrange them according to the block position and structural order, summarize them to form the basic data content of the fusion processing, and obtain the dominant image fusion content set.

[0012] The beneficial effects of the technical solutions provided in the embodiments of the present invention include at least the following: In this invention, the source of an image is labeled by the sum of the grayscale values ​​of image patches, establishing structural correspondences between multimodal images and improving the traceability of image content. Patch paths are analyzed according to chronological order, eliminating discontinuous items and highlighting the continuity of regional changes in the image sequence. Path direction consistency is extracted to enhance the ability to determine the direction of dynamic patch distribution. By combining source classification with spatial positional offset comparison, slowly shifting regions are selected as fusion references, enhancing the structural coordination and information fusion matching degree between patches in the fused image. Attached Figure Description

[0013] Figure 1 This is a flowchart of the method of the present invention; Figure 2 This is a flowchart illustrating the process of obtaining the set of grayscale sum values ​​and corresponding tile positions in this invention. Figure 3 This is a flowchart illustrating the process of obtaining the grayscale centroid path sequence of the image blocks in this invention. Figure 4 This is a flowchart illustrating the process of obtaining the continuous tile path data group of the present invention. Figure 5 This is a flowchart illustrating the process of obtaining inter-modal region offset tendency information in images according to the present invention. Figure 6 This is a flowchart illustrating the process of acquiring the dominant image fusion content set in this invention. Detailed Implementation

[0014] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0015] refer to Figures 1 to 6 The terahertz imaging and visible light image fusion detection method includes the following steps: S1: Acquire images from the terahertz imaging mode and visible light imaging mode during the flight mission, divide the entire image into equal-area patches, read the grayscale values ​​of the pixels in the patches one by one, sum the read values, use the corresponding positions of the patches as association points, complete the annotation of the image source information of each patch, and obtain the set of grayscale sum values ​​and patch positions. S2: Call the tile data in the continuous image frames corresponding to the gray-level sum value and tile position, extract the spatial position of the tile in each frame in time sequence, sort out the continuous tile paths, remove the tiles that are not displayed continuously, and string the remaining tile positions together into a change trajectory to obtain the tile gray-level centroid path sequence. S3: Based on the path direction in the gray-scale centroid path sequence of the tiles, extract the relative movement trend between tiles along each path direction line, filter out paths with consistent direction changes, identify cases of continuous tile movement, extract and retain tile data with a clear direction of movement, and obtain a continuous tile path data group. S4: Call the image source of the tile involved in the continuous tile path data group, divide the tile into two groups: terahertz source and visible light source, check the distribution of the corresponding positions of the two groups in the original image, compare the offset performance of the two sources in the spatial plane, and obtain the regional offset tendency information between image modes. S5: Based on the modalities showing relatively gentle shifts in the inter-modal region shift tendency information, the entire modal image is imported into the image fusion processing stage. Corresponding patch features are introduced according to the patch structure in the image. The patch features are then processed centrally and organized into the content on which the image fusion operation is based, resulting in the dominant image fusion content set.

[0016] The set of grayscale sum values ​​corresponding to tile positions includes tile spatial positioning parameters, total grayscale index of tiles, and image modality category label. The tile grayscale centroid path sequence includes path continuity identifier, tile grayscale center distribution, and tile time series information. The continuous tile path data set includes stable movement direction, path duration, and tile sequence number. The image modality inter-regional offset tendency information includes modality source distribution range, offset direction attribute, and relative displacement value between modalities. The dominant image fusion content set includes tile feature extraction results, fusion reference structure, and dominant modality image baseline.

[0017] Please see Figure 2The steps to obtain the set of corresponding grayscale sum values ​​and tile positions are as follows: S101: Based on the terahertz imaging images and visible light imaging images acquired during the flight mission, each image is divided into multiple patches according to a unified area standard. Boundary positioning information is extracted in the horizontal and vertical directions of the image, and pixel regions are divided according to the spacing to ensure that the pixel range covered by each patch is consistent, thus obtaining the patch pixel region distribution set. The processor retrieves the terahertz imaging data stream and visible light imaging data stream from the flight mission memory, defining these two sets of image data in different frequency bands as the first modal image source and the second modal image source, respectively. It then initiates an image preprocessing program, resampling the pixel matrices of both images to a uniform 2048x2048 size using bilinear interpolation according to a unified area standard, while simultaneously quantizing the pixel bit depth to a standard grayscale format. Based on this, the processor performs spatial segmentation according to preset grid partitioning instructions, setting both the horizontal and vertical segmentation steps to 64 pixels, and establishing a precise grid indexing system in the image coordinate system. This system starts from the origin and cuts the image at equal intervals along the horizontal and vertical directions, dividing the entire image into 1024 independent rectangular segments, assigning a unique index identifier to each segment for subsequent data addressing. Taking the tile with index 50 as an example, the processor calculates its specific pixel coverage range using a grid coordinate mapping algorithm, determining that the tile's position in the image matrix lies within a rectangular interval of horizontal coordinates from 1088 to 1151 and vertical coordinates from 64 to 127. A region consistency verification program is then initiated, traversing all 1024 tile indices and checking the total number of pixels actually covered by each tile to verify that it remains constant at 4096 pixels, aiming to eliminate potentially incomplete regions at image edges. After confirming that the geometric dimensions of all retained regions are highly consistent, the processor binds and stores the index of each tile with its corresponding set of pixel coordinate ranges, constructing a tile pixel region distribution set containing complete spatial positioning information.

[0018] S102: Based on the pixel range of each region in the pixel distribution set of the image patch, extract the grayscale content of the corresponding region from the image item by item, read all pixel grayscale values ​​in sequence and perform accumulation processing, complete the grayscale calculation within the region block by block, and obtain the cumulative combination of image patch grayscale values; Based on the pixel coordinate range of each region determined by the pixel distribution set of the image patches, the processor activates the grayscale feature extraction engine to extract the grayscale content of the corresponding region from the terahertz image and the visible light image item by item. This process adopts a row-first scanning method. The processor sequentially reads the grayscale values ​​of all pixels in the specified region matrix and sends the read values ​​to the accumulation register to perform numerical accumulation processing to obtain the overall radiation energy or reflection intensity characterization of the region. For the image patch with index 50, the processor reads the grayscale values ​​of 4096 pixels in the terahertz mode and completes the accumulation operation. Since the region contains high emissivity targets sensitive to the terahertz band, the calculated grayscale accumulation value reaches 614400. At the same time, the processor performs the same accumulation operation on the region at the same coordinate position in the visible light mode. Due to the limitations of lighting conditions and material reflection characteristics, the calculated grayscale accumulation value is 450560. Following this standardized computational logic, the processor sequentially calculates the total grayscale value within each of the 1024 patches in the entire image, generating a list containing the total grayscale energy of all patches under both modes. This list comprehensively records the energy response characteristics of each local region in the image under different imaging mechanisms. By associating these values ​​with the corresponding patch indices, the cumulative grayscale combination of the patches is obtained.

[0019] S103: Call the image coordinate information of the corresponding region and the cumulative gray-level combination of the patch, extract the modal content of the original image, add the image source category on the basis of the coordinates, complete the pairing operation between the region location, image category and gray-level calculation, and obtain the set of gray-level sum value and patch location correspondence.

[0020] The processor calls upon the accumulated grayscale values ​​of image tiles and the corresponding image coordinates of the regions to perform multidimensional data association and encapsulation operations. First, based on the metadata header information of the original image data, the processor identifies the image source to which each accumulated grayscale value belongs and adds a clear image source category label, identifying it as either terahertz type or visible light type, to distinguish different modal data attributes. Based on the tile's grid index, a reverse mapping calculation is performed, converting the tile's vertex coordinates into geometric center coordinates to achieve a single-point description of the region's location. For example, for a tile with index 50, the processor calculates its geometric center to be located at x-coordinate 1120 and y-coordinate 96. Based on this, the processor combines the calculated accumulated grayscale values, geometric center coordinates, modal category labels, and tile index identifiers to construct a standardized data unit. This data unit logically binds the originally scattered energy, spatial, and category information, achieving a precise pairing between region location, image category, and grayscale calculation results. The processor performs this operation on all tiles, generating a list containing detailed attributes of all tiles, thus obtaining a set of tile grayscale values ​​and modal location information.

[0021] Please see Figure 3 The steps to obtain the grayscale centroid path sequence of the image patch are as follows: S201: Call the data of the corresponding tile region in each image frame of the tile grayscale value and modal position information set, extract the region position in each frame in chronological order, connect the coordinate information of the same tile index in each frame along the time axis to form a linear path, and obtain the tile temporal movement path line segment set; The system retrieves data from the corresponding tile regions in each image frame from the tile grayscale value and modal position information set, and initiates the temporal feature extraction program. The system retrieves 10 consecutive frames of image data acquired within a set time period. In each frame, it sequentially extracts the grayscale distribution data of a specific tile region in chronological order, and calculates the grayscale centroid coordinates of that region within each frame using a weighted average algorithm. Due to the slight relative motion that may exist between the imaging target or platform, the grayscale centroid exhibits dynamic changes between consecutive frames. Taking a tile with index 50 as an example, the processor calculates that its centroid coordinates are located at x-coordinate 1120.5 and y-coordinate 95.2 in the first frame. As time progresses, it gradually moves in subsequent frames until reaching a position of x-coordinate 1124.0 and y-coordinate 98.5 in the tenth frame. The processor sequentially connects the coordinate information of the same tile index in each frame along the time axis to construct a linear path describing the trajectory of the tile's positional change over time. By performing this time-series tracking operation on all tiles, the system transforms static spatial location information into dynamic trajectory data, thereby obtaining a set of time-series movement path segments for the tiles.

[0022] S202: Based on whether there are missing tile positions in each path of the tile time-series moving path line segment set within consecutive frames, extract the tile path index of the frame sequence interruption, filter out broken paths according to the continuity requirement, and retain the remaining uninterrupted paths to the next stage to obtain the continuous area moving path set. Based on the data status of each path in the time-series movement path segment set, the processor executes path continuity verification logic to eliminate paths with incomplete data caused by external interference. The processor iterates through the coordinate data of each path within consecutive frames, checking for missing or empty data due to occlusion, sensor blind spots, or data transmission packet loss. If a path fails to extract valid centroid coordinates in any frame, the processor immediately extracts the path's index, marks it as a broken path, and removes it from the dataset. Only when a path has complete and valid coordinate data in all consecutive frames is it considered a valid path and retained. For example, in the initially acquired 1024 paths, after rigorous continuity verification, the processor identifies and removes some broken paths, retaining 950 continuous, unbroken regional movement paths on the timeline. The processor saves these filtered unbroken paths and their corresponding complete coordinate sequences to the next processing stage, thus obtaining a set of continuous regional movement paths.

[0023] S203: Based on the position index of the corresponding tile in each frame of each path in the continuous area movement path set, the coordinates of the same tile in each frame are sequentially connected in time order to extract the position change trend, and the tiles are arranged in the order of the frame marked path structure to obtain the region gray-scale centroid change sequence.

[0024] Based on the position index of the corresponding tile in each frame of the continuous region movement path set, the processor concatenates the coordinates of the same tile in each frame in a temporal sequence and initiates a motion trend quantification analysis program. The processor calculates the displacement vector between two adjacent frames in the path and measures the average movement speed of the entire path by accumulating the displacement magnitude and dividing by the time interval. Taking the tile with index 50 as an example, the processor calculates the displacement components between adjacent frames based on its coordinate sequence and then measures its average movement speed to be 2.405 pixels per frame. The processor compares this calculation result with a preset static noise threshold. Since 2.405 pixels is significantly higher than the preset 0.5 pixel threshold, the system determines that the tile exhibits obvious motion characteristics and marks it as a "dynamic feature block"; conversely, if the speed is lower than the threshold, it is marked as a "background block". Based on this, the processor arranges the frame-marked path structure in order and encapsulates the position coordinate sequence, displacement vector sequence, and dynamic attribute markers to obtain the region grayscale centroid change sequence.

[0025] Please see Figure 4 The steps for obtaining continuous tile path data groups are as follows: S301: Based on the gray-scale centroid path sequence of the map, extract the gray-scale centroid coordinates of the map in the path, calculate the direction change of the corresponding direction line, filter out paths with discontinuous direction changes, and obtain the direction-consistent path angle sequence. Based on the grayscale centroid path sequence of the map tiles, the processor further extracts the grayscale centroid coordinates of the tiles in the path and initiates a directional stability analysis program to calculate the change in the direction of the corresponding direction lines. For each path, the processor calculates the change in the angle between adjacent displacement vectors to assess the consistency of the object's motion direction. For example, for a specific path, the processor calculates that its direction angle change during continuous motion is only 3.18 degrees. The system compares this value with a preset direction continuity threshold of 30 degrees. Since the change is much smaller than the threshold, it indicates that the path maintains a highly stable direction during motion, without drastic backtracking or random jitter, consistent with the motion patterns of real objects. Conversely, if the direction change of a path exceeds the threshold, it is considered an invalid path caused by noise interference and is discarded. The processor filters all paths according to this standard, removing paths with discontinuous direction changes and retaining only valid paths with consistent directions, thus obtaining a sequence of directionally consistent path angles.

[0026] S302: Based on the tile number in the direction-consistent path angle sequence and the relative position of the tile in the image, calculate the movement trend along the path direction, extract continuous movement path segments, and obtain the continuous path displacement change trend. Based on the tile numbers in the directional consistency path angle sequence and the relative positions of the tiles in the image, the processor calculates the movement trend along the path direction and extracts continuous movement path segments. In this step, the processor uses the least squares method to linearly fit all coordinate points in the path, constructs a motion trend line, and calculates the variance of the Euclidean distance from all actual path points to this trend line to quantify the smoothness and linearity of the path. Taking a retained path as an example, the processor calculates that the variance of the distance between its path points and the fitted line is 1.2. The system compares this variance value with a preset linearity threshold of 5.0. Since 1.2 is significantly less than 5.0, it indicates that the path's movement trajectory is very smooth and has a clear directionality, conforming to the physical characteristics of continuous movement. Therefore, it is determined to be a valid path and retained. For messy paths with variances exceeding the threshold, the system considers them invalid interference and discards them. Through this process, the processor filters out a set of paths with clear displacement trends and obtains the displacement change trend of continuous paths.

[0027] S303: Based on the directional information in the displacement change trend of the continuous path, extract the block sequence with consistent direction, import the corresponding path number and coordinate content, and obtain the continuous block path data group.

[0028] Based on the directional information in the displacement trend of continuous paths, the processor performs path clustering and filtering operations to extract tile sequences with consistent orientation. The processor first extracts the total displacement vector of each retained path and assigns it to the corresponding motion quadrant according to the vector direction. It counts the number of paths in each direction and identifies the dominant direction cluster containing the most paths, for example, determining that most tiles are moving towards a specific quadrant. The processor imports and packages all tile path numbers belonging to this dominant direction cluster along with their corresponding full-time coordinate sequences, while removing outlier paths whose orientation is inconsistent with the dominant trend to eliminate the impact of local abnormal motion on overall registration. The filtered path numbers and their corresponding coordinate contents are standardized and integrated to construct a dataset with high directional consistency and motion continuity, resulting in a continuous tile path data set.

[0029] Please see Figure 5 The steps for obtaining inter-modal region offset tendency information in images are as follows: S401: Call the image source information of the tiles involved in the continuous tile path data group, determine the type of the tile based on the naming identifier of the original image corresponding to the tile, divide the tile into two categories: terahertz source and visible light source, and obtain the distribution information of tile source type. The system retrieves image source information for tiles from the continuous tile path data set. Based on the naming identifier of the original image corresponding to each tile, it initiates a source classification program to determine the type of each tile. The system divides all tiles into two main categories: terahertz sources and visible light sources, and counts the number of valid tiles in each set. The terahertz set contains 400 valid path tiles, while the visible light set contains 420. To find a registration reference, the processor further compares the tile indices in these two sets, identifying common tiles with valid paths in both modes. 380 common tile indices existing in both sets are extracted; these tiles represent the same physical region successfully detected and tracked in both imaging modes. The processor then obtains the tile source type distribution information based on this.

[0030] S402: Based on the classification results in the distribution information of tile source type, extract the coordinate position of each type of tile in the original image, and count the coverage area and relative position of each type of tile in the image space to obtain the spatial location distribution characteristics of the tiles. Based on the classification results in the tile source type distribution information, the processor extracts the corresponding coordinate positions of each type of tile in the original image and calculates the coverage area and relative positions of each type of tile in the image space. Specifically, the processor calculates the average centroid positions of these 380 common tiles in the terahertz image sequence and the visible light image sequence, respectively. For example, for a specific common tile, the processor measures its average centroid in the terahertz image to be at x-coordinate 1130 and y-coordinate 106, while in the visible light image, the average centroid is located at x-coordinate 1135 and y-coordinate 109. The processor further calculates the spatial bounding boxes of the two sets of points and determines the approximate overlapping area of ​​the two images by comparing the overlap range of the bounding boxes. This process transforms the discrete tile coordinates into macroscopic spatial distribution features, obtaining the spatial location distribution features of the tiles.

[0031] S403: Based on the relative position data in the spatial distribution characteristics of the image patches, calculate the coordinate offset relationship between the terahertz image patches and the visible light image patches, classify the offset directions in each region, and obtain the regional offset tendency information between image modes.

[0032] Based on the relative position data in the spatial distribution characteristics of image tiles, the processor performs detailed calculations on the coordinate offset relationship between terahertz tiles and visible light tiles, and categorizes the offset directions in each region. The processor calculates the coordinate difference vector for each common tile in both modes and performs statistical averaging analysis on the difference vectors of all common tiles. The calculations show that the global average offset vector is 4.8 pixels horizontally and 2.9 pixels vertically. Simultaneously, the processor calculates the standard deviation of the offset and finds it to be less than 1.0, indicating that the offset has a high degree of consistency across the entire image, belonging to a systematic parallax offset rather than a local random deformation. The processor establishes this statistical result as the regional offset tendency information between image modalities.

[0033] Please see Figure 6 The steps for obtaining the dominant image fusion content set are as follows: S501: Based on the slow offset behavior in the inter-modal region offset tendency information of the image, extract the complete image content of the corresponding modal image, and combine it with the original tile path range to determine the tile index and corresponding region in the image that should participate in subsequent processing, and obtain the imported image tile index information. Based on the slow offset behavior in the inter-modal region offset trend information of the image, the processor extracts the complete image content of the corresponding modality and, combined with the original tile path range, determines the tile index and corresponding region in the image that should participate in subsequent processing. Based on the determined global offset vector, the processor reverse-corrects the reading coordinates of the visible light image, using the terahertz image as a reference. Specifically, the system adjusts the reading window of the visible light image by 5 pixels horizontally (rounded to 4.8) and 3 pixels vertically (rounded to 2.9) to compensate for the misalignment caused by physical parallax. Through this coordinate mapping adjustment, the processor redefines the effective region corresponding to the terahertz tiles in the visible light image, filters out the effective tile indexes located within the image overlap region, and removes invalid tiles that exceed the image boundary after translation, thus obtaining the imported image tile index information.

[0034] S502: Call the tile position in the imported image tile index information, extract the edge shape, gray level distribution and texture structure information of the corresponding tile in the original image, and organize the various features in the tile according to the position order to obtain the tile feature combination structure; The processor retrieves the tile locations from the imported image tile index information and extracts the edge morphology, grayscale distribution, and texture structure information of the corresponding tiles from the original terahertz and visible light images. It then organizes the various features within each tile according to their positional order. The processor calculates multidimensional feature indices for tiles in both modes, including edge intensity using gradient operators, grayscale mean and variance using statistical moments, and texture energy and contrast using the gray-level co-occurrence matrix. For example, for tiles at the same spatial location, the processor measures a texture contrast value of 1.5 in the terahertz mode and 3.2 in the visible light mode. The processor then structurally encapsulates these feature parameters according to the tile index, forming a feature vector sequence containing rich details, thus obtaining the tile feature combination structure.

[0035] S503: Based on the various block feature contents in the block feature combination structure, allocate them to the input channels in the image fusion processing flow, arrange them according to the block position and structural order, summarize the basic data content that constitutes the fusion processing, and obtain the dominant image fusion content set.

[0036] Based on the various tile features in the tile feature combination structure, the processor assigns them to the input channels in the image fusion processing flow, arranging them according to tile position and structural order. The processor uses a "feature saliency maximization" strategy to calculate fusion weights, dynamically allocating the contribution of each channel based on the numerical ratio of texture contrast. Based on the aforementioned extracted feature data, since the texture contrast of visible light tiles is significantly higher than that of terahertz tiles, the processor assigns a high weight of 0.68 to the visible light channel and a weight of 0.32 to the terahertz channel through numerical ratio calculation, ensuring that the fused image can retain high-contrast texture details to the maximum extent. The processor combines the calculated weight pairs with the original image data block by block, performs a weighted summation operation, and summarizes the basic data content constituting the fusion processing to obtain the dominant image fusion content set.

[0037] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A terahertz imaging and visible light image fusion detection method, characterized in that, Includes the following steps: S1: Acquire images from the terahertz imaging mode and visible light imaging mode during the flight mission, divide the images into equal-area patches, read the pixel grayscale values ​​within each patch sequentially and sum them, use the patch positions as association points, complete the annotation of image source information, and obtain the set of grayscale sum values ​​and patch positions corresponding to each other. S2: Call the patch data in the continuous image frames corresponding to the gray sum value and patch position, extract the spatial position of each frame patch according to time, sort out the continuous patch path, remove interrupted patches, and connect the remaining patch positions as the change trajectory to obtain the patch gray centroid path sequence. S3: Based on the path direction in the gray-scale centroid path sequence of the blocks, extract the movement trend between blocks along the path direction line, filter out paths that maintain consistency in direction changes, identify continuous movement of blocks, extract block data with a clear movement direction, and obtain a continuous block path data group. S4: Call the image source of the tile in the continuous tile path data group, classify the tiles into two groups: terahertz and visible light, view the position distribution of the two groups in the image, compare the spatial offset performance between the sources, and obtain the regional offset tendency information between image modes.

2. The terahertz imaging and visible light image fusion detection method according to claim 1, characterized in that: The grayscale summation value and the corresponding set of tile positions include tile spatial positioning parameters, tile grayscale total index, and image modality category label. The tile grayscale centroid path sequence includes path continuity identifier, tile grayscale center distribution, and tile time series information. The continuous tile path data group includes stable movement direction, path duration, and tile sequence number. The image modality inter-regional offset tendency information includes modality source distribution range, offset direction attribute, and inter-modality relative displacement value.

3. The terahertz imaging and visible light image fusion detection method according to claim 1, characterized in that: The steps for obtaining the set of corresponding grayscale sum values ​​and tile positions are as follows: S101: Based on the terahertz imaging images and visible light imaging images acquired during the flight mission, each image is divided into multiple patches according to a unified area standard. Boundary positioning information is extracted in the horizontal and vertical directions of the image, and pixel regions are divided according to the spacing to ensure that the pixel range covered by each patch is consistent, thus obtaining the patch pixel region distribution set. S102: Based on the pixel range of each region in the pixel region distribution set of the image blocks, extract the grayscale content of the corresponding region from the image item by item, read all pixel grayscale values ​​in sequence and perform accumulation processing, complete the grayscale calculation in the region block by block, and obtain the cumulative combination of image block grayscale values; S103: Call the image coordinate information of the corresponding region and the accumulated gray level combination of the image patch, extract the modal content of the original image, add the image source category on the basis of the coordinates, complete the pairing operation between the region position, image category and gray level calculation, and obtain the image patch gray level value and modal position information set.

4. The terahertz imaging and visible light image fusion detection method according to claim 1, characterized in that: The steps for obtaining the grayscale centroid path sequence of the image patch are as follows: S201: Call the data of the corresponding block region of each image frame in the block gray value and modal position information set, extract the region position in each frame in chronological order, connect the coordinate information of the same block index in each frame along the time axis to form a linear path, and obtain the block temporal movement path line segment set; S202: Based on whether there are missing tile positions in each path of the tile time-series moving path segment set within consecutive frames, extract the tile path index of the frame sequence interruption, filter out broken paths according to the continuity requirement, and retain the remaining uninterrupted paths to the next stage to obtain a continuous area moving path set. S203: Based on the position index of the corresponding map block in each frame of the continuous area movement path set, the coordinates of the same map block in each frame are sequentially connected in time order to extract the position change trend, and the map block is arranged in the order of the frame marked path structure to obtain the region gray-scale centroid change sequence.

5. The terahertz imaging and visible light image fusion detection method according to claim 1, characterized in that: The steps for obtaining the continuous tile path data group are as follows: S301: Based on the gray-scale centroid path sequence of the image blocks, extract the gray-scale centroid coordinates of the image blocks in the path, calculate the direction change of the corresponding direction lines, filter out paths with discontinuous direction changes, and obtain a path angle sequence with consistent direction. S302: Based on the tile number in the direction-consistent path angle sequence and the relative position of the tile in the image, calculate the movement trend along the path direction, extract continuous movement path segments, and obtain the continuous path displacement change trend. S303: Based on the directional information in the displacement change trend of the continuous path, extract the block sequence with consistent direction, and import the corresponding path number and coordinate content to obtain the continuous block path data group.

6. The terahertz imaging and visible light image fusion detection method according to claim 1, characterized in that: The steps for obtaining the image intermodal region offset tendency information are as follows: S401: Call the image source information of the tiles involved in the continuous tile path data group, determine the type of the tile according to the naming identifier of the original image corresponding to the tile, divide the tile into two categories: terahertz source and visible light source, and obtain the distribution information of tile source type; S402: Based on the classification results in the distribution information of the source type of the image patch, extract the coordinate position of each type of image patch in the original image, and count the coverage area and relative position of each type of image patch in the image space to obtain the spatial location distribution characteristics of the image patch. S403: Based on the relative position data in the spatial distribution characteristics of the image blocks, calculate the coordinate offset relationship between the terahertz image blocks and the visible light image blocks, classify the offset directions in each region, and obtain the regional offset tendency information between image modes.

7. The terahertz imaging and visible light image fusion detection method according to claim 1, characterized in that, The method further includes step S5: S5: Based on the modality with a relatively gentle shift in the inter-modal region shift tendency information of the image, the modality image is imported into the fusion processing stage, and corresponding block features are introduced according to the block structure. The block features are centrally processed and organized into the content on which the image fusion operation is based, thus obtaining the dominant image fusion content set. The dominant image fusion content set includes patch feature extraction results, fusion reference structure, and dominant modality image benchmark.

8. The terahertz imaging and visible light image fusion detection method according to claim 7, characterized in that: The steps for obtaining the dominant image fusion content set are as follows: S501: Based on the slow offset behavior in the inter-modal region offset tendency information of the image, extract the complete image content of the corresponding modal image, and combine it with the original tile path range to determine the tile index and corresponding region in the image that should participate in subsequent processing, and obtain the imported image tile index information; S502: Call the tile position in the imported image tile index information, extract the edge shape, gray scale distribution and texture structure information of the corresponding tile in the original image, and organize the various features in the tile according to the position order to obtain the tile feature combination structure; S503: Based on the various block feature contents in the block feature combination structure, allocate them to the input channels in the image fusion processing flow, arrange them according to the block position and structural order, summarize them to form the basic data content of the fusion processing, and obtain the dominant image fusion content set.

Images