Target detection method and system based on co-aperture multi-band feature fusion

Abstract

The present application relates to the technical field of multi-band image detection, and discloses a target detection method and system based on common-aperture multi-band feature fusion, which comprises the following steps: in view of the problem that it is difficult to select complementary edges according to spatial relationship for the same field of view visible light image and infrared image under different field illuminations, the two images are placed in a common spatial coordinate and coded to form a common fusion image at a corresponding fusion scale; visible light edge images and infrared edge images are obtained from the visible light coded image and the infrared coded image, common edge areas, visible light unique edge areas, infrared unique edge areas and common point areas are determined, connected sections with two ends contacting different common point areas are screened as visible light bridge sections and infrared bridge sections, simple closed loops containing visible light bridge sections and infrared bridge sections and switching bands only in common point areas are searched, visible light coded images are selected in visible light bridge sections, infrared coded images are selected in infrared bridge sections, the rest are common fusion images, a relay fusion image is formed and detected, and no accepted closed loop is kept as a common fusion image.

Description

Technical Field

[0001] This invention relates to the field of multi-band image detection technology, and more specifically, to a target detection method and system based on co-aperture multi-band feature fusion. Background Technology

[0002] In fixed-point monitoring of park perimeters, road entrances / exits, and warehouse boundaries, target detection typically employs visible light imaging, or a combination of visible light and infrared imaging. Common-aperture imaging devices can simultaneously acquire visible light and infrared images within the same field of view, and through channel correction and spatial registration, place the two images in a common spatial coordinate system. Visible light images typically exhibit clearer textures and edges under high illumination conditions, while infrared images provide additional target and background response information under low illumination or reduced visibility. Therefore, both bands are often used simultaneously for fixed-scene detection.

[0003] Existing dual-band detection workflows typically begin by unifying and normalizing the image size, then encoding the visible light and infrared branches separately to obtain visible light and infrared encoded images at the corresponding fusion scale. Subsequently, a common fusion image is formed through stitching, convolution, weighting, or other direct fusion methods. This common fusion image is then fed into the neck network and the target detection head, outputting the target category, bounding box location, and confidence score. Some methods also obtain edge positions from the visible light and infrared encoded images to assist in registration, fusion, or candidate region selection.

[0004] In the above process, direct fusion typically processes the visible light coded image and the infrared coded image uniformly according to their spatial location, making it difficult to determine whether a single-band edge is spatially supported by another band. Single-band edges may originate from occlusion, reflection, background texture, or differences in sensor response; if such edges participate in local fusion without checking connectivity, ordinary fused images may lack edge information based on cross-band criteria. On the other hand, processing methods that set fusion weights based on the entire image or adjust based on global conditions make it difficult to retain the complementary information of the visible light and infrared images at different locations.

[0005] Existing edge processing methods may only perform positional matching, connected segment filtering, or loop closure search. However, these methods may not simultaneously ensure that both ends of a connected segment are connected to a common region, nor may they ensure that a closed path simultaneously includes both visible light and infrared bridge segments, or restrict band switching to the common region. Therefore, a multi-band target detection method oriented towards shared spatial coordinates is still needed. This method should be able to filter connected segments that can participate in fusion based on the spatial connectivity of edge lines, select the appropriate encoded map at locations that meet the dual-band loop closure condition, and maintain the normal fusion map when no accepted loop is formed. Summary of the Invention

[0006] This invention provides a target detection method and system based on co-aperture multi-band feature fusion, which solves the technical problems mentioned in the background art.

[0007] To achieve the above-mentioned objectives, the present invention provides the following technical solution.

[0008] Target detection methods based on co-aperture multi-band feature fusion include: The visible light image and infrared image of the same field of view are acquired simultaneously by a common aperture imaging device and placed in a common spatial coordinate system to form an aligned visible light image and an aligned infrared image. Encode the two separately to obtain a visible light encoded map and an infrared encoded map at the corresponding fusion scale. Then fuse the visible light encoded map and the infrared encoded map to obtain a normal fusion map. The fusion does not depend on the edges and closed loops. Obtain visible light edge maps and infrared edge maps respectively, and determine the common edge area, the visible light-specific edge area, and the infrared-specific edge area; A set of common areas is formed by connecting segments in the common boundary area that contact the ends of the visible light-specific boundary area or the infrared-specific boundary area. Connecting segments that contact different common areas at both ends are identified as visible light bridge segments and infrared bridge segments according to their wavelength. Using the common edge continuous path, visible light bridge segment and infrared bridge segment as path units connecting the common point area, search for a simple closed loop that returns from the common point area to the common point area and whose path unit and intermediate common point area do not overlap. The closed loop contains visible light bridge segment and infrared bridge segment and switches bands only in the common point area. For each accepted closed loop, visible light encoded images and infrared encoded images are taken at the visible light bridge segment and infrared bridge segment positions, respectively, and the rest are ordinary fusion images, which are then used to form a relay fusion image. If no closed loop is accepted, the relay fusion graph is a normal fusion graph. Detect the relay fusion image and output the target category, target bounding box position and confidence score.

[0009] Compared with the prior art, the present invention has the following substantial features and significant progress: This invention simultaneously acquires visible light and infrared images of the same field of view using a common aperture imaging device and places them in a common spatial coordinate system. Visible light coded images and infrared coded images are formed at corresponding fusion scales, respectively. A standard fused image is obtained using a reference fusion independent of edge lines and closed loops. Then, based on the visible light and infrared edge lines, common edge lines, visible light-specific edge lines, infrared-specific edge lines, and common point areas are determined. Visible light bridge segments and infrared bridge segments are formed using connected segments that contact different common point areas at both ends. A closed loop containing both visible light and infrared bridge segments, switching bands only in the common point area, is searched using path units. The visible light coded image is selected only at the visible light bridge segment location, and the infrared coded image is selected at the infrared bridge segment location; the standard fused image is maintained at other locations before target detection is performed. Therefore, the condition for a single-band edge line to enter the fusion result can be limited to a spatial closed loop with cross-band connection basis, reducing the interference of edge lines without connection basis on the fused image and preserving the complementary information provided by the two bands at different locations. If no closed loop is accepted, it reverts to a standard fused image, maintaining the continuity of the detection process. Attached Figure Description

[0010] Figure 1 This is a system functional architecture diagram of the present invention; Figure 2 This is a flowchart of the method of the present invention. Detailed Implementation

[0011] The following description is provided in conjunction with the accompanying drawings. It should be understood that the following embodiments are for illustrative purposes only and are not intended to limit the invention; those skilled in the art can make equivalent substitutions or combinations for the specific implementations.

[0012] This embodiment relates to target detection at fixed monitoring points. The common aperture imaging device 111 simultaneously acquires visible light and infrared images of the same field of view in the same detection cycle. The processor completes alignment, encoding, and source selection in a common spatial coordinate system and inputs the resulting relay fused image into the target detection head.

[0013] This implementation method is applicable to monitoring scenarios such as fixed park perimeters, road entrances and exits, and warehouse boundaries. The common aperture imaging device 111 is fixedly installed, with the visible light channel and infrared channel observing the same monitoring field of view through the same incident aperture. When the ambient light is high, the visible light channel provides target contour information; when the ambient light decreases or the visible distance shortens, the infrared channel provides target contour information. The response intensities of the two channels may differ, but their positional correspondence in a common spatial coordinate system remains consistent.

[0014] like Figure 1As shown, the common aperture imaging device 111 provides visible light and infrared images to the spatial alignment module 112. The spatial alignment module 112 outputs aligned visible light and infrared images, and the dual-band encoding module 113 outputs visible light encoded images and infrared encoded images, respectively. The general fusion module 114 outputs a general fusion image, the edge partitioning module 115 outputs visible light edge images and infrared edge images, the co-point area, bridge segment, and accepted loop closure determination module 116 outputs path determination results, the local source selection and relay fusion module 117 outputs a relay fusion image, and the multi-scale convergence and target detection module 118 outputs detection results. .

[0015] The scene input interface is connected to the processor. It reads the ambient illuminance, visibility distance, maximum monitoring distance, and camera reference illuminance; the maximum monitoring distance and camera reference illuminance are written to the fixed-point configuration, while the ambient illuminance and visibility distance are updated periodically. The processor, memory, and output interface are used to execute and transmit the above data. Modules are connected in the order of image input, spatial processing, source selection, and detection output.

[0016] At the start of the detection cycle, the common aperture imaging device 111 activates the visible light channel and the infrared channel by the same trigger signal, generating a pair of images within the same detection cycle. These images are denoted as the visible light image and the infrared image, respectively. The processor matches the outputs of the two channels into a single frame pair based on the image timestamps, and this frame pair is used for subsequent spatial processing.

[0017] The visible light image retains the pixel arrangement of the visible light channel, and the infrared image retains the pixel arrangement of the infrared channel. The bit depth and number of channels can be different for the two channels. The common aperture imaging device 111 directly provides the visible light image and the infrared image. The target detection process uses the image information from both channels and the common spatial coordinates, without adding radar, lidar, or other edge sensors.

[0018] The scene input interface reads the ambient illuminance and visible distance within the same detection cycle, and reads the furthest monitoring distance and camera reference illuminance from the fixed point configuration. The timestamps of the four quantities correspond to the image timestamps. When all four quantities are valid, the processor compares the ambient illuminance with the camera reference illuminance, and compares the visible distance with the furthest monitoring distance; if any quantity is missing or has no value, the processor sets the search order of the path units to search the infrared segment first and then the visible light segment.

[0019] The spatial alignment module 112 performs distortion correction according to the intrinsic parameters of the two imaging channels, and then uses the two-dimensional transformation obtained from the pre-deployment calibration to map the infrared image to the common spatial coordinates of the visible light image. In this embodiment, the two-dimensional transformation is a homography transformation. The calibration object is observed jointly by both channels. The processor selects at least four pairs of non-collinear corresponding positions from the calibration object to obtain the homography transformation, and retains at least one pair of corresponding positions that were not involved in the solution for inspection.

[0020] Residual edge deviation is represented by the larger of the lateral and longitudinal coordinate differences at the corresponding positions. When all residual edge deviations at the corresponding positions do not exceed one input pixel, the current calibration parameters are used to form common spatial coordinates. When the position, focal length, or optical axis orientation of the common aperture imaging device 111 changes, the homography transformation corresponding to the current installation state is re-acquired.

[0021] The camera reference illuminance is determined under the current exposure and gain configuration of the visible light channel. A standard target is placed at the location specified in the calibration procedure. When the ambient illuminance reaches a level where the visible light map clearly distinguishes the target's outline, the ambient illuminance is read and written into the camera reference illuminance. The camera reference illuminance is redefined after any changes to the exposure or gain configuration. Two-dimensional transformations are used to establish spatial coordinate correspondences but are not used to generate target categories or bounding box positions.

[0022] The spatial alignment module 112 uses a common output size, coordinate origin, crop position, padding position, and flip direction to process two channels. The processor determines the two transformed images as an aligned visible light image and an aligned infrared image. The upper left corner of the input image is the common spatial coordinate origin, with the horizontal coordinate increasing along the image width and the vertical coordinate increasing along the image height.

[0023] Normalization is performed after spatial alignment is complete. In the normalization expression below, the right side... and Specifically, this refers to the pixel or channel values ​​read from the corresponding spatial positions in the visible light and infrared images after spatial alignment. The original acquisition array is no longer used as input to the dual-band coding module 113 after spatial alignment is completed. The normalized numerical expression is: This represents the visible light image value read from the normalization operation. This represents the infrared image value read from the normalization operation.

[0024] This represents the normalized, aligned visible light image. This represents the normalized aligned infrared image.

[0025] 255 is the maximum integer pixel value for an 8-bit visible light image, and 16383 is the maximum integer pixel value for a 14-bit infrared image. The two division operations are applied to all channels at the corresponding spatial locations.

[0026] After entering the dual-band coding module 113, the input pixels are converted to feature grids for each fusion scale according to the network pixel step size. The network pixel step size is the input image pixel interval corresponding to the current fusion scale; the converted spatial coordinates are integer coordinates in the feature grid. The origin, horizontal direction, and vertical direction of the spatial coordinates remain consistent across all fusion scales.

[0027] The residual edge deviation of the input image is first converted according to the network pixel step size of the current fusion scale, and then used to determine the correspondence at the spatial coordinates of the current fusion scale. The conversion uses the nearest integer coordinates, and positions that cross the boundary of the current fusion scale are truncated from the neighborhood expansion results. The unit of alignment tolerance is locked to the spatial coordinate unit of the current fusion scale.

[0028] The dual-band coding module 113 includes a visible light branch and an infrared branch. After alignment, the visible light image enters the visible light branch to obtain a visible light coded image; after alignment, the infrared image enters the infrared branch to obtain an infrared coded image. The visible light coded image and the infrared coded image have the same spatial width, spatial height, and number of channels at the same fusion scale.

[0029] Indicates the first Visible light coding map at multiple fusion scales. Indicates the first Infrared coded map at multiple fusion scales.

[0030] This indicates the fusion scale number. The spatial coordinates and channel arrangement of one fusion scale are not directly merged with those of another fusion scale.

[0031] The general fusion module 114 performs preprocessing for each selected fusion scale in the edge partitioning module 115, fusing the visible light coded image and the infrared coded image reference into a general fused image. The channel arrangement of the visible light coded image precedes the channel arrangement of the infrared coded image, and the two coded images are stitched together along the channel direction. The stitched features are processed using a step size of 1. Convolution, with space padding set to 0, and the number of output channels set to the number of channels at the current fusion scale.

[0032] Indicates the first A standard fused map at a fusion scale. The standard fused map has the same spatial width, spatial height, and number of channels as the visible light coded map and the infrared coded map.

[0033] This indicates that the images are spliced ​​along the channel direction, with the visible light coded image first and the infrared coded image second. The convolution kernel spatial size is A convolution operation with a stride of 1 and an output channel number equal to the current fusion scale channel number.

[0034] The general fusion graph covers all spatial locations at the current fusion scale. The formation of the general fusion graph uses only feature values ​​from the visible light coded graph and the infrared coded graph. Boundary partitioning, common point region formation, bridge segment determination, and accepted loop closure search are performed after the general fusion graph is formed.

[0035] Within the same fusion scale, the three encoding maps use the same spatial index. The convolution operation combines the channel values ​​of the two encoding maps at each spatial location, outputting the channel values ​​of the ordinary fused map; subsequent source selection copies the complete channel vector according to the spatial location.

[0036] The fourth step is to obtain the visible light edge map and the infrared edge map. The edge partitioning module 115 performs edge extraction operations on the visible light encoded image and the infrared encoded image respectively. At each spatial location, the processor calculates the arithmetic mean of the absolute values ​​of all channels at that location to obtain a single-channel response; the single-channel response is convolved with the Sobel horizontal kernel and the Sobel vertical kernel respectively to obtain the horizontal gradient and the vertical gradient; the processor obtains the gradient magnitude based on the square root of the sum of the squares of the two gradients.

[0037] This implementation uses the following Sobel horizontal and Sobel vertical kernels: This represents the Sobel horizontal kernel, where the integers in the matrix are the kernel coefficients of the horizontal gradient convolution.

[0038] This represents the Sobel vertical kernel, where the integers in the matrix are the kernel coefficients of the vertical gradient convolution.

[0039] When the convolution window exceeds the boundary of the current fusion scale, it is padded with zeros. The convolution result retains the spatial width and spatial height of the current fusion scale. The visible light encoded map and the infrared encoded map use the same convolution boundary rules.

[0040] The processor obtains the minimum and maximum values ​​of the gradient magnitude within each fusion scale. When the maximum value is greater than the minimum value, the minimum value is mapped to 0 and the maximum value is mapped to 1, and the remaining gradient magnitudes are mapped according to the same linear range. When the maximum value is equal to the minimum value, neither the visible light edge map nor the infrared edge map records the edge position. The position where the normalized gradient magnitude reaches the edge binarization threshold is recorded as the edge position.

[0041] In this implementation, the edge binarization threshold is set to 0.35. The processor converts the spatial locations registered in the edge graph into corresponding sets. When the state value of the edge graph is 1, the spatial location is added to the set; when the state value of the edge graph is 0, the spatial location is not added to the set.

[0042] The set of spatial locations obtained by converting the visible light edge map is denoted as The set of spatial locations obtained by converting the infrared edge map is denoted as Edge extraction operation The process is performed separately at each fusion scale, and the extreme values, edge binarization thresholds, and spatial location sets at the current fusion scale are not propagated to other fusion scales.

[0043] The processor calculates the Chebyshev distance between the spatial coordinates of the visible light edge map and the infrared edge map within the same fusion scale. When the distance between two spatial coordinates is no greater than the alignment tolerance, the two spatial positions are determined to be corresponding positions. When the alignment tolerance is 1, adjacent feature mesh positions in the horizontal, vertical, and diagonal directions are all considered corresponding. Eight neighboring domains.

[0044] The alignment tolerance neighborhood expansion operation takes a set of spatial locations as input and, within a finite spatial coordinate domain of the current fusion scale, identifies all locations whose Chebyshev distance from at least one location in the input set is no greater than the alignment tolerance. The expansion result is truncated within the spatial width and height range of the current fusion scale, and the output locations still use integer spatial coordinates.

[0045] No. The common boundary region of each fusion scale is represented as: Indicates the first A common boundary area of ​​multiple fusion scales.

[0046] This represents the set of spatial locations in a visible light edge diagram. This represents the set of spatial locations in the infrared edge diagram.

[0047] This indicates the alignment tolerance neighborhood expansion operation. Indicates alignment tolerance; The output domain is the spatial coordinate domain of the current fusion scale.

[0048] Spatial locations in the visible light edge diagram that do not have a corresponding position within the alignment tolerance neighborhood of the infrared edge diagram constitute a unique visible light edge region: Indicates the first Each visible light-specific edge region is integrated across multiple scales. The set difference only groups spatial locations; the edge positions within the visible light-specific edge regions are retained.

[0049] Spatial locations in the infrared edge map that do not have a corresponding position within the alignment tolerance neighborhood of the visible light edge map form a unique infrared edge region: Indicates the first A unique infrared edge region with a fusion scale. and All elements are spatial locations in the original edge graph; locations generated by neighborhood expansion are not separately registered as edge locations.

[0050] The shared edge region, the visible light-specific edge region, and the infrared-specific edge region all use the same fusion scale for spatial position coordinates. The shared edge region stores edge positions that correspond to each other in both bands, while the visible light-specific edge region and the infrared-specific edge region store edge positions that do not correspond to each other.

[0051] The processor performs eight-neighbor connectivity analysis on both the visible light-specific edge region and the infrared-specific edge region. When a location is adjacent to another location of the same type within its eight-neighborhood, the two locations belong to the same connected component. The processor counts the number of adjacent locations of the same type for each location within each connected component.

[0052] Positions with one identical adjacent position are designated as endpoints of a connected segment. Positions with more than two identical adjacent positions are designated as branching positions. The processor traces unvisited identical adjacent positions from each endpoint or branching position until it reaches another endpoint or branching position, resulting in an ordered connected segment. Branching positions are recorded as endpoints in each adjacent branch, not as internal positions within any branch; branching positions are not paired between adjacent branches.

[0053] When a connected component lacks endpoints, it does not form a connected segment suitable for bridge segment determination. Each ordered connected segment stores the starting endpoint, the ending endpoint, and a complete spatial position sequence from the starting endpoint to the ending endpoint. When the endpoint coordinates are the same distance, the recording direction of the connected segment is determined by the order of the horizontal coordinates followed by the vertical coordinates.

[0054] The processor performs eight-neighbor connectivity analysis on the common edge region and uses the original common edge connected cells as the skeleton to be decomposed. For each unique edge connected segment end, the processor searches for contact positions in the common edge skeleton where the Chebyshev distance to that end is no greater than the alignment tolerance. Contact positions that are mutually eight-neighbored and serve the same unique edge end are merged into a common area candidate.

[0055] The processor first registers candidate common areas, then marks these candidates as path nodes from the common edge skeleton to be decomposed. Each consecutive candidate common area is considered a connected cell within a common edge area, and all common areas form a common area set. The candidate concurrent regions only include the positions of common edges that are in contact with the ends of the segments connected to unique edges, and do not include the positions of common edges between two concurrent regions into any concurrent region.

[0056] The processor extracts the eight-neighborhood continuous position sequence between nodes from the common edge skeleton after marking path nodes, obtaining a continuous common edge path. A continuous common edge path stores the complete spatial position sequence between two common area nodes and is treated as an independent path unit. Therefore, the same original common edge connecting the interior of a cell can be divided into multiple common areas and multiple continuous common edge paths through node segmentation, with each segmented object having a unique node and path affiliation.

[0057] In the visible light-only boundary region, an ordered connected segment whose two ends spatially contact two different common points is registered as a visible light bridge segment. In the infrared-only boundary region, an ordered connected segment is registered as an infrared bridge segment under the same conditions. A connected segment with one end suspended and both ends contacting the same common point, or whose ends belong to the same common point, is not registered as a bridge segment.

[0058] The processor examines the complete spatial position sequence of a connected segment. Except for the common regions corresponding to the two ends, when the sequence passes through a third common region, the connected segment is segmented at that third common region. The sub-segments between each adjacent common region store the two ends, and each sub-segment is checked to see if the two ends contact different common regions. Multiple third common regions are segmented sequentially according to the order of the connected segment's spatial position sequence.

[0059] The contact between an end point and a common point region is determined by the minimum Chebyshev distance between their spatial positions. When an end point contacts multiple common point regions simultaneously, the minimum distances are compared first. If the minimum distances are the same, the region is assigned a position according to the order of the x-coordinate and y-coordinate of the position represented by the common point region. Contact relationships that are not selected are not registered separately.

[0060] The complete position sequence of visible light bridge segments constitutes the visible light bridge segment set. The complete location sequence of infrared bridge segments constitutes the infrared bridge segment set. .

[0061] Indicates the first The set of visible light bridge segments that have been checked for double-ended contact and third common point region in the fusion scale. This represents the corresponding set of infrared bridge segments. The position sequence of the bridge segments, the two ends, and the common areas corresponding to the two ends are all stored together.

[0062] The processor treats common points as path connection nodes and registers continuous paths along common edges, visible light bridge segments, and infrared bridge segments as path units. A continuous path along a common edge connects two common points, and visible light and infrared bridge segments also connect their respective end-point common points. Each path unit stores the starting common point, ending common point, complete spatial location sequence, and source type.

[0063] The search begins with a common point region. The processor connects the beginning and end of the path unit, recording each path unit as it is expanded, along with any intermediate common points regions traversed. The expansion process does not reuse recorded path units or re-enter recorded intermediate common points regions; when the common point region terminates and returns to the starting common point region, a candidate loop closure is obtained.

[0064] A candidate loop is registered as an accepted loop if it meets the following criteria: the starting and ending common areas are the same; each path unit appears once in the candidate loop; intermediate common areas are not repeated; each path unit contains at least one visible light bridge segment and one infrared bridge segment; and the source switching between the visible light coded map and the infrared coded map occurs in the common area. Continuous paths along the common edge only serve as connections between common areas, and the source type is maintained within each bridge segment.

[0065] The processor identifies all common points as starting common points based on their x and y coordinates. A depth-first search is performed on each starting common point. The expansion order of path units is determined first by the order of competing bridge segments based on scene quantity, and then by the spatial coordinates of the starting and ending points of the path unit. After finding an accepted closed loop, the processor continues the search along the unchecked path units.

[0066] When both visible light and infrared bridge segments exist simultaneously between the same pair of common areas, and all four quantities are valid, the on-site illuminance is not lower than the camera reference illuminance, and the on-site visible distance is not less than the farthest monitoring distance, the processor will first search for the visible light bridge segment; otherwise, it will first search for the infrared bridge segment. This order only determines the search order and does not change the source type of the visible light and infrared bridge segments.

[0067] The depth-first search records the current common area, the path units already used, the intermediate common areas already traversed, and the current source type. Upon returning to the starting common area, the processor sequentially checks the path unit repetition status, the intermediate common area repetition status, the presence status of visible light and infrared bridge segments, and the source switching location. If any condition is not met, the last extended path unit is withdrawn, and the processor continues to try the next path unit at the same level.

[0068] Continuous paths along a common edge participate in duplicate checks as a complete path unit. Position sequences within a bridge segment are not split into multiple path units; source switching only occurs in the common area connecting adjacent path units. Multiple accepted loop closures are processed according to the completion order of the depth-first search; positions of different loop closures are merged into the corresponding inbound bridge segment set, positions with already registered sources retain their earlier registered sources, and other non-conflicting positions continue to be registered.

[0069] The processor aggregates the actual visible light bridge segment locations traversed in the accepted closed loop into an in-loop visible light bridge segment set, and aggregates the actual infrared bridge segment locations into an in-loop infrared bridge segment set. Continuous paths along common edges are used to connect common points; these paths do not enter either of the two in-loop bridge segment sets. Bridge segment locations that do not form an accepted closed loop also do not enter either of the two in-loop bridge segment sets.

[0070] Indicates the first The set of visible light bridge segments entering the loop at a fusion scale, that is, the set of visible light bridge segment locations actually used in the accepted closed loop.

[0071] Indicates the first The set of in-loop infrared bridge segments at each fusion scale is the set of infrared bridge segment locations actually used in the accepted closed loop. and Conflicts are removed based on spatial location during generation.

[0072] processor by spatial coordinates Source selection is performed. The relay fusion diagram is represented as follows: Indicates the first The relay fusion map at various fusion scales is located in the spatial coordinates. The channel vector at that location.

[0073] , and These represent the channel vectors of the visible light coded image, infrared coded image, and ordinary fused image at the same spatial location, respectively.

[0074] and It is a set of locations, and the elements of the two sets are mutually exclusive; spatial locations that do not belong to the union of the two sets are taken as ordinary merged graphs.

[0075] When no loop closure is accepted at a fusion scale, both inbound bridge segments are empty, and the relay fusion graph is taken from the ordinary fusion graph for all spatial locations at that fusion scale. When multiple accepted loop closures conflict in location, the source that was registered first is retained according to the order of loop closure completion, and the non-conflicting locations are added to the corresponding inbound bridge segment set.

[0076] The dual-band coding module 113 outputs visible light coded maps and infrared coded maps at at least three corresponding fusion scales. Each fusion scale independently completes the formation of ordinary fusion maps, edge partitioning, common point region formation, bridge segment determination, path unit search, and relay fusion map formation.

[0077] The relay fusion graphs at various fusion scales are input into the neck network. The neck network performs scale transformation, lateral connections, and convolutions on the relay fusion graphs of different spatial sizes to obtain a multi-scale convergence graph. In this embodiment, the neck network uses PAN-FPN, and channel-matched convolution is used before lateral connections.

[0078] Indicates the first Multi-scale convergence diagram of fusion scales in the neck network. It is obtained by converging the relay fusion diagrams of each fusion scale through the neck network.

[0079] A multi-scale convergence map is input to the object detection head. The object detection head is a one-stage object detection head without anchor points, and it outputs candidate boxes, object categories, and confidence scores. The confidence score of each candidate box is the confidence score output by the object detection head, and the object category is bound to the candidate box and its confidence score. The candidate boxes use a left-top-right-bottom coordinate format, with the four coordinates representing the left boundary, top boundary, right boundary, and bottom boundary, respectively.

[0080] The processor first deletes files with a confidence level lower than the confidence threshold. The candidate boxes are then processed in descending order of confidence. The cross-union ratio (CUR) between candidate boxes is suppressed to reach a non-maximum threshold. When the confidence levels are the same, the candidate boxes processed earlier are retained and the conflicting candidate boxes processed later are deleted. When the confidence levels are the same, the order of processing is determined by the x and y coordinates of the top left and bottom right coordinates of the candidate boxes.

[0081] This indicates the confidence level screening threshold. This indicates the non-maximum suppression cross-union ratio (CUI) threshold. The processor will retain the target category, target location, and confidence score of the candidate boxes to form the detection result. .

[0082] The processor restores the candidate box coordinates to the original image coordinates. The restoration process follows the coordinate origin and coordinate direction determined by the spatial alignment module 112, and maintains that the left boundary is no greater than the right boundary and the upper boundary is no greater than the lower boundary. Candidate boxes that do not reach the non-maximum suppression cross-union ratio (CUI) threshold are retained, while candidate boxes that reach the threshold undergo non-maximum suppression in order of confidence.

[0083] During the model preparation phase, paired visible light and infrared images are acquired, and the same target label is configured for the same frame pair. The same geometric enhancement is performed on both channels to maintain the correspondence between the spatial coordinates before and after enhancement. Trainable parameters in the visible light branch, infrared branch, general fusion module 114, neck network, and target detection head are updated based on the target detection output.

[0084] After training, the processor saves the network parameters of the dual-band coding module 113, the general fusion module 114, the neck network, and the target detection head, as well as the spatial alignment parameters corresponding to the current installation state. The fixed point configuration saves the farthest monitoring distance and camera reference illuminance, and the scene quantity input interface reads the on-site illuminance and on-site visibility distance in each detection cycle.

[0085] During deployment, the processor receives visible light and infrared images from the same detection cycle and uses the saved spatial alignment parameters to form aligned visible light and infrared images. Subsequently, relay fusion maps are formed according to each fusion scale, the neck network outputs a multi-scale converged map, and the target detection head outputs the detection results. .

[0086] Example 1: The following example uses pedestrian and vehicle detection at fixed perimeter points in a park to illustrate data processing and spatial determination within a detection cycle. The implementation example provides three fusion scales; the first fusion scale lists the local spatial location and channel sub-vectors, while the second and third fusion scales list the scale processing results.

[0087] At the start of the detection cycle, the common aperture imaging device 111 acquires a visible light image and an infrared image within the same field of view. The network input width is 640 pixels and the input height is 512 pixels, with no additional padding. The visible light image is a three-channel 8-bit image, and the infrared image is a single-channel 14-bit effective grayscale image; both channels belong to the same frame pair. The furthest monitoring distance at the fixed point is taken as... Scene quantity input interface reads the on-site illuminance. Visibility distance Camera reference illumination .

[0088] Indicates the furthest monitoring distance. Indicates the on-site illuminance. Indicates the visible distance at the scene. This indicates the camera reference illuminance. When the ambient illuminance is lower than the camera reference illuminance and the visible distance is less than the furthest monitoring distance, and there is competition between the same pair of common points, the path unit first searches for the infrared segment.

[0089] The installation calibration yielded a homography transformation from the infrared channel to the common spatial coordinates. Checking the corresponding positions not involved in the solution revealed a residual edge deviation of 0.7 input pixels, which does not exceed the calibration condition of 1 input pixel. The three fusion scales used network pixel strides of 8, 16, and 32, respectively, with alignment tolerance set at 1 spatial coordinate unit in all three fusion scales. .

[0090] The aligned visible light and infrared images are normalized according to the normalization expression described above. The number of output channels for the five-level encoder is 32, 64, 128, 256, and 512, respectively, and the network pixel stride is 2, 4, 8, 16, and 32, respectively. In this example, three fusion scales corresponding to strides of 8, 16, and 32 are selected. The spatial size and number of channels for the first fusion scale are as follows: The second fusion scale is The third fusion scale is .

[0091] The standard fusion module 114 first stitches the visible light coded image along the channel direction at each fusion scale, and then stitches the infrared coded image, using a step size of 1. The convolution output is a normal fused graph with the same number of channels as the current fusion scale.

[0092] This example calculates the arithmetic mean of the absolute values ​​of all channels at each spatial location across three fusion scales, and then performs edge extraction. The gradient magnitude is calculated using the aforementioned Sobel horizontal and vertical kernels. Each fusion scale independently performs normalization from minimum to 0 and from maximum to 1, with a boundary binarization threshold of 0.35. Fusion scales with equal maximum and minimum values ​​do not register boundary positions.

[0093] The spatial dimensions of the first fusion scale are: According to the common edge skeleton decomposition rules, four single-location concurrent regions are formed in the local window. The representative locations of the four concurrent regions are... , , and These locations are first marked as path nodes from the original common edge connected cells as common point area candidates, and then continuous common edge paths are extracted from the remaining common edge locations between the nodes.

[0094] Continuous paths along common edges to The internal position sequence is as follows: , , , , , , , , , , , Continuous paths along common edges to The internal position sequence is as follows: , , , , , , , , , , , The positions of the two consecutive paths mentioned above are within the path between nodes and are not registered as common points.

[0095] The visible light-specific edge region includes a connecting line. and The upper connected segment, its complete spatial position sequence is as follows: , , , , , , , , , , First position and The Chebyshev distance is 1, and the final position is... The Chebyshev distance is 1.

[0096] The infrared-specific edge area includes a connection and The competing connected segments, whose complete spatial location sequence is as follows: , , , , , , , , , , First position and The Chebyshev distance is 1, and the final position is... The Chebyshev distance is 1. The ordinate of the corresponding position of this competing connected segment differs from that of the upper visible light connected segment by 2, therefore the Chebyshev distance at the corresponding position is 2, which is greater than 1. The two connected segments are each retained in two unique edge regions.

[0097] The infrared-specific edge area also includes a connecting line and The lower connected segment, its complete spatial position sequence is as follows: , , , , , , , , , , First position and The Chebyshev distance is 1, and the final position is... The Chebyshev distance is 1.

[0098] There is another visible light-specific edge segment in the local window. , , This connected segment only has its first position connected to the first position. The Chebyshev distance is 1, and there is no contact point region at the other end, so it does not qualify for a visible light bridge segment.

[0099] Under the aforementioned positional relationships, the four common regions are each in contact with the ends of adjacent unique edge connectivity segments. Although the common edge skeleton may belong to the same connected cell to be decomposed in the original eight-neighbor connectivity analysis, node labeling and path segmentation are performed before the registration of continuous paths along the common edge. to Each node is stored as an independent common point region, and the continuous positions between nodes are stored as a continuous path along a common edge.

[0100] Because the ambient light level is lower than the camera's reference light level and the visible distance is less than the furthest monitoring distance, the processor first searches for connections. and The infrared competing connected segment. Then, it is expanded sequentially. to Continuous paths along common edges to The infrared lower side connector and to A continuous path sharing a common edge. The path returns to... Each of the four path units is used once, and the common points in the middle are not repeated, but the path only contains infrared bridge segments. Therefore, it is registered as a candidate closed loop but not as an accepted closed loop.

[0101] The processor continues to attempt connections at the same search level. and The visible light upper connected segment. The resulting path passes sequentially through the visible light bridge segment, to The common edge continuous path, infrared lower bridge segment and to The common edge of the continuous path eventually returns to Each path unit is used once, the intermediate common area is not repeated, and it contains both visible light and infrared bridge segments; both source switchings occur in the common area, and therefore it is registered as an accepted closed loop.

[0102] The positions of the visible light upper connected segments in the accepted closed loop form the inbound visible light bridge segment set, and the positions of the infrared lower connected segments form the inbound infrared bridge segment set. The internal positions of the common edge continuous path are only used as a connection basis. The suspended visible light unique edge connected segments and the infrared competing connected segments do not enter the inbound bridge segment set.

[0103] The first fusion scale has 128 channels. The following explicit expression represents the first 8 channels of the 128-channel tensor. (Subscripts in the notation are used.) This represents a channel slice from channel 1 to channel 8; channels not shown are processed according to the same source selection rules. Normal fused image, visible light coded image, and infrared coded image are shown in... The first 8 channels are: This indicates the spatial location of the ordinary fused graph at the first fused scale. The first 8 channel slices at a given location have values ​​that are the channel values ​​of the normal fused image at that location.

[0104] This represents the first 8 channel slices of the visible light encoded map at this location.

[0105] This represents the first 8 channels of the infrared encoded image at this location. Because... Belonging to the set of visible light bridge segments entering the loop, the relay fusion diagram at this location uses visible light encoded diagrams for all 128 channels. The first 8 channels are represented as follows: This represents the first 8 channel slices of the relay fusion diagram at this location. The equation indicates that this slice is taken from... Channels 9 through 128 will use the same source selection.

[0106] Infrared lower bridge segment spatial position The three first 8 channel slices at that location are: Indicates the ordinary fusion graph in The first 8 channels of the slice.

[0107] Indicating visible light coding pattern in The first 8 channels of the slice.

[0108] Indicates the infrared coded image in The first 8 channels are sliced ​​at this location. This location belongs to the in-loop infrared bridge segment set. The relay fusion image at this location uses infrared coded images for all 128 channels. The first 8 channels are represented as follows: This represents the first 8 channel slices of the relay fusion diagram at this location. The equation indicates that this slice is taken from... Channels 9 through 128 will use the same source selection.

[0109] Right common sideline position If the location is not within either of the two inbound bridge segments, the complete channel vector at that position is taken from the normal fusion graph: , These represent the complete channel vectors of the relay fusion map and the normal fusion map at this position in the first fusion scale, respectively.

[0110] Unique edge position of suspended visible light If the channel vector at that location does not enter the accepted closed loop, the complete channel vector is taken from the normal fusion graph: and These represent the complete channel vectors of the relay fusion graph and the normal fusion graph at this position, respectively.

[0111] The spatial dimensions and number of channels in the second fusion scale are: The processor performs edge extraction, edge binarization, alignment tolerance neighborhood expansion, common point region formation, and bridge segment determination at this scale, and searches for path cells. At this scale, there exist accepted closed loops that simultaneously contain visible light and infrared bridge segments. Locations with concentrated visible light bridge segments entering the loop are represented by visible light encoded maps, locations with concentrated infrared bridge segments entering the loop are represented by infrared encoded maps, and other spatial locations are represented by ordinary fused maps.

[0112] The spatial dimensions and number of channels in the third fusion scale are: This scale completes edge location acquisition, spatial partitioning, and bridge segment checking, but does not contain any accepted closed loops that simultaneously include both visible light and infrared bridge segments; therefore, the two in-loop bridge segment sets at this scale are empty, and the relay fusion graph is equal to the ordinary fusion graph. The relay fusion diagram representing the third fusion scale. This represents the ordinary fusion map at the third fusion scale. This equation applies to the complete spatial location and complete number of channels at the third fusion scale.

[0113] The relay fusion maps at three fusion scales are input into the PAN-FPN. The neck network performs scale transformation and lateral connections on the three spatial dimensions, outputting multi-scale convergence maps with 128, 256, and 512 channels, respectively. The multi-scale convergence maps are input into a one-stage target detection head without anchors, and the target detection head outputs candidate boxes, target categories, and confidence scores.

[0114] In this example, the confidence threshold is set to... Non-maximum suppression crossover ratio threshold The object detection head outputs three candidate boxes: the pedestrian candidate box is... The confidence level is 0.91; the vehicle candidate box is... The confidence level is 0.88; the candidate boxes near the fence are The confidence level is 0.21.

[0115] The processor first deletes candidate boxes near the fence with a confidence level of 0.21. The intersection-union (IoU) ratios of the pedestrian and vehicle candidate boxes do not reach the non-maximum suppression IoU threshold, so both candidate boxes are retained. Detection results. Including pedestrian target category and target bounding box location With a confidence level of 0.91, and the vehicle target category and target bounding box location. And confidence level 0.88.

[0116] This example uses 12,000 pairs of registered and labeled visible light and infrared images, with 9,600 pairs in the training set, 1,200 pairs in the validation set, and 1,200 pairs in the test set. Paired images are simultaneously subjected to random horizontal flipping with a flip probability of 0.50, and geometric augmentation maintains the spatial correspondence between the two channels. Both training and inference employ the same general fusion image formation, edge partitioning, co-point region formation, bridge segment determination, and accepted loop closure rules.

[0117] The optimizer used was AdamW with an initial learning rate of 0.001, weight decay of 0.0005, a batch size of 16, and 120 training epochs. The first three epochs performed linear warm-up, after which the learning rate was reduced to 0.01 times the initial learning rate using a cosine-based approach. The weights for box regression loss were 7.5, object presence loss was 1.0, and class loss was 0.5. The training parameters were used to obtain the network parameters for the dual-band coding module 113, the general fusion module 114, the neck network, and the object detection head. The spatial partitioning rules and loop closure rules remained unchanged.

[0118] After one detection cycle is completed, the processor outputs the detection results. Test results Each target category, target bounding box position, and confidence score in the image comes from the same retained candidate bounding box; the candidate bounding boxes have undergone confidence filtering and non-maximum suppression, and the target bounding box positions follow the original image coordinate orientation.

[0119] Although embodiments of this application have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of this application, and all such modifications and variations fall within the scope defined by the appended claims.

Claims

1. A target detection method based on co-aperture multi-band feature fusion, characterized in that, include: The visible light image and infrared image of the same field of view are acquired simultaneously by a common aperture imaging device and placed in a common spatial coordinate system to form an aligned visible light image and an aligned infrared image. Encode the two separately to obtain a visible light encoded map and an infrared encoded map at the corresponding fusion scale. Then fuse the visible light encoded map and the infrared encoded map to obtain a normal fusion map. The fusion does not depend on the edges and closed loops. Obtain visible light edge maps and infrared edge maps respectively, and determine the common edge area, the visible light-specific edge area, and the infrared-specific edge area; A set of common areas is formed by connecting segments in the common boundary area that contact the ends of the visible light-specific boundary area or the infrared-specific boundary area. Connecting segments that contact different common areas at both ends are identified as visible light bridge segments and infrared bridge segments according to their wavelength. Using the common edge continuous path, visible light bridge segment and infrared bridge segment as path units connecting the common point area, search for a simple closed loop that returns from the common point area to the common point area and whose path unit and intermediate common point area do not overlap. The closed loop contains visible light bridge segment and infrared bridge segment and switches bands only in the common point area. For each accepted closed loop, visible light encoded images and infrared encoded images are taken at the visible light bridge segment and infrared bridge segment positions, respectively, and the rest are ordinary fusion images, which are then used to form a relay fusion image. If no closed loop is accepted, the relay fusion graph is a normal fusion graph. Detect the relay fusion image and output the target category, target bounding box position and confidence score.

2. The target detection method based on co-aperture multi-band feature fusion according to claim 1, characterized in that, Determining the common edge region, the visible light-specific edge region, and the infrared-specific edge region includes: The positions of the edges in the visible light edge map and the infrared edge map are represented by spatial coordinates. The positions where the distance between the spatial coordinates of the two edge positions is not greater than the alignment tolerance are determined as the corresponding positions. The corresponding positions are determined as the common edge area. The positions that do not correspond in the visible light edge map are determined as the visible light-specific edge area. The positions that do not correspond in the infrared edge map are determined as the infrared-specific edge area.

3. The target detection method based on co-aperture multi-band feature fusion according to claim 2, characterized in that, The set of common points is composed of connected cells in the common edge area. The connected cells are in contact with the end of the connected segment in the visible light unique edge area or the end of the connected segment in the infrared unique edge area, and the interior of the connected segment of the visible light bridge segment and the infrared bridge segment does not pass through a third common point area.

4. The target detection method based on common aperture multi-band feature fusion according to claim 1, characterized in that, The continuous path of the common edge connects two common regions, and the path unit forms the simple closed loop by connecting the beginning and end of the common regions.

5. The target detection method based on co-aperture multi-band feature fusion according to claim 1, characterized in that, When both visible light and infrared bridge segments exist simultaneously between the same pair of common areas, the search order of the path units is determined based on the comparison results of the on-site illuminance and the camera reference illuminance, as well as the comparison results of the on-site visible distance and the farthest monitoring distance. When the on-site illuminance is not lower than the camera reference illuminance and the on-site visible distance is not less than the farthest monitoring distance, the visible light bridge segment is searched first; otherwise, the infrared bridge segment is searched first.

6. The target detection method based on co-aperture multi-band feature fusion according to claim 1, characterized in that, Obtaining the visible light edge map and the infrared edge map includes: The arithmetic mean of the absolute values ​​of the channels at each spatial location of the visible light encoded image and the infrared encoded image is calculated. The gradient magnitude is calculated using the Sobel horizontal kernel and the Sobel vertical kernel. The gradient magnitude is normalized to a value range of zero to one and then binarized according to the edge binarization threshold.

7. The target detection method based on co-aperture multi-band feature fusion according to claim 1, characterized in that, Visible light encoded maps and infrared encoded maps are obtained at at least three corresponding fusion scales. Relay fusion maps of each fusion scale corresponding to the visible light encoded maps and infrared encoded maps are obtained. The relay fusion maps of each fusion scale are input into the neck network to obtain a multi-scale convergence map, and the multi-scale convergence map is input into the target detection head.

8. The target detection method based on co-aperture multi-band feature fusion according to claim 1, characterized in that, The target detection performed on the force fusion map includes: Delete candidate boxes with confidence scores below the confidence score screening threshold, and perform non-maximum suppression when the cross-union ratio (CUP) between candidate boxes reaches the non-maximum suppression CUP threshold.

Images