A training method for a Time-of-Flight (TOF) image object detection model based on structure-guided terms

Abstract

This application provides a method for training a Time-of-Flight (TOF) image target detection model based on a structure-guided term. The method includes: preprocessing infrared and depth images to form a multi-channel input tensor; inputting the input tensor into the target detection model and fusing features from the infrared and depth images through a structure-aware attention module in the model to generate an intermediate feature map; inputting the intermediate feature map into subsequent network layers of the target detection model for processing to obtain an output feature map, thus constructing a structure-guided loss; combining the structure-guided loss as a regularization term with the basic detection loss to form a total loss function, and updating the parameters of the target detection model by optimizing the total loss function until convergence. The technical problem addressed by this application is that existing methods fail to effectively utilize the spatial structure information provided by depth images, leading to problems such as misidentification and inaccurate localization in complex scenes with large lighting variations, strong background interference, or object occlusion.

Description

Technical Field

[0001] This invention relates to the field of intelligent vehicles, and more specifically, to a method for training a Time-of-Flight (TOF) image target detection model based on structure-guided terms. Background Technology

[0002] In the field of intelligent vehicles, in-vehicle visual perception systems rely on IR and depth image pairs provided by TOF cameras for target detection. In existing technologies, model training methods typically use only IR images as input, converting them into grayscale or pseudo-RGB images for training.

[0003] However, the relevant technologies have at least one of the following problems: existing methods fail to effectively utilize the spatial structure information provided by depth images, which makes the models prone to misidentification and inaccurate localization in complex scenes such as large changes in lighting, strong background interference, or object occlusion. Summary of the Invention

[0004] The technical problem solved by this invention is that existing methods fail to effectively utilize the spatial structure information provided by depth images, which leads to problems such as misidentification and inaccurate localization in complex scenes with large changes in lighting, strong background interference, or object occlusion.

[0005] To address the aforementioned problems, this invention provides a method for training a Time-of-Flight (TOF) image target detection model based on a structure-guided term. The method includes: acquiring infrared and depth images, and preprocessing the infrared and depth images to form a multi-channel input tensor; inputting the input tensor into a target detection model, and fusing features from the infrared and depth images through a structure-aware attention module in the model to generate an intermediate feature map; inputting the intermediate feature map into subsequent network layers of the target detection model for processing to obtain an output feature map; constructing a structure-guided loss by calculating the difference between the edge gradients of the output feature map and the edge gradients of the depth image; combining the structure-guided loss as a regularization term with the basic detection loss to form a total loss function, and updating the parameters of the target detection model by optimizing the total loss function until convergence.

[0006] Compared with existing technologies, the technical effects achieved by this solution are as follows: This application effectively solves the misidentification problem caused by the failure to utilize depth image structural information in traditional methods, and improves the localization accuracy of the target detection model under complex lighting conditions. Dynamic feature fusion is achieved through a structure-aware attention mechanism, enhancing the model's ability to represent target boundaries. The structure-guided loss, used as a regularization term to constrain the model learning process, enables the network to maintain sensitivity to geometric structures while optimizing the detection task, significantly reducing the false negative rate in background interference and partially occluded scenes.

[0007] In one embodiment of the present invention, an infrared image and a depth image are acquired, and the infrared image and depth image are preprocessed to form a multi-channel input tensor, including: S11, performing geometric alignment and size unification processing on the infrared image and depth image; S12, normalizing the processed infrared image and depth image respectively; S13, stitching the normalized infrared image channel and the depth image channel together to form a multi-channel input tensor.

[0008] Compared with existing technologies, the technical effects achieved by this solution are as follows: This solution ensures spatial consistency of two modal data through geometric alignment, eliminates sensor differences through normalization, and achieves preliminary fusion of structural and texture information through channel stitching, providing richer feature inputs for subsequent networks.

[0009] In one embodiment of the present invention, an input tensor is input into a target detection model, and features of infrared images and depth images are fused through a structure-aware attention module in the model to generate an intermediate feature map. This includes: extracting structural features based on the depth image and generating an attention map; using texture features extracted from the infrared image as query features and fusing them with structural features as keys and values ​​across modalities; and weighting the features fused from the infrared image and depth image according to the attention map to output the intermediate feature map.

[0010] Compared with existing technologies, the technical effects achieved by this solution are as follows: This application can effectively improve the recognition accuracy of the target detection model in scenarios with changes in illumination, background interference, and object occlusion. The structural information provided by the depth image guides the fusion process of infrared texture features through an attention mechanism, thereby enhancing the target boundary features and reducing false detections and false negatives caused by complex environmental interference. The dynamic fusion mechanism of cross-modal features enhances the model's ability to collaboratively utilize multi-source information and improves the robustness of feature representation.

[0011] In one embodiment of the present invention, extracting structural features based on a depth image and generating an attention map includes: performing gradient calculation on the depth image to extract its edge structure information; and generating an attention map based on the edge structure information to enhance the response of the target boundary region.

[0012] Compared with existing technologies, the technical effects achieved by adopting this technical solution are as follows: This application can enhance the model's ability to perceive the geometric boundaries of the target, and significantly reduce false detections and false negatives caused by edge blurring in scenes with changing lighting or complex backgrounds, thereby improving the stability and accuracy of detection results.

[0013] In one embodiment of the present invention, cross-modal fusion is achieved through the following formula 1: ;in, Features derived from the fusion of infrared and depth images. For attention maps, Texture features of infrared images, These are the structural features of a depth image.

[0014] Compared with existing technologies, the technical effects achieved by adopting this technical solution are as follows: This application can enhance the object detection model's ability to identify object boundaries in complex scenes, reduce false detections caused by changes in lighting or background interference, and avoid the problem of excessive suppression of single modal features, thereby improving the model's detection robustness in occluded scenes.

[0015] In one embodiment of the present invention, the intermediate feature map is input into a subsequent network layer of the target detection model for processing to obtain an output feature map, including: inputting the intermediate feature map into a subsequent network layer containing a multi-scale feature fusion structure; performing feature enhancement and scale integration on the intermediate feature map through the multi-scale feature fusion structure to generate an output feature map for target classification and localization.

[0016] Compared with existing technologies, the technical effects achieved by this solution are as follows: This application effectively improves the model's detection accuracy for occluded targets and multi-scale targets. In scenarios with drastic lighting changes or densely packed targets, the classification confidence and bounding box localization accuracy of the output feature map are significantly improved. The collaborative optimization mechanism of multi-scale features enables the model to simultaneously process high-resolution details in the foreground and low-resolution contour information in the background, reducing the probability of false positives and false negatives in complex environments.

[0017] In one embodiment of the present invention, the structure-guided loss is combined with the basic detection loss as a regularization term to form a total loss function, and the parameters of the object detection model are updated by optimizing the total loss function until convergence. This includes: weighting the structure-guided loss and the basic detection loss using adjustable hyperparameters to form the total loss function; guiding the model to execute a dual learning path from global localization to edge structure using the total loss function; controlling the hyperparameters using a dynamic adjustment mechanism to balance the training weights between the basic detection task and the structure-guided term; and updating the model parameters by optimizing the total loss function, forcing the model to explicitly learn the edge gradient features of the target in the depth image until the model converges.

[0018] Compared with existing technologies, the technical effects achieved by this approach are as follows: This approach introduces structure-guided loss as a regularization term to construct a multi-task joint optimization framework, enabling the model to simultaneously learn target semantic features and geometric structural features. Existing technologies often use fixed-weight loss combinations, which can easily lead to training instability. The dynamic adjustment mechanism employed in this approach adaptively adjusts the loss weights according to the training stage, avoiding interference from initial structural constraints on the convergence of the basic tasks while ensuring that structural information is fully integrated into the model parameters later. Furthermore, existing methods lack explicit supervision of the edge structure of deep images at the feature extraction layer, while this approach forces the model to maintain structural consistency in the feature space through gradient difference calculation, significantly enhancing its ability to represent target boundaries in complex scenes.

[0019] In one embodiment of the present invention, a multi-scale structural consistency mechanism is further included to maintain structural alignment between feature maps of different scales during the feature fusion stage. The multi-scale structural consistency mechanism includes: extracting edge gradients from intermediate layer feature maps of multiple scales in the target detection network; applying structural consistency constraints between intermediate layer feature maps of adjacent scales; and calculating difference loss after making the gradient map resolution consistent through upsampling operations.

[0020] Compared with existing technologies, the technical effects achieved by this solution are as follows: This application effectively solves the structural misalignment problem between multi-scale feature maps, enabling the model to maintain stable edge perception capabilities even in complex scenes with target occlusion, background interference, or lighting changes. Feature maps of different scales complement each other under structural constraints, improving the detection accuracy of small targets and blurred boundaries, while enhancing the consistency of the model's representation of the physical structure of objects.

[0021] In one embodiment of the present invention, the total loss function further includes a multi-scale structural consistency loss, the combination of which is given by the following formula 2: Formula 2: ;

[0022] For the total loss function, Based on the detection of loss, For structure-guided loss, λ1 and λ2 are the multi-scale structural consistency loss; where λ1 and λ2 are the adjustment coefficients used to balance the weights of each loss.

[0023] Compared with existing technologies, the technical effects achieved by this solution are as follows: This application can significantly improve the edge localization accuracy of the target detection model in complex scenarios such as occlusion and lighting changes, and reduce misidentification caused by inconsistencies in the multi-scale feature map structure. Simultaneously, the dynamic adjustment mechanism ensures that the model's performance does not degrade due to multi-task conflicts during optimization, enabling the structure guidance term and the basic detection task to achieve synergistic optimization.

[0024] In one embodiment of the present invention, the basic detection loss includes classification loss, bounding box regression loss, and confidence loss.

[0025] Compared with existing technologies, the technical effects achieved by this solution are as follows: By introducing confidence loss and combining it with a multi-task joint optimization mechanism, this solution enables the model to dynamically adjust the reliability of detection results. At the same time, by utilizing the complementarity of bounding box regression loss and structure-guided loss, the sensitivity to the edge structure of the target is enhanced, thereby achieving more stable detection performance in complex scenarios.

[0026] By adopting the technical solution of the present invention, the following technical effects can be achieved:

[0027] (1) By fusing structural information from depth images, the challenges brought by changes in illumination, background interference and object occlusion are effectively overcome, and the false recognition and false detection rates are significantly reduced.

[0028] (2) By using the structure-aware attention mechanism and structure-guided loss, the model is forced to explicitly learn the geometric contours of objects, thereby achieving more accurate bounding box localization.

[0029] (3) Through the multi-scale structural consistency mechanism and dynamic loss adjustment, the model can maintain stable edge perception for targets of different sizes, balance multi-task learning, and ensure the convergence and performance of the training process. Attached Figure Description

[0030] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings to be used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0031] Figure 1 This is a flowchart illustrating a method for training a TOF image target detection model based on structure-guided terms, as provided in an embodiment of the present invention.

[0032] Figure 2 This is a flowchart illustrating the specific process of step S1;

[0033] Figure 3 This is a schematic diagram of the specific process for step S2;

[0034] Figure 4 This is a flowchart illustrating the specific process of step S4;

[0035] Figure 5 This is a flowchart illustrating the specific process of step S5. Detailed Implementation

[0036] The following will refer to the appendix to this application. Figures 1 to 5 The technical solutions in this application are clearly and completely described. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. The components of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application. It should be noted that similar reference numerals and letters in the following drawings indicate similar items; therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. Furthermore, in the description of this application, the terms "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0037] In existing technologies, in-vehicle visual perception systems rely on infrared and depth images acquired by Time-of-Flight (TOF) cameras for target detection. Traditional methods typically use only infrared images as input, converting them to grayscale or pseudo-RGB images for model training. This single-modal data processing approach prevents the model from effectively utilizing the spatial structure information in the depth image, easily leading to target misidentification and positioning errors in scenarios with drastic lighting changes, complex background interference, or partial object occlusion. For example, in nighttime driving environments, the strong glare from oncoming vehicle headlights can cause infrared images to be overexposed, making it difficult to accurately identify pedestrian outlines relying solely on infrared data.

[0038] To address the aforementioned issues, the inventors discovered that edge gradient information in depth images can effectively characterize the geometric structure of targets, but existing models lack a mechanism to integrate this information into the detection process. By analyzing the characteristics of multimodal data, they recognized the complementarity between infrared and depth images in terms of texture and spatial structure features. Further investigation revealed that establishing a cross-modal association mechanism during the feature fusion stage and constraining the model's learning of structural features from depth images through a loss function can enhance the model's ability to perceive target boundaries. Based on this, they proposed introducing a structure-aware attention module into model training to achieve dynamic feature fusion, while simultaneously constructing a structure-guided loss function as a regularization term, forming a dual-supervision mechanism.

[0039] This invention provides a method for training a Time-of-Flight (TOF) image target detection model based on a structure-guided term, comprising: acquiring infrared and depth images, and preprocessing the infrared and depth images to form a multi-channel input tensor; inputting the input tensor into a target detection model, and fusing the features of the infrared and depth images through a structure-aware attention module in the model to generate an intermediate feature map; inputting the intermediate feature map into subsequent network layers of the target detection model for processing to obtain an output feature map; constructing a structure-guided loss by calculating the difference between the edge gradient of the output feature map and the edge gradient of the depth image; combining the structure-guided loss as a regularization term with the basic detection loss to form a total loss function, and updating the parameters of the target detection model by optimizing the total loss function until convergence.

[0040] Specifically, the multi-channel input tensor refers to a multi-dimensional data matrix formed by concatenating the channels of geometrically aligned infrared and depth images. This feature ensures that the model simultaneously receives both texture and structural information.

[0041] Furthermore, the structure-aware attention module refers to a neural network component that generates attention weights through cross-modal feature interaction, specifically employing a self-attention mechanism to achieve interactive computation between query features and key-value features. This module dynamically adjusts the feature fusion weights to enhance the feature response of the target boundary region.

[0042] Furthermore, the structure-guided loss refers to a supervisory signal constructed by comparing the edge gradient differences between the output feature map and the original depth image. Specifically, the mean squared error is calculated after extracting the gradient map using the Sobel operator. This loss function forces the model to maintain sensitivity to the target geometry.

[0043] Specifically, infrared and depth images are geometrically aligned and normalized to form a multi-channel input, providing the model with complementary texture and spatial information. The structure-aware attention module establishes cross-modal associations during feature extraction, dynamically allocating feature weights from the two data sources through an attention mechanism, allowing the network to focus on key regions such as target boundaries. Intermediate feature maps are processed by a multi-scale feature fusion network to generate output feature maps containing semantic information. The structure-guided loss constrains the structural consistency between the model output and the depth image through gradient difference constraints, serving as a regularization term to optimize network parameters along with the detection loss. This dual-supervision mechanism enables the model to explicitly learn edge structure features in the depth image while completing the target localization task, thereby improving detection robustness in complex scenes.

[0044] In one specific embodiment, firstly, for network input processing: the input image is a pair of IR (infrared) and Depth (depth) images from a pulse TOF camera. After geometric alignment, size unification, and normalization of the two images, they are stitched together by channel to form a dual-channel or triple-channel input (such as IR+Depth+Edge).

[0045] Among them, the Time-of-Flight (TOF) camera is an active depth imaging device. Its working principle involves emitting high-intensity, short-duration infrared pulses into the scene and calculating the distance (depth) from each pixel to the camera by measuring the round-trip time of the pulse from emission to reflection from the target. This principle can be described by the following formula:

[0046] ;

[0047] Where d represents the distance (i.e., depth value) from the target pixel to the camera. c t represents the speed of light (approximately 3 x 10^8 m / s); t represents the round-trip time difference between the emission of the light pulse and its reflection back to the receiver.

[0048] The mathematical expression for IR-Depth image pairs, assuming the image resolution is H×W, is as follows:

[0049] ;

[0050] ;

[0051] These are the RGB pixel values; These are the pixel values ​​of the depth map; Let be the set of positive real numbers.

[0052] After stitching, the IR image and the Depth image are pixel-aligned, constructing a two-channel input tensor:

[0053] ;

[0054] For multi-channel input tensors at pixel positions The vector at that location; For infrared images in pixels The brightness value at that location; For depth images in pixels The depth value at that location.

[0055] The formula represents the feature vector of the input tensor T at the spatial position (x, y), and the final tensor shape is: C×H×W, where C=2 and H / W is the uniform input size.

[0056] The model structure is then modified. For example, YOLO's Backbone feature extraction network is responsible for extracting multi-scale, high-semantic feature representations from the input image layer by layer. As the layers deepen, the resolution gradually decreases while the number of channels gradually increases. At this point, an attention mechanism can be inserted at mid-to-high levels, where spatial details and semantic information are relatively balanced, to guide the model's ability to recognize structural boundaries and target shapes, capture local spatial geometric relationships, and improve the accuracy of spatial perception. When the model fuses features at multiple scales, the features at each scale are connected across layers, resulting in inconsistent spatial resolution. Modification is needed to ensure consistency across the multi-scale structure. The following effects can be achieved using a local attention mechanism:

[0057] 1. Enhance cross-scale structural consistency and reduce feature breaks between scales;

[0058] 2. Apply spatial location encoding to the fused feature map to guide the model to focus on key spatial structures and improve detection accuracy;

[0059] 3. By combining the spatial geometric information provided by the depth map, features at different scales can carry structure-aware contextual information, promoting the structural consistency and robustness of the fused features.

[0060] The LSAM (Structure Aware Attention Module) receives intermediate outputs from two feature streams. When extracting IR and Depth features, it introduces spatial location awareness and structural consistency constraints, embedding them into the structure-guided loss function and the multi-scale feature fusion part of the model. This enables the model to understand the spatial geometric relationships between objects more accurately, and is expected to show stronger target localization capabilities in complex structures inside vehicles.

[0061] Traditional object detection losses (such as those of the YOLO series) typically include classification losses, bounding box regression losses, and confidence losses. The core of these losses is optimization based on calculating the difference between the model's predicted values ​​and the ground truth labels. This type of loss primarily relies on pixel information and label boxes from RGB or IR images, lacking explicit constraints on spatial geometry. Introducing depth information means predicting not only two-dimensional boxes and categories, but also requiring the model to learn and understand the spatial structure of the depth map. Therefore, based on the original supervised loss, a structure-guided term is introduced as a regularization term, defining the loss function as follows:

[0062] ;

[0063] is the structure-guided loss; H is the height of the feature map; W is the width of the feature map; This represents the value of the model's output feature map at pixel (x, y); This represents the value of the corresponding pixel in the depth map; This represents gradient operators (such as Sobel, Scharr, etc.) used to extract the edge structure of an image;

[0064] The square norm is used to measure the Euclidean distance between two gradient vectors.

[0065] This loss encourages the model's output features to align with the depth map on the edge structure, thereby enhancing the ability to perceive target boundaries.

[0066] In one embodiment of the present invention, an infrared image and a depth image are acquired, and the infrared image and depth image are preprocessed to form a multi-channel input tensor, including: S11, performing geometric alignment and size unification processing on the infrared image and depth image; S12, normalizing the processed infrared image and depth image respectively; S13, stitching the normalized infrared image channel and the depth image channel together to form a multi-channel input tensor.

[0067] Specifically, geometric alignment and size unification processing refers to eliminating spatial misalignment between two images through affine or perspective transformations. This can be achieved by extracting key points from the images using a feature point matching algorithm and then calculating the transformation matrix. This process ensures the pixel-level correspondence between the two modal data, avoiding spatial deviations during cross-modal feature fusion.

[0068] Furthermore, normalization refers to linearly mapping image pixel values ​​to a preset numerical range, which can be achieved using maximum-minimum normalization or standard deviation normalization methods. This process eliminates the influence of differences in data distribution from different sensors on model training, providing standardized input for subsequent feature fusion.

[0069] Furthermore, channel stitching refers to stacking two single-channel images along the channel dimension, which can be achieved using tensor stitching operations. This process achieves a preliminary fusion of the texture information of the infrared image and the structural information of the depth image, forming complementary feature representations.

[0070] Specifically, the geometric alignment and size unification stage establishes a spatial correspondence between the two images through a feature point matching algorithm and adjusts the image geometry using a transformation matrix to ensure complete spatial alignment between the infrared and depth images. The normalization stage independently standardizes the data for both images, eliminating differences in brightness and depth values ​​caused by sensor variations. The channel stitching stage combines the standardized infrared and depth images into a dual-channel tensor, enabling subsequent networks to simultaneously extract texture and structural features. These three processing steps form a progressive data processing chain: spatial alignment eliminates intermodal misalignment, data standardization unifies the input distribution, and channel stitching achieves feature fusion, collectively addressing the problem of effectively integrating multimodal data input.

[0071] Through the above technical solutions, this application effectively improves target detection accuracy in complex scenarios of intelligent vehicles. When illumination changes drastically, the structural information provided by the depth image can compensate for the lack of texture features in the infrared image; when background interference is strong, the spatial information of the depth channel helps distinguish between foreground and background; and when objects are occluded, structural features can assist in locating the edge contours of the occluded target. The multi-channel input tensor generated by the preprocessing step enables the model to simultaneously utilize texture and structural information, enhancing detection robustness in complex environments.

[0072] In one embodiment of the present invention, an input tensor is input into a target detection model, and features of infrared images and depth images are fused through a structure-aware attention module in the model to generate an intermediate feature map. This includes: extracting structural features based on the depth image and generating an attention map; using texture features extracted from the infrared image as query features and fusing them with structural features as keys and values ​​across modalities; and weighting the features fused from the infrared image and depth image according to the attention map to output the intermediate feature map.

[0073] Specifically, the structure-aware attention module refers to a neural network component that uses the spatial structure information of depth images to generate attention weights, which is used to enhance the feature response of the target boundary region.

[0074] In one embodiment, the Structure-Aware Attention Module (LSAM) is designed to be embedded in the mid-to-high-level stages of the object detection network, primarily acting on the feature layer that possesses a certain semantic understanding capability while still retaining some spatial structural information.

[0075] Taking the YOLO series of networks as an example, LSAM can be inserted after layers C3 or C4. These two layers correspond to low-to-medium resolution and high-to-medium resolution feature map outputs, respectively. The feature maps at this stage already have strong semantic abstraction capabilities, but have not completely lost spatial location details, making them suitable for introducing additional structural information to enhance the modeling of object edges and geometric contours. Through LSAM, the model is guided to enhance responses to object contours, boundaries, and abrupt structural changes, while suppressing background or texture interference areas, thereby improving the target resolution capability in scenes with occlusion or similar backgrounds.

[0076] Furthermore, cross-modal fusion refers to establishing the feature interaction relationship between infrared images and depth images through an attention mechanism, using infrared texture features as query vectors and depth structural features as key and value vectors to achieve information complementarity between modalities.

[0077] Furthermore, the attention map refers to the weight distribution matrix that reflects the importance of the target edge structure, and is used to dynamically adjust the intensity of feature fusion.

[0078] Specifically, after gradient calculation to extract edge structure information from the depth image, an attention map with spatial guidance is generated. Texture features extracted from the infrared image through convolutional layers are used as a query vector, and this vector is multiplied by the key-value pairs formed by the structural features of the depth image to calculate the attention weight distribution. This weight distribution acts on the fused feature map, enhancing the feature responses of target boundary regions and suppressing features of background interference regions. Through this mechanism, the thermal radiation information of the infrared image and the spatial structure information of the depth image synergize in the feature space, enhancing the model's ability to represent target objects in complex scenes.

[0079] In one embodiment of the present invention, extracting structural features based on a depth image and generating an attention map includes: performing gradient calculation on the depth image to extract its edge structure information; and generating an attention map based on the edge structure information to enhance the response of the target boundary region.

[0080] Gradient calculation refers to the operation of edge detection on depth images using mathematical operators, specifically the Sobel operator, which extracts the contour information of the target by calculating the gradient magnitude of the depth image in the horizontal and vertical directions.

[0081] Specifically, in depth images, the spatial structure information of objects is explicitly extracted through gradient calculation. The Sobel operator is used to calculate the gradient components in the x and y directions of the image, and the results are merged to obtain an edge structure map. Subsequently, the gradient magnitude of the edge structure map is input into an activation function to generate an attention map, which exhibits high weight values ​​in the target boundary region. During feature fusion, the attention map spatially modulates the features of the infrared image, thereby enhancing the model's feature response in the target contour region and improving the accuracy of boundary localization.

[0082] In one embodiment of the present invention, cross-modal fusion is achieved through the following formula 1: Formula 1: ;in, Features derived from the fusion of infrared and depth images. For attention maps, Texture features of infrared images, These are the structural features of a depth image.

[0083] Cross-modal fusion: It uses an attention mechanism to use the texture features extracted from the IR image as the query and the structural features extracted from the depth image as the key and value, thereby enhancing the attention response in the boundary region. The fused features carry the geometric structure information of the object and are particularly suitable for target detection tasks in complex scenes such as occlusion and lighting changes.

[0084] In one embodiment of the present invention, the intermediate feature map is input into a subsequent network layer of the target detection model for processing to obtain an output feature map, including: inputting the intermediate feature map into a subsequent network layer containing a multi-scale feature fusion structure; performing feature enhancement and scale integration on the intermediate feature map through the multi-scale feature fusion structure to generate an output feature map for target classification and localization.

[0085] Specifically, multi-scale feature fusion structures refer to network modules that can integrate feature information at different levels. They can be implemented using feature pyramids or path enhancement structures to extract semantic information from different receptive fields to enhance the model's adaptability to changes in target size.

[0086] Furthermore, feature enhancement refers to weighting key regions of the feature map, which can be achieved through spatial attention mechanisms or channel attention mechanisms to improve the feature response intensity of target edges and detailed regions.

[0087] Furthermore, scale integration refers to spatial alignment and semantic fusion of feature maps at different resolutions. Specifically, it can be achieved by upsampling and feature concatenation operations to eliminate semantic differences between multi-scale features and establish cross-layer feature associations.

[0088] Specifically, after the intermediate feature maps are input into subsequent network layers, feature representations at different levels are first obtained through cross-layer connections. In the feature pyramid architecture, deep features are spatially aligned with shallow high-resolution features through upsampling operations, and then feature fusion is achieved through element-wise addition or channel concatenation. During the fusion process, an attention mechanism is applied to the channel dimension or spatial location of the feature maps, for example, amplifying the feature activation values ​​of target boundary regions. After multiple rounds of feature interaction and resolution adjustment, a final output feature map with unified semantic representation capabilities is generated. This feature map simultaneously preserves fine-grained local details and high-level semantic information, providing robust feature support for subsequent classification and localization tasks.

[0089] In one embodiment of the present invention, the structure-guided loss is combined with the basic detection loss as a regularization term to form a total loss function, and the parameters of the object detection model are updated by optimizing the total loss function until convergence. This includes: weighting the structure-guided loss and the basic detection loss using adjustable hyperparameters to form the total loss function; guiding the model to execute a dual learning path from global localization to edge structure using the total loss function; controlling the hyperparameters using a dynamic adjustment mechanism to balance the training weights between the basic detection task and the structure-guided term; and updating the model parameters by optimizing the total loss function, forcing the model to explicitly learn the edge gradient features of the target in the depth image until the model converges.

[0090] Specifically, structure-guided loss is a loss term constructed by calculating the difference between the edge gradients of the output feature map and the edge gradients of the depth image. The Sobel operator is used to extract the edge gradients, and the difference between the two is calculated using the L1 norm. Its purpose is to constrain the feature map output by the model to maintain consistency with the depth image in terms of edge structure, thereby enhancing the model's ability to perceive the geometric features of the target.

[0091] The Structure-Guided Loss function aims to address the following key issues in the training of existing object detection networks:

[0092] Traditional detection loss methods (such as YOLO and Faster R-CNN) mainly focus on overall target information such as class confidence and bounding box localization, but lack the ability to model the detailed structure of the target, especially the edge contour and object shape boundary.

[0093] In real-world scenarios, such as uneven lighting, complex backgrounds, and occlusion between targets, accurately distinguishing the boundary between the target and the background becomes crucial to the detection accuracy. Therefore, this invention designs a method to guide the model to focus on the edge gradient features of the target in the depth map from a "structural level". By introducing a gradient consistency regularization term, the model is prompted to explicitly learn the geometric contour and spatial boundary of the target during the training phase, thereby enhancing the model's structural perception ability and small target recognition ability.

[0094] Implementation path:

[0095] Structural alignment constraint: This step is used to measure the structural difference between the model output feature map and the original depth map in the edge region. First, the same gradient operator (such as Sobel) is used to extract the edge response for both the model output feature map and the original depth map. Then, the structural guidance loss term is defined as the sum of squared pixel-level differences between the two gradient maps.

[0096] Regularization: This difference is used as an additional structural guide for the loss term, guiding the model to explicitly learn the edge contour of the target.

[0097] Dynamic adjustment mechanism: An adjustable hyperparameter balancing structure is introduced to guide the training weights between the model and the basic detection task. This design guides the model through a dual learning path from the whole to the edge, which improves the localization accuracy and enhances the model's ability to identify complex boundaries, especially for small objects, occlusions, and multi-target contact areas, with significant optimization effects.

[0098] The specific form of the loss function is as follows:

[0099] ;

[0100] And combined with the basic detection loss, it becomes:

[0101] ;

[0102] This is the total loss function; Based on loss detection; This is the adjustment coefficient for the structure-guided term. For structure-guided loss.

[0103] Furthermore, the dynamic adjustment mechanism refers to the strategy of adaptively adjusting the weights of the structure-guided loss and the basic detection loss according to the training phase. Its role is to avoid the structural constraints being too strong in the early stage of training due to fixed weights, which would affect the convergence of the basic detection task, while ensuring that the structural information is fully integrated into the model in the later stage of training.

[0104] Furthermore, adjustable hyperparameters refer to coefficients used to control the relative importance of different loss terms. Their role is to establish a multi-task optimization framework, enabling the model to balance target localization accuracy and structural feature learning.

[0105] Specifically, during model training, the basic detection loss and the structure-guided loss are first combined with initial weights to form the total loss function. The basic detection loss supervises target classification and localization accuracy, while the structure-guided loss guides the model to focus on the target's edge structure through gradient difference constraints. In the early stages of training, a dynamic adjustment mechanism gradually increases the weight of the structure-guided loss, allowing the model to strengthen structure learning after initially mastering target localization capabilities. As training progresses and the basic detection task stabilizes, the structural information of the depth image is further encoded into the feature extraction layer through gradient backpropagation. During this process, model parameter updates are simultaneously constrained by both target localization error and structural differences, forcing the feature extraction network to actively capture geometric features in the depth image. For example, in occluded scenarios, even if the target texture information in the infrared image is incomplete, the model can still learn the edge gradient patterns of the occluded target through the structure-guided term, thereby improving detection robustness.

[0106] Through the above technical solutions, this application can effectively solve the problem of infrared image texture blurring caused by sudden changes in illumination in the in-vehicle visual perception scenario of intelligent vehicles. For example, when the illumination conditions inside and outside the vehicle change drastically, the edge features of the depth image learned by the model through the structure-guided term can help identify the target contours interfered with by strong light. In scenarios where the target is partially occluded, the model can combine the spatial structure information provided by the depth image to accurately infer the boundary position of the occluded area, reducing false detections and false negatives. At the same time, the dynamic adjustment mechanism ensures that the model gradually adapts to multi-task optimization during training, avoiding the decrease in positioning accuracy caused by excessive structural constraints, and ultimately achieving stable target detection in complex environments.

[0107] In one embodiment of the present invention, a multi-scale structural consistency mechanism is further included to maintain structural alignment between feature maps of different scales during the feature fusion stage. The multi-scale structural consistency mechanism includes: extracting edge gradients from intermediate layer feature maps of multiple scales in the target detection network; applying structural consistency constraints between intermediate layer feature maps of adjacent scales; and calculating difference loss after making the gradient map resolution consistent through upsampling operations.

[0108] Specifically, the multi-scale structural consistency mechanism refers to maintaining the geometric alignment between feature maps of different scales in the feature pyramid network through cross-level constraints. Its role is to eliminate structural misalignment when features of different scales are fused.

[0109] Furthermore, edge gradient extraction refers to obtaining the target edge response from the feature map, specifically using the Sobel operator to capture the physical structure information in features at different levels of abstraction.

[0110] Furthermore, structural consistency constraints refer to forcing the edge distribution of feature maps at adjacent scales to tend to be consistent, in order to maintain the structural continuity of features across scales.

[0111] Furthermore, upsampling refers to scaling up a low-resolution gradient map to a high resolution to unify the spatial dimensions of gradient maps at different scales for matching calculations.

[0112] To improve the model's ability to model edge structure information in multi-scale scenarios, especially its robustness in detecting small targets, this invention further proposes a multi-scale structural consistency mechanism to maintain structural alignment between feature maps of different scales during the feature fusion stage.

[0113] In target detection networks based on Feature Pyramid Network (FPN) or Path Enhancement Network (PAN) structures, feature maps span multiple resolution levels. Without structural consistency constraints, issues such as broken edge information and blurred boundaries may occur between different scales, especially noticeable in small targets.

[0114] Core Mechanism:

[0115] Cross-scale alignment: Edge gradient extraction is performed on feature maps at different scales, and structural consistency constraints are applied between adjacent scales;

[0116] Upsampling alignment: Upsampling is used to move high-level feature maps up to the low-level scale so that the gradient maps have the same resolution before alignment.

[0117] Multi-scale structural consistency loss is defined as follows:

[0118]

[0119] in, This represents the loss of structural consistency across multiple scales. This represents the feature map at the L-th scale. This represents the edge gradient map of the feature map at layer L; Upsampling operations are typically bilinear interpolation or deconvolution. To represent upsampling the high-level feature gradient at low resolution, so that it is similar to the low-level feature gradient... Alignment.

[0120] In summary, the final training loss function can be integrated as follows:

[0121] ;

[0122] For the total loss function, Based on the detection of loss, For structure-guided loss, This represents the loss of structural consistency across multiple scales.

[0123] Specifically, in the feature pyramid of the object detection network, feature maps at different levels generate corresponding gradient maps through the edge gradient extraction module. Lower-level feature maps are upsampled to align their resolution with the gradient maps of adjacent higher-level feature maps, and then the difference loss between them is calculated. This loss function is backpropagated to the network, forcing feature maps at each scale to maintain consistency in edge structure during fusion. By layer-wise constraining gradient matching between adjacent scales, deep semantic features and shallow detail features achieve synergistic optimization in geometric structure, thereby suppressing target boundary blurring or misalignment caused by scale jumps.

[0124] In one embodiment of the present invention, the total loss function further includes a multi-scale structural consistency loss, the combination of which is given by the following formula 2: Formula 2:

[0125] ;

[0126] For the total loss function, Based on the detection of loss, For structure-guided loss, λ1 and λ2 are the multi-scale structural consistency loss; where λ1 and λ2 are the adjustment coefficients used to balance the weights of each loss.

[0127] Specifically, multi-scale structural consistency loss refers to a loss term that forces the model to maintain cross-scale structural alignment by imposing edge gradient consistency constraints between feature maps at different scales. This can be achieved by extracting edge gradients from multiple intermediate layer feature maps and then calculating the difference loss after unifying the resolution through upsampling. This loss term suppresses edge localization deviations caused by the attenuation of structural information between feature pyramid levels by constraining the structural consistency between feature maps at different scales.

[0128] Here, the adjustment coefficients λ1 and λ2 are trainable parameters used to dynamically adjust the weights of the structure guidance loss and the multi-scale structure consistency loss in the total loss function. Specifically, this can be achieved by dynamically adjusting the weights based on the convergence speed of each loss term during training. By adjusting these coefficients, the basic detection task can be optimized first in the early stages of training, and the structural constraint strength can be gradually increased in the later stages of training, thus achieving a balance between the optimization objectives.

[0129] Specifically, during the training of the object detection model, the total loss function is constructed as a weighted combination of basic detection loss, structure guidance loss, and multi-scale structural consistency loss. The multi-scale structural consistency loss works as follows: first, intermediate feature maps are extracted from different levels of the feature pyramid network, and edge gradients are calculated for each layer; then, low-resolution gradient maps are aligned with adjacent high-resolution gradient maps through bilinear upsampling; finally, the mean squared error between the two is calculated as the structural consistency loss. This loss term is added to the total loss function, forming a multi-level structural constraint together with the structure guidance loss. The adjustment coefficients λ1 and λ2 are dynamically adjusted according to the training stage; for example, they are set to smaller values ​​in the early stages of training to prioritize the detection task, and gradually increased in the later stages of training to strengthen structural alignment. Thus, the model, during optimization, not only ensures the core performance of object classification and localization but also improves edge localization accuracy through cross-scale gradient consistency constraints.

[0130] In one embodiment of the present invention, the basic detection loss includes classification loss, bounding box regression loss, and confidence loss.

[0131] Specifically, classification loss refers to a loss function that optimizes the model's classification ability by calculating the difference between the predicted class and the true label, and is used to constrain the model's ability to distinguish the target class.

[0132] Furthermore, bounding box regression loss is a loss function that measures the positional deviation between the predicted bounding box coordinates and the true labeled box, and is used to improve the accuracy of target localization.

[0133] Furthermore, confidence loss refers to the loss function that evaluates the probability confidence of whether the prediction box contains the target object, and is used to suppress the interference of low-quality prediction results.

[0134] The three elements are jointly optimized to form a multi-task training framework, which establishes a synergistic effect between target recognition, localization accuracy and detection reliability. At the same time, the structure-guided loss constrains the edge gradient, enabling the model to achieve a balance between overall localization and local structural feature learning.

[0135] Through the above technical solution, this application effectively solves the problems of misidentification and positioning deviation caused by ignoring depth image structure information in the prior art. In scenarios with drastic changes in lighting, background interference, or partial occlusion of objects, it can simultaneously optimize the target classification accuracy, positioning accuracy, and detection confidence, significantly reduce the false detection rate and false negative rate, and enhance the model's ability to perceive the target edge structure, thereby improving the comprehensive detection robustness in complex environments.

[0136] In summary, the training process of a Time-of-Flight (TOF) image object detection model based on structure-guided terms is as follows:

[0137] S1. Acquire infrared and depth images, and preprocess the infrared and depth images to form a multi-channel input tensor;

[0138] S2. Input the input tensor into the target detection model, and fuse the features of the infrared image and the depth image through the structure-aware attention module in the model to generate an intermediate feature map;

[0139] S3. Input the intermediate feature map into the subsequent network layers of the object detection model for processing to obtain the output feature map;

[0140] S4. Construct a structure-guided loss by calculating the difference between the edge gradients of the output feature map and the edge gradients of the depth image;

[0141] S5. Combine the structure-guided loss as a regularization term with the basic detection loss to form the total loss function, and update the parameters of the object detection model by optimizing the total loss function until convergence.

[0142] The following is the detailed process of step S1:

[0143] S11. Perform geometric alignment and size unification processing on infrared and depth images;

[0144] S12. Normalize the processed infrared image and depth image respectively;

[0145] S13. The normalized infrared image channel and the depth image channel are stitched together to form a multi-channel input tensor.

[0146] The following is the detailed process of step S2:

[0147] S21. Extract structural features from depth images and generate attention maps;

[0148] S22. Use the texture features extracted from the infrared image as query features and perform cross-modal fusion with the structural features that serve as keys and values;

[0149] S23. Based on the attention map, weight the features after fusing the infrared image and the depth image, and output the intermediate feature map.

[0150] The following is the detailed process for step S4:

[0151] S41. Input the intermediate feature map into the subsequent network layer containing the multi-scale feature fusion structure;

[0152] S42. The intermediate feature map is enhanced and scaled by a multi-scale feature fusion structure to generate an output feature map for target classification and localization.

[0153] The following is the detailed process for step S5:

[0154] S51. The structural guidance loss and the basic detection loss are weighted and combined using adjustable hyperparameters to form the total loss function;

[0155] S52. Guide the model to execute a dual learning path from overall localization to edge structure through the total loss function;

[0156] S53. Use a dynamic adjustment mechanism to control hyperparameters to balance the training weights between the basic detection task and the structure guidance term;

[0157] S54. Update the model parameters by optimizing the total loss function, forcing the model to explicitly learn the edge gradient features of the target in the depth image until the model converges.

[0158] While the present invention has been disclosed above, it is not limited thereto. Any person skilled in the art can make various modifications and alterations without departing from the spirit and scope of the invention; therefore, the scope of protection of the present invention should be determined by the scope defined in the claims.

Claims

1. A method for training a TOF image target detection model based on structure-guided items, characterized in that, The method comprises the following steps: acquiring an infrared image and a depth image, and pre-processing the infrared image and the depth image to form a multi-channel input tensor; inputting the input tensor into a target detection model, and fusing the features of the infrared image and the depth image through a structure perception attention module in the model to generate an intermediate feature map; inputting the intermediate feature map into a subsequent network layer of the target detection model for processing to obtain an output feature map; constructing a structure guiding loss by calculating the difference between the edge gradient of the output feature map and the edge gradient of the depth image; combining the structure guiding loss as a regularization term with a basic detection loss as a total loss function, and updating the parameters of the target detection model by optimizing the total loss function until convergence; the inputting the input tensor into a target detection model, and fusing the features of the infrared image and the depth image through a structure perception attention module in the model to generate an intermediate feature map comprises: extracting structure features based on the depth image, and generating an attention map; extracting texture features of the infrared image as query features, and cross-modally fusing the texture features with the structure features as keys and values; weighting the features fused from the infrared image and the depth image according to the attention map, and outputting the intermediate feature map; the cross-modal fusion is realized through the following formula 1: formula 1: ； Wherein, the is a feature after the infrared image and the depth image are fused, the is the attention map, the is the texture feature of the infrared image, the is the structure feature of the depth image. 2.The TOF image object detection model training method of claim 1, wherein, the acquiring an infrared image and a depth image, and pre-processing the infrared image and the depth image to form a multi-channel input tensor comprises: S11, geometric alignment and size unification processing are performed on the infrared image and the depth image; S12, the processed infrared image and the depth image are normalized respectively; S13, the normalized infrared image channel and the depth image channel are spliced to form the multi-channel input tensor. 3.The TOF image object detection model training method of claim 1, wherein, the extracting structure features based on the depth image, and generating an attention map comprises: gradient calculation is performed on the depth image to extract its edge structure information; the attention map for enhancing the response of the target boundary region is generated according to the edge structure information. 4.The TOF image object detection model training method of claim 1, wherein, the inputting the intermediate feature map into a subsequent network layer of the target detection model for processing to obtain an output feature map comprises: the intermediate feature map is inputted into the subsequent network layer containing a multi-scale feature fusion structure; the intermediate feature map is subjected to feature enhancement and scale integration through the multi-scale feature fusion structure to generate the output feature map for target classification and positioning. 5.The TOF image object detection model training method of claim 1, wherein, the combining the structure guiding loss as a regularization term with a basic detection loss as a total loss function, and updating the parameters of the target detection model by optimizing the total loss function until convergence comprises: the structure guiding loss and the basic detection loss are weighted and combined through an adjustable hyperparameter to constitute the total loss function; the model is guided to perform a dual learning path from overall positioning to edge structure through the total loss function; the hyperparameter is controlled by a dynamic adjustment mechanism to balance the training weight between the basic detection task and the structure guiding term; The model parameters are updated by optimizing the total loss function, so as to force the model to explicitly learn the edge gradient features of the target in the depth image until the model converges. 6.The TOF image object detection model training method of claim 1, wherein, Further comprising: A multi-scale structure consistency mechanism for maintaining structure alignment between feature maps of different scales in a feature fusion stage, the multi-scale structure consistency mechanism comprising: extracting edge gradients from intermediate layer feature maps of multiple scales in the target detection network; applying a structure consistency constraint between the intermediate layer feature maps of adjacent scales, and calculating a difference loss after the gradient map resolution is made consistent through an upsampling operation. 7.The TOF image object detection model training method of claim 1, wherein, The total loss function further comprises a multi-scale structure consistency loss, which is combined in the form of formula 2 as follows: Formula 2: ； The is the total loss function, the is the base detection loss, the is the structure guidance loss, the is the multi-scale structure consistency loss; Wherein, λ1 and λ2 are adjustment coefficients for balancing the loss weights of each term.

8. The TOF image target detection model training method according to claim 1, characterized in that, The basic detection loss includes a classification loss, a bounding box regression loss, and a confidence loss.

Images