Image fusion driven anti-collision recognition method and system

Abstract

The present invention relates to the field of image processing technology, and in particular to an image fusion-driven anti-collision recognition method and system. The method comprises the following steps: acquiring multi-source image data, and performing semantically guided image fusion based on the multi-source image data to obtain image fusion data; performing scene segmentation based on the image fusion data to obtain scene segmentation data, and performing target detection on the scene segmentation data to obtain scene target data; performing confidence-layered spatial positioning based on the scene target data to obtain target positioning data, and performing motion trajectory prediction based on the target positioning data to obtain motion trajectory prediction data; performing collision area propagation risk assessment based on the motion trajectory prediction data to obtain collision risk data. By fusing image perception depth, semantic understanding dimension and trajectory evolution modeling, the present invention has stronger environmental adaptability, weak target recovery capability and system interpretability.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an image fusion-driven anti-collision recognition method and system. Background Art

[0002] With the continuous expansion of application scenarios such as autonomous driving, intelligent robots, and unmanned inspection equipment, visual-based environmental perception and collision risk identification technologies have become core components of intelligent system safety control. Existing technologies typically use a single image source (such as RGB or infrared images) for target detection and collision assessment. These are susceptible to factors such as lighting variations, occlusion interference, and image blur, leading to inaccurate target recognition and missed detection of weak targets, which in turn reduces the system's robustness in diverse environments. Summary of the Invention

[0003] In order to solve the above technical problems, the present invention proposes an image fusion driven anti-collision recognition method and system to solve at least one of the above technical problems.

[0004] This application provides an image fusion driven anti-collision recognition method, comprising the following steps:

[0005] Step S1: acquiring multi-source image data, and performing semantic-guided image fusion based on the multi-source image data to obtain image fusion data;

[0006] Step S2: performing scene segmentation based on the image fusion data to obtain scene segmentation data, and performing target detection on the scene segmentation data to obtain scene target data;

[0007] Step S3: performing confidence-level hierarchical spatial positioning based on the scene target data to obtain target positioning data, and performing motion trajectory prediction based on the target positioning data to obtain motion trajectory prediction data;

[0008] Step S4: perform collision area propagation risk assessment based on the motion trajectory prediction data to obtain collision risk data.

[0009] In this invention, step S1 introduces a semantically guided image fusion mechanism, improving the fusion quality of multi-source images in terms of structural features, clarity, and semantic consistency, providing more stable visual input for scene analysis. Step S2 combines the fused images for scene segmentation and target detection, enabling accurate identification of multiple targets and multi-semantic entities within an environment. Step S3 introduces confidence-based spatial positioning and motion trajectory prediction based on target detection results, improving the tracking stability of key targets and enhancing the system's ability to model target behavior in dynamic scenarios. Step S4 further utilizes the predicted trajectories to construct a collision zone propagation model, enabling visual diffusion reasoning and risk level determination of risk areas, significantly improving the response accuracy and foresight of the collision avoidance system in multi-target, multi-path interaction scenarios. This invention boasts the advantages of lightweight, efficient, and intelligently evolving fusion recognition and dynamic prediction, making it suitable for a variety of scenarios, including intelligent driving (such as real-time monitoring of tower cranes and vehicles in their operating environments), unmanned equipment, and security inspections.

[0010] Preferably, step S1 is specifically:

[0011] Acquire multi-source image data;

[0012] Extracting edge maps based on multi-source image data to obtain edge map data;

[0013] Perform multi-scale convolution blurring on the edge map data to obtain convolution blurred data, and perform sharpening on the convolution blurred data to obtain fused edge map data;

[0014] Perform semantic region mapping on edge graph data to obtain graph semantic region data;

[0015] Calculate the edge density of the fused edge graph data based on the graph semantic region data to obtain the structural weight data;

[0016] Perform high-frequency response calculation based on multi-source image data to obtain image clarity data;

[0017] The multi-source image data is weightedly fused according to the structural weight data and the image clarity data to obtain image fusion data.

[0018] This method enhances the key structural contours and detail responses of multi-source images by extracting edge maps and performing multi-scale convolution blurring and sharpening. It also introduces a semantic region mapping mechanism, combined with edge maps, to assess the edge density of specific semantic regions, thereby constructing semantically perceptual structural weight data. It also extracts image clarity metrics through high-frequency responses, helping to identify information-dense and blurred areas within the image. By fusing structural weight data with image clarity data and adaptively weighting multi-source images based on regional importance, the method significantly improves the fused image's structural integrity, detail preservation, and semantic continuity.

[0019] Preferably, the semantic area mapping is specifically:

[0020] Extracting local color features and local texture features from edge image data to obtain image color feature data and image texture feature data respectively;

[0021] Use the preset lightweight image semantic recognition model to perform low-resolution semantic reasoning on the image color feature data and image texture feature data to obtain semantic category label data and semantic confidence data;

[0022] Constructing a proximity graph based on the semantic category label data and the semantic confidence data to obtain proximity graph data;

[0023] The edge graph data is mapped to category credibility based on the proximity graph data to obtain graph semantic region data.

[0024] The present invention achieves efficient and structure-sensitive semantic assignment of regions in edge maps by constructing a lightweight semantic region mapping mechanism without relying on a deep semantic segmentation model. By jointly extracting local color features and local texture features from the edge map, the adaptability of semantic reasoning to structural details is enhanced; a lightweight image semantic recognition model is used to perform low-resolution semantic reasoning on color and texture features, and while maintaining computational efficiency, a semantic candidate map with category labels and confidence information is generated. A neighboring relationship graph is constructed based on semantic labels and confidence, and the structured propagation of semantic information in edge regions is achieved by introducing dual constraints of edge spatial adjacency and semantic consistency. The edge map is semantically enhanced and annotated through a category credibility mapping mechanism guided by the neighboring relationship graph, effectively improving the ability to identify and control weights of semantically important regions during image fusion. Compared with existing methods that rely on large-scale semantic segmentation models or unified full-image reasoning, this step significantly reduces the demand for computing resources, improves the ability to model local semantics on the edge, and has stronger real-time performance, regional adaptability, and embedded deployment advantages.

[0025] Preferably, step S2 is specifically:

[0026] Perform hierarchical semantic segmentation based on image fusion data to obtain semantic segmentation data;

[0027] Perform boundary consistency reconstruction based on semantic segmentation data to obtain boundary reconstruction data;

[0028] Perform entity segmentation on the image fusion data according to the boundary reconstruction data to obtain scene segmentation data;

[0029] Perform regional priority detection and sorting based on scene segmentation data to obtain detection sorting data;

[0030] Cross-modal feature enhancement target detection is performed based on the detection ranking data and image fusion data to obtain scene target data.

[0031] The present invention realizes a high-precision semantic perception process from scene structure analysis to key target identification by constructing a four-level processing chain of semantic drive-boundary reconstruction-entity segmentation-cross-modal reinforcement. Hierarchical semantic segmentation is performed based on image fusion data to effectively divide the semantic areas in the scene, providing a semantic basis for structural analysis; through the boundary consistency reconstruction mechanism, the boundaries of the preliminary semantic areas are refined and corrected, solving the problems of edge blur and entity breakage in traditional segmentation methods, and improving spatial consistency and segmentation accuracy; on this basis, the fused image is segmented at the entity level in combination with boundary information to generate scene segmentation data with complete structure and clear semantics. A regional priority detection and sorting mechanism is designed to sort each candidate area by indicators such as spatial proportion, semantic weight or edge activity, effectively focusing detection resources on key target areas; finally, cross-modal feature reinforcement detection is performed using image fusion data, and the recognition and enhancement of detail areas in the target detection stage are achieved through the joint input of structural features and semantic channels. Compared with existing technologies that only rely on a single path (such as semantic detection or instance segmentation), the present invention enhances detection accuracy and regional focus distinction capabilities while preserving the integrity of the segmentation structure, and has higher adaptability and practicality in multi-target, semi-occluded or semantically blurred scenarios.

[0032] Preferably, the hierarchical semantic segmentation is specifically:

[0033] Perform multi-resolution division according to the image fusion data to obtain image division data;

[0034] Performing hierarchical semantic perception on the image segmentation data to obtain hierarchical semantic perception data;

[0035] Constructing hierarchical semantic paths based on hierarchical semantic perception data to obtain hierarchical semantic path data;

[0036] The boundaries of the image segmentation data are reconstructed according to the hierarchical semantic path data to obtain semantic segmentation data.

[0037] The present invention introduces multi-resolution division and hierarchical semantic perception mechanisms to construct a hierarchical semantic segmentation method for the coordination of structural details and semantic levels, which significantly improves the clarity of semantic boundaries and the integrity of segmentation structures in diverse scenes. By performing multi-resolution division on image fusion data, the hierarchical expansion of the image in spatial scale is achieved, and the structural texture and contextual information at different scales are retained; hierarchical semantic perception is performed on regions with different resolutions, and local significant features and global semantic associations are effectively extracted to form hierarchical semantic perception data. By constructing hierarchical semantic paths, semantic information at different levels is orderly associated and aggregated, so that the semantic relationship between adjacent regions can be retained and traceable, thereby enhancing semantic consistency; hierarchical semantic paths are used to reconstruct image boundaries, repair semantic breaks and detail jumps, and output semantic segmentation data with continuous structure and complete semantics. Compared with traditional semantic segmentation methods that use uniform scale input and single-layer semantic feature extraction, the present invention integrates multi-scale structural expression and semantic path construction to make the model more context-sensitive and boundary-aware, and exhibits stronger adaptability and segmentation accuracy in scenes with multiple objects overlapping, large scale differences or regional semantic transitions.

[0038] Preferably, the cross-modal feature enhanced target detection is specifically:

[0039] Perform spatial attention calculation based on the detection sorting data and image fusion data to obtain spatial attention data;

[0040] The detection ranking data is labeled according to the spatial attention data to obtain the detection weight data;

[0041] Perform multimodal feature extraction based on graph semantic region data and image fusion data to obtain multimodal feature data;

[0042] Perform regional attention enhancement on the multimodal feature data according to the detection weight data to obtain feature enhancement data;

[0043] The preset target recognition model is used to identify the feature enhanced data to obtain the scene target data.

[0044] The present invention calculates spatial attention weights based on detection sorting data and image fusion data, dynamically focuses on areas with structural significance and semantic priority in the image, and thus constructs a spatial guidance mechanism for the attention area; then, the spatial attention results are used for detection sorting annotation to generate importance tags for the target area, forming detection weight data, which enhances regional selectivity and information utilization efficiency in the detection process; combines graph semantic region data with image fusion data to extract multimodal feature information, integrates edge structure, semantic labels and multi-source image responses, and constitutes multimodal feature data with high-dimensional information expression; regional attention enhancement is implemented on multimodal features based on detection weight data, so that the feature expression of key target areas can be amplified in the feature space, effectively suppressing background interference and fuzzy boundaries; the enhanced features are input into the target recognition model for detection and classification, and accurate scene target data is output. Compared with existing single-modality or non-guidance mechanism detection methods, the present invention not only has cross-modal expression capabilities, but also introduces regional weights and attention control, which improves the detection accuracy and stability of the system in scenes with multiple targets, occlusions, weak boundaries or high inter-class similarity, and has stronger engineering application value and generalization ability.

[0045] Preferably, step S3 is specifically:

[0046] Confidence-layered target filtering is performed based on scene target data to obtain confidence-layered data, where the confidence-layered data includes high-confidence target data, low-confidence target data, and structure-assisted candidate data. High-confidence target data refers to target areas whose confidence scores are significantly higher than a preset threshold during the detection process. Low-confidence target data refers to target areas whose confidence scores are significantly lower than a preset threshold during the detection process. Structure-assisted candidate data refers to target areas that are not clearly marked as targets by the preset detection model, but which exhibit geometric or semantic features similar to those of the target in the structure perception layer (such as the edge map).

[0047] Performing spatiotemporal density enhancement on low-confidence target data in confidence layered data to obtain layered enhanced data;

[0048] Calculate the edge area overlap of the layered enhanced data according to the fused edge map data to obtain edge area overlap data;

[0049] The matching degree is calculated based on the image semantic area data and the edge area overlap data to obtain the target positioning data;

[0050] Motion vector features are extracted based on the target positioning data to obtain motion vector feature data, and motion function fitting is performed based on the motion vector feature data to obtain motion trajectory prediction data.

[0051] The present invention performs confidence hierarchical filtering on scene target data and divides the detection results into three types of areas: high-confidence targets, low-confidence targets, and structure-assisted candidates, which significantly improves the system's perception coverage of weak targets or non-explicit targets; among them, the structure-assisted candidate data compensates for the omission of low-saliency targets by traditional detection models through edge map and semantic structure discrimination. Secondly, by performing spatiotemporal density enhancement on low-confidence areas and combining the spatial stability and local aggregation trend of continuous frames, the dynamic recognizability of weak targets is improved. Furthermore, by combining the fused edge map data to perform edge area overlap analysis and calculate the semantic matching degree with the map semantic area, the spatial consistency and semantic rationality of target positioning are effectively improved. Finally, by extracting motion vector features from the target position sequence and performing dynamic function fitting, structured motion trajectory prediction data is obtained. Compared with existing target detection and tracking methods that mostly rely solely on high-confidence targets for trajectory modeling, the present invention takes into account both structural assistance and semantic reasoning, and has stronger target completeness, accuracy robustness and the ability to recover from occluded / blurred targets. It is particularly suitable for high-reliability anti-collision recognition systems in dynamic, unstructured or low-contrast scenarios.

[0052] Preferably, the spatiotemporal density enhancement is specifically:

[0053] Perform spatial clustering on low-confidence target data in confidence stratified data to obtain repeated target data;

[0054] Perform cross-frame trajectory tracking according to repeated target data to obtain cross-frame trajectory tracking data;

[0055] The spatiotemporal density of low-confidence target data is calculated based on cross-frame trajectory tracking data and repeated target data to obtain target spatiotemporal density data;

[0056] Low-confidence target data in the confidence layered data are screened according to the target spatiotemporal density data to obtain layered enhanced data.

[0057] The present invention spatially clusters low-confidence target data to identify target regions that appear repeatedly or are located close to each other in the same frame or adjacent frames, forming preliminary repeated target data. Cross-frame trajectory tracking is performed based on the repeated target data to construct a time series correlation path for low-confidence targets, forming cross-frame trajectory tracking data. This solves the problem of traditional detection models lacking weak target consistency modeling between frames. Combining the frequency of occurrence, inter-frame continuity, and trajectory stability of repeated targets, the target spatiotemporal density is calculated to form a quantitative target presence confidence index. Low-confidence targets are then screened based on this index, retaining real target regions that are structurally persistent and consistent. The output layered enhanced data not only significantly improves target coverage but also provides more reliable input for edge overlap analysis and positioning reasoning. Compared to existing methods that typically directly eliminate or simply score low-confidence targets, the present invention introduces multi-frame consistency enhancement and spatial density quantization strategies in low-confidence regions, achieving structural recovery and dynamic credible extraction of weak targets. This method has higher target perception integrity and motion modeling accuracy, making it suitable for application scenarios with severe occlusion, inter-frame discontinuity, or large image quality fluctuations.

[0058] Preferably, step S4 is specifically:

[0059] Generate a trajectory intersection area based on the motion trajectory prediction data to obtain trajectory intersection area data;

[0060] Construct a collision propagation map based on the trajectory intersection area data to obtain collision propagation map data;

[0061] Risk propagation and diffusion simulation is performed based on the collision propagation map data to obtain collision risk data.

[0062] The present invention combines motion trajectory prediction data to generate trajectory intersection area data, effectively capturing spatial intersection areas with potential collision risks between multiple targets and accurately locating key spatial nodes that may cause interference or conflict. A collision propagation graph is constructed based on the trajectory intersection area data, in which nodes represent potential collision objects and edges represent risk propagation factors such as trajectory intersection intensity, temporal proximity, or directional consistency, thereby forming a graph model with clear structural expression and weighted quantification. A propagation simulation algorithm is used to diffuse and evolve the collision propagation graph, combining factors such as velocity vector, intersection probability, and path density to derive the collision risk level and its diffusion trend that evolve over time, and output collision risk data. Unlike traditional methods that only judge collisions based on the current frame or a single intersection point, the present invention models the dynamic interaction of multi-target trajectories as a graph structure and performs propagation simulation, which can predict the possible spatial range and risk level of collisions in future time periods, significantly improving the foresight, stability, and interpretability of the collision avoidance system in high-density, multi-path, and multi-target dynamic environments.

[0063] Preferably, the present application further provides an image fusion driven anti-collision recognition system for executing the image fusion driven anti-collision recognition method described above, the image fusion driven anti-collision recognition system comprising:

[0064] The semantic fusion perception module is used to obtain multi-source image data and perform semantic-guided image fusion based on the multi-source image data to obtain image fusion data;

[0065] The target semantic parsing module is used to perform scene segmentation based on the image fusion data to obtain scene segmentation data, and perform target detection on the scene segmentation data to obtain scene target data;

[0066] The layered positioning and trajectory prediction module is used to perform confidence-level layered spatial positioning based on scene target data to obtain target positioning data, and to perform motion trajectory prediction based on the target positioning data to obtain motion trajectory prediction data;

[0067] The collision risk inference module is used to evaluate the collision area propagation risk based on the motion trajectory prediction data to obtain collision risk data.

[0068] The beneficial effects of the present invention are as follows: Step S1 introduces a semantically guided image fusion mechanism and uses structural weights and clarity responses to perform weighted processing on multi-source images, thereby enhancing the scene structure features and semantic consistency and providing high-quality perception input for the task; Step S2 combines hierarchical semantic segmentation with cross-modal feature-enhanced target detection technology to achieve high-precision identification of key areas while maintaining the integrity of the target structure; Step S3 uses a confidence layering mechanism and an edge semantic-guided positioning strategy to perform structural repair and dynamic trajectory fitting on low-saliency targets, effectively improving target coverage and trajectory continuity; Step S4 further constructs a collision propagation map based on the predicted trajectory, simulates the potential interaction relationship and risk diffusion process between multiple targets, and outputs collision risk results with clear risk levels and clear evolution paths. Compared with traditional solutions that only rely on high-confidence targets or rule judgments, the present invention has stronger environmental adaptability, weak target recovery capabilities and system interpretability by integrating image perception depth, semantic understanding dimensions and trajectory evolution modeling, and is suitable for the deployment of high-reliability collision avoidance systems in high-density, unstructured or weakly perceived scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Other features, objects and advantages of the present application will become more apparent upon reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0070] Figure 1 A flowchart showing the steps of an image fusion driven anti-collision recognition method according to one embodiment is shown;

[0071] Figure 2A flowchart showing the steps of a semantic fusion perception method according to an embodiment is shown;

[0072] Figure 3 A flowchart showing the steps of a target semantic parsing method according to an embodiment is shown;

[0073] Figure 4 A flowchart showing the steps of a hierarchical positioning and trajectory prediction method according to an embodiment is shown;

[0074] Figure 5 A flowchart of a collision risk reasoning method according to an embodiment is shown. DETAILED DESCRIPTION

[0075] The following is a clear and complete description of the technical method of the present invention in conjunction with the accompanying drawings. It is obvious that the embodiments described are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making any creative efforts are within the scope of protection of the present invention.

[0076] Furthermore, the accompanying drawings are merely schematic illustrations of the present invention and are not necessarily drawn to scale. Functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor and / or microcontroller approaches.

[0077] It should be understood that although the terms "first," "second," and the like may be used herein to describe various elements, these elements should not be limited by these terms. These terms are used solely to distinguish one element from another. For example, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element, without departing from the scope of the exemplary embodiments. The term "and / or" as used herein includes any and all combinations of one or more of the listed associated items.

[0078] See also Figures 1 to 5 , the present application provides an image fusion driven anti-collision recognition method, comprising the following steps:

[0079] Step S1: acquiring multi-source image data, and performing semantic-guided image fusion based on the multi-source image data to obtain image fusion data;

[0080] Specifically, image data from different viewpoints or sensors (such as RGB cameras, infrared cameras, and lidar projections) are obtained and unified into a fixed size (such as 640×480). All image data is time-synchronized and geometrically corrected to eliminate viewpoint distortion and resolution inconsistency. A pre-trained semantic segmentation model (such as DeepLabv3+) is used to perform pixel-level semantic annotation on each image to obtain a semantic label map. The semantic weight matrix is ​​set. , assign different fusion priorities to semantic categories (such as pedestrians, vehicles, and background). For example: pedestrian = 1.0, vehicle = 0.8, background = 0.2. Perform weighted fusion: , The fused image at pixel position The fused pixel value at is the image order term, is the total number of image sources, For the The weight value of the road image at this position guided by the semantic category, For the Road image at pixel location The semantically guided weighting mechanism described above can preserve the target region information with high semantic value (such as pedestrians and vehicles) in the multi-source images during the fusion process, while suppressing low-value regions (such as the background).

[0081] Step S2: performing scene segmentation based on the image fusion data to obtain scene segmentation data, and performing target detection on the scene segmentation data to obtain scene target data;

[0082] Specifically, for the fused image Use a semantic segmentation network (such as SegFormer) to process and output per-pixel category probabilities , that is, pixels Belong to category The probability of . Use CRF (conditional random field) to optimize edge details to ensure the accuracy of the target boundary. Generate label map , To fuse the image at position The semantic category label of For all categories The predicted probability Maximize selection. Pixel Belong to category The predicted probability value of For categories, is the pixel horizontal coordinate, is the pixel vertical coordinate. Afterwards, the region of interest is filtered based on the continuous semantic regions in the image (e.g. the detection box is the minimum bounding rectangle of the semantic block). Detection algorithms such as YOLOv7 or Faster-RCNN are used to further detect the target location and type on the fused image, and output a set of target boxes. , is the set of all detected targets, For the Detection target boxes, is the target index, the range is , is the total number of detected targets, each target contains location information, including the center coordinates of the target ,width and height ; Category label, including the semantic category to which the target belongs, such as vehicle, pedestrian, traffic sign, etc.; detection confidence, recorded as , which indicates the credibility of the target box or the classification confidence score value.

[0083] Step S3: performing confidence-level hierarchical spatial positioning based on the scene target data to obtain target positioning data, and performing motion trajectory prediction based on the target positioning data to obtain motion trajectory prediction data;

[0084] Specifically, for each detection box , divided into levels according to confidence, such as high confidence (conf > 0.8), directly treated as a reliable target; medium confidence (0.5 < conf ≤ 0.8), if met, then the target is verified to see whether it appears continuously in the current frame and the adjacent frames before and after, to determine whether it has trajectory continuity; low confidence (conf ≤ 0.5) is temporarily not involved in trajectory prediction and is only stored as redundant reference information. After obtaining the confidence level, the spatial three-dimensional coordinates of each target position are restored, and the depth value is estimated from the fused image using a depth estimation network (such as Monodepth2) , represents the relative depth value corresponding to each pixel in the image. Combined with the camera intrinsic parameter K, according to the monocular camera imaging model, using the pixel coordinate With depth value , calculate the three-dimensional space coordinates of the point in the camera coordinate system: , is the three-dimensional spatial position of the pixel in the camera coordinate system, Pixel The depth estimate of is the inverse matrix of the camera's intrinsic parameter matrix, including the focal length and principal point offset, is the homogeneous coordinate vector of the pixel point. Motion trajectory prediction: Use the Kalman filter or LSTM sequence model to predict the trajectory of the target position sequence for the past N frames: , is the predicted position vector of the target at the next moment, For the current or The position of the frame target, For the previous or The position of the frame target, For the best or The position of the frame target, It is a prediction function, which can be implemented by Kalman filter, LSTM or other time series models.

[0085] Step S4: perform collision area propagation risk assessment based on the motion trajectory prediction data to obtain collision risk data.

[0086] Specifically, for any two targets, predict the trajectory 、 , determine whether there is a spatial overlap area: ,in For some selected time points, For the current moment, is the predicted future time window (e.g. 2 seconds), is the collision threshold (such as 0.5 meters), Target In time The predicted position (3D coordinates), Target In time The predicted location (3D coordinates) of each target. To systematically describe the risk transmission relationship between targets, a collision risk graph is constructed, where nodes are targets, edges are potential collision relationships, and edge weights are predicted collision probabilities. A propagation algorithm (such as a PageRank-based risk diffusion model, which iteratively scores the entire graph structure to obtain a risk level score for each target) is used to dynamically propagate high-risk targets and assess the scope and intensity of their impact on surrounding targets. The risk level is output, and the collision level (high / medium / low) is labeled for each target, along with the expected collision time and area. ,in Target The collision level function, For high risk, For medium risk, For low risk, is the collision probability.

[0087] Preferably, step S1 is specifically:

[0088] Step S11: Acquire multi-source image data;

[0089] Specifically, the system collects image data from multiple cameras arranged at different viewing angles in the target area, and the viewing angles include but are not limited to front-view cameras, side-view cameras, and overhead cameras. In order to ensure the consistency of multi-source image data in the time dimension, the system inputs the image data into a time synchronization module for frame-level time alignment. The time synchronization module is used to perform timestamp calibration and synchronization processing on image frames from different cameras according to the system master clock signal, thereby ensuring that multi-view images are comparable at the same time point. Furthermore, in order to ensure the consistency of the spatial geometric structure of the image, the system performs spatial geometric distortion correction on each image. The correction method may include dedistortion correction of fisheye images, epipolar geometric correction of binocular images, back-projection correction of lens distortion parameters, etc. The above-mentioned geometric correction operation is used to restore the spatial projection relationship of real objects in the image, ensuring that the spatial correspondence between images captured by different cameras is accurate.

[0090] Step S12: extracting edge maps based on the multi-source image data to obtain edge map data;

[0091] Specifically, the system processes each acquired image data Apply edge extraction algorithm, preferably Canny edge detection algorithm. The Canny algorithm includes Gaussian filtering denoising, image gradient amplitude and direction calculation, non-maximum suppression processing, and edge connection determination based on dual threshold strategy. In actual processing, two edge detection thresholds are set for each image, namely low threshold and low threshold. and high threshold , used to control the screening process of edge intensity response: if the gradient value of a pixel is higher than , it is directly retained as a strong edge; if and If the edges are between , and connected to strong edges, they are retained as weak edges; otherwise, they are removed. images , extract the corresponding edge map ,The edge map represents the spatial distribution of edge pixels in binary form.

[0092] Step S13: performing multi-scale convolution blurring on the edge map data to obtain convolution blurred data, and performing sharpening processing on the convolution blurred data to obtain fused edge map data;

[0093] Specifically, multiple Gaussian kernels (such as 3×3, 5×5, 7×7) are used to perform convolution on each edge map to enhance the structural texture features. ,in For the The fuzzy enhancement result of the image is is the Gaussian kernel order term, is the nuclear fusion coefficient (can be set to decreasing weights, such as 0.5, 0.3, 0.2), For each edge graph , using multiple Gaussian convolution kernels Perform convolution operation, For the edge image, For the Gaussian kernel. Convolution blurs the image Perform sharpening operation and use Laplace operator for sharpening, which is defined as follows , is the image after sharpening, For the The fuzzy enhancement result of the image is is the sharpening strength coefficient, the value range can be set to 0.8 to 1.2, For the image The Laplace transform applied, i.e. second-order derivative enhancement, is used to extract edge change information of the image. Perform weighted fusion to obtain the fused edge image The fusion adopts a weight mechanism based on the image edge information entropy, assuming that each sharpened image The corresponding fusion weight is The weight value can be dynamically adjusted according to the edge information entropy of each image. The higher the information entropy, the more significant the representation and the greater the weight. The fusion formula is: , is the fused edge map, is the order term for sharpening the image, For images The fusion weight of For the A sharpened image.

[0094] Step S14: performing semantic region mapping on the edge graph data to obtain graph semantic region data;

[0095] Specifically, a pre-trained semantic segmentation model (such as DeepLabv3+) is used to perform pixel-level segmentation on the original image to obtain the corresponding semantic region map, which contains semantic categories including but not limited to pedestrians, vehicles, obstacles and road surfaces. Subsequently, the semantic region map and edge map data are aligned and fused pixel by pixel to construct the edge semantic mapping matrix. The generation rule of this mapping matrix is: when the first When the value of the edge map of the channel at a certain pixel is 1, that is, when there is an edge feature, the corresponding semantic category in the semantic area map is extracted and assigned to the mapping matrix; if the pixel value of the edge map is 0, it is marked as an invalid value (None) in the mapping matrix.

[0096] Step S15: performing edge density calculation on the fused edge graph data according to the graph semantic region data to obtain structural weight data;

[0097] Specifically, for each semantic region , count the sum of the response values ​​of all the pixels in the fused edge map, and divide it by the total number of pixels in the area to calculate the edge density value per unit area, , is the edge density value, which represents the average edge strength value within each unit semantic area. is the pixel coordinate pair within the semantic area, representing the pixel point Belongs to the semantic area , is the pixel value of the fused edge map, indicating that in the fused edge map, the position The edge response strength at For the Semantic regions are the blocks of regions corresponding to the semantic categories identified by the semantic segmentation model. The structural weights are obtained by normalization: , is the structural weight value, which represents the significance weight of the semantic region structural information, and its range is normalized to [0,1]. is the edge density value, which represents the average edge strength value within each unit semantic area. is the maximum edge density value in all semantic regions. Output structure weight map , where each semantic region is assigned a structural weight value. The higher the edge density of the region, the greater its weight, reflecting that the structural information of the region is richer.

[0098] Step S16: performing high frequency response calculation based on the multi-source image data to obtain image clarity data;

[0099] Specifically, for images Applying the Laplace filter to extract high-frequency information in the image is equivalent to calculating the second-order derivative response of the image in the grayscale domain, which is recorded as ; Calculate the grayscale variance of the Laplace response result, recorded as The result is defined as the clarity index of the image The variance value reflects the overall distribution of edge responses in the image. The higher the value, the clearer the edges in the image, the stronger the contrast, and the higher the image focus. The system forms a set of clarity indicators for all images. .

[0100] Step S17: performing weighted fusion on the multi-source image data according to the structural weight data and the image clarity data to obtain image fusion data.

[0101] Specifically, for each source image, at each pixel in the image The corresponding fusion weight is calculated. The fusion weight takes two factors into consideration: one is the structural weight, that is, the structural significance of the image at this pixel; the other is the clarity level of the overall image. , For the Source image in pixels The fusion weight at is the fusion coefficient of structure weight and clarity weight, ranging from 0 to 1, and is used to adjust the relative contribution of structural features and overall quality. For the Image in pixels The structural significance weight at , indicates the structural sensitivity of the pixel in the image. For the The overall clarity evaluation index of an image can be quantified by image gradient, contrast or frequency domain energy. is the image order term, For the The overall clarity evaluation index of the image. During the fusion process, for each pixel position, based on the corresponding pixel values ​​of each source image and their comprehensive fusion weights, a weighted sum is performed to obtain the pixel value of the fused image at that position.

[0102] Preferably, the semantic area mapping is specifically:

[0103] Extracting local color features and local texture features from edge image data to obtain image color feature data and image texture feature data respectively;

[0104] Specifically, input edge map data , original image , for each edge point Extract its neighborhood window (such as a 9×9 area); if the edge density of the local area where the current pixel is located (i.e. the ratio of the number of edge pixels to the total number of pixels in the window) is lower than a preset threshold (such as 0.1), skip the feature extraction step of the pixel point; local color feature extraction includes calculating the color histogram features of the window (such as HSV three channels, each channel is divided into 5 levels, a total of =125 dimensions); or calculate color statistics such as mean, standard deviation, skewness, kurtosis, etc. Local texture feature extraction includes calculating gray level co-occurrence matrix (GLCM) features, including contrast (intensity of grayscale changes), homogeneity (similarity of grayscale values ​​of adjacent pixels), entropy (complexity of texture information) or extracting LBP local binary pattern features. Output image color feature data Image texture feature data .

[0105] Use the preset lightweight image semantic recognition model to perform low-resolution semantic reasoning on the image color feature data and image texture feature data to obtain semantic category label data and semantic confidence data;

[0106] Specifically, extract the color feature vector of each pixel in the image and texture feature vector , and concatenate the two in the feature dimension to form a joint feature representation vector The feature vector is used as the input of the lightweight semantic recognition model. The model is a pre-trained small convolutional neural network, which can adopt MobileNetV2, ShuffleNet or other custom lightweight structures. The output of the semantic recognition model includes the position of each pixel. Semantic category labels , whose values ​​belong to several predefined categories (e.g. {vehicle,pedestrian, road, background, ...}); and the corresponding semantic confidence scores , whose value range is [0,1], is used to indicate the credibility of the current predicted label. In the model inference stage, in order to improve the computational efficiency, the input image is first subjected to 4 spatial dimensions. 4-scale downsampling enables the model to complete semantic reasoning at a coarse-grained resolution, and the obtained low-resolution semantic results are then mapped back to the original image resolution through the nearest neighbor interpolation method. The model training phase is based on supervised learning based on the annotated semantic image dataset. The loss function includes multi-class cross entropy loss and label smoothing regularization terms to improve the stability and generalization ability of the model in the class boundary area. The system output includes the semantic category label data of each image pixel. Corresponding semantic confidence data , which can serve as the basic input for subsequent graph semantic region mapping, structural weight modeling and fusion guidance.

[0107] Constructing a proximity graph based on the semantic category label data and the semantic confidence data to obtain proximity graph data;

[0108] Specifically, a region segmentation operation is performed on the semantic label map, using a watershed algorithm, connected domain extraction or superpixel algorithm (such as SLIC) to divide the image into multiple semantic candidate region blocks. Each block A node in the graph structure, whose attributes include the main semantic category and average semantic confidence If any two regions and If there is a boundary contact relationship in the image, a connecting edge is established in the image The weight of the edge Defined as: , For the edge The weight value indicates the strength of regional semantic consistency. is the natural exponential term, To control the weight factor of semantic difference, Semantic category and The category distance function between (such as 0 or 1), For the region The main semantic category label, For the region The main semantic category label, is the weight factor for controlling the confidence difference, For the region The average semantic confidence value of For the region Output proximity graph data ,The proximity relationship graph structure is composed of three parts: node set, edge set, and edge weight set.

[0109] The edge graph data is mapped to category credibility based on the proximity graph data to obtain graph semantic region data.

[0110] Specifically, for each edge point in the edge map , if there is a valid edge response at the pixel (i.e. , the edge image at position Is it an edge point (1 means edge)? Then perform the following processing to find the image semantic segmentation block to which the edge point belongs, which is recorded as the block area. ; Assign initial semantic labels to edge points ,in For the region The corresponding semantic category label; at the same time, set its initial semantic confidence ,in Indicates area The semantic credibility score of the graph is obtained based on the maximum softmax value of the classification model output. A graph structure is constructed with all edge points as nodes, and there is a spatial or feature similarity adjacency relationship between nodes. A semantic confidence diffusion iteration is performed on the graph to enhance the transferability of semantic information between adjacent regions. For any edge node in the graph, , update its semantic confidence value , the formula is as follows: , For nodes The semantic confidence value obtained after diffusion iteration update, For nodes The set of neighbor nodes of For nodes Adjacent nodes , For nodes With node The edge weights between them represent the semantic or spatial similarity between them. is the node order item, For nodes The sum of the weights of all connected edges, For nodes The current semantic confidence value. After completing a diffusion, if the semantic confidence of an edge point is If the confidence threshold is less than 0.5, the semantics of the point is considered uncertain and it is marked or downgraded during processing. The semantic state is encapsulated as a triple: , is the semantic category label, is the semantic confidence after diffusion, It is a Boolean value, indicating whether the point is a semantically credible area (when ≥0.5 is true, otherwise false). Output graph semantic region data .

[0111] Preferably, step S2 is specifically:

[0112] Step S21: performing hierarchical semantic segmentation based on the image fusion data to obtain semantic segmentation data;

[0113] Specifically, from image fusion data Constructing a three-layer image pyramid , the scaling ratios of each layer are 1.0, 0.5, and 0.25 respectively; in each scale image, a semantic segmentation model that supports multi-scale feature fusion is used, such as SegFormer, BiSeNet, etc.; a three-layer prediction map is output , each layer output includes pixel semantic labels C(x,y) and confidence maps S(x,y). The fused semantic labels are: , is the fused semantic label, To select the category with the largest score among all categories, is the order of the image layers, is the scale weight, for example , It is an indicator function, which takes the value 1 when the condition in the brackets is true, and 0 otherwise. For the Layer semantic category label map, For the Layer semantic confidence map, the value range is [0,1]. Output high confidence semantic segmentation map , corresponding to the semantic probability distribution map .

[0114] Step S22: Reconstruct boundary consistency based on the semantic segmentation data to obtain boundary reconstruction data;

[0115] Specifically, edge detection algorithms (such as Sobel operator, Laplacian operator or Canny algorithm) are used to extract the boundaries of the semantic segmentation map and generate a boundary response map. On this basis, the semantic labels of the adjacent pixels on both sides of the boundary are , , , , etc. to perform co-occurrence detection. If the detected boundary area is smaller than the set threshold If there are isolated semantic regions, boundary reconstruction is performed. The reconstruction strategy includes using morphological closing operations to close the boundaries and remove boundary breaks or burrs; combining the conditional random field (CRF) model to refine the label distribution at the boundaries; introducing the graph cut method to optimize the local region labels and output the reconstructed semantic boundary map. .

[0116] Step S23: performing entity segmentation on the image fusion data according to the boundary reconstruction data to obtain scene segmentation data;

[0117] Specifically, the connected region analysis is used to obtain the closed region; each closed region is defined as a candidate entity block, and each region is recorded as , and record attributes such as region boundaries, main categories, region areas, and shape factors. Set several filtering conditions to filter the extracted candidate regions, including but not limited to setting a minimum area threshold , minimum aspect ratio and other parameters to filter out abnormally small areas and non-physical structure areas; filter out noise areas with shapes similar to random patches. For each valid entity area retained Generate a mask image and extract the corresponding original image block (in the fused image). The output contains a set of all valid entity regions, denoted as , the mask data of each region is recorded as .

[0118] Step S24: performing regional priority detection and sorting according to the scene segmentation data to obtain detection sorting data;

[0119] Specifically, the system segments multiple semantic regions in the image based on the scene segmentation data obtained in the previous step. Priority scoring and sorting are performed to achieve efficient allocation of target detection resources. The sorting strategy is designed to prioritize areas with high semantic confidence, large spatial size, and close to the center of the image, thereby improving overall detection efficiency and response speed. : , is the weight coefficient of the regional area, is the area of ​​the region, is the weight coefficient of the average semantic confidence, is the average semantic confidence, is the weight coefficient of the proximity of the region center to the image center (prioritizes the region close to the center), is the distance between the center of the region and the center of the image (the area close to the center is preferred). According to the score, the candidate sequence of detection is generated from high to low. , output the sorted detection area list .

[0120] Step S25: Perform cross-modal feature enhancement target detection based on the detection sorting data and the image fusion data to obtain scene target data.

[0121] Specifically, for each detection area, feature data consisting of multiple channels is extracted from the image fusion map. The channels may include but are not limited to visible light channels (RGB), edge map channels (such as Canny edge response maps), depth map channels (such as relative depth maps generated by depth estimation models or lidar), etc. The above multimodal channels are uniformly stacked into regional image blocks and constructed into a set of regional feature tensors. Let The feature tensor of a region is recorded as , whose size is ,in and Respectively The height and width of the region; is the number of channels after fusion; The multimodal feature representation of the region is represented. The channel attention mechanism is used for feature fusion. Preferably, the SE module or CBAM module can be used to weight the importance of each channel to enhance the semantic sensitive channel. The feature tensor after attention fusion is input into the region detector for target recognition. The detector can use advanced target detection networks such as YOLOv8 or Deformable DETR to perform the target location and category recognition task. The detector performs feature fusion on each region. Output the following target information, such as location frame information, including center coordinates ,width and height ; Target category label , such as vehicle, pedestrian, etc.; detection confidence score ∈[0,1], indicating the reliability of the detection result. After the detection is completed, the system further performs boundary fine-tuning to ensure the boundary consistency between the detection box and the semantic region. If there is a large deviation between the detection box and the previous semantic segmentation boundary, the following two methods can be used to correct it: boundary alignment strategy, adjust the outline of the target detection box to be more consistent with the edge outline of the semantic region; or, contour reconstruction mechanism, call the instance segmentation model (such as Instance Segmentation network) to refit the contour boundary to obtain a more detailed description of the target shape. Output target detection dataset , is the set of all detection targets, For the The location information of the target, including , For the The category labels of the targets, For the The detection confidence score of each target, is the target sequence item, is the total number of detected targets.

[0122] Preferably, the hierarchical semantic segmentation is specifically:

[0123] Perform multi-resolution division according to the image fusion data to obtain image division data;

[0124] Specifically, the system performs multi-level downsampling operations on the image fusion data to construct a three-level image resolution image sequence ,in is the original resolution image, corresponding to the scaling ratio = 1.0, which means retaining the original image size; It is a medium-resolution image obtained by downsampling the original image to 0.5 times; is a low-resolution image obtained by downsampling the original image to 0.25 times. All through the image scaling function Build, where Indicates the scaling factor of the current resolution layer. Represents the pyramid level number (numbered sequentially from high resolution to low resolution). To enhance the edge perception of images at each resolution, the system performs edge feature extraction on each layer of image $I^L$ and smoothes the edge results, thereby improving the adaptability of multi-resolution images in structural modeling tasks.

[0125] Performing hierarchical semantic perception on the image segmentation data to obtain hierarchical semantic perception data;

[0126] Specifically, semantic features are extracted from image data at different resolution levels. , using a lightweight semantic segmentation network (such as the MobileNet backbone network and the DeepLabHead decoder structure), ,in is the semantic label prediction for each pixel, is the semantic confidence map, For the The semantic segmentation sub-network model of the layer, For each layer image. All prediction images Upsample to original size; normalize confidence map: , For the Layer at pixel location The semantic confidence value of is the number of layers. Output layered semantic label atlas , Hierarchical Semantic Confidence Atlas .

[0127] Constructing hierarchical semantic paths based on hierarchical semantic perception data to obtain hierarchical semantic path data;

[0128] Specifically, for each layer of semantic perception graph (where L = 1, 2, 3 corresponds to high, medium, and low resolution respectively) Perform connected region labeling and extract region sets Calculate the overlap between regions of different resolutions. If the intersection over union (IoU) between the medium / low resolution region and the high resolution region exceeds the set threshold, , then a semantic path connection relationship is established to form a multi-scale corresponding path structure . Calculate the confidence for each path: , is the path confidence, is the average confidence of the semantic region at the first layer resolution, is the average confidence of the semantic region at the second layer resolution, is the average confidence of the semantic region at the third layer resolution. The path structure is screened according to the confidence, redundant or low-confidence paths are filtered out, and a hierarchical semantic path set is output.

[0129] The boundaries of the image segmentation data are reconstructed according to the hierarchical semantic path data to obtain semantic segmentation data.

[0130] Specifically, for the generated hierarchical semantic path data, the system performs weighted fusion on the multi-scale semantic label maps by fusing the semantic consistency information between the paths, and constructs a semantic image with clear boundaries and cross-scale semantic consistency. The system introduces a path voting fusion strategy. This strategy is based on the semantic label prediction map under each scale image (such as the original image, 0.5 times zoom image, 0.25 times zoom image) and its corresponding path confidence map, and performs a path voting fusion on each pixel position. The labels of are multi-scale fusion calculated. The label map at the layer scale is , and its corresponding semantic confidence graph is , the fusion weight is , then the fusion label of the pixel is , the calculation rules are as follows: for each candidate category , calculate the weighted sum of the consistent confidence of the category at all scales, and take the category with the largest weighted sum as the label: , is an indicator function, if the label is equal to , the value is 1, otherwise it is 0. When abrupt changes such as jumps, tears, and breaks are detected at the region boundary, the system will perform boundary closure compensation, including morphological expansion and closure operations; perform structural repair on the fusion result, fill internal segmentation holes, and remove isolated small spots to avoid pseudo-target contamination. Output semantic segmentation data ,Semantic graph with hierarchical enhancement results.

[0131] Preferably, the cross-modal feature enhanced target detection is specifically:

[0132] Perform spatial attention calculation based on the detection sorting data and image fusion data to obtain spatial attention data;

[0133] Specifically, the system is based on the detection and sorting data and image fusion data , performs spatial attention modeling operations to improve the model's ability to identify local salient target areas. In this process, the system detects and sorts each area. , firstly, the image fusion data Crop the corresponding image sub-block , as the input of attention evaluation. Based on the intermediate feature output in the neural network structure (such as the penultimate feature map of the convolutional network), perform the following operations to generate candidate regions Spatial saliency map of ,right Apply a global average pooling (GAP) operation to extract the channel average response; then pass The convolution operation integrates the feature channels and uses the Softmax function to normalize the response to generate the spatial attention map of the candidate area. , indicating the importance of each pixel in the region to the semantic target. The system converts each local spatial attention map Map back to the full image space. The mapping method is based on its original detection area coordinates Perform the back projection operation, that is, Resample to The corresponding area position is filled to the original image size. All attention maps The inverse mapping results are weighted superposition or maximum fusion to obtain the full-image spatial attention map , the figure is based on The continuous values ​​in the range represent the saliency score for each pixel location in the image.

[0134] The detection ranking data is labeled according to the spatial attention data to obtain the detection weight data;

[0135] Specifically, the detection area sorting is weighted based on spatial attention and visual priority is introduced. , calculate the mean spatial attention value in the region: , For the region The original attention mean weight of For the Detection box area, For the region The pixel coordinates of is the pixel position The corresponding spatial attention value, all detection box weights are normalized to , mark each area with a detection weight . Output detection weight dataset , bound to the sorting detection data.

[0136] Perform multimodal feature extraction based on graph semantic region data and image fusion data to obtain multimodal feature data;

[0137] Specifically, the fused image data, edge map, clarity map and semantic region label map (using one-hot encoding) are stacked in the channel dimension to construct the input feature tensor. , perform region cropping operations on the multi-channel feature tensor to extract multimodal input segments containing visual features, structural features, and semantic features of the region, denoted as , For the The multimodal input tensor of the region, is the region tensor clipping operation, is the image fusion data (RGB or YUV format), is the edge map data, which indicates the edge strength of each pixel. is the clarity map data, indicating the clarity score of each pixel, is the semantic region label map, which has been encoded into one-hot multi-channel format; each Input multimodal convolutional network (multi-path branching structure can be used); output feature tensor , output multimodal feature dataset .

[0138] Perform regional attention enhancement on the multimodal feature data according to the detection weight data to obtain feature enhancement data;

[0139] Specifically, the multimodal feature tensor of each detection area Perform attention weighting on the channel dimension. The feature tensor consists of image fusion channels (such as RGB), edge channels, clarity maps or depth map channels, etc., and its size is ,in and are the height and width of the region, The system uses the channel attention mechanism (CA), preferably the SE module or the CBAM module. This mechanism uses global average pooling Compress the spatial response of each channel into a single value and then pass it through the fully connected layer Activate with the Sigmoid function to obtain the importance weight of each channel The specific operation is . Call the previously generated spatial attention map , and according to the detection box area Crop the corresponding positions to generate a regional-level spatial attention map This graph is used to measure the saliency response of each pixel in the region. The higher the value, the more important the position. , construct the attention map in the corresponding area , perform pixel-by-pixel multiplication, , is the normalized detection area weight, output feature enhancement dataset .

[0140] The preset target recognition model is used to identify the feature enhanced data to obtain the scene target data.

[0141] Specifically, based on the input data after feature enhancement processing, the system calls the preset high-performance target detection model to identify and infer potential targets in the enhanced image area, and outputs structured scene target recognition results. In terms of model selection, according to the computing power constraints and real-time requirements of the application deployment environment, the following types of mainstream target recognition models can be flexibly adopted, such as YOLOv8: suitable for edge device deployment, with lightweight structure and high precision; DeformableDETR: supports feature alignment mechanism, suitable for processing targets with significant scale changes; RT-DETR: has fast reasoning capabilities, especially suitable for end-side real-time detection scenarios. The above models all use deep neural networks to perform spatial structure modeling and semantic feature classification reasoning on images. In the reasoning stage, the system will each enhanced target area (denoted as ) and its location information are passed into the model as input data. The model performs forward reasoning on the region and outputs the following structured target information, target bounding box location, including target center coordinates ( , ),width and height , forming a bounding box ; Target category label , identified semantic categories; classification confidence , the target is the probability score of the category, and the value range is The system performs post-processing on all model outputs, including confidence filtering: removing targets with confidence scores below a set threshold (e.g., 0.3); and target overlap suppression: when multiple predicted target bounding boxes overlap, a non-maximum suppression (NMS) strategy is used to retain the highest-scoring target to avoid duplicate detections.

[0142] Preferably, step S3 is specifically:

[0143] Step S31: Confidence-layered target filtering is performed based on the scene target data to obtain confidence-layered data, wherein the confidence-layered data includes high-confidence target data, low-confidence target data, and structure-assisted candidate data. The high-confidence target data is a target area whose confidence score is significantly higher than a preset threshold during the detection process. The low-confidence target data is a target area whose confidence score is significantly lower than the preset threshold during the detection process. The structure-assisted candidate data is a target area that is not clearly marked as a target by a preset detection model, but which exhibits geometric or semantic features similar to the target in the structure perception layer (such as an edge map).

[0144] Specifically, the scene target data is classified according to confidence, and structure-assisted compensation is introduced. The scene target data set is input. , is the scene target dataset, For the The bounding box position of each object, is the category label of the target, is the confidence score of the target. The preset threshold is defined as the high confidence threshold: ; Low confidence threshold: The stratification rule is a high confidence target, which means that during the target detection process, its confidence score is higher than or equal to the high confidence threshold. The target area is expressed formally as: , For high confidence targets, A single detection target data item; a low confidence target refers to a target whose confidence score is lower than or equal to the low confidence threshold during the target detection process The target area is expressed formally as: ;Structure-assisted candidate data recognition, scan all unrecognized areas; if the area is in the fusion edge map Middle edge density > threshold , and the graph semantic region graph is a potential target category, then it is included in the structure auxiliary candidate set: , is the structure auxiliary candidate data, To scan all areas in the image that are not marked by the detection model, is the edge density, is the edge density threshold, To query the semantic category of the region in the graph semantic region graph, is the target category set, output three types of target sets .

[0145] Step S32: performing spatiotemporal density enhancement on low-confidence target data in the confidence layered data to obtain layered enhanced data;

[0146] Specifically, for each low-confidence target in the current frame , search for similar targets in the previous N frames (such as 3 frames) to see if they are in similar positions (overlapping areas) ) appears repeatedly; at the same time, whether there are similar candidates in the current image spatial neighborhood (within 5×5 pixels). Combining the two types of features, a spatiotemporal density scoring function is constructed. , for, is the time continuity weight factor, Target Frequency score in consecutive frames, For the Low confidence target data, is the spatial redundancy weight factor, is the redundancy score of the target in the spatial neighborhood in the current frame, and the obtained ,like (like is 0.6), then include it in the enhanced set: , For the target set after spatiotemporal enhancement, the low-confidence target after spatiotemporal enhancement is returned to the low-confidence target data in the confidence layered data to replace it, and the layered enhanced data is obtained. is a set of high-confidence targets, The current frame low confidence targets, is a set of low-confidence targets, Enhanced priority for goals, is the decision threshold, which controls the acceptance boundary of the enhanced target (such as 0.6) and outputs the hierarchical enhanced data .

[0147] Step S33: calculating edge region overlap of the layered enhanced data according to the fused edge map data to obtain edge region overlap data;

[0148] Specifically, from the fusion edge map Extract all pixels with non-zero edge responses; for each detection box , calculate its internal edge pixel density: , For the The edge pixel density of the target area, For the detection box The total number of pixels within The point is in the target box Inside, For location The edge response strength value of With edge density threshold Compare and construct edge matching confidence, , For the The edge matching confidence of each target, For the detection targets, is the minimum function, ensuring that the score result does not exceed 1.0, For the The edge pixel density of the target area, The edge density scoring threshold can be set based on experimental experience (e.g. 0.3) to output edge area overlap data. , is the edge area overlap data set, For the detection targets, Give it an edge overlap confidence score.

[0149] Step S34: Calculate the matching degree based on the image semantic area data and the edge area overlap data to obtain target positioning data;

[0150] Specifically, the edge and semantic matching information are fused to determine the exact correspondence of the target in the semantic graph. , extract its position in the graph semantic area graph The main tab in ,like and the detection model predicts the category If the semantic match is consistent, the semantic match score is assigned 1, otherwise it is assigned 0 or a penalty value is decreased (such as 0.6). The calculation is based on semantic consistency and edge matching: , Target The matching degree, For the Enhanced target, is a weighting coefficient used to adjust the relative importance of structural edge and semantic consistency, with a value range of [0.5, 0.8]. For the The structural overlap score between the target boundary and the edge map is in the range [0,1], For the The semantic label matching score is 1, which means it is consistent, and 0.6 means it is inconsistent but acceptable. ,like , then it is confirmed as a valid target, its spatial positioning center and bounding box coordinates are output, and the target positioning data set is output: , is the output target positioning dataset, is the horizontal coordinate of the target center (positioning result), is the target center ordinate (positioning result), is the target bounding box (which may include center coordinates and size), is the target semantic category label.

[0151] Step S35: extracting motion vector features based on the target positioning data to obtain motion vector feature data, and performing motion function fitting based on the motion vector feature data to obtain motion trajectory prediction data.

[0152] Specifically, the historical position change trend is extracted and the motion path is fitted to predict the future position; , collect its positioning points in nearly N frames (such as 5 frames) and the next moment's positioning point , calculate the velocity vector between every two frames: , if high-order modeling is enabled, the acceleration term can be: , is the acceleration, For the The velocity vector at time t, For the Velocity vector at the moment, trajectory fitting is performed using polynomial regression or Kalman filter model: Or use LSTM / GRU neural network to predict the next moment Predict the target position for the next K frames and output the predicted trajectory sequence: , For the The predicted trajectory sequence of a target, For the The target horizontal coordinate at the moment, For the The target vertical coordinate at the moment, is the predicted horizontal position of the target in the kth frame in the future, Output the motion trajectory prediction dataset as the predicted vertical position of the target in the kth frame in the future .

[0153] Preferably, the spatiotemporal density enhancement is specifically:

[0154] Perform spatial clustering on low-confidence target data in confidence stratified data to obtain repeated target data;

[0155] Specifically, the input is a set of all low-confidence targets in the current frame , is the set of low-confidence targets in the current frame, Target The central abscissa of Target The central ordinate of Target The bounding box width, Target The bounding box height of Target The category label, Target Detection confidence; construct spatial distance metric and category consistency constraint: , ,in Target 、 The spatial similarity value of For the low confidence targets, For the low confidence targets, is the category consistency indicator function, Target The category label, Target The category label, is the natural exponential term, Target The center point coordinates, Target The center point coordinates, for, Control the clustering range (e.g., 20 pixels); use DBSCAN or MeanShift clustering to identify repeated target clusters and set the minimum number of clusters , the maximum neighborhood distance px, output repeated target dataset.

[0156] Perform cross-frame trajectory tracking according to repeated target data to obtain cross-frame trajectory tracking data;

[0157] Specifically, determine whether these repeated targets have consistent motion trajectory support between consecutive frames, input the repeated targets after clustering in the current frame , low confidence targets in historical frames ,…,in For the The set of low-confidence targets in the frame, For the The set of low confidence targets in the frame. , find matching targets in the first N frames , For the Frame candidate historical targets, satisfying , , is the intersection-over-union ratio of two bounding boxes, The current frame Repeated targets, The current frame Repeated targets, for The semantic category label corresponding to the target, for The semantic category label corresponding to the target. Use SORT or Deep SORT lightweight tracker to give each target in the cluster a track ID. , For the The cross-frame motion trajectory of a target, For the goal The horizontal coordinate of the center position in the frame, For the goal The vertical coordinate of the center position in the frame, For the goal The horizontal coordinate of the center position in the frame, For the goal The vertical coordinate of the center position in the frame, For the goal The vertical coordinate of the center position in the frame, For the goal The vertical coordinate of the center position in the frame, output cross-frame trajectory tracking data set , is a cross-frame trajectory tracking data set, which contains all target trajectory sequences. For the Cross-frame motion trajectory of a target.

[0158] The spatiotemporal density of low-confidence target data is calculated based on cross-frame trajectory tracking data and repeated target data to obtain target spatiotemporal density data;

[0159] Specifically, spatial repetition and temporal continuity are integrated into a spatiotemporal density index to screen whether weak targets are credible. The spatial density is calculated as follows: , calculate its local density: , For the The spatial density of target clusters, For the target clusters, is the value of pi, which is approximately 3.1416. is the radius of the circumscribed circle of the cluster, and the temporal stability calculation is as follows: The number of occurrences in the frame is , then the time density is , Target Time density score, Target in the past The number of times it was detected in the frame, is the total number of frames in the time window, and the spatiotemporal density fusion index is calculated as follows , Target The spatiotemporal credibility score of For the low confidence targets, is the fusion weight coefficient of spatial density, is the normalized spatial density, is the fusion weight coefficient of time density, is the normalized time density, =0.6, =0.4, output the spatiotemporal density score table for each low-confidence target: , we1, For the low confidence targets, Score the spatiotemporal density fusion for this target.

[0160] Low-confidence target data in the confidence layered data are screened according to the target spatiotemporal density data to obtain layered enhanced data.

[0161] Specifically, the confidence threshold for spatiotemporal density screening is set as , used to determine whether a low-confidence target has a basis for credible reinforcement. Preferably, the threshold can be set to , which requires that the target should show above medium spatiotemporal density in terms of spatial aggregation and temporal consistency. For all low confidence targets ,like , it is considered as an enhanced target and added to the new detection set: , To filter the enhanced target detection result set, The target set whose confidence is originally higher than the threshold is directly regarded as a credible target. is the union operation of sets, For the low confidence targets, For all low-confidence target sets (e.g., confidence ≤ 0.5), For the The spatiotemporal density score of each target, with a value range of [0,1], is the screening threshold, set to 0.6, and output the hierarchical enhanced data set , containing the reinforced low-confidence target.

[0162] Preferably, step S4 is specifically:

[0163] Step S41: generating a track intersection area according to the motion track prediction data to obtain track intersection area data;

[0164] Specifically, find the spatial region where the motion trajectories of any two targets overlap in the future, i.e., the potential collision region. Input the trajectory prediction data of each target: , For motion trajectory prediction data, Target In the future time step The two-dimensional horizontal coordinate position of Target In the future time step The two-dimensional vertical coordinate position of To predict the start time, Predict the number of steps for the trajectory, is the initial time. The overlap judgment rule is that for any two target trajectories , , perform moment-by-moment crossover judgment: if there is a time step ,satisfy: , Target In the future time step The two-dimensional horizontal coordinate position of Target In the future time step The two-dimensional horizontal coordinate position of Target In the future time step The two-dimensional horizontal coordinate position of Target In the future time step The two-dimensional horizontal coordinate position of For collision threshold (such as target radius and + safety margin, e.g. 0.5m), expand the intersection position into an elliptical / circular safety zone , and record the time window , each set of trajectory intersections constitutes a potential collision region object: , is the trajectory intersection area object, It is the spatial buffer area. is the time parameter, is the starting time of the crossover duration interval, The end time of the crossing duration interval, output trajectory crossing area dataset .

[0165] Step S42: constructing a collision propagation graph based on the trajectory intersection area data to obtain collision propagation graph data;

[0166] Specifically, a graph structure is established to represent the collision propagation chain relationship between multiple targets due to trajectory intersection, and each moving target is a node in the graph. , if the target and There is a track intersection area , then add an edge . : Calculate edge weights based on trajectory overlap, duration, and speed direction difference: , Target point to The intensity of risk transmission, is the weight of the angle conflict between the two target motion directions, is the ratio of the duration of the intersection of the two target trajectories to the prediction time window, is the weight of the angle conflict between the two target motion directions, is the degree of angular conflict between the two target motion directions, is the normalized score of the included angle (the larger the included angle, the higher the score, and angles above 90° are significantly weighted), is the weight of the area ratio between the intersection area and the joint bounding box of the two targets, is the area ratio between the intersection area and the joint bounding box of the two targets. The collision propagation graph structure is to construct a directed graph , For nodes, For connection relationship, Is the edge weight, which represents the impact strength of the collision. Output collision propagation graph data .

[0167] Step S43: Perform risk propagation and diffusion simulation based on the collision propagation graph data to obtain collision risk data.

[0168] Specifically, the diffusion trend of collision risk in the propagation graph is simulated to obtain the dynamic risk level of each target, and the nodes directly involved in the trajectory intersection are The node is marked as the initial risk source, and its risk value is set to 1.0; the initial value of other nodes is 0. The propagation model uses a PageRank-style propagation model or a Bayesian risk diffusion model: , For the Round time target node The risk value, For the moving target nodes, For the moving target nodes, is the risk propagation control factor, which determines the ratio of the initial value to the influence of neighbors and is set to 0.05. For nodes The initial risk value (0 or 1), For nodes The set of neighbor nodes of For adjacent nodes Towards The edge weight of the spread (risk weight), is the node order item, For nodes The sum of the edge weights of all outgoing edges, For nodes In the Risk score in the iteration, For nodes In the Round node risk value. To propagate the control factor, the propagation process is iterated until the risk value converges or reaches a fixed number of rounds (such as 10 rounds). Setting level: =High risk, 0.4∼0.7: Medium risk, Output the collision risk level and prediction time window of each target. The output structure is , For collision risk data collection, record the risk score and level of each target, For the moving target nodes, is the risk level classification result of the target node (high, medium, low), For nodes Risk score after proliferation has ceased.

[0169] Preferably, the present application further provides an image fusion driven anti-collision recognition system for executing the image fusion driven anti-collision recognition method described above, the image fusion driven anti-collision recognition system comprising:

[0170] The semantic fusion perception module is used to obtain multi-source image data and perform semantic-guided image fusion based on the multi-source image data to obtain image fusion data;

[0171] The target semantic parsing module is used to perform scene segmentation based on the image fusion data to obtain scene segmentation data, and perform target detection on the scene segmentation data to obtain scene target data;

[0172] The layered positioning and trajectory prediction module is used to perform confidence-level layered spatial positioning based on scene target data to obtain target positioning data, and to perform motion trajectory prediction based on the target positioning data to obtain motion trajectory prediction data;

[0173] The collision risk inference module is used to evaluate the collision area propagation risk based on the motion trajectory prediction data to obtain collision risk data.

[0174] Therefore, no matter from which point of view, the embodiments should be regarded as illustrative and non-restrictive, the scope of the present invention is limited by the attached application documents rather than the above description, and it is intended that all changes that fall within the meaning and scope of equivalent elements of the application documents are included in the present invention.

[0175] The foregoing description is intended only to provide specific embodiments of the present invention, which will enable those skilled in the art to understand and implement the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not intended to be limited to the embodiments shown herein, but is to be construed in the widest possible manner consistent with the principles and novel features disclosed herein.

Claims

1. An image fusion driven anti-collision recognition method, characterized in that: The following steps are involved: Step S1: acquiring multi-source image data, and performing semantic-guided image fusion based on the multi-source image data to obtain image fusion data; Step S2: performing scene segmentation based on the image fusion data to obtain scene segmentation data, and performing target detection on the scene segmentation data to obtain scene target data; Step S3: Confidence-layered target filtering is performed based on the scene target data to obtain confidence-layered data, wherein the confidence-layered data includes high-confidence target data, low-confidence target data, and structure-assisted candidate data. The high-confidence target data is a target area whose confidence score is significantly higher than a preset threshold during the detection process. The low-confidence target data is a target area whose confidence score is significantly lower than the preset threshold during the detection process. The structure-assisted candidate data is a target area that is not clearly marked as a target by a preset detection model, but the area exhibits geometric or semantic features similar to the target in the edge map data. The edge map data is data obtained by edge map extraction based on multi-source image data. The low-confidence target data in the confidence-layered data is subjected to spatiotemporal density enhancement to obtain layered enhanced data. Calculate the edge area overlap of the layered enhanced data according to the fused edge map data to obtain edge area overlap data; The matching degree is calculated based on the semantic area data and the edge area overlap data to obtain the target positioning data, and the motion trajectory is predicted based on the target positioning data to obtain the motion trajectory prediction data; Step S4: performing collision area propagation risk assessment based on motion trajectory prediction data to obtain collision risk data; The specific enhancement of space-time density is: Perform spatial clustering on low-confidence target data in confidence stratified data to obtain repeated target data; Perform cross-frame trajectory tracking according to repeated target data to obtain cross-frame trajectory tracking data; The spatiotemporal density of low-confidence target data is calculated based on cross-frame trajectory tracking data and repeated target data to obtain target spatiotemporal density data; Low-confidence target data in the confidence layered data are screened according to the target spatiotemporal density data to obtain layered enhanced data.

2. The method according to claim 1, characterized in that Step S1 is specifically as follows: Acquire multi-source image data; Extracting edge maps based on multi-source image data to obtain edge map data; Perform multi-scale convolution blurring on the edge map data to obtain convolution blurred data, and perform sharpening on the convolution blurred data to obtain fused edge map data; Perform semantic region mapping on edge graph data to obtain graph semantic region data; Calculate the edge density of the fused edge graph data based on the graph semantic region data to obtain the structural weight data; Perform high-frequency response calculation based on multi-source image data to obtain image clarity data; Perform weighted fusion on multi-source image data according to structural weight data and image clarity data to obtain image fusion data; The semantic area mapping is specifically as follows: Extracting local color features and local texture features from edge image data to obtain image color feature data and image texture feature data respectively; Use the preset lightweight image semantic recognition model to perform low-resolution semantic reasoning on the image color feature data and image texture feature data to obtain semantic category label data and semantic confidence data; Constructing a proximity graph based on the semantic category label data and the semantic confidence data to obtain proximity graph data; The edge graph data is mapped to category credibility based on the proximity graph data to obtain graph semantic region data.

3. The method according to claim 2, characterized in that Step S2 is specifically as follows: Perform hierarchical semantic segmentation based on image fusion data to obtain semantic segmentation data; Perform boundary consistency reconstruction based on semantic segmentation data to obtain boundary reconstruction data; Perform entity segmentation on the image fusion data according to the boundary reconstruction data to obtain scene segmentation data; Perform regional priority detection and sorting based on scene segmentation data to obtain detection sorting data; Perform cross-modal feature enhancement target detection based on detection ranking data and image fusion data to obtain scene target data; The cross-modal feature reinforcement target detection is specifically as follows: Perform spatial attention calculation based on the detection sorting data and image fusion data to obtain spatial attention data; The detection ranking data is labeled according to the spatial attention data to obtain the detection weight data; Perform multimodal feature extraction based on graph semantic region data and image fusion data to obtain multimodal feature data; Perform regional attention enhancement on the multimodal feature data according to the detection weight data to obtain feature enhancement data; The preset target recognition model is used to identify the feature enhanced data to obtain the scene target data.

4. The method according to claim 3, characterized in that The hierarchical semantic segmentation is specifically as follows: Perform multi-resolution division according to the image fusion data to obtain image division data; Performing hierarchical semantic perception on the image segmentation data to obtain hierarchical semantic perception data; Constructing hierarchical semantic paths based on hierarchical semantic perception data to obtain hierarchical semantic path data; The boundaries of the image segmentation data are reconstructed according to the hierarchical semantic path data to obtain semantic segmentation data.

5. The method according to claim 1, characterized in that Step S4 is specifically as follows: Generate a trajectory intersection area based on the motion trajectory prediction data to obtain trajectory intersection area data; Construct a collision propagation map based on the trajectory intersection area data to obtain collision propagation map data; Risk propagation and diffusion simulation is performed based on the collision propagation map data to obtain collision risk data.

6. An image fusion driven anti-collision recognition system, characterized in that: For executing the image fusion driven anti-collision recognition method according to claim 1, the image fusion driven anti-collision recognition system comprises: The semantic fusion perception module is used to obtain multi-source image data and perform semantic-guided image fusion based on the multi-source image data to obtain image fusion data; The target semantic parsing module is used to perform scene segmentation based on the image fusion data to obtain scene segmentation data, and perform target detection on the scene segmentation data to obtain scene target data; The layered positioning and trajectory prediction module is used to perform confidence layered target filtering based on scene target data to obtain confidence layered data, wherein the confidence layered data includes high-confidence target data, low-confidence target data and structure-assisted candidate data. The high-confidence target data is the target area whose confidence score is significantly higher than the preset threshold during the detection process, the low-confidence target data is the target area whose confidence score is significantly lower than the preset threshold during the detection process, and the structure-assisted candidate data is the target area where the preset detection model does not clearly mark the target, but the area shows similar geometric or semantic features to the target in the edge map data. The edge map data is the data obtained by edge map extraction based on multi-source image data; the low-confidence target data in the confidence layered data is subjected to spatiotemporal density enhancement to obtain layered enhanced data. ; Calculate the edge area overlap of the layered enhanced data according to the fused edge map data to obtain edge area overlap data; calculate the matching degree according to the map semantic area data and the edge area overlap data to obtain target positioning data, and predict the motion trajectory according to the target positioning data to obtain motion trajectory prediction data; wherein the spatiotemporal density enhancement is specifically as follows: spatially cluster the low-confidence target data in the confidence layered data to obtain repeated target data; perform cross-frame trajectory tracking according to the repeated target data to obtain cross-frame trajectory tracking data; calculate the spatiotemporal density of the low-confidence target data according to the cross-frame trajectory tracking data and the repeated target data to obtain target spatiotemporal density data; screen the low-confidence target data in the confidence layered data according to the target spatiotemporal density data to obtain layered enhanced data; The collision risk inference module is used to evaluate the collision area propagation risk based on the motion trajectory prediction data to obtain collision risk data.

Images