Image fusion driven anti-collision identification method and system

Abstract

The invention relates to the technical field of image processing, in particular to an image fusion driven anti-collision identification method and system. The method comprises the following steps: acquiring multi-source image data, and performing semantic guidance image fusion according to the multi-source image data to obtain image fusion data; performing scene segmentation according to the image fusion data to obtain scene segmentation data, and performing target detection on the scene segmentation data to obtain scene target data; performing confidence hierarchical space positioning according to the scene target data to obtain target positioning data, and performing motion trail prediction according to the target positioning data to obtain motion trail prediction data; and performing collision area propagation risk assessment according to the motion trail prediction data to obtain collision risk data. By fusing image perception depth, semantic understanding dimension and trajectory evolution modeling, the method has higher environmental adaptability, weak target recovery capability and system interpretability.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and particularly to an anti-collision recognition method and system driven by image fusion. Background Art

[0002] With the continuous expansion of application scenarios such as autonomous driving, intelligent robots, and unmanned inspection equipment, the environmental perception and collision risk recognition technology based on visual information has become the core component of the safety control of intelligent systems. In the prior art, a single image source (such as an RGB image or an infrared image) is usually used for target detection and collision judgment, which is easily affected by factors such as illumination change, occlusion interference, or image blur, resulting in inaccurate target recognition and easy omission of weak targets, thereby reducing the robustness of the system in different environments. Summary of the Invention

[0003] In order to solve the above technical problems, the present invention proposes an anti-collision recognition method and system driven by image fusion to solve at least one of the above technical problems.

[0004] The present application provides an anti-collision recognition method driven by image fusion, including the following steps: Step S1: Obtain multi-source image data, and perform semantic-guided image fusion according to the multi-source image data to obtain image fusion data; Step S2: Perform scene segmentation according to the image fusion data to obtain scene segmentation data, and perform target detection on the scene segmentation data to obtain scene target data; Step S3: Perform confidence-level hierarchical spatial positioning according to the scene target data to obtain target positioning data, and perform motion trajectory prediction according to the target positioning data to obtain motion trajectory prediction data; Step S4: Perform collision area propagation risk assessment according to the motion trajectory prediction data to obtain collision risk data.

[0005] In the present invention, step S1 improves the fusion quality of multi-source images in terms of structural features, clarity, and semantic consistency by introducing a semantic-guided image fusion mechanism, providing a more stable visual input for scene analysis; step S2 combines the fused image for scene segmentation and target detection, enabling accurate recognition of multiple targets and multi-semantic entities in the environment; step S3 introduces confidence-level stratified spatial localization and motion trajectory prediction based on the target detection results, not only improving the tracking stability of key targets but also enhancing the system's target behavior modeling ability in dynamic scenarios; step S4 further constructs a collision area propagation model using the predicted trajectory, achieving visual diffusion reasoning and risk level determination of risk areas, thus significantly improving the response accuracy and forward-looking ability of the anti-collision system in multi-target and multi-path interaction scenarios. The present invention has the advantages of lightweight, high efficiency, intelligent evolution in fusion recognition and dynamic prediction, and is applicable to various scenarios such as intelligent driving (e.g., real-time monitoring of tower cranes and working vehicles in the operating environment), unmanned equipment, and security patrol.

[0006] Preferably, step S1 is specifically as follows: Obtain multi-source image data; Extract an edge map based on the multi-source image data to obtain edge map data; Perform multi-scale convolutional blurring on the edge map data to obtain convolutional blurring data, and perform sharpening processing on the convolutional blurring data to obtain fused edge map data; Perform semantic region mapping on the edge map data to obtain graph semantic region data; Calculate the edge density of the fused edge map data according to the graph semantic region data to obtain structural weight data; Perform high-frequency response calculation based on the multi-source image data to obtain image clarity data; Perform weighted fusion on the multi-source image data according to the structural weight data and the image clarity data to obtain image fusion data.

[0007] In the present invention, by performing edge map extraction and multi-scale convolutional blurring-sharpening processing on multi-source images, the key structural contours and detail responses of the images are enhanced; a semantic region mapping mechanism is introduced, and the edge density of specific semantic regions is evaluated in combination with the edge map, thereby constructing structural weight data with semantic perception ability; the image clarity index is extracted through high-frequency response, which helps to identify information-dense regions and blurred regions in the image. By fusing the structural weight data and the image clarity data, the multi-source images are adaptively weighted and fused according to the importance of the regions, significantly improving the performance of the fused images in terms of structural integrity, detail retention, and semantic continuity.

[0008] Preferably, the semantic region mapping is specifically as follows: Extract local color features and local texture features from the edge map data to obtain the map color feature data and the map texture feature data respectively; Use a preset lightweight image semantic recognition model to perform low-resolution semantic inference on the map color feature data and the map texture feature data to obtain semantic category label data and semantic confidence data; Construct an adjacent relationship graph based on the semantic category label data and the semantic confidence data to obtain adjacent relationship graph data; Perform category credibility mapping on the edge map data according to the adjacent relationship graph data to obtain map semantic region data.

[0009] In the present invention, by constructing a lightweight semantic region mapping mechanism, efficient and structure-sensitive semantic assignment to regions in the edge map is achieved without relying on a deep semantic segmentation model. By jointly extracting local color features and local texture features from the edge map, the adaptability of semantic inference to structural details is enhanced; a lightweight image semantic recognition model is used to perform low-resolution semantic inference on color and texture features, generating a semantic candidate map with class label and confidence information while maintaining computational efficiency. An adjacent relationship graph is constructed based on semantic labels and confidence, and through the introduction of dual constraints of edge space adjacency and semantic consistency, structured propagation of semantic information within the edge region is realized. Through the class credibility mapping mechanism guided by the adjacent relationship graph, semantic enhancement annotation is performed on the edge map, effectively improving the recognition and weight control capabilities of semantic important regions during the image fusion process. Compared with the existing methods that rely on large-scale semantic segmentation models or unified full-map inference, this step significantly reduces the computational resource requirements, improves the edge local semantic modeling ability, and has stronger real-time performance, regional adaptability, and embedded deployment advantages.

[0010] Preferably, step S2 is specifically as follows: Perform hierarchical semantic segmentation on the image fusion data to obtain semantic segmentation data; Perform boundary consistency reconstruction according to the semantic segmentation data to obtain boundary reconstruction data; Perform entity segmentation on the image fusion data according to the boundary reconstruction data to obtain scene segmentation data; Perform region priority detection and sorting according to the scene segmentation data to obtain detection and sorting data; Perform cross-modal feature reinforcement target detection according to the detection and sorting data and the image fusion data to obtain scene target data.

[0011] In the present invention, a four-level processing chain of semantic-driven - boundary reconstruction - entity segmentation - cross-modal reinforcement is constructed to achieve a high-precision semantic perception process from scene structure analysis to key target recognition. Hierarchical semantic segmentation is performed based on image fusion data to effectively divide the semantic regions in the scene, providing a semantic basis for structure analysis; through the boundary consistency reconstruction mechanism, the boundaries of the preliminary semantic regions are refined and corrected, solving the problems of blurred edges and entity breaks in traditional segmentation methods and improving spatial consistency and segmentation accuracy; on this basis, entity-level segmentation of the fusion image is performed in combination with boundary information to generate scene segmentation data with complete structure and clear semantics. A regional priority detection and sorting mechanism is designed to sort each candidate region according to indicators such as spatial occupancy, semantic weight, or edge activity, effectively focusing the detection resources on the key target regions; finally, cross-modal feature reinforcement detection is performed using image fusion data, and through the joint input of structural features and semantic channels, the recognition of detail regions is enhanced in the target detection stage. Compared with the prior art that only relies on a single path (such as semantic detection or instance segmentation), the present invention enhances the detection accuracy and the ability to distinguish key regions while retaining the integrity of the segmentation structure, and has higher adaptability and practicality in multi-target, semi-occluded, or semantic boundary blurred scenarios.

[0012] Preferably, the hierarchical semantic segmentation is specifically as follows: Perform multi-resolution division on the image fusion data to obtain image division data; Perform hierarchical semantic perception on the image division data to obtain hierarchical semantic perception data; Construct a hierarchical semantic path based on the hierarchical semantic perception data to obtain hierarchical semantic path data; Perform boundary reconstruction on the image division data according to the hierarchical semantic path data to obtain semantic segmentation data.

[0013] In the present invention, by introducing a multi-resolution division and hierarchical semantic perception mechanism, a hierarchical semantic segmentation method for coordinating structural details and semantic levels is constructed, significantly improving the clarity of semantic boundaries and the integrity of segmentation structures in diverse scenarios. Through multi-resolution division of image fusion data, hierarchical expansion of the image in the spatial scale is achieved, and structural textures and context information at different scales are retained; hierarchical semantic perception is performed on different resolution regions to effectively extract local significant features and global semantic associations, forming hierarchical semantic perception data. By constructing a hierarchical semantic path, orderly association and aggregation of semantic information at different levels are carried out, enabling the semantic relationship between adjacent regions to be retained and traced, thereby enhancing semantic consistency; the hierarchical semantic path is used to reconstruct the image boundary, repair semantic breaks and detail jumps, and output semantic segmentation data with continuous structure and complete semantics. Compared with the traditional semantic segmentation method that uses a unified scale input and single-layer semantic feature extraction, the present invention makes the model more context-sensitive and boundary-aware by fusing multi-scale structural expressions and semantic path construction, and shows stronger adaptability and segmentation accuracy in scenarios with multi-object overlap, large scale differences or regional semantic transitions.

[0014] Preferably, the cross-modal feature reinforcement object detection is specifically as follows: Perform spatial attention calculation based on the detection ranking data and the image fusion data to obtain spatial attention data; Annotate the detection ranking data according to the spatial attention data to obtain detection weight data; Perform multi-modal feature extraction based on the graph semantic region data and the image fusion data to obtain multi-modal feature data; Enhance the regional attention of the multi-modal feature data according to the detection weight data to obtain feature enhancement data; Use a preset object recognition model to recognize the feature enhancement data to obtain scene object data.

[0015] In the present invention, spatial attention weights are calculated based on detection sorting data and image fusion data to dynamically focus on regions with structural saliency and semantic priority in the image, thereby constructing a spatial guidance mechanism for the region of interest. Subsequently, the spatial attention results are used for detection sorting annotation to generate importance markers for the target region, forming detection weight data, which enhances the region selectivity and information utilization efficiency during the detection process. Multimodal feature information is extracted by combining graph semantic region data and image fusion data, integrating edge structures, semantic labels, and multi-source image responses to form multimodal feature data with high-dimensional information expression ability. Regional attention enhancement is performed on the multimodal features according to the detection weight data, so that the feature expression of the key target region is amplified in the feature space, effectively suppressing background interference and blurred boundaries. The enhanced features are input into the target recognition model for detection and classification to output accurate scene target data. Compared with the existing single-modal or unguided detection methods, the present invention not only has cross-modal expression ability, but also introduces regional weights and attention control, improving the detection accuracy and stability of the system in scenarios with multiple targets, occlusions, weak boundaries, or high inter-class similarity, and has stronger engineering application value and generalization ability.

[0016] Preferably, step S3 is specifically as follows: Confidence-level hierarchical target filtering is performed on the scene target data to obtain confidence-level hierarchical data, where the confidence-level hierarchical data includes high-confidence target data, low-confidence target data, and structure-assisted candidate data. The high-confidence target data is the target region whose confidence score is significantly higher than the preset threshold during the detection process. The low-confidence target data is the target region whose confidence score is significantly lower than the preset threshold during the detection process. The structure-assisted candidate data is the target region where the preset detection model does not clearly mark the target, but this region shows geometric or semantic features similar to the target in the structure perception layer (such as the edge map). Spatial-temporal density enhancement is performed on the low-confidence target data in the confidence-level hierarchical data to obtain hierarchical enhancement data. Edge region overlap degree calculation is performed on the hierarchical enhancement data according to the fused edge map data to obtain edge region overlap degree data. Matching degree calculation is performed according to the graph semantic region data and the edge region overlap degree data to obtain target location data. Motion vector feature extraction is performed according to the target location data to obtain motion vector feature data, and motion function fitting is performed according to the motion vector feature data to obtain motion trajectory prediction data.

[0017] In the present invention, confidence-level hierarchical filtering is performed on scene target data, and the detection results are divided into three types of regions: high-confidence targets, low-confidence targets, and structure-assisted candidates, significantly improving the system's perception and coverage capabilities for weak or non-explicit targets. Among them, structure-assisted candidate data is discriminated through edge maps and semantic structures, making up for the omission of low-significance targets by traditional detection models. Secondly, spatio-temporal density enhancement is performed on low-confidence regions, combining the spatial stability and local aggregation trend of consecutive frames to improve the dynamic recognizability of weak targets. Further, by performing edge-region overlap analysis on fused edge-map data and calculating the semantic matching degree with graph semantic regions, the spatial consistency and semantic rationality of target localization are effectively improved. Finally, by extracting motion vector features from the target position sequence and performing dynamic function fitting, structured motion trajectory prediction data is obtained. Compared with existing object detection and tracking methods that mostly rely only on high-confidence targets for trajectory modeling, the present invention takes into account structure assistance and semantic reasoning, has stronger object completeness, accuracy robustness, and the ability to recover occluded / blurred targets, and is particularly suitable for highly reliable anti-collision recognition systems in dynamic, unstructured, or low-contrast scenarios.

[0018] Preferably, the spatio-temporal density enhancement is specifically as follows: Perform spatial clustering on the low-confidence target data in the confidence-level hierarchical data to obtain duplicate target data; Perform cross-frame trajectory tracking based on the duplicate target data to obtain cross-frame trajectory tracking data; Calculate the spatio-temporal density of the low-confidence target data based on the cross-frame trajectory tracking data and the duplicate target data to obtain target spatio-temporal density data; Screen the low-confidence target data in the confidence-level hierarchical data based on the target spatio-temporal density data to obtain hierarchical enhancement data.

[0019] In the present invention, spatial clustering is performed on low-confidence target data to identify target regions that repeatedly appear or are close in position in the same frame or adjacent frames, constituting preliminary duplicate target data; cross-frame trajectory tracking is performed based on the duplicate target data to construct a time-series association path for low-confidence targets, forming cross-frame trajectory tracking data, which solves the problem that traditional detection models lack weak target consistency modeling between frames. By combining the appearance frequency of duplicate targets, inter-frame continuity, and trajectory stability, target spatio-temporal density is calculated to form a quantitative target presence confidence index, and based on this, low-confidence targets are screened to retain real target regions with persistence and consistency in structure. The output hierarchical enhanced data not only significantly improves target coverage but also provides more reliable input for edge overlap analysis and positioning inference. Compared with existing methods that usually directly eliminate low-confidence targets or simply score them, the present invention introduces a multi-frame consistency enhancement and spatial density quantization strategy in low-confidence regions, realizing the structural recovery and dynamic reliable extraction of weak targets, with higher target perception integrity and motion modeling accuracy, and is applicable to application scenarios with severe occlusion, discontinuous frames, or large fluctuations in image quality.

[0020] Preferably, step S4 is specifically as follows: Generate a trajectory intersection region based on the motion trajectory prediction data to obtain trajectory intersection region data; Construct a collision propagation graph based on the trajectory intersection region data to obtain collision propagation graph data; Perform a risk propagation diffusion simulation based on the collision propagation graph data to obtain collision risk data.

[0021] In the present invention, trajectory intersection region data is generated by combining motion trajectory prediction data, effectively capturing the spatial intersection region where there is a potential collision risk between multiple targets, and accurately positioning the key spatial nodes that may cause interference or conflict; a collision propagation graph is constructed based on the trajectory intersection region data, where the nodes in the graph represent potential collision objects, and the edges represent risk propagation factors such as trajectory intersection intensity, time proximity, or direction consistency, thus forming a graph model with clear structural expression and quantifiable weights; a propagation simulation algorithm is used to perform diffusion evolution on the collision propagation graph, and by combining factors such as velocity vectors, intersection probabilities, and path densities, the collision risk level and its diffusion trend evolving over time are deduced, and collision risk data is output. Different from traditional methods that only judge collisions based on the current frame or a single intersection point, the present invention models the dynamic interaction of multi-target trajectories as a graph structure and performs propagation simulation, which can predict the possible spatial range and risk degree of collisions in the future period, significantly improving the forward-looking, stability, and interpretability of the anti-collision system in a high-density, multi-path, and multi-target dynamic environment.

[0022] Preferably, the present application further provides an anti-collision recognition system driven by image fusion, which is used to execute the anti-collision recognition method driven by image fusion as described above. The anti-collision recognition system driven by image fusion includes: A semantic fusion perception module, which is used to obtain multi-source image data and perform semantic-guided image fusion based on the multi-source image data to obtain image fusion data; A target semantic parsing module, which is used to perform scene segmentation on the image fusion data to obtain scene segmentation data, and perform target detection on the scene segmentation data to obtain scene target data; A hierarchical positioning and trajectory prediction module, which is used to perform confidence-level hierarchical spatial positioning on the scene target data to obtain target positioning data, and perform motion trajectory prediction based on the target positioning data to obtain motion trajectory prediction data; A collision risk inference module, which is used to perform collision area propagation risk assessment on the motion trajectory prediction data to obtain collision risk data.

[0023] The beneficial effects of the present invention are as follows: In step S1, by introducing a semantic-guided image fusion mechanism and using structural weights and clarity responses to perform weighted processing on multi-source images, the scene structure features and semantic consistency are enhanced, providing high-quality perceptual input for the task; in step S2, by combining hierarchical semantic segmentation and cross-modal feature reinforcement target detection technology, high-precision recognition of key areas is achieved while maintaining the integrity of the target structure; in step S3, through a confidence-level hierarchical mechanism and an edge semantic-guided positioning strategy, structural repair and dynamic trajectory fitting are performed on low-significance targets, effectively improving the target coverage rate and trajectory continuity; in step S4, further based on the predicted trajectory, a collision propagation map is constructed to simulate the potential interaction relationships and risk diffusion processes among multiple targets, and a collision risk result with clear risk levels and explicit evolution paths is output. Compared with traditional solutions that only rely on high-confidence targets or rule judgments, the present invention has stronger environmental adaptability, weak target recovery ability, and system interpretability by fusing image perception depth, semantic understanding dimension, and trajectory evolution modeling, and is suitable for the deployment of highly reliable anti-collision systems in high-density, unstructured, or weakly-perceived scenarios. Description of the Drawings

[0024] By reading the detailed description of the non-restrictive embodiments with reference to the following drawings, other features, purposes, and advantages of the present application will become more obvious: Figure 1 Shows the step flow chart of an anti-collision recognition method driven by image fusion according to an embodiment; Figure 2 Shows the step flow chart of a semantic fusion perception method according to an embodiment; Figure 3 Shows the step flow chart of a target semantic parsing method according to an embodiment; Figure 4 The flowchart of the steps of a hierarchical positioning and trajectory prediction method according to an embodiment is shown; Figure 5 The flowchart of the steps of a collision risk reasoning method according to an embodiment is shown. Detailed implementation manners

[0025] The technical method of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the scope of protection of the present invention.

[0026] In addition, the accompanying drawings are only schematic diagrams of the present invention and are not necessarily drawn to scale. The functional entities can be implemented in the form of software, or in one or more hardware modules or integrated circuits, or in different networks and / or processor methods and / or microcontroller methods.

[0027] It should be understood that although the terms "first", "second", etc. may be used here to describe various units, these units should not be limited by these terms. These terms are only used to distinguish one unit from another. For example, without departing from the scope of the exemplary embodiments, the first unit may be called the second unit, and similarly the second unit may be called the first unit. The term "and / or" used here includes any and all combinations of one or more of the listed related items.

[0028] Please refer to Figures 1 to 5 , this application provides a collision prevention recognition method driven by image fusion, including the following steps: Step S1: Obtain multi-source image data, and perform semantic-guided image fusion according to the multi-source image data to obtain image fusion data; Specifically, obtain image data from different perspectives or sensors (such as RGB cameras, infrared cameras, lidar projection maps), and unify them to a fixed size (such as 640×480). Perform time synchronization and geometric correction on all image data to eliminate perspective distortion and resolution inconsistency problems. Use a pre-trained semantic segmentation model (such as DeepLabv3+) to perform pixel-level semantic annotation on each image to obtain a semantic label map. Set a semantic weight matrix , and assign different fusion priorities to semantic categories (such as pedestrians, vehicles, backgrounds). For example: pedestrians = 1.0, vehicles = 0.8, backgrounds = 0.2. Perform weighted fusion: , For the fused image at the pixel position The fused pixel value at is the image sequence item, is the total number of image sources, is the weight value obtained by guiding the th image at this position by the semantic category, and is the original pixel value of the th image at the pixel position . Through the above semantic-guided weighting mechanism, it is possible to retain the information of target regions with high semantic value (such as pedestrians, vehicles) in multi-source images during the fusion process, while suppressing low-value regions (such as backgrounds).

[0029] Step S2: Perform scene segmentation based on the image fusion data to obtain scene segmentation data, and perform object detection on the scene segmentation data to obtain scene object data; Specifically, process the fused image using a semantic segmentation network (such as SegFormer) to output the class probability per pixel , that is, the probability that the pixel belongs to the class . Use CRF (conditional random field) to optimize the edge details to ensure accurate object boundaries. Generate the label map , is the semantic class label of the fused image at the position , is the predicted probability for all classes , is to perform a maximum selection, is the pixel point belongs to the class predicted probability value, is the class, is the abscissa of the pixel, is the ordinate of the pixel. After obtaining the semantic label map , filter the regions of interest based on the continuous semantic regions in the image (such as the minimum bounding rectangle of the semantic block for the detection box). Use detection algorithms such as YOLOv7 or Faster-RCNN to further detect the object positions and types in the fused image, and output the target box set , is the set of all detected objects, is the th detection target box, is the target index, with a range of , is the total number of detected objects, and each object contains position information, including the center coordinates of the object, the width and the height ; a class label, including the semantic class to which the target belongs, such as vehicle, pedestrian, sign, etc.; a detection confidence level, denoted as , indicating the credibility of the target box or the classification confidence score value.

[0030] Step S3: Perform confidence-level stratified spatial positioning based on the scene target data to obtain target positioning data, and perform motion trajectory prediction based on the target positioning data to obtain motion trajectory prediction data; Specifically, for each detection box , grades are divided according to the confidence level. For example, for high confidence (conf > 0.8), it is directly treated as a reliable target; for medium confidence (0.5 < conf ≤ 0.8), if it is satisfied, it is verified whether the target continuously appears in the current frame and the adjacent frames before and after, and it is judged whether it has trajectory continuity; for low confidence (conf ≤ 0.5), it does not participate in trajectory prediction temporarily and is only stored as redundant reference information. After obtaining the confidence-level stratification, the three-dimensional spatial coordinates of each target position are restored, and a depth estimation network (such as Monodepth2) is used to estimate the depth value from the fused image , representing the relative depth value corresponding to each pixel point in the image. Combining the camera internal parameter K, according to the monocular camera imaging model, using the pixel coordinate and the depth value , calculate the three-dimensional spatial coordinates of this point in the camera coordinate system: , is the three-dimensional spatial position of the pixel point in the camera coordinate system, is the pixel point 's depth estimation value, is the inverse matrix of the camera internal parameter matrix, including the focal length and the principal point offset, is the homogeneous coordinate vector of the pixel point. Motion trajectory prediction: Use a Kalman filter or an LSTM sequence model to perform trajectory prediction on the target position sequence of the recent N frames: , is the predicted position vector of the target at the next moment, is the position of the current or the th frame target, is the position of the previous or the th frame target, is the previous or the th frame target's position, is the prediction function, which can be implemented by a Kalman filter, an LSTM or other time series models.

[0031] Step S4: Perform a collision area propagation risk assessment based on the motion trajectory prediction data to obtain collision risk data.

[0032] Specifically, for any two target prediction trajectories , , determine whether there is a spatially overlapping area: , where is a selected set of time points, is the current time, is the predicted future time window (e.g., 2 seconds), is the collision threshold (e.g., 0.5 meters), is the target at time 's predicted position (3D coordinates), is the target at time 's predicted position (3D coordinates). To systematically describe the risk conduction relationship between targets, construct a collision risk graph, where the nodes are targets, the edges are potential collision relationships, and the edge weights are predicted collision probabilities. Use a propagation algorithm (such as a risk diffusion model based on PageRank) to perform iterative scoring on the entire graph structure to obtain the risk level scores of each target. Output the risk level, label the collision level (high / medium / low) for each target, and mark the predicted collision time point and area, , where is the collision level function of target , is high risk, is medium risk, is low risk, is the collision probability.

[0033] Preferably, step S1 is specifically: Step S11: Obtain multi-source image data; Specifically, the system acquires image data from multiple cameras deployed at different perspective positions in the target area. The perspectives include, but are not limited to, front-view cameras, side-view cameras, and top-view cameras, etc. To ensure the consistency of multi-source image data in the time dimension, the system inputs the image data into a time synchronization module for frame-level time alignment. The time synchronization module is used to calibrate and synchronize the timestamps of image frames from different cameras according to the system master clock signal, so as to ensure the comparability of multi-perspective images at the same time point. Further, to ensure the consistency of the images in the spatial geometric structure, the system performs spatial geometric distortion correction on each image. The correction methods may include de-distortion correction of fisheye images, epipolar geometry correction of binocular images, back-projection correction of lens distortion parameters, etc. The above geometric correction operations are used to restore the spatial projection relationship of real objects in the images and ensure the accurate spatial correspondence of images collected by different cameras.

[0034] Step S12: Extract edge maps based on the multi-source image data to obtain edge map data; Specifically, the system applies an edge extraction algorithm to each acquired image data Preferably, the Canny edge detection algorithm is used. The Canny algorithm includes Gaussian filtering for denoising, calculation of image gradient magnitude and direction, non-maximum suppression processing, and edge connection determination based on a double-threshold strategy. In actual processing, two edge detection thresholds are set for each image, namely a low threshold and a high threshold , which are used to control the screening process of edge intensity responses: if the gradient value of a certain pixel is higher than , it is directly retained as a strong edge; if it is between and and is connected to a strong edge, it is retained as a weak edge; otherwise, it is excluded. For the th image , its corresponding edge map is extracted. The edge map represents the spatial distribution of edge pixels in binary form.

[0035] Step S13: Perform multi-scale convolutional blurring on the edge map data to obtain convolutional blurring data, and perform sharpening processing on the convolutional blurring data to obtain fused edge map data; Specifically, multiple Gaussian kernels (such as 3×3, 5×5, 7×7) are used to perform convolution on each edge map to enhance the structural texture features, , where is the blurring enhancement result of the th image, is the Gaussian kernel sequence term, is the nuclear fusion coefficient (which can be set as a decreasing weight, such as 0.5, 0.3, 0.2). For each edge map , multiple Gaussian convolution kernels are used for convolution operation. is the th edge image. is the th Gaussian kernel. The blurred convolution image is sharpened. The Laplace operator is used for sharpening, which is defined as follows . is the sharpened image. is the blurred enhancement result of the th image. is the sharpening intensity coefficient, and its value range can be set from 0.8 to 1.2. is the Laplace transform applied to the image , that is, second-order derivative enhancement, which is used to extract the edge change information of the image. The images sharpened at multiple scales are weighted and fused to obtain the fused edge image. The fusion adopts a weight mechanism based on the edge information entropy of the image. Let the fusion weight corresponding to each sharpened image be . The weight value can be dynamically adjusted according to the edge information entropy of each image. The higher the information entropy, the more significant the representation and the greater the weight. The fusion formula is . is the fused edge map. is the sharpened image sequence term. is the image 's fusion weight. is the th sharpened image.

[0036] Step S14: Perform semantic region mapping on the edge map data to obtain the graph semantic region data; Specifically, a pre-trained semantic segmentation model (such as DeepLabv3+) is used to perform pixel-level segmentation on the original image to obtain the corresponding semantic region map. The semantic categories included are but not limited to pedestrians, vehicles, obstacles, and road surfaces. Subsequently, the semantic region map and the edge map data are aligned and fused pixel by pixel to construct an edge semantic mapping matrix. The generation rule of this mapping matrix is: when the value of the edge map in the th channel at a certain pixel point is 1, that is, there is an edge feature, the corresponding semantic category in the semantic region map is extracted and assigned to the mapping matrix; if the edge image pixel value is 0, it is marked as an invalid value (None) in the mapping matrix.

[0037] Step S15: Calculate the edge density of the fused edge map data based on the graph semantic region data to obtain the structure weight data; Specifically, for each semantic region , sum the response values of all pixels within it in the fused edge map, and divide by the total number of pixels in this region, thereby calculating the edge density value per unit area. , is the edge density value, representing the average edge intensity value per unit semantic region. is the pixel coordinate pair within the semantic region, representing the pixel point belonging to the semantic region , is the pixel value of the fused edge image, representing the edge response intensity at position in the fused edge map. is the th semantic region, which is the region block corresponding to the semantic category recognized by the semantic segmentation model. After normalization processing, the structure weight is obtained: , is the structure weight value, representing the significance weight of the semantic region structure information, and the range is normalized to [0,1]. is the edge density value, representing the average edge intensity value per unit semantic region. is the maximum edge density value among all semantic regions. Output the structure weight map , where each semantic region is assigned a structure weight value. The higher the edge density of a region, the greater its weight, reflecting that the structure information of this region is richer.

[0038] Step S16: Calculate the high-frequency response based on the multi-source image data to obtain the image sharpness data; Specifically, apply the Laplace filter to the image to extract the high-frequency information in the image. This operation is equivalent to calculating the second-order derivative response of the image in the gray domain, denoted as ; calculate the gray variance of the Laplace response result, denoted as , and the obtained result is defined as the sharpness index of this image . The variance value reflects the overall distribution of edge responses in the image. The higher its value, the clearer the edges and stronger the contrast in the image, indicating a higher degree of image focus. The system forms a set of the sharpness indices of all images.

[0039] Step S17: Perform weighted fusion on the multi-source image data according to the structure weight data and the image sharpness data to obtain the image fusion data.

[0040] Specifically, for each source image, at each pixel point of the image At a position, calculate its corresponding fusion weight. The fusion weight takes into account two factors. One is the structural weight, that is, the structural saliency of the image at this pixel; the other is the clarity level of the overall image. , Let be the fusion weight of the th source image at pixel be the fusion coefficient of the structural weight and the clarity weight, with a value range between 0 and 1, which is used to adjust the relative contributions of the structural features and the overall quality. Let be the structural saliency weight of the th image at pixel indicating the structural sensitivity of this pixel point in the image. Let be the overall clarity evaluation index of the th image, which can be quantified by means of image gradient, contrast or frequency domain energy, etc. be the overall clarity evaluation index of the

[0041] Preferably, the semantic region mapping is specifically as follows: Extract the local color features and local texture features from the edge map data to obtain the map color feature data and the map texture feature data respectively; Specifically, input the edge map data , the original image , for each edge point extract its neighborhood window (such as a 9×9 region); if the edge density (i.e., the ratio of the number of edge pixels to the total number of window pixels) in the local region where the current pixel is located is lower than a preset threshold (such as 0.1), then skip the feature extraction step of this pixel point; the local color feature extraction includes calculating the color histogram features of the window (such as the HSV three channels, with 5 bins for each channel, a total of =125 dimensions); or calculating color statistics such as mean, standard deviation, skewness, kurtosis, etc. The local texture feature extraction includes calculating the gray-level co-occurrence matrix (GLCM) features, including contrast (the intensity of gray-level changes), homogeneity (the similarity degree of adjacent pixel gray-level values), entropy (the complexity of texture information), or extracting the LBP local binary pattern features. Output the map color feature data and the map texture feature data .

[0042] Perform low-resolution semantic inference on the graph color feature data and the graph texture feature data using a preset lightweight image semantic recognition model to obtain semantic category label data and semantic confidence data; Specifically, extract the color feature vector of each pixel point in the image and the texture feature vector , and splice the two in the feature dimension to form a joint feature representation vector . This feature vector serves as the input to the lightweight semantic recognition model. The model is a pre-trained small convolutional neural network, and lightweight structures such as MobileNetV2, ShuffleNet, or other custom ones can be used. The output of this semantic recognition model includes the semantic category label at each pixel position , whose value belongs to several predefined categories (e.g., {vehicle, pedestrian, road, background,...}); and the corresponding semantic confidence score , whose value range is [0,1] and is used to represent the credibility of the current predicted label. In the model inference stage, to improve the computational efficiency, the input image is first downsampled at a ratio of 4 4 in the spatial dimension, so that the model can complete semantic inference at a coarse-grained resolution, and the obtained low-resolution semantic results are then mapped back to the original image resolution by the nearest neighbor interpolation method. In the model training stage, supervised learning is performed based on the labeled semantic image dataset, and the loss function includes multi-class cross-entropy loss and label smoothing regularization terms to improve the stability and generalization ability of the model in the class boundary region. The system output includes the semantic category label data of each image pixel point and the corresponding semantic confidence data, which can be used as the basic input for subsequent graph semantic region mapping, structure weight reconstruction modeling, and fusion guidance.

[0043] Construct a proximity relationship graph based on the semantic category label data and the semantic confidence data to obtain proximity relationship graph data; Specifically, perform a region division operation on the semantic label graph, using algorithms such as the watershed algorithm, connected component extraction, or superpixel algorithm (such as SLIC) to divide the image into multiple semantic candidate region blocks . Each region block constitutes a node in the graph structure, and its attributes include the main semantic category and the average semantic confidence . If any two regions and have a boundary contact relationship in the image, then a connecting edge is established in the graph. The weight value of this edge is defined as: , is the weight value of side , representing the strength of regional semantic consistency. is the natural exponential term. is the weight factor for controlling semantic differences. is the semantic category and is the category distance function (such as 0 or 1) between them. is the region 's main semantic category label. is the region 's main semantic category label. is the weight factor for controlling confidence differences. is the region 's average semantic confidence value. is the region 's main semantic category label. Output adjacent relationship graph data , and the adjacent relationship graph structure consists of three parts: a node set, an edge set, and an edge weight set.

[0044] Perform category credibility mapping on the edge graph data according to the adjacent relationship graph data to obtain graph semantic region data.

[0045] Specifically, for each edge point in the edge graph, if there is an effective edge response at this pixel (i.e., , whether the edge image is an edge point at position (1 indicates an edge)), then perform the following processing: find the image semantic segmentation block to which this edge point belongs, denoted as block region ; assign an initial semantic label to the edge point, where is the semantic category label corresponding to the region ; at the same time, set its initial semantic confidence , where represents the semantic credibility score of the region (which can be obtained based on the softmax maximum value output by the classification model). Construct a graph structure with all edge points as nodes, and there is a spatial or feature similarity adjacency relationship between the nodes. Perform a semantic confidence diffusion iteration on this graph to enhance the transitivity of semantic information between adjacent regions. For any edge node in the graph, update its semantic confidence value according to the following formula: , is the semantic confidence value updated after the diffusion iteration of node , is the set of neighbor nodes of node . is the node adjacent to node . , is the edge weight between node and node , representing the semantic or spatial similarity between them. is the node sequence item. is the sum of the weights of all connecting edges of node . is the current semantic confidence value of node . After one diffusion, if the semantic confidence of a certain edge point is less than the confidence threshold (such as 0.5), it is considered that the semantics of this point is uncertain and is specially marked or weighted down during processing. Enclose the semantic state of each edge point in a triple: , is the semantic category label, is the semantic confidence after diffusion, is a boolean value indicating whether this point is a semantically credible region (when ≥0.5, it is true; otherwise, it is false). Output the graph semantic region data .

[0046] Preferably, step S2 is specifically as follows: Step S21: Perform hierarchical semantic segmentation on the image fusion data to obtain semantic segmentation data; Specifically, construct a three-layer image pyramid from the image fusion data , with the scaling ratios of each layer being 1.0, 0.5, and 0.25 respectively; for each scale image, use a semantic segmentation model that supports multi-scale feature fusion, such as semantic segmentation models like SegFormer and BiSeNet that support multi-scale feature fusion; output three prediction maps , and the output of each layer includes the pixel semantic label C(x,y) and the confidence map S(x,y). The fused semantic label is: , , is the fused semantic label, is the category with the largest score selected from all categories, is the image layer sequence item, is the scale weight, for example , is the indicator function, which takes the value of 1 when the condition in the parentheses is true and 0 otherwise, is the semantic category label map of the th layer, is the Layer semantic confidence map, with a value range of [0, 1]. Output a semantic segmentation map with high confidence , corresponding to the semantic probability distribution map .

[0047] Step S22: Perform boundary consistency reconstruction based on the semantic segmentation data to obtain boundary reconstruction data; Specifically, use an edge detection algorithm (such as the Sobel operator, Laplacian operator, or Canny algorithm) to extract the boundary of the semantic segmentation map and generate a boundary response map . On this basis, for the semantic labels of adjacent pixels on both sides of the boundary , , , , etc., perform co-occurrence detection. If it is detected that the boundary area contains isolated semantic regions with an area smaller than the set threshold , then perform boundary reconstruction operations. The reconstruction strategies include using morphological closing operations to close the boundary, removing boundary breaks or burrs; combining the conditional random field (CRF) model to refine the label distribution at the boundary; introducing the graph cut (GraphCut) method to optimize the label of the local area, and output the reconstructed semantic boundary map .

[0048] Step S23: Perform entity segmentation on the image fusion data according to the boundary reconstruction data to obtain scene segmentation data; Specifically, use connected component analysis to obtain closed regions; each closed region is defined as a candidate entity block, and each region is denoted as , and record attributes such as the region boundary, main category, region area, and shape factor. Set several filtering conditions to screen the extracted candidate regions, including but not limited to setting a minimum area threshold , minimum aspect ratio and other parameters, to filter out abnormally small regions and non-physical structure regions; filter out noise regions with a shape approximately like a random patch. For each remaining valid entity region generate a mask map, and extract the corresponding original image block (in the fused image). Output a set containing all valid entity regions, denoted as , and the mask data of each region is denoted as .

[0049] Step S24: Perform region priority detection and sorting according to the scene segmentation data to obtain detection and sorting data; Specifically, the system uses the scene segmentation data obtained in the previous steps to process multiple semantic regions in the image Perform priority scoring and sorting to achieve efficient allocation of object detection resources. The sorting strategy aims to prioritize the processing of regions with high semantic confidence, large spatial extent, and proximity to the center of the image, thereby enhancing the overall detection efficiency and response speed. For example, calculate : , is the weight coefficient of the region area, is the region area, is the weight coefficient of the average semantic confidence, is the average semantic confidence, is the weight coefficient of the proximity of the region center to the image center (prioritize the center region), is the proximity of the region center to the image center (prioritize the center region). Calculate the scores of all ; Generate a detection candidate sequence in descending order of scores, and output the sorted list of detection regions .

[0050] Step S25: Perform cross-modal feature reinforcement object detection based on the detection sorting data and the image fusion data to obtain the scene object data.

[0051] Specifically, for each detection region, extract the feature data composed of multiple channels from the image fusion map. The channels may include, but are not limited to, the visible light channel (RGB), the edge map channel (such as the Canny edge response map), the depth map channel (such as the relative depth map generated by a depth estimation model or lidar), etc. The above multi-modal channels are uniformly stacked into region image blocks and constructed into a set of region feature tensors. Let the feature tensor of the th region be denoted as , and its size is , where and are the height and width of the th region respectively; is the number of fused channels; represents the multi-modal feature representation of the region. Use the channel attention mechanism for feature fusion. Preferably, the SE module or the CBAM module can be used to weight and adjust the importance of each channel respectively, thereby enhancing the semantic sensitive channels. The feature tensor after attention fusion is input into the region detector for object recognition. The detector can adopt advanced object detection networks, such as YOLOv8 or Deformable DETR, to perform the recognition tasks of object position and category. The detector outputs the following object information for the feature of each region, such as the position bounding box information, including the center coordinates , width and height ; Target class label , such as vehicle, pedestrian, etc.; Detection confidence score ∈ [0, 1], indicating the reliability of the detection result. After the detection is completed, the system further performs boundary fine-tuning to ensure the boundary consistency between the detection box and the semantic region. If there is a large deviation between the detection box and the previous semantic segmentation boundary, the following two methods can be used for correction. The boundary alignment strategy adjusts the contour of the target detection box to be more fitting with the edge contour of the semantic region; or, the contour reconstruction mechanism calls an instance segmentation model (such as the Instance Segmentation network) to refit the contour boundary to obtain a more refined description of the target shape. Output the target detection dataset , is the set of all detected targets, is the position information of the -th target, including , is the class label of the -th target, is the detection confidence score of the -th target, is the target ordinal term, is the total number of detected targets.

[0052] Preferably, the hierarchical semantic segmentation is specifically as follows: Perform multi-resolution partitioning on the image fusion data to obtain image partitioning data; Specifically, the system performs multi-level downsampling operations on the image fusion data to construct a three-layer image resolution image sequence , where is the original resolution image, corresponding to the scaling ratio = 1.0, that is, the original image size is retained; is the medium resolution image, obtained by downsampling the original image to 0.5 times; is the low resolution image, obtained by downsampling the original image to 0.25 times. Each of the above images is constructed through the image scaling operation function , where represents the scaling coefficient of the current resolution layer, represents the pyramid level number (numbered sequentially from high resolution to low resolution). To enhance the feature expression of each resolution image in edge perception, the system performs edge feature extraction operations on each layer of image $I^L$ and performs smoothing filtering on the edge results, thereby improving the adaptability of the multi-resolution image in the structure modeling task.

[0053] Perform hierarchical semantic perception on the image partition data to obtain hierarchical semantic perception data; Specifically, perform semantic feature extraction on the image partition data at different resolution levels. For each layer , use a lightweight semantic segmentation network (such as the MobileNet backbone network and the DeepLabHead decoder structure), , where is the semantic label prediction per pixel, is the semantic confidence map, is the -th layer semantic segmentation sub-network model, is the image of each layer. All prediction maps are upsampled to the original size; normalize the confidence map: , is the semantic confidence value of the -th layer at pixel position , is the number of layers. Output the hierarchical semantic label map set , the hierarchical semantic confidence map set .

[0054] Construct a hierarchical semantic path based on the hierarchical semantic perception data to obtain hierarchical semantic path data; Specifically, perform connected component labeling on each layer of semantic perception map (where \(L = 1, 2, 3\) respectively correspond to high, medium, and low resolutions) to extract the region set . Calculate the overlap degree between regions of different resolutions. If the intersection over union (IoU) of the medium / low resolution region and the high resolution region exceeds the set threshold , then establish a semantic path connection relationship to form a multi-scale corresponding path structure . Calculate the confidence for each path: , is the path confidence, is the average confidence of the semantic region at the first layer resolution, is the average confidence of the semantic region at the second layer resolution, is the average confidence of the semantic region at the third layer resolution. Filter the path structure according to this confidence, filter redundant or low-confidence paths, and output the hierarchical semantic path set.

[0055] Reconstruct the boundary of the image partition data based on the hierarchical semantic path data to obtain semantic segmentation data.

[0056] Specifically, for the generated hierarchical semantic path data, the system constructs a semantic image with clear boundaries and cross-scale semantic consistency by fusing the semantic consistency information between paths and performing weighted fusion on multi-scale semantic label maps. The system introduces a path voting fusion strategy. This strategy is based on the semantic label prediction maps and their corresponding path confidence maps at each scale of the image (such as the original image, 0.5x scaled image, 0.25x scaled image), and performs multi-scale fusion calculations on the labels at each pixel position Let the label map at the -th layer scale be , its corresponding semantic confidence map be , the fusion weight be , then the fused label of the pixel point is , and its calculation rule is as follows. For each candidate category , calculate the weighted sum of the consistency confidence of the category appearing at all scales, and take the category with the largest weighted sum as the label: , is an indicator function. If the label is equal to , the value is 1, otherwise it is 0. When detecting mutations such as jumps, tears, and breaks at the region boundary, the system will perform boundary closure compensation, including morphological dilation and closing operations; perform structural repair on the fusion result, fill internal segmentation holes, and remove isolated small speckle regions to avoid false target contamination. Output semantic segmentation data , a semantic map with hierarchical enhancement results.

[0057] Preferably, the cross-modal feature reinforcement object detection is specifically as follows: Calculate spatial attention data based on the detection sorting data and the image fusion data; Specifically, the system is based on the detection sorting data and the image fusion data , and performs a spatial attention modeling operation, aiming to improve the model's recognition ability of local significant target regions. In this process, for each detection sorting region , the system first crops the corresponding image sub-block from the image fusion data as the input for attention evaluation. Based on the intermediate feature output in the neural network structure (such as the second-to-last layer feature map of the convolutional network), perform the following operations to generate the spatial saliency map of the candidate region . Apply global average pooling (GAP) operation to to extract the channel average response; then pass through The convolution operation integrates the feature channels and normalizes the response using the Softmax function to generate the spatial attention map of the candidate region , indicating the importance of each pixel in the region for the semantic target. The system maps each local spatial attention map back to the full image space. The mapping method is based on its original detection region coordinates to perform a back-projection operation, that is, resamples to the corresponding region position and fills it to the original image size. The reflected mapping results of all attention maps are weighted and superimposed or maximum fused to obtain the full image spatial attention map , which represents the saliency score of each pixel position in the image with continuous values within the range of .

[0058] Annotate the detection sorting data according to the spatial attention data to obtain the detection weight data; Specifically, based on the spatial attention, weight the detection region sorting to introduce visual priority. For each detection box , calculate the mean value of the spatial attention within the region: , is the original attention mean weight of the region , is the th detection box region, is the pixel coordinate belonging to the region , is the spatial attention value corresponding to the pixel position . The weights of all detection boxes are normalized to , and each region is marked with a detection weight . Output the detection weight data set , which is bound to the sorted detection data.

[0059] Extract multi-modal features according to the graph semantic region data and the image fusion data to obtain multi-modal feature data; Specifically, stack the fused image data, edge map, sharpness map, and semantic region label map (using one-hot encoding) on the channel dimension to construct the input feature tensor. For each detection region , perform a region cropping operation on the multi-channel feature tensor to extract the multi-modal input segment containing the visual features, structural features, and semantic features of the region, denoted as , is the multi-modal input tensor of the th region, is the region tensor cropping operation, is image fusion data (in RGB or YUV format), is edge map data, representing the edge intensity of each pixel, is sharpness map data, representing the sharpness score of each pixel, is a semantic region label map, encoded in one-hot multi-channel format; each input is fed into a multi-modal convolutional network (which can use a multi-path branch structure); the output is a feature tensor , and a multi-modal feature dataset is output .

[0060] Perform region attention enhancement on the multi-modal feature data according to the detection weight data to obtain feature-enhanced data; Specifically, perform attention weighting on the multi-modal feature tensor of each detection region in the channel dimension. This feature tensor consists of image fusion channels (such as RGB), edge channels, sharpness map or depth map channels, etc., with a size of , where and are the height and width of the region respectively, is the number of channels. The system adopts a channel attention mechanism (CA), and preferably, an SE module or a CBAM module can be selected. This mechanism compresses the spatial response of each channel into a single value through global average pooling , and then passes through a fully connected layer and is activated by the Sigmoid function to obtain the importance weight of each channel. The specific operation is . Call the previously generated spatial attention map , and crop it according to the corresponding position of the detection box region to generate a region-level spatial attention map . This map is used to measure the saliency response of each pixel within the region, and the higher the value, the more important the position. Use the aforementioned spatial attention map to construct an intra-region attention map , perform pixel-by-pixel weighting, , is the normalized detection region weight, and the feature-enhanced dataset is output .

[0061] Use a preset object recognition model to identify the feature-enhanced data to obtain scene object data.

[0062] Specifically, based on the input data after feature enhancement processing, the system calls a preset high-performance target detection model to identify and reason about potential targets in the enhanced image region, and outputs a structured scene target recognition result. In terms of model selection, according to the computing power constraints and real-time requirements of the application deployment environment, the following types of mainstream target recognition models can be flexibly adopted, such as YOLOv8: suitable for edge device deployment, with a lightweight structure and high accuracy; Deformable DETR: supports a feature alignment mechanism and is suitable for processing targets with significant scale changes; RT-DETR: has fast inference ability and is especially suitable for end-side real-time detection scenarios. The above models all perform spatial structure modeling and semantic feature classification reasoning on images through deep neural networks. In the inference stage, the system takes each enhanced target region (denoted as ) and its location information as input data and passes it into the model. The model performs forward inference on this region and outputs the following structured target information: the position of the target bounding box, including the target center coordinates ( , ), width and height , which form the bounding box ; the target class label , the recognized semantic class; the classification confidence , the probabilistic score that the target belongs to the class, with a value range of . The system performs post-processing operations on the output results of all models, specifically including confidence filtering: eliminating targets with a confidence lower than the set threshold (e.g., 0.3); target overlap suppression. For the case where there is overlap between multiple predicted target bounding boxes, a non-maximum suppression (NMS) strategy is adopted to retain the one with the highest score to avoid duplicate detections.

[0063] Preferably, step S3 is specifically as follows: Step S31: Perform confidence-level target filtering on the scene target data to obtain confidence-level data, where the confidence-level data includes high-confidence target data, low-confidence target data, and structure-assisted candidate data. High-confidence target data are target regions with a confidence score significantly higher than the preset threshold during the detection process. Low-confidence target data are target regions with a confidence score significantly lower than the preset threshold during the detection process. Structure-assisted candidate data are target regions where the preset detection model does not clearly mark the target, but this region shows geometric or semantic features similar to the target in the structure perception layer (such as an edge map). Specifically, classify the scene target data by confidence and introduce structure-assisted compensation, input the scene target data set, , is the scene target data set, is the position of the bounding box of the rd target, is the class label of the target, and is the confidence score of the target. The preset thresholds are defined as the high confidence threshold: ; and the low confidence threshold: . The stratification rule for high confidence targets means that during the target detection process, the confidence score is higher than or equal to the high confidence threshold for the target region. Formally expressed as: , is a high confidence target, is a single detection target data item; for low confidence targets, it means that during the target detection process, the confidence score is lower than or equal to the low confidence threshold for the target region. Formally expressed as: ; for structure-assisted candidate data recognition, scan all unrecognized regions; if the edge density in the fused edge map > threshold , and it is a potential target class in the graph semantic region map, then include it in the structure-assisted candidate set: , is the structure-assisted candidate data, is to scan all regions in the image that are not labeled by the detection model, is the edge density, is the edge density threshold, is to query the semantic class of the region in the graph semantic region map, is the set of target classes, and output three types of target sets .

[0064] Step S32: Perform spatio-temporal density enhancement on the low confidence target data in the confidence-stratified data to obtain stratified enhanced data; Specifically, for each low confidence target in the current frame , retrieve whether the same type of target appears repeatedly in similar positions (overlapping regions ) in the previous N frames (such as 3 frames); at the same time, check whether there are similar candidates in the current image spatial neighborhood (within 5×5 pixels). Combining these two types of features, construct a spatio-temporal density scoring function , where, is the temporal continuity weight factor, is the target 's frequency score of appearance in consecutive frames, is the th low confidence target data, is the spatial redundancy weight factor, is the redundancy degree score of the target in the spatial neighborhood in the current frame, and the obtained , if (such as being 0.6), then incorporate it into the enhancement set: , is the target set after spatio-temporal enhancement. Return the low-confidence targets after spatio-temporal enhancement to replace the low-confidence target data in the confidence-level stratified data to obtain stratified enhancement data, is the high-confidence target set, is the th low-confidence target in the current frame, is the low-confidence target set, is the target enhancement priority, is the decision threshold, controlling the acceptance boundary of the enhanced target (such as 0.6), and output the stratified enhancement data .

[0065] Step S33: Calculate the edge region overlap degree for the stratified enhancement data based on the fused edge map data to obtain edge region overlap degree data; Specifically, extract all pixel points with non-zero edge responses from the fused edge map ; for each detection box , calculate its internal edge pixel density: , is the edge pixel density of the th target region, is the total number of pixels within the detection box , is that this point is within the target box , is the position 's edge response intensity value; compare with the edge density threshold to construct the edge matching confidence, , is the edge matching confidence of the th target, is the th detected target, is the minimum value function to ensure that the scoring result does not exceed 1.0, is the th target region's edge pixel density, is the edge density scoring threshold, which can be set according to experimental experience (for example, 0.3), and output the edge region overlap degree data , is the edge region overlap degree data set, is the th detected target, is its edge overlap confidence score.

[0066] Step S34: Calculate the matching degree according to the graph semantic region data and the edge region overlap degree data to obtain the target positioning data; Specifically, fuse the edge and semantic matching information to determine the exact correspondence of the target in the semantic graph. For each enhanced target , extract its main label at its position in the graph semantic region graph . If is consistent with the detection model prediction category , assign a semantic matching score of 1, otherwise 0 or a penalty value (such as 0.6), and calculate based on semantic consistency and edge matching degree: , , is the matching degree of the target , is the th enhanced target, is the weighting coefficient, used to adjust the relative importance of the structural edge and semantic consistency, and the value range is [0.5, 0.8], is the th structural overlap degree score between the target boundary and the edge graph, range [0,1], is the th semantic label matching score, 1 means consistent, 0.6 means inconsistent but acceptable, , if , then confirm it as a valid target, output its spatial positioning center and bounding box coordinates, and output the target positioning data set: , is the output target positioning data set, is the abscissa of the target center (positioning result), is the ordinate of the target center (positioning result), is the target bounding box (which can include the center coordinates and dimensions), is the target semantic category label.

[0067] Step S35: Extract the motion vector features according to the target positioning data to obtain the motion vector feature data, and fit the motion function according to the motion vector feature data to obtain the motion trajectory prediction data.

[0068] Specifically, extract the historical position change trend and fit the motion path to predict the future position; for the target , collect its positioning points in the recent N frames (such as 5 frames) and the positioning point at the next moment, and calculate the velocity vector between every two frames: , if high-order modeling is enabled, the acceleration term can be added: , is the acceleration, is the velocity vector at the moment, is the velocity vector at the moment, and trajectory fitting is performed using polynomial regression or a Kalman filter model: Or use an LSTM / GRU neural network to predict the next moment Predict the target positions in the next K frames and output the predicted trajectory sequence: , is the predicted trajectory sequence of the th target, is the abscissa of the target at the moment, is the ordinate of the target at the moment, is the predicted horizontal position of the target in the k-th frame in the future, is the predicted vertical position of the target in the k-th frame in the future, and output the motion trajectory prediction data set .

[0069] Preferably, the spatio-temporal density enhancement is specifically: Perform spatial clustering on the low-confidence target data in the confidence-stratified data to obtain duplicate target data; Specifically, the input is the set composed of all low-confidence targets in the current frame , is the set of low-confidence targets in the current frame, is the target 's central abscissa, is the target 's central ordinate, is the target 's bounding box width, is the target 's bounding box height, is the target 's class label, is the target 's detection confidence; construct a spatial distance metric and a class consistency constraint: , , where is the spatial similarity value of the target and , is the th low-confidence target, is the th low-confidence target, is the class consistency indicator function, is the target The class label, is the target The class label, is the natural exponential term, is the target The center point coordinates of, is the target The center point coordinates of, is, Control the clustering range (e.g., 20 pixels); use DBSCAN or MeanShift clustering to identify duplicate target clusters and set the minimum number of clusters , the maximum neighborhood distance px, and output the duplicate target dataset.

[0070] Perform cross-frame trajectory tracking based on the duplicate target data to obtain cross-frame trajectory tracking data; Specifically, determine whether these duplicate targets have consistent motion trajectories between consecutive frames. Input the duplicate targets after clustering in the current frame , the low-confidence targets in the historical frame , …, where is the set of low-confidence targets in the th frame, is the set of low-confidence targets in the th frame. For each , find matching targets in the previous N frames , is the th candidate historical target in the th frame, satisfying , , is the intersection over union of two bounding boxes, is the th duplicate target in the current frame, is the th duplicate target in the current frame, is The semantic class label corresponding to the target, is The semantic class label corresponding to the target. Use the SORT or Deep SORT lightweight tracker to assign a trajectory ID to each target in the cluster, , is the cross-frame motion trajectory of the th target, is the abscissa of the center position of the target in the th frame, is the ordinate of the center position of the target in the th frame, is the target in the The abscissa of the central position in the frame, is the ordinate of the central position of the target in the frame, is the ordinate of the central position of the target in the frame, is the ordinate of the central position of the target in the frame, and the cross-frame trajectory tracking data set is output , is the cross-frame trajectory tracking data set, including all target trajectory sequences, is the th cross-frame motion trajectory of the target.

[0071] Calculate the spatio-temporal density of the low-confidence target data based on the cross-frame trajectory tracking data and the repeated target data to obtain the target spatio-temporal density data; Specifically, integrate spatial repetition and temporal continuity into a spatio-temporal density index to screen whether the weak target is credible. The spatial density calculation is as follows: for each repeated target cluster , calculate its local density: , is the spatial density of the th target cluster, is the th target cluster, is pi, with a value of approximately 3.1416, is the radius of the circumscribed circle of the clustering cluster. The time stability calculation is as follows: if the target appears times in the frame, then the time density is as , is the time density score of the target , is the target in the past frames and is detected times. , is the spatio-temporal credibility score of the target , is the th low-confidence target, is the fusion weight coefficient of the spatial density, is the standardized spatial density, is the fusion weight coefficient of the time density, is the standardized time density, =0.6, =0.4, and output the spatio-temporal density score table for each low-confidence target: , we1, is the th low-confidence target, and is the spatio-temporal density fusion score for this target.

[0072] Filter the low-confidence target data in the confidence stratified data according to the target spatio-temporal density data to obtain stratified enhanced data.

[0073] Specifically, set the confidence threshold for spatio-temporal density screening, denoted as , which is used as the basis for judging whether a low-confidence target has credible reinforcement. Preferably, this threshold can be set to , that is, it is required that the target shows a spatio-temporal density above medium in both the spatial aggregation and temporal consistency dimensions. For all low-confidence targets , if , then it is regarded as an enhanced target and added to the new detection set: , is the set of target detection results after screening and strengthening, is the set of targets whose original confidence is higher than the threshold and is directly regarded as credible targets, is the union operation of the sets, is the th low-confidence target, is the set of all low-confidence targets (such as confidence ≤ 0.5), is the th spatio-temporal density score of the target, and the numerical range is [0,1], is the screening threshold, set to 0.6 in the middle, and the stratified enhanced data set is output, which contains the strengthened low-confidence targets.

[0074] Preferably, step S4 is specifically: Step S41: Generate a trajectory intersection area according to the motion trajectory prediction data to obtain trajectory intersection area data; Specifically, find the spatial area where the motion trajectories of any two targets overlap in the future time period, that is, the potential collision area. Input the trajectory prediction data of each target: , is the motion trajectory prediction data, is the target at the future time step in the two-dimensional horizontal coordinate position, is the target at the future time step in the two-dimensional vertical coordinate position, is the prediction start time, is the number of trajectory prediction steps, is the initial time. The overlapping judgment rule is for any two target trajectories , , perform moment-by-moment intersection determination: If there exists a certain time step , satisfying: , is the target at the future time step in the two-dimensional horizontal coordinate position, is the target at the future time step in the two-dimensional horizontal coordinate position, is the target at the future time step in the two-dimensional horizontal coordinate position, is the target at the future time step in the two-dimensional horizontal coordinate position, where is the collision threshold (such as the target radius and + safety redundancy, for example, 0.5 meters). Expand the intersection position into an elliptical / circular safety area , and record the time window , each set of trajectory intersections constitutes a potential collision area object: , is the trajectory intersection area object, is the spatial buffer area, is the time parameter, is the start time of the intersection duration interval, is the end time of the intersection duration interval, output the trajectory intersection area dataset .

[0075] Step S42: Construct a collision propagation graph based on the trajectory intersection area data to obtain the collision propagation graph data; Specifically, establish a graph structure to represent the possible collision propagation chain relationship between multiple targets. Each moving target is a node in the graph , if the target and have a trajectory intersection area , then add an edge . Calculate the edge weight according to the trajectory overlap degree, duration, and velocity direction difference: , is the risk propagation intensity from the target pointing to , is the weight of the angle conflict degree between the motion directions of the two targets, is the proportion of the trajectory intersection duration of the two targets in the predicted time window, is the weight of the angle conflict degree between the moving directions of two targets. is the angle conflict degree between the moving directions of two targets, which is the normalized score of the included angle (the larger the included angle, the higher the score, and the score is significantly weighted above 90°). is the weight of the area ratio between the intersection area and the combined bounding box of two targets. is the area ratio between the intersection area and the combined bounding box of two targets. The collision propagation graph structure is to construct a directed graph , is a node, is a connection relationship, is an edge weight, and the edge weight represents the collision influence intensity. Output the collision propagation graph data .

[0076] Step S43: Perform a risk propagation and diffusion simulation based on the collision propagation graph data to obtain the collision risk data.

[0077] Specifically, simulate the diffusion trend of the collision risk in the propagation graph, obtain the dynamic risk levels of each target, and mark the nodes directly participating in the trajectory intersection as the initial risk sources, and set their risk values to 1.0; the initial values of other nodes are 0. The propagation model uses a PageRank-style propagation model or a Bayesian risk diffusion model: , is the risk value of the target node at the th round of time, is the th moving target node, is the th moving target node, is the risk propagation control factor, which determines the proportion of the initial value and the neighbor influence, and is set to 0.05. is the initial risk value (0 or 1) of node , is the set of neighbor nodes of node , is the adjacent node propagating to 's edge weight (risk weight), is the node sequence term, is the node 's sum of the edge weights of all outgoing edges, is the node 's risk score in the th round of iteration, is the node 's risk value at the th round of node risk. is the propagation control factor, and the propagation process is iterated until the risk value converges or reaches a fixed number of rounds (such as 10 rounds). According to the risk value Set the level: is high risk, 0.4∼0.7: medium risk, is low risk. Output the collision risk level and prediction time window of each target, and the output structure is , is the collision risk data set, recording the risk scores and levels of each target, is the th moving target node, is the risk level classification result (high, medium, low) of the target node, is the node 's risk score after the diffusion termination.

[0078] Preferably, the present application also provides an image fusion-driven anti-collision recognition system for performing the image fusion-driven anti-collision recognition method as described above. The image fusion-driven anti-collision recognition system includes: A semantic fusion perception module for acquiring multi-source image data and performing semantic-guided image fusion according to the multi-source image data to obtain image fusion data; A target semantic parsing module for performing scene segmentation according to the image fusion data to obtain scene segmentation data, and performing target detection on the scene segmentation data to obtain scene target data; A hierarchical positioning and trajectory prediction module for performing confidence-level hierarchical spatial positioning according to the scene target data to obtain target positioning data, and performing motion trajectory prediction according to the target positioning data to obtain motion trajectory prediction data; A collision risk inference module for performing collision area propagation risk assessment according to the motion trajectory prediction data to obtain collision risk data.

[0079] Therefore, from any point of view, the embodiments should be regarded as exemplary and non-limiting. The scope of the present invention is defined by the appended application documents rather than the above description. Therefore, it is intended to cover all changes falling within the meaning and scope of the equivalent elements of the application documents within the present invention.

[0080] The above are only specific embodiments of the present invention, enabling those skilled in the art to understand or implement the present invention. Various modifications to these embodiments will be obvious to those skilled in the art. The general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to these embodiments shown herein, but rather to the broadest scope consistent with the principles and novel features invented herein.

Claims

1. An image fusion-driven anti-collision recognition method, characterized in that, It includes the following steps: Step S1: Obtain multi-source image data, and perform semantic-guided image fusion based on the multi-source image data to obtain image fusion data; Step S2: Perform scene segmentation based on the image fusion data to obtain scene segmentation data, and perform object detection on the scene segmentation data to obtain scene object data; Step S3: Perform confidence-level hierarchical spatial positioning based on the scene object data to obtain target positioning data, and perform motion trajectory prediction based on the target positioning data to obtain motion trajectory prediction data; Step S4: Perform collision area propagation risk assessment based on the motion trajectory prediction data to obtain collision risk data.

2. The method according to claim 1, wherein Specifically, step S1 is as follows: Obtain multi-source image data; Extract edge map data based on the multi-source image data; Perform multi-scale convolutional blurring on the edge map data to obtain convolutional blurring data, and perform sharpening processing on the convolutional blurring data to obtain fused edge map data; Perform semantic region mapping on the edge map data to obtain graph semantic region data; Calculate the edge density of the fused edge map data according to the graph semantic region data to obtain structure weight data; Perform high-frequency response calculation based on the multi-source image data to obtain image sharpness data; Perform weighted fusion on the multi-source image data according to the structure weight data and the image sharpness data to obtain image fusion data.

3. The method according to claim 2, wherein Among them, the semantic region mapping is specifically as follows: Extract local color features and local texture features from the edge map data to obtain graph color feature data and graph texture feature data respectively; Use a preset lightweight image semantic recognition model to perform low-resolution semantic inference on the graph color feature data and the graph texture feature data to obtain semantic category label data and semantic confidence data; Construct an adjacent relationship graph according to the semantic category label data and the semantic confidence data to obtain adjacent relationship graph data; Perform category credibility mapping on the edge map data according to the adjacent relationship graph data to obtain graph semantic region data.

4. The method according to claim 2, wherein Specifically, step S2 is as follows: Perform hierarchical semantic segmentation based on the image fusion data to obtain semantic segmentation data; Perform boundary consistency reconstruction based on the semantic segmentation data to obtain boundary reconstruction data; Perform entity segmentation on the image fusion data according to the boundary reconstruction data to obtain scene segmentation data; Perform region priority detection and sorting on the scene segmentation data to obtain detection sorting data; Perform cross-modal feature reinforcement object detection according to the detection sorting data and the image fusion data to obtain scene object data.

5. The method according to claim 4, wherein Among them, the hierarchical semantic segmentation is specifically as follows: Perform multi-resolution division on the image fusion data to obtain image division data; Perform hierarchical semantic perception on the image division data to obtain hierarchical semantic perception data; Construct hierarchical semantic paths according to the hierarchical semantic perception data to obtain hierarchical semantic path data; Perform boundary reconstruction on the image division data according to the hierarchical semantic path data to obtain semantic segmentation data.

6. The method according to claim 4, characterized in that, Among them, the cross-modal feature reinforcement object detection is specifically as follows: Perform spatial attention calculation according to the detection sorting data and the image fusion data to obtain spatial attention data; Annotate the detection sorting data according to the spatial attention data to obtain detection weight data; Perform multi-modal feature extraction based on the graph semantic region data and the image fusion data to obtain multi-modal feature data; Enhance the regional attention of the multi-modal feature data according to the detection weight data to obtain feature-enhanced data; Use a preset target recognition model to recognize the feature-enhanced data to obtain scene target data.

7. The method according to claim 2, wherein Step S3 is specifically as follows: Perform confidence-level stratified target filtering based on the scene target data to obtain confidence-level stratified data, where the confidence-level stratified data includes high-confidence target data, low-confidence target data, and structure-assisted candidate data. The high-confidence target data is the target area where the confidence score is significantly higher than the preset threshold during the detection process, the low-confidence target data is the target area where the confidence score is significantly lower than the preset threshold during the detection process, and the structure-assisted candidate data is the target area where the preset detection model does not clearly mark the target, but this area shows geometric or semantic features similar to the target in the edge map data; Perform spatio-temporal density enhancement on the low-confidence target data in the confidence-level stratified data to obtain stratified enhanced data; Calculate the edge region overlap degree of the stratified enhanced data according to the fused edge map data to obtain edge region overlap degree data; Calculate the matching degree according to the graph semantic region data and the edge region overlap degree data to obtain target localization data; Extract the motion vector features according to the target localization data to obtain motion vector feature data, and perform motion function fitting according to the motion vector feature data to obtain motion trajectory prediction data.

8. The method according to claim 7, wherein Among them, the spatio-temporal density enhancement is specifically as follows: Perform spatial clustering on the low-confidence target data in the confidence-level stratified data to obtain duplicate target data; Perform cross-frame trajectory tracking according to the duplicate target data to obtain cross-frame trajectory tracking data; Calculate the spatio-temporal density of the low-confidence target data according to the cross-frame trajectory tracking data and the duplicate target data to obtain target spatio-temporal density data; Filter the low-confidence target data in the confidence-level stratified data according to the target spatio-temporal density data to obtain stratified enhanced data.

9. The method according to claim 1, wherein Step S4 is specifically as follows: Generate a trajectory intersection area according to the motion trajectory prediction data to obtain trajectory intersection area data; Construct a collision propagation graph according to the trajectory intersection area data to obtain collision propagation graph data; Perform a collision risk propagation and diffusion simulation according to the collision propagation graph data to obtain collision risk data.

10. An image fusion-driven anti-collision recognition system, characterized in that, For implementing the image fusion-driven anti-collision recognition method as described in claim 1, the image fusion-driven anti-collision recognition system includes: A semantic fusion perception module, configured to obtain multi-source image data and perform semantic-guided image fusion according to the multi-source image data to obtain image fusion data; A target semantic parsing module, configured to perform scene segmentation according to the image fusion data to obtain scene segmentation data, and perform target detection on the scene segmentation data to obtain scene target data; A hierarchical localization and trajectory prediction module, configured to perform confidence-level stratified spatial localization according to the scene target data to obtain target localization data, and perform motion trajectory prediction according to the target localization data to obtain motion trajectory prediction data; A collision risk inference module, which is used to evaluate the risk of collision area propagation based on the predicted data of the movement trajectory and obtain the collision risk data.

Images