Anti-collision monitoring system based on depth estimation and instance segmentation fusion three-dimensional model

Abstract

The application discloses a collision prevention monitoring system based on depth estimation and instance segmentation fusion three-dimensional model, and particularly relates to the field of intelligent driving security, and is used for solving the target recognition and collision prevention problem in the environment. The efficient feature extraction of global semantics and local texture is realized through a multi-branch network structure, so that the accuracy and comprehensiveness of target segmentation are ensured. In combination with a cross-frame identity association and conflict detection mechanism, the consistency of the target identity is effectively maintained, and the mismatch problem caused by occlusion or similar appearance is reduced. A multi-scale space-time occlusion mode analysis and boundary motion consistency verification are adopted, the occlusion complexity and segmentation robustness are comprehensively evaluated by using a fusion algorithm, a high-risk occlusion area is identified and secondary difference correction is performed, and the spatial positioning information of the target is significantly optimized. Finally, the optimized segmentation and depth data are integrated into a global coordinate system, a high-precision and coherent three-dimensional semantic scene is constructed, and stable and high-quality input data are provided for collision detection and trajectory prediction.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of intelligent driving security, more particularly, the present application relates to a collision prevention monitoring system based on depth estimation and instance segmentation fusion three-dimensional model. BACKGROUND

[0002] In complex environments, dense target scenes widely exist in the fields of autonomous driving, robot group operation and intelligent security, etc. These scenes are usually accompanied by high occlusion, irregular distribution and complexity of appearance features of targets. In this environment, three-dimensional modeling technology based on depth estimation and instance segmentation is widely used to achieve accurate identification and real-time modeling of static and dynamic objects in the scene. This technology provides important support for dynamic environment perception and collision risk prediction, but its effectiveness depends on accurate target segmentation, stable identity labeling and high-quality depth estimation results.

[0003] Existing instance segmentation algorithms are prone to problems such as overlapping of segmentation boundaries and jumping of target identities in situations of severe occlusion or highly similar target appearances. Since instance segmentation mainly relies on feature extraction and discrimination of single-frame images, when local areas are occluded or targets are densely distributed, the algorithm's ability to identify target boundaries decreases significantly, leading to identity mismatch or segmentation errors between targets. These mismatching information is amplified in subsequent three-dimensional modeling and collision prediction, which may cause model reconstruction distortion or motion trajectory prediction deviation, ultimately affecting the accuracy and real-time performance of the system's collision prevention warning. This problem is particularly prominent in multi-target dynamic scenes, and an improved technical path is needed to achieve more robust instance segmentation and target identification. In order to solve the above problems, the present application provides a technical solution. SUMMARY

[0004] In order to overcome the above-mentioned defects of the prior art, the embodiments of the present application provide a collision prevention monitoring system based on depth estimation and instance segmentation fusion three-dimensional model, which realizes efficient feature extraction of global semantics and local textures through a multi-branch network structure, ensuring the accuracy and comprehensiveness of target segmentation; combines cross-frame identity association and conflict detection mechanism to effectively maintain the consistency of target identity and reduce the mismatching problems caused by occlusion or similar appearance; uses multi-scale spatio-temporal occlusion pattern analysis and boundary motion consistency verification, uses fusion algorithm to comprehensively evaluate the occlusion complexity and segmentation robustness, identifies high-risk occlusion areas and performs secondary difference correction, significantly optimizes the spatial positioning information of the target; finally, the optimized segmentation and depth data are integrated into the global coordinate system to construct a high-precision and continuous three-dimensional semantic scene, providing stable and high-quality input data for collision detection and trajectory prediction, to solve the problems raised in the background art.

[0005] To achieve the above purpose, the present application provides the following technical scheme:

[0006] The anti-collision monitoring system based on depth estimation and instance segmentation fusion three-dimensional model comprises a feature extraction module, an identity matching module, an occlusion correction module and a three-dimensional integration module.

[0007] The feature extraction module: based on a multi-branch network structure, performs feature extraction of global semantics and local texture on an input image, generates target segmentation mask and category information, and stores the results in a data structure The data structure is transmitted to the identity matching module.

[0008] The identity matching module: uses the segmentation mask and category information in the data structure , combines target tracking and multi-frame recognition strategy of adjacent frames to perform cross-frame identity matching and conflict detection, and uses re-identification mechanism to correct boundary jump or identity mismatch caused by occlusion, and the updated segmentation result and identity are stored in the data structure The data structure is transmitted to the occlusion correction module.

[0009] The occlusion correction module: fuses the corrected segmentation result in the data structure and depth estimation data, uses multi-scale space-time occlusion mode analysis and boundary motion consistency verification, and through a nonlinear fusion algorithm, comprehensively evaluates the occlusion complexity and segmentation robustness, performs secondary difference correction on high-risk occlusion areas, and outputs optimized spatial position information, which is stored in the data structure The data structure is transmitted to the three-dimensional integration module.

[0010] The three-dimensional integration module: integrates the three-dimensional coordinates and segmentation labels of all targets into a global coordinate system, constructs a complete three-dimensional semantic scene, and stores the integrated three-dimensional model in the data structure , which provides stable and high-quality input data for collision detection and trajectory prediction.

[0011] In a preferred embodiment, the feature extraction module has the following specific processing logic:

[0012] In specific processing, first, a global backbone network and a local branch network are established for the input image , let represent the global semantic feature mapping, and let represent the local texture feature mapping; then without changing the original resolution of the global semantic feature mapping and the local texture feature mapping, a position homomorphism transformation operator performs multi-scale local calibration in the feature coordinate system, and calculates the following fusion feature mapping: ; wherein representing characteristic coordinates, performing equidistant projection and differential translation in local texture feature mapping according to the learnable pose bias matrix, representing the fused multi-dimensional features, i.e., the fusion feature map; then using a multi-level classification decoder on the fusion feature map to obtain the target segmentation mask and category information , wherein the multi-level classification decoder filters the global features in the semantic dimension and distinguishes edge differences in the texture dimension, and finally outputs the candidate segmentation region; finally, the target segmentation mask and category information and their coordinate mapping relationship are stored in the data structure in the form of .

[0013] In a preferred embodiment, the identity matching module specifically processes the following logic:

[0014] For each target instance stored in the data structure , extract its spatio-temporal feature vector , wherein represents the unique identifier of the target; the feature extraction is realized by a multi-modal feature encoder;

[0015] Using the dynamic time warping algorithm, compare the target feature vector in the previous frame with the target feature vector in the current frame to calculate the feature sequence similarity score .

[0016] For target pairs with a feature sequence similarity score below the matching threshold, use the re-identification module to re-determine using the high-dimensional fusion vector of the global semantic feature and the local texture feature; the re-identification process optimizes the discriminability of the target feature through a generative adversarial network; the re-identification recalculates the feature sequence similarity score according to the optimized high-dimensional fusion vector, and re-matches the target identity according to the new score.

[0017] In a preferred embodiment, the identity matching module specifically processes the following logic:

[0018] For cases where there are still multiple pairs of matching scores above the matching threshold after the re-identification mechanism, a multi-target matching algorithm based on graph optimization is used to construct a target matching graph and extract the unique identity correspondence through the maximum matching subgraph; the specific steps include:

[0019] Building graph nodes: each node represents a target instance;

[0020] Building graph edges: connecting nodes according to the feature sequence similarity score and .

[0021] Apply the maximum weight matching algorithm to extract the optimal target identity correspondence. The formula is as follows: ;in, is the final matching set; the target information corrected by identity matching and conflict detection is updated to the data structure ,in: is the corrected segmentation mask; For updated category information; is the new coordinate mapping relationship; Target identity.

[0022] In a preferred embodiment, the specific processing logic of the occlusion correction module is as follows:

[0023] S3.1, first, the segmentation mask Corresponding depth data Perform multi-scale spatiotemporal analysis; capture the occlusion dynamics of target objects at different time frames and spatial scales through spatiotemporal occlusion density evaluation; specifically, for each target , in the time interval In each frame The degree of occlusion To evaluate, the calculation formula is as follows: ;in, is the spatial occlusion density weight function, which reflects the influence of different spatial positions on the occlusion degree; through multi-scale analysis, the occlusion pattern sequence of the target at each time frame and spatial scale is obtained. ;

[0024] S3.2, based on the occlusion pattern sequence, using the holographic cointegration transformation operator The spatiotemporal occlusion is dynamically converted into a high-dimensional vector representation; the sequence is convolved and pooled through a multi-scale temporal convolutional network to capture the consistency and complexity of occlusion at different time scales and generate an occlusion holographic cointegration vector. ; The specific calculation process is: ;in, and Represent multi-scale convolution and multi-scale pooling operations respectively.

[0025] In a preferred embodiment, the specific processing logic of the occlusion correction module further includes the following:

[0026] S3.3, to evaluate the robustness of segmentation boundaries in dynamic environments, define the dynamic boundary continuous tensor , by segmenting the boundaries The curvature change and motion continuity of the target are used to extract multi-level spatial features; specifically, for each target , in each frame , calculate its boundary curvature rate of change and boundary motion consistency score , as follows: ; wherein, denotes the target boundary curvature at frame , and is the boundary motion consistency score, is a small constant to prevent division by zero; the curvature rate of change and the motion consistency score are integrated by a multi-level fusion operator to generate a dynamic boundary continuity tensor : .

[0027] In a preferred embodiment, the occlusion correction module specific processing logic further includes the following:

[0028] S3.4, the occlusion holographic co-integration vector is calculated with the dynamic boundary continuity tensor to generate a cross-domain harmonic kernel matrix ; an adaptive threshold mechanism and a nonlinear mapping function are used in the fusion process to dynamically adjust the weight distribution under different occlusion complexity and boundary robustness scenarios, as follows: ; wherein, and are learnable adjustment parameters, is an offset constant; based on the classification evaluation of the cross-domain harmonic kernel matrix and the risk threshold, the target is classified as a high-risk occlusion region: if the cross-domain harmonic kernel matrix is greater than the risk threshold, the occlusion region target is evaluated as high risk;

[0029] S3.5, for the target of the occlusion region evaluated as high risk, secondary difference correction is performed to optimize its spatial position information; the segmentation mask and its mapping position in the depth map are micro-refined, as follows: ; wherein, is the refinement coefficient, denotes the gradient operation on the cross-domain harmonic kernel matrix, which is used to guide the adjustment direction and amplitude of the segmentation mask; through the secondary difference correction, the optimized segmentation mask and the corresponding spatial position information are generated and stored in the data structure .

[0030] In a preferred embodiment, the three-dimensional integration module specific processing logic is as follows:

[0031] S4.1, first, for each target in the data structure ​ Spatial location information Global coordinate system conversion: set the global coordinate system and use the known sensor position parameters to convert the spatial position of each target into the global coordinate system : ;in, Indicates the target The three-dimensional position vector in the global coordinate system;

[0032] S4.2, using depth data and optimized segmentation mask , generating each target 3D point cloud ; The specific operations are as follows: ;in, is the image pixel coordinate, is the corresponding depth value; then, the 3D point clouds of all targets are aligned through global coordinate transformation and fused into the global point cloud middle: ;in, Represents spatial transformation operations on a point cloud.

[0033] The specific processing logic of the 3D integration module also includes the following:

[0034] S4.3, the data structure The category information in the global point cloud is integrated with the semantic labels of the points; the specific operation is to , according to their target The category information is annotated to generate a 3D point cloud with semantic labels : ;in, represents the semantic category label; then, a spatial consistency optimization algorithm is applied Optimize the 3D point cloud with semantic labels to eliminate noise points, fill data holes, and ensure the consistency and coherence of semantic labels in 3D space: ; Generate 3D semantic point cloud through spatial consistency optimization algorithm ;

[0035] S4.4, the optimized 3D semantic point cloud and the corresponding semantic labels and spatial coordinate information are stored in a structured global data structure ,in Represents the set of semantic labels for all target instances.

[0036] The technical effects and advantages of the anti-collision monitoring system based on depth estimation and instance segmentation fusion 3D model of the present invention are as follows:

[0037] The application realizes efficient feature extraction of global semantics and local texture through a multi-branch network structure, ensures the accuracy and comprehensiveness of target segmentation, effectively maintains the consistency of target identity by combining cross-frame identity association and conflict detection mechanism, reduces the mismatch problem caused by occlusion or similar appearance, uses multi-scale space-time occlusion mode analysis and boundary motion consistency verification, comprehensively evaluates the occlusion complexity and segmentation robustness by using a nonlinear fusion algorithm, identifies high-risk occlusion areas and performs secondary difference correction, significantly optimizes the spatial positioning information of the target, and finally integrates the optimized segmentation and depth data into a global coordinate system to construct a high-precision and continuous three-dimensional semantic scene, providing stable and high-quality input data for collision detection and trajectory prediction. The spatial perception ability and early warning accuracy of the system in a complex dynamic environment are improved, the real-time performance and reliability of the anti-collision monitoring system are enhanced, and the system is widely applicable to the fields of autonomous driving, robot navigation and intelligent security, and significantly improves the adaptability and practicality of the system. BRIEF DESCRIPTION OF DRAWINGS

[0038] Figure 1 FIG. 1 is a structural schematic diagram of an anti-collision monitoring system based on depth estimation and instance segmentation fusion three-dimensional model according to the present application. DETAILED DESCRIPTION

[0039] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0040] Embodiment 1: Figure 1 The anti-collision monitoring system based on depth estimation and instance segmentation fusion three-dimensional model according to the present application comprises a feature extraction module, an identity matching module, an occlusion correction module and a three-dimensional integration module.

[0041] The feature extraction module: based on a multi-branch network structure, the feature extraction module performs feature extraction of global semantics and local texture on input images, generates target segmentation masks and class information, and stores the results in a data structure The data structure is transmitted to the identity matching module.

[0042] The identity matching module: using the segmentation masks and class information in the data structure , combining target tracking and multi-frame recognition strategies of adjacent frames, performing cross-frame identity matching and conflict detection, and using re-identification mechanism to correct boundary jumps or identity mismatches caused by occlusion, the updated segmentation results and identity labels are stored in a data structure The data structure Passed to the occlusion correction module;

[0043] Occlusion Correction Module: Fusion Data Structure The corrected segmentation results and depth estimation data in the dataset are analyzed by multi-scale spatiotemporal occlusion pattern analysis and boundary motion consistency verification. The occlusion complexity and segmentation robustness are comprehensively evaluated through nonlinear fusion algorithm. Secondary differential correction is performed on high-risk occlusion areas. The optimized spatial position information is output and stored in the data structure. , the data structure Transfer to the 3D integration module;

[0044] 3D integration module: integrates the 3D coordinates and segmentation annotations of all targets into the global coordinate system to build a complete 3D semantic scene. The integrated 3D model is stored in the data structure , providing stable and high-quality input data for collision detection and trajectory prediction.

[0045] In dynamic environments with dense scenes, the distribution of target objects is often highly complex and diverse, including issues such as occlusion, overlap, and blurred feature boundaries. In this context, to provide a reliable segmentation foundation for subsequent cross-frame recognition and correction, it is essential to fully exploit the image's global semantics and local texture features, integrating multi-scale information to ensure the accuracy and completeness of the initial object mask and category annotation. This step is the initial step in the segmentation and recognition task within the entire system, and its results directly determine the stability and accuracy of subsequent processing modules.

[0046] The specific processing logic of the feature extraction module is as follows:

[0047] In the specific processing, first for the input image Building a global backbone network With a local branch network ,make Represents the global semantic feature map, let Represents the local texture feature map; then, without changing the original resolution of the global semantic feature map and the local texture feature map, the position homomorphic transformation operator Perform multi-scale local calibration in the feature coordinate system and calculate the following fused feature map: ;in represents the feature coordinates, Performs isometric projection and differential translation on the local texture feature map according to a learnable pose bias matrix, Represents the fused multidimensional features, that is, the fused feature map; then a multi-level classification decoder is used on the fused feature map Get the target segmentation mask and category information wherein the multi-level classification decoder filters the global features in the semantic dimension and distinguishes edge differences in the texture dimension, and finally outputs candidate segmentation regions; finally, the target segmentation mask and category information and its coordinate mapping relationship are stored in the form of to the data structure , for calling by the cross-frame target tracking and conflict detection unit in the identity matching module; after completion, the preliminary object segmentation that may exist in the dense environment has been identified with the mask and category information, providing identification reference for subsequent occlusion correction and three-dimensional modeling, wherein the preliminary target mask and category prediction represent the preliminary target distribution in the scene in the spatial dimension and the semantic dimension, respectively.

[0048] The position homomorphism transformation operator is an advanced mathematical tool for aligning global semantic features and local texture features in a multi-branch network. Its core technical features include multi-scale geometric correction, pose adjustment, and equidistant projection. The operator first receives local texture feature maps from a local branch network, performs geometric correction and positional offset on the local features through a learnable rotation matrix and translation vector, ensuring alignment with global semantic features in the same coordinate system. Next, the position homomorphism transformation operator applies convolution kernels at multiple scales to refine the adjusted local feature maps, enhancing their robustness and expressiveness and reducing feature misalignment caused by scale differences. Finally, the geometrically corrected and pose-adjusted local feature maps are fused with global semantic feature maps in the feature coordinate system, generating a unified fused feature map, thereby achieving consistency and complementarity between global and local features in the spatial dimension.

[0049] Through joint extraction and correction of global semantic and local texture features, a target segmentation mask and category information with complete preliminary segmentation information are generated, covering the preliminary representation of target distribution in the spatial dimension and the semantic dimension. These data are stored in a structured form, serving as important inputs for subsequent cross-frame target tracking and occlusion correction. The results of this module ensure the accuracy and robustness of preliminary object segmentation in complex background conditions.

[0050] In a multi-target dense and dynamic environment, the identity consistency of target objects and the accuracy of segmentation boundaries are crucial for ensuring the accuracy of subsequent three-dimensional modeling and collision prediction. Existing instance segmentation algorithms have difficulty effectively maintaining the continuity of target identities when processing single frames, especially in cases of target occlusion and high appearance similarity, making it more prone to mismatch and boundary jumping. Therefore, cross-frame identity association and conflict detection must be combined with multi-frame recognition strategies to ensure the continuity of target identities and the stability of segmentation boundaries. This process not only requires accurate target tracking but also utilizes re-identification mechanisms to dynamically correct identity mismatches caused by occlusion, ensuring the accuracy of subsequent three-dimensional modeling and the reliability of collision prediction.

[0051] The specific processing logic of the identity matching module is as follows:

[0052] The preliminary object segmentation mask and category information generated by the feature extraction module are thoroughly analyzed to extract the spatiotemporal features of each object instance. These features, including the object's shape, color, texture, and spatial position, are comprehensively encoded using a multimodal feature encoder. The encoded feature vector fully reflects the multidimensional properties of the object, ensuring high discrimination and uniqueness of different objects in the feature space.

[0053] Stored in data structures For each target instance in , extract its spatiotemporal feature vector ,in Represents the unique identifier of the target. Feature extraction is done through a multimodal feature encoder The implementation formula is as follows: ;in, Target The segmentation mask, For its category information, The spatiotemporal feature vector comprehensively considers the target's shape, color, texture, spatial position and other multi-dimensional information to ensure high discrimination and uniqueness of the features.

[0054] Using the dynamic time warping algorithm, compare the previous frame The target feature vector in With the current frame The target feature vector in , calculate the feature sequence similarity score The Dynamic Time Warping algorithm effectively processes the target's motion trajectory and feature changes across different time frames. By setting a similarity threshold, it selects highly similar target pairs and completes cross-frame identity matching. This process ensures target identity consistency across consecutive frames and reduces mismatches caused by motion or occlusion.

[0055] For target pairs whose feature sequence similarity scores are lower than the matching threshold, the re-identification module is used , using the high-dimensional fusion vector of global semantic features and local texture features Re-identification is performed. The re-identification process optimizes the discriminative ability of the target features by generating an adversarial network. The formula is as follows: ; The re-identification module recalculates the feature sequence similarity score based on the optimized high-dimensional fusion vector and re-matches the target identity based on the new score.

[0056] If there are still multiple matching scores higher than the matching threshold after the re-identification mechanism, a multi-target matching algorithm based on graph optimization is used to construct a target matching graph and extract unique identity correspondences through the maximum matching subgraph. The specific steps include:

[0057] Construct graph nodes: each node represents a target instance.

[0058] Constructing graph edges: scoring based on feature sequence similarity and Connect the nodes.

[0059] in, Represents the similarity score after the re-ID mechanism optimization. This score is recalculated based on the high-dimensional fused feature vector optimized by the generative adversarial network. It aims to improve the accuracy of target matching and correct mismatches caused by occlusion or high appearance similarity in the initial matching.

[0060] Apply the maximum weight matching algorithm to extract the optimal target identity correspondence. The formula is as follows: ;in, The final matching set.

[0061] Update the target information corrected by identity matching and conflict detection to the data structure ,in:

[0062] is the corrected segmentation mask.

[0063] This is the updated category information.

[0064] is the new coordinate mapping relationship.

[0065] Target identity.

[0066] The data structure It will serve as the input of the subsequent occlusion correction module to ensure the consistency of target identity and the accuracy of segmentation boundaries, providing a reliable data foundation for high-precision 3D modeling and collision prediction.

[0067] Through cross-frame identity association and conflict detection, the consistency of target identity and dynamic correction of segmentation boundaries are achieved. The dynamic time warping algorithm is used for preliminary matching, and the re-identification module and graph optimization algorithm are used to solve the problems of low similarity and multiple matching conflicts, ensuring the accurate correspondence of target identity and the stability of segmentation boundaries. Finally, the updated data structure The target information including identity calibration and segmentation correction is provided, and a solid foundation is provided for subsequent depth estimation and spatial position information optimization, and the recognition accuracy and robustness of the anti-collision monitoring system in a complex dynamic scene are significantly improved.

[0068] In the identity matching module of the application, only the target identity is introduced in the identity matching module, and the identity is not included in the data structure of other modules. Specifically, the identity matching module is the core link of cross-frame identity association and conflict detection, and is responsible for ensuring the consistency of the target identity and the accuracy of the segmentation boundary. Therefore, it is necessary to introduce the target identity to track and correct the identity of each target in the continuous frames. However, in the subsequent occlusion correction module and three-dimensional integration module, although the target identity is not explicitly mentioned in the data structure, the identity consistency is actually maintained indirectly through the segmentation mask , category information and spatial position information . Specifically, the optimized segmentation mask and spatial position information have been adjusted and corrected based on the identity matching result of the identity matching module, so that the identity information can be implicitly transmitted and applied in the three-dimensional integration process. Therefore, this design ensures the continuity and consistency of the identity in the overall process of the system, and there is no need to repeatedly store the identity in the data structure of each step, thereby optimizing the simplicity and processing efficiency of the data structure.

[0069] After completing the cross-frame identity association and conflict detection of the identity matching module, the data structure already contains the corrected target segmentation mask, category information, coordinate mapping relationship and target identity. However, in a dense and dynamic scene, complex occlusion between target objects and small changes in segmentation boundaries may still cause inaccurate spatial position information. In order to further improve the accuracy of three-dimensional model construction, it is necessary to combine depth estimation data to analyze and correct the segmentation result more carefully. Through multi-scale spatiotemporal occlusion pattern analysis and boundary motion consistency verification, a nonlinear fusion algorithm is used to comprehensively evaluate the occlusion complexity and segmentation robustness, and secondary difference correction is performed on high-risk occlusion areas, thereby outputting more accurate spatial position information and ensuring the high reliability of three-dimensional model construction and collision prediction.

[0070] The specific processing logic of the occlusion correction module is as follows:

[0071] S3.1, first, the segmentation mask output by the identity matching module is subjected to multi-scale spatiotemporal analysis with the corresponding depth data . By introducing spatiotemporal occlusion density evaluation, the occlusion dynamics of target objects are captured at different time frames and spatial scales. Specifically, for each target , the time interval In each frame The degree of occlusion To evaluate, the calculation formula is as follows: ;in, is the spatial occlusion density weight function, which reflects the influence of different spatial positions on the occlusion degree. Through multi-scale analysis, the occlusion pattern sequence of the target at each time frame and spatial scale is obtained. , providing a basis for subsequent occlusion dynamic capture and quantification.

[0072] Among them, depth data refers to the depth information about each pixel or spatial position in the scene obtained through depth estimation technology. Specifically, depth data can be obtained using a variety of sensor technologies, such as depth cameras, lidar or stereo vision systems. These sensors can capture the position and distance of objects in the scene relative to the camera or sensor, thereby generating a two-dimensional image or three-dimensional point cloud data containing depth values. In the specific implementation process, depth data is usually stored in the form of a two-dimensional depth map or a three-dimensional point cloud, and each data point contains the depth value of the corresponding pixel or spatial position. The depth value corresponding to each pixel in the depth map represents the distance from the point to the sensor. In the three-dimensional point cloud, each point represents its specific coordinate position in three-dimensional space. Through the precise acquisition and processing of depth data, the system can achieve high-precision spatial perception and modeling of complex dynamic scenes, providing reliable spatial information support for the anti-collision monitoring system.

[0073] S3.2, based on the occlusion pattern sequence, using the holographic cointegration transformation operator The spatiotemporal occlusion is dynamically converted into a high-dimensional vector representation. This transformation performs convolution and pooling operations on the sequence through a multi-scale temporal convolutional network, capturing the consistency and complexity of occlusion at different time scales and generating an occlusion holographic cointegration vector. The specific calculation process is: ;in, and They represent multi-scale convolution and multi-scale pooling operations, respectively, to ensure that the features of occlusion dynamics at different time scales are effectively captured and integrated.

[0074] S3.3, to evaluate the robustness of segmentation boundaries in dynamic environments, define the dynamic boundary continuous tensor , by segmenting the boundaries The curvature change and motion continuity of the target are used to extract multi-level spatial features. Specifically, for each target , in each frame , calculate the rate of change of its boundary curvature and boundary motion consistency score , the formula is as follows: ;in, Indicates the target At the boundary of a frame the curvature of the boundary, the boundary motion consistency score, is a small constant to prevent division by zero.

[0075] Where the segmentation boundary is a set of pixel points in the target segmentation mask, which are located at the boundary of the target region and the background or other target regions, defining the spatial range of the target instance. The segmentation boundary is generated by extracting the transition points between pixels with values of 1 (representing the target region) and 0 (representing the background or other regions) in the segmentation mask. These boundaries not only describe the shape of the target, but also provide the local geometric characteristics of the target in the scene. In dynamic scenes, the changes in segmentation boundaries between consecutive frames are key elements in measuring target motion consistency and boundary robustness. For example, by analyzing the curvature of the segmentation boundary, abnormal changes caused by occlusion or motion can be identified, and the basis for further segmentation optimization can be provided.

[0076] The curvature change rate and the motion consistency score are integrated through a multi-level fusion operator to generate a dynamic boundary continuity tensor : The tensor comprehensively reflects the continuity and stability of the target segmentation boundary in a multi-frame dynamic environment.

[0077] Where the multi-level fusion operator is a mathematical tool that realizes multi-dimensional feature comprehensive analysis through hierarchical decomposition, cross-layer correlation and dynamic weighted integration. The specific operation includes dividing the input features (such as boundary curvature change rate and motion consistency score) into multiple levels (such as low-level local features and high-level global features) according to their attributes. After extracting the feature representation of each layer, the correlation between different levels is analyzed using inner product, convolution or adaptive weighting methods. The low-level and high-level features are dynamically weighted and fused using a nonlinear mapping function. Finally, a unified representation is generated that combines local fine information and global trends, providing basic support for segmentation boundary robustness evaluation or occlusion dynamic capture in complex scenes; the formula is as follows: Where and represent low-level and high-level features, is a cross-layer correlation analysis function, is a nonlinear activation function, is the fusion result.

[0078] S3.4, the occlusion holographic co-integration vector and the dynamic boundary continuity tensor are comprehensively calculated to generate a cross-domain harmonic kernel matrix In this fusion process, an adaptive threshold mechanism and a nonlinear mapping function are used to dynamically adjust the weight distribution in different occlusion complexity and boundary robustness scenarios. The formula is as follows: ; wherein, and are learnable adjustment parameters, is an offset constant, ensuring the adaptability and stability of the fused cross-domain harmonic kernel matrix under different scenarios. Based on the classification evaluation of the cross-domain harmonic kernel matrix and the risk threshold, the target is classified as a high-risk or low-risk occlusion region: if the cross-domain harmonic kernel matrix is greater than the risk threshold, the occlusion region target is evaluated as high-risk.

[0079] S3.5, for the target of the occlusion region evaluated as high-risk , perform secondary differential correction to optimize its spatial position information. Micro-refine the segmentation mask and its mapping position in the depth map, as follows: ; wherein, is a refinement coefficient, represents a gradient operation on the cross-domain harmonic kernel matrix, used to guide the adjustment direction and amplitude of the segmentation mask. Through secondary differential correction, the optimized segmentation mask and the corresponding spatial position information are generated and stored in the data structure , providing high-precision input data for the three-dimensional modeling and collision prediction of the subsequent three-dimensional integration module.

[0080] Through multi-scale spatiotemporal occlusion pattern quantification and boundary motion consistency verification, the occlusion correction module effectively combines the corrected instance segmentation results and depth estimation data, and uses holographic cointegration vectors and boundary continuous tensors for complexity and robustness evaluation. Through a nonlinear fusion algorithm, a cross-domain harmonic kernel matrix is generated, dynamically adjusting the weight distribution under the scenarios of occlusion complexity and segmentation robustness, achieving accurate discrimination and secondary differential correction of high-risk occlusion regions. Finally, the optimized spatial position information is systematically stored, providing a highly accurate and reliable data foundation for three-dimensional model construction and collision prediction in the three-dimensional integration module. The spatial perception ability and early warning accuracy of the anti-collision monitoring system in complex dynamic scenarios are significantly improved, ensuring the efficiency and accuracy of three-dimensional model construction and collision prediction.

[0081] After completing the secondary differential correction of high-risk occlusion regions in the occlusion correction module, the data structure contains the optimized segmentation mask , category information , and accurate spatial position information . However, these optimized segmentation and position information alone are not enough to build a complete and consistent three-dimensional semantic scene. In order to achieve high-precision three-dimensional model construction and ensure the reliability of collision detection and trajectory prediction, it is necessary to integrate these optimized target information into the global coordinate system to form a unified and coherent three-dimensional semantic scene. Through precise coordinate transformation, data alignment and fusion processing, the accurate positioning and semantic consistency of each target in the three-dimensional space are ensured, thereby providing stable and high-quality input data for subsequent collision detection and trajectory prediction.

[0082] The specific processing logic of the 3D integration module is as follows:

[0083] S4.1, first, the data structure Each target in Spatial location information Global coordinate system conversion. Set the global coordinate system and use the known sensor position parameters (including translation vector and the rotation matrix ), converting the spatial position of each target into global coordinates : ;in, Indicates the target The 3D position vector in the global coordinate system. This transformation ensures that all target instances are positioned in a unified spatial reference frame, laying the foundation for subsequent 3D modeling.

[0084] S4.2, using depth data and optimized segmentation mask , generating each target 3D point cloud The specific operations are as follows: ;in, is the image pixel coordinate, is the corresponding depth value.

[0085] Then, the 3D point clouds of all targets are aligned through global coordinate transformation and fused into the global point cloud. middle: ;in, Represents the spatial transformation operation of the point cloud. The fused global point cloud covers the accurate position and shape of all target instances in the global coordinate system, providing a detailed data foundation for the construction of 3D semantic scenes.

[0086] S4.3, the data structure The category information in the global point cloud is integrated with the semantic labels of the points. The specific operation is to , according to their target The category information is annotated to generate a 3D point cloud with semantic labels : ; wherein, represents the semantic class label.

[0087] Subsequently, a spatial consistency optimization algorithm is applied Further optimization is performed on the three-dimensional point cloud with semantic labels to eliminate noise points, fill data gaps, and ensure the consistency and coherence of semantic labels in three-dimensional space: ; Through the spatial consistency optimization algorithm, high-quality three-dimensional semantic point cloud is generated , providing accurate and stable spatial semantic information for collision detection and trajectory prediction.

[0088] The spatial consistency optimization algorithm is an advanced processing technology for improving the coherence and accuracy of three-dimensional semantic point cloud data in spatial dimensions. The algorithm ensures the consistency and stability of the position and semantic labels of each target instance in the global coordinate system through a multi-step process. First, the algorithm applies a graph-based filtering technique to identify and remove noise points in the point cloud, ensuring data cleanliness. Subsequently, using spatial neighborhood clustering methods, adjacent and semantically identical point clouds are clustered into continuous entities to eliminate segmentation breaks caused by sensor errors or dynamic environmental changes. In addition, the algorithm introduces geometric constraints, through curvature analysis and normal vector consistency checks, to further correct minor deviations in target boundary, ensuring the smoothness and coherence of the boundary. To handle moving targets in dynamic scenes, the algorithm combines time series information and applies motion compensation techniques to correct spatial displacement caused by target movement, maintaining the dynamic consistency of the three-dimensional model. Finally, the spatial consistency optimization algorithm integrates semantic labels and spatial position information through multi-level feature fusion and optimization strategies, generating high-quality, coherent, and semantically consistent three-dimensional semantic point clouds.

[0089] S4.4, the optimized three-dimensional semantic point cloud and corresponding semantic labels and spatial coordinate information are stored in a structured global data structure , wherein represents the semantic label set of all target instances. This data structure ensures the ordered storage of all three-dimensional spatial information and semantic labels, facilitating efficient calling and processing by subsequent collision detection and trajectory prediction modules.

[0090] Through accurate coordinate system conversion, multi-target three-dimensional point cloud generation and fusion, semantic label integration and spatial consistency optimization, the three-dimensional integration module successfully integrates the optimized target segmentation and depth estimation data into the global coordinate system, and constructs a complete and high-precision three-dimensional semantic scene. The structured storage of three-dimensional semantic point cloud and its semantic label set provides stable and detailed input data for subsequent collision detection and trajectory prediction. Through strict data processing and fusion logic, the module ensures the accuracy and consistency of the three-dimensional model construction, significantly improves the spatial perception ability and early warning precision of the anti-collision monitoring system in complex dynamic scenes, and ensures the efficiency and reliability of the system in practical application.

[0091] The above formulas are dimensionless numerical calculations, and the formulas are obtained by software simulation of a large amount of data to obtain a formula of the most recent real situation, and the preset parameters in the formula are set by a person skilled in the art according to the actual situation.

[0092] The above only describes some exemplary embodiments of the present application by way of illustration, and it is self-evident that those skilled in the art can modify the described embodiments in various ways without departing from the spirit and scope of the present application. Therefore, the above figures and descriptions are illustrative in nature and should not be construed as limiting the scope of protection of the claims of the present application.

[0093] It should be noted that in this paper, if there are relationship terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between the entities or operations. Moreover, the terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. Without more limitations, the element defined by the sentence "includes a" does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0094] The above is only a specific embodiment of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical range disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A collision avoidance monitoring system based on the fusion of depth estimation and instance segmentation with a three-dimensional model, characterized in that, Comprise: The feature extraction module, identity matching module, occlusion correction module and three-dimensional integration module; The feature extraction module: based on the multi-branch network structure, the global semantic and local texture feature extraction of the input image is carried out, the target segmentation mask and the category information are generated, and the results are stored in the data structure The data structure is transmitted to the identity matching module; Identity matching module: utilize the segmentation mask and category information in the data structure , combine adjacent frame target tracking and multi-frame identification strategy to perform cross-frame identity matching and conflict detection, and utilize re-identification mechanism to correct boundary jump or identity mismatch caused by occlusion, and store the updated segmentation result and identity label in the data structure , and pass the data structure to the occlusion correction module; The occlusion correction module: fusion data structure The occlusion correction module: fusion data structure The occlusion correction module: fusion data structure The occlusion correction module: fusion data structure Three-dimensional integration module: integrate three-dimensional coordinates of all targets and segmentation annotations into a global coordinate system, construct a complete three-dimensional semantic scene, and store the integrated three-dimensional model in a data structure To provide stable and high-quality input data for collision detection and trajectory prediction. 2.The anti-collision monitoring system based on fusion of depth estimation and instance segmentation with three-dimensional model according to claim 1, characterized in that, The feature extraction module specific processing logic as follows: In the specific processing, firstly, for the input image establish a global backbone network and local branch network , let represent the global semantic feature mapping, and let represent the local texture feature mapping; Then, without changing the original resolution of global semantic feature map and local texture feature map, the position homomorphism transformation operator Multi-scale local calibration is performed in the feature coordinate system, and the following fusion feature map is calculated: ; wherein represents the feature coordinates, According to the learnable pose bias matrix, equidistant projection and differential translation are performed in the local texture feature map, represents the fused multi-dimensional feature, i.e., the fusion feature map; then, the multi-level classification decoder is used on the fusion feature map to obtain the target segmentation mask and the category information , wherein the multi-level classification decoder filters the global feature in the semantic dimension and identifies the edge difference in the texture dimension, and finally outputs the candidate segmentation region; finally, the target segmentation mask, the category information, and the coordinate mapping relationship thereof are stored in the data structure in the form of . 3.The anti-collision monitoring system based on fusion of depth estimation and instance segmentation with three-dimensional model according to claim 2, characterized in that, Identity matching module specific processing logic as follows: Stored in data structures For each target instance in , extract its spatiotemporal feature vector ,in Indicates the unique identifier of the target; feature extraction is achieved through a multimodal feature encoder; The dynamic time warping algorithm is used to compare the target feature vector in the previous frame with the target feature vector in the current frame to calculate a feature sequence similarity score ; For the feature sequence similarity score below the matching threshold of the target pair, using the re-identification module, using the high-dimensional fusion vector of global semantic features and local texture features to re-determine; The re-identification process through the generation of adversarial network optimization target feature discriminant ability; re-identification according to the optimized high-dimensional fusion vector re-computed feature sequence similarity score, and according to the new score re-matching target identity. 4.The anti-collision monitoring system based on fusion of depth estimation and instance segmentation with three-dimensional model according to claim 3, characterized in that, Identity matching module specific processing logic also includes the following content: For the case that still exists after the re-identification mechanism of multiple pairs of matching score above the matching threshold, using the multi-target matching algorithm based on graph optimization, construct target matching graph, and through the maximum matching subgraph extraction unique identity corresponding relationship; The specific steps include: Build graph nodes: each node represents a target instance; Construct graph edges: based on feature sequence similarity scores With Connection nodes; The maximum weight matching algorithm is applied to extract the optimal target identity corresponding relationship, and the formula is as follows: ; wherein, is the final matching set; the target information after the identity matching and conflict detection correction is updated to the data structure , wherein: is the corrected segmentation mask; is the updated category information; is the new coordinate mapping relationship; is the target identity. 5.The anti-collision monitoring system based on fusion of depth estimation and instance segmentation with three-dimensional model according to claim 4, characterized in that, The occlusion correction module specific processing logic as follows: S3.1, first, the segmentation mask Corresponding depth data Perform multi-scale spatiotemporal analysis; capture the occlusion dynamics of target objects at different time frames and spatial scales through spatiotemporal occlusion density evaluation; specifically, for each target , in the time interval In each frame The degree of occlusion To evaluate, the calculation formula is as follows: ;in, is the spatial occlusion density weight function, which reflects the influence of different spatial positions on the occlusion degree; through multi-scale analysis, the occlusion pattern sequence of the target at each time frame and spatial scale is obtained. ; S3.2, based on the sequence of occlusion patterns, adopt holographic co-integration transform operator The spatio-temporal occlusion dynamics is dynamically converted into a high-dimensional vector representation; through a multi-scale time convolution network, the sequence is convolved and pooled to capture the consistency and complexity of the occlusion at different time scales, generating an occlusion holographic co-integration vector ; The specific calculation process is as follows: ; Wherein, and respectively represent multi-scale convolution and multi-scale pooling operation. 6.The anti-collision monitoring system based on fusion of depth estimation and instance segmentation with three-dimensional model according to claim 5, characterized in that, The occlusion correction module specific processing logic also includes the following content: S3.3, to evaluate the robustness of segmentation boundaries in dynamic environments, define the dynamic boundary continuous tensor , by segmenting the boundaries The curvature change and motion continuity of the target are used to extract multi-level spatial features; specifically, for each target , in each frame , calculate the rate of change of its boundary curvature and boundary motion consistency score , the formula is as follows: ;in, Indicates the target In frame The boundary curvature of Score the boundary motion consistency, To prevent the small constant from dividing by zero; the curvature change rate and motion consistency score are combined through a multi-level fusion operator Integration, generating dynamic boundary continuous tensors : . 7.The anti-collision monitoring system based on fusion of depth estimation and instance segmentation with three-dimensional model according to claim 6, characterized in that, The occlusion correction module specific processing logic also includes the following content: S3.4, the occlusion holographic co-integration vector is calculated with the dynamic boundary continuous tensor to generate the cross-domain harmonic kernel matrix ; an adaptive threshold mechanism and a nonlinear mapping function are used in the fusion process to dynamically adjust the weight distribution under different occlusion complexity and boundary robustness scenarios, as follows: ; wherein, and are learnable adjustment parameters, is an offset constant; based on the classification evaluation of the cross-domain harmonic kernel matrix and the risk threshold, the target is classified as a high-risk occlusion region: if the cross-domain harmonic kernel matrix is greater than the risk threshold, the occlusion region target is evaluated as high risk; S3.5, for the target of the occluded region assessed as high risk , performing a second-order difference correction to optimize its spatial position information; micro-refining the segmentation mask and its mapped position in the depth map, as follows: ; wherein, is the refinement coefficient, denotes a gradient operation on the cross-domain harmonic kernel matrix, for guiding the adjustment direction and amplitude of the segmentation mask; through the second-order difference correction, an optimized segmentation mask and the corresponding spatial position information are generated and stored to the data structure . 8.The anti-collision monitoring system based on fusion of depth estimation and instance segmentation with three-dimensional model according to claim 7, characterized in that, Three-dimensional integration module specific processing logic as follows: S4.1, first, the spatial position information of each target in the data structure global coordinate system conversion;​​ A global coordinate system is set, and the spatial position of each target is converted into the global coordinate by using the known sensor position parameters : ; wherein, represents a target a three-dimensional position vector in the global coordinate system; S4.2, using depth data and optimized segmentation mask , generating each target 3D point cloud ; The specific operations are as follows: ;in, is the image pixel coordinate, is the corresponding depth value; then, the 3D point clouds of all targets are aligned through global coordinate transformation and fused into the global point cloud middle: ;in, Represents spatial transformation operations on a point cloud. 9.The anti-collision monitoring system based on fusion of depth estimation and instance segmentation with three-dimensional model according to claim 8, characterized in that, Three-dimensional integration module specific processing logic also includes the following content: S4.3, integrating the category information in the data structure with the points in the global point cloud with semantic label; specifically, for each point , according to the category information of the target to which it belongs, it is labeled to generate a three-dimensional point cloud with semantic labels : ; wherein represents a semantic category label; subsequently, a spatial consistency optimization algorithm is applied to optimize the three-dimensional point cloud with semantic labels, eliminate noise points, fill data gaps, and ensure the consistency and continuity of semantic labels in three-dimensional space: ; through the spatial consistency optimization algorithm, a three-dimensional semantic point cloud is generated; S4.4, store the optimized three-dimensional semantic point cloud and the corresponding semantic labels and spatial coordinate information into a global data structure wherein denotes a set of semantic labels of all target instances.

Images