Automatic driving complex scene sensing method based on multi-modal fusion

Abstract

The invention relates to the technical field of multi-modal fusion, in particular to an automatic driving complex scene sensing method based on multi-modal fusion, which comprises the following steps of: acquiring an image bounding box and a point cloud cluster center position, identifying and pairing space overlapping targets, analyzing direction change to extract a track difference frame segment, and extracting a track difference frame segment; the edge trend and point cluster arrangement are linked to divide modal sources, contact areas and modal attribution are concluded according to side lines, and a perceptual path sorting framework is constructed according to an arrangement fusion sequence. According to the method, a cross-modal target association relationship is constructed through spatial overlapping pairing of an intra-frame image and a point cloud modal target, track difference fragments are extracted through a continuous frame direction change sequence, behavior change positions are distinguished, and modal sources are divided through conjoint analysis of an edge trend and a point cluster arrangement trend; and establishing an association matching relationship between a modal and a scene component through extraction of a contact area, arranging a fusion sequence through contact expression of the modal in a difference frame segment, and constructing a perceptual path sorting framework with distinguishing capability.

Description

Technical Field

[0001] This invention relates to the field of multimodal fusion technology, and in particular to a method for perceiving complex scenes in autonomous driving based on multimodal fusion. Background Technology

[0002] The field of multimodal fusion technology involves the joint modeling and information complementarity of data from different types of sensors, such as image information, radar point clouds, lidar scanning, sound signals, and location information. The core aspects of this technology include spatiotemporal alignment of heterogeneous data sources, feature extraction, feature fusion, and semantic understanding. Its purpose is to enhance the cognitive ability of complex environments by fusing data from multiple modalities. It is widely used in fields such as autonomous driving, robot perception, human-computer interaction, and intelligent security. In autonomous driving, multimodal fusion can effectively improve the stability and robustness of perception under different weather, lighting, and obstacle occlusion conditions.

[0003] Among them, the traditional method for perceiving complex scenes in autonomous driving based on multimodal fusion refers to using image and point cloud data from sensors such as cameras and LiDAR in the autonomous driving environment, using region proposal networks for target detection, and achieving data fusion through joint convolution operations of feature maps. Common methods include constructing three-dimensional sparse tensors based on voxelization to uniformly process image and point cloud information, or using graph neural networks based on attention mechanisms to perform feature matching and supplementation of different modal information, aiming to achieve the perception and recognition of elements such as pedestrians, vehicles, road structures and traffic signs in complex traffic scenes.

[0004] Existing technologies focus on feature space fusion modeling, but do not explicitly pair heterogeneous modal targets within a frame in terms of spatial location. This can easily lead to ambiguity in the matching relationship between modalities. In the behavior analysis stage, the role of the order of directional changes in consecutive frames is ignored, making it difficult to identify trajectory change conflicts caused by modal differences. Edge trends and point cluster arrangement trends are not used as the basis for determining modal origins, affecting the accurate screening of modal attribution. The lack of a clear contact correspondence between scene structure and perception results leads to a lack of effective links between modal information and environmental components. The order of modal processing during fusion is unclear, affecting the responsiveness and coordination of data integration in dynamic scenes. Summary of the Invention

[0005] The purpose of this invention is to address the shortcomings of existing technologies by proposing a multimodal fusion-based method for perceiving complex scenes in autonomous driving.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: a multimodal fusion-based method for perceiving complex scenes in autonomous driving, comprising the following steps:

[0007] S1: Obtain the positions of the center of the image modal bounding box and the center of the point cloud modal reflection point cluster, identify spatially overlapping targets within the same frame, pair the image and point cloud targets, extract the intra-frame correlation, and obtain the corresponding set of image and point cloud targets;

[0008] S2: Based on the paired targets in the target correspondence set of the image point cloud, analyze the order of directional changes in the continuous frames of the modality, identify the alternating frame segments in the trajectory movement, divide them into trajectory difference regions, and obtain the conflict segment identification book;

[0009] S3: Based on the frame segments in the conflict segment identifier book, analyze the image edge trend and point cloud arrangement changes, identify output frame groups with consistent edge directions and no jump in point cluster arrangement, divide the modal source according to the frame segment position, and obtain modal output groups;

[0010] S4: Based on the target position in the modal output group, read the position of the four sides of the image border, compare the spatial position with the road markings and obstacle edges, summarize the contact area and modal source, and obtain the modal and scene structure mapping set;

[0011] S5: Based on the contact situation between the modal and the scene structure mapping set, analyze the continuous contact performance of the modal with road components in the differentiated frame segments, arrange the perception fusion order according to the spatial distribution relationship, and obtain the fusion perception path sorting framework.

[0012] As a further embodiment of the present invention, the image point cloud target correspondence set includes the center position of the bounding box, the center position of the reflection point cluster, the positional overlap combination, and the intra-frame pairing relationship; the conflict segment identifier book includes the direction change order, the directionally inconsistent segments, and the trajectory behavior markers; the modal output grouping includes the edge extension direction, the point cluster arrangement amplitude, the modal identifier, and the frame segment position range; the modal and scene structure mapping set includes the spatial contact area, the structural association relationship, the target edge position, and the road component edge; and the fusion perception path sorting framework includes the contact frequency, the fusion priority order, the number of modal occurrences, and the number of boundary markers.

[0013] As a further aspect of the present invention, the specific steps of S1 are as follows:

[0014] S101: Obtain the center position of the vehicle bounding box in each frame of the image modality and the center position of the reflection point cluster in each frame of the point cloud modality. Pair the image target position with the point cloud target position frame by frame according to the frame number, identify the target point pairs that appear in the same spatial region, and obtain the modal spatial association dataset.

[0015] S102: Based on the modal space association dataset, analyze the regional coverage relationship between image targets and point cloud targets, determine whether the line connecting the center points falls within the common area of ​​the image border and the edge of the point cloud distribution, filter target combinations with continuous contact features, and obtain modal pairing coverage relationship groups.

[0016] S103: Based on the modal pairing coverage relationship group, arrange the pairing relationship between image targets and point cloud targets in frame order, extract the target combination data with spatial contact features in each frame, and obtain the image point cloud target correspondence set.

[0017] As a further aspect of the present invention, the specific steps of S2 are as follows:

[0018] S201: Based on the paired target entities in the target correspondence set of the image point cloud, obtain the target position sequence of the image mode and the point cloud mode in consecutive frames, extract the motion direction vector of each frame, and arrange the direction change order of the same target in the differentiated mode according to the frame order to obtain the direction order sequence group.

[0019] S202: Based on the direction sequence group, calculate the proportion of direction switching in consecutive frames in the image modality and point cloud modality, divide the rhythm difference between frame segments by comparing the rhythm of direction change, and obtain the motion direction rhythm offset segment.

[0020] S203: Based on the motion direction rhythm offset segment, screen the target combination where the image modality and point cloud modality have inconsistent motion directions in the same frame segment, and map the segments with continuously changing rhythm differences to the target entity position sequence to obtain the conflict segment identification book.

[0021] As a further aspect of the present invention, the formula for calculating the proportion of direction switching in consecutive frames in the image modality and point cloud modality is as follows:

[0022]

[0023] Among them, P d The x represents the proportion of orientation switching in consecutive frames within the image modality and point cloud modality, where m represents the number of consecutive frames involved in the computation, and x represents the proportion of orientation switching within the image modality and point cloud modality. j y represents the motion direction encoding value of the image modality in the j-th frame. j w represents the motion direction encoding value of the point cloud mode in frame j. j δ represents the motion confidence weight of the j-th frame, δ represents the smoothing constant when the direction value sum is zero, and λ represents the square root term adjustment value.

[0024] As a further aspect of the present invention, the specific steps of S3 are as follows:

[0025] S301: Based on the frame segments recorded in the conflict segment identifier book, extract the bounding box edge coordinates of adjacent frames in the image modality, determine the extension direction of the edge line segments between consecutive frames according to the frame order, filter out the frame segments whose direction keeps changing continuously, and obtain the edge extension direction sequence.

[0026] S302: Based on the edge extension direction sequence, call the reflection point cluster position data of the corresponding frame segment in the point cloud modality, calculate the angle offset value of the point cluster arrangement direction between consecutive frames, and identify the frame segments where the angle offset variation range tends to be consistent within the continuous segment to obtain the arrangement trend consistent segment.

[0027] S303: Based on the matching segments of the arrangement trend, extract the modal source identifier corresponding to each frame in the segment, classify and merge the modal identifiers according to the frame segment range correspondence, and obtain modal output groups.

[0028] As a further aspect of the present invention, the formula for calculating the angular offset value of the arrangement direction of the continuous inter-frame point clusters is as follows:

[0029]

[0030] in, S represents the angular offset between the cluster arrangement direction of frame v and the previous frame (v-1). v S represents the sum of the principal direction sequence projection values ​​of the reflection point cluster in frame v. v-1 α represents the sum of the principal direction sequence projection values ​​of the reflection point cluster in frame v-1, α represents the proportionality coefficient of the previous frame's order value in the current angle offset calculation, β represents the positive offset adjustment constant used to stabilize the denominator result, and γ represents the adjustment factor for the angular deflection dispersion of the reflection point cluster. This represents the deflection magnitude of the m-th cluster of reflection points in the v-th frame. N represents the average deflection amplitude of the reflection point cluster in frame v. v This represents the number of point clusters participating in the statistics in the v-th frame, where v represents the current lidar point cloud sequence frame number, and m represents the m-th reflection point cluster participating in the calculation in the current frame.

[0031] As a further aspect of the present invention, the specific steps of S4 are as follows:

[0032] S401: Based on the target position of each frame sequence in the modal output group, read the linear markers and obstacle boundary layer data in the road image in the corresponding frame, extract the top, bottom, left and right edge position coordinates of the image target border, list the spatial position information of the border edge in each frame, and obtain the border edge position dataset.

[0033] S402: Based on the bounding box edge position dataset, obtain the coordinate range of lane markings and obstacle boundaries, read the positioning information of bounding box edge coordinates and component edge coordinates in each frame, calculate the coverage ratio of the intersection area of ​​bounding box edge and component edge in the image plane, and obtain the edge contact feature combination.

[0034] S403: Based on the edge contact feature combination, mark the frame order of the intersection of the border edge and the road component edge position, and associate the modal source identification information in each frame with the relationship between the spatial position and modal information in each frame to obtain the modal and scene structure mapping set.

[0035] As a further aspect of the present invention, the formula for calculating the coverage ratio of the intersection area between the border edge and the component edge line in the image plane is as follows:

[0036]

[0037] Where, γ ez p represents the proportion of the intersection area between the e-th border edge and the z-th component edge in the image plane. ek q represents the lateral coordinate value of the e-th border edge in the k-th frame image. zk p represents the vertical coordinate value of the z-th component edge in the k-th frame image. en q represents the horizontal coordinate value of the e-th border edge in the current frame. zn Δp represents the vertical coordinate value of the z-th component edge in the current frame. en Δq represents the lateral offset of the e-th border edge relative to the mean in the current frame. zn Δ represents the vertical offset of the z-th component's edge relative to the mean in the current frame. e zn represents the pixel coordinate difference between the border edge and the component edge in the current frame, ω is the coordinate difference adjustment factor, φ is the compensation constant, k represents the index variable from frame 1 to frame n, n is the current frame number, e represents the index number of the border edge, and z represents the index number of the component edge.

[0038] As a further aspect of the present invention, the specific steps of S5 are as follows:

[0039] S501: Based on the contact positions recorded in the modality and scene structure mapping set, filter out the modality identifiers of continuous frame segment handover phenomena, and classify the repeated handover information of the same modality in multiple frame sequences into a single index item to obtain modality repeated contact combination.

[0040] S502: Based on the frame segment information of the mode appearing in the modal repetitive contact combination, track the corresponding scene handover features according to the frame segment arrangement order, index the segments in which the differentiated mode appears in consecutive frames and intersects with road components, and obtain the modal handover sequence index table;

[0041] S503: Based on the segment identifier of the modality in the modality handover sequence index table, the handover performance of the corresponding modality in the differentiated frame segments is marked into the fusion sorting process node according to the scene order to obtain the fusion perception path sorting framework.

[0042] Compared with the prior art, the advantages and positive effects of the present invention are as follows:

[0043] In this invention, cross-modal target association is constructed by spatially overlapping and pairing intra-frame images with point cloud modal targets. Trajectory difference segments are extracted by the sequence of directional changes in consecutive frames to distinguish the location of behavioral changes. Modal sources are divided by joint analysis of edge trends and point cluster arrangement trends. The association and matching relationship between modal and scene components is established by extracting contact areas. The fusion order is arranged by the contact performance of modal in difference frame segments to construct a perceptual path sorting framework with distinguishing capabilities. Attached Figure Description

[0044] Figure 1 This is a flowchart of the method steps of the present invention. Detailed Implementation

[0045] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0046] In the description of this invention, it should be understood that the terms "length," "width," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicating orientation or positional relationships, are based on the orientation or positional relationships shown in the accompanying drawings and are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. Furthermore, in the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0047] Please see Figure 1 This invention provides a technical solution: a method for perceiving complex scenes in autonomous driving based on multimodal fusion, comprising the following steps:

[0048] S1: Call the center position of the vehicle bounding box in each frame of the image modality and the center position of the reflection point cluster in the point cloud modality, identify the target regions located in the same area in the modality, pair the positions of image targets and point cloud targets in the same frame one by one, extract the target combinations with overlapping and covered positions, summarize the pairing relationships in each frame, and obtain the corresponding set of image point cloud targets.

[0049] S2: Based on the paired target entities in the target correspondence set of the image point cloud, extract the changes in the movement direction within consecutive frames, compare the order of the direction changes under the differentiated modalities, and identify whether there are continuous alternating changes. When there are inconsistent records of the direction between consecutive frames, mark the target corresponding frame segment as a segment with inconsistent trajectory behavior to obtain the conflict segment identifier book.

[0050] S3: Based on the frame segments recorded in the conflict fragment identifier book, extract the extension direction of the bounding box edge trend between adjacent frames from the image modality, analyze the arrangement direction of the reflection points in the continuous frames from the point cloud modality, classify the modality records with consistent edge extension direction and no shift in the change of point cluster arrangement direction, extract the modality identifier by combining the start and end positions of the corresponding frame segments, and obtain the modality output group.

[0051] S4: Based on the target position of each frame sequence in the modal output group, retrieve the linear markings and obstacle boundaries on the road in the corresponding frame, read the position values ​​of the four sides of the target bounding box in the image output, and compare the position values ​​with the position values ​​of the lane markings and obstacle boundary lines point by point. When the target bounding box edge line and the road component edge line appear in the same spatial area, record the corresponding spatial position and correlation to obtain the modal and scene structure mapping set.

[0052] S5: Based on the contact records in the modality and scene structure mapping set, screen the number of contacts of the same modality with road markings and obstacle boundaries in differentiated frames, list the modality recognition results with continuous contact records as the priority source for fusion, sort them according to the modality occurrence frequency and the number of markings, arrange the processing order in the multimodal fusion process, and obtain the fusion perception path sorting framework.

[0053] The image point cloud target correspondence set includes the bounding box center position, the reflection point cluster center position, position overlap combination, and intra-frame pairing relationship. The conflict segment identifier book includes the direction change order, inconsistent direction segments, and trajectory behavior markers. The modal output grouping includes the edge extension direction, point cluster arrangement amplitude, modal identifier, and frame segment position range. The modal and scene structure mapping set includes the spatial contact area, structural association relationship, target edge position, and road component edge. The fusion perception path ranking framework includes contact frequency, fusion priority order, modal occurrence frequency, and number of boundary markers.

[0054] The specific steps of S1 are as follows:

[0055] S101: Obtain the center position of the vehicle bounding box in each frame of the image modality and the center position of the reflection point cluster in each frame of the point cloud modality. Pair the image target position with the point cloud target position frame by frame according to the frame number, identify the target point pairs that appear in the same spatial region, and obtain the modal spatial association dataset.

[0056] First, extract the position parameters of the target bounding box from each frame of the image modality. Record the coordinates of the top left and bottom right corners and then calculate the center coordinates. For example, if there are two vehicle bounding boxes in frame 28, extract their corresponding center positions and label them A1 and A2 respectively. Then, extract the spatial positions of all point clusters in that frame from the point cloud modality. Locate the geometric center of each point cluster by finding the median of its boundary coordinates. For example, obtain the center positions of point clusters B1, B2, and B3 and label them with numbers. Next, align the point cloud frame numbers with the image frame numbers and match the image target and the point cloud target in each frame. Determine if there is a spatial relationship between them that is close or overlapping in the image plane. If the distance between the boundary centers is less than the set spatial distance, the target is considered to be in the point cloud. The scope is limited. For example, in frame 28, if A1 and B2 are close in position, they are identified as paired targets. The relationship between their image and point cloud numbers is recorded, and the frame number and paired target identifier are stored. This judgment process is repeated for each frame to find all matching targets. Further checks are made to see if there are multiple image targets corresponding to a single point cloud target. If so, the optimal pairing relationship is selected according to the proximity of the boundary center positions. Priority screening is performed within the offset range between adjacent targets. Finally, the pairing results are stored by frame number and archived as a set of target combination relationship records. This record is used to support the entity reference basis in subsequent modal difference alignment and conflict determination, and finally, a modal space association dataset is obtained.

[0057] S102: Based on the modal space association dataset, analyze the regional coverage relationship between image targets and point cloud targets, determine whether the line connecting the center points falls within the common area of ​​the image border and the edge of the point cloud distribution, filter the target combinations with continuous contact features, and obtain modal pairing coverage relationship groups;

[0058] First, the coordinate range of the four sides and the position of the center point of each image target bounding box are extracted. Then, the outer contour edge points and center points of the point clusters are obtained from the corresponding point cloud data. Next, a spatial connection is constructed between the center of the image target and the center of the point cloud target. This connection is used as the reference path for spatial contact determination. The intersection of this path in the coordinate regions of the image bounding box and the point cloud cluster edge is analyzed segment by segment. When the connection between the center of the image target and the center of the point cloud target has continuous path segments in both regions, it is determined that there is regional contact between the two. For example, if the center coordinates of the image bounding box G1 in frame 42 are (240, 160), and the point cloud... If the center coordinates of point cluster C1 are (238, 158), a line can be drawn connecting the two points. This line passes through the lower left and upper right corners of the image bounding box and falls within the coverage area of ​​the point cloud cluster. This indicates that G1 and C1 constitute a contact pair and the contact state is marked as 1. Conversely, if the line only intersects with the boundary area of ​​one of the points, the contact relationship is not recognized, and the next pair is screened. The target combinations that satisfy the region overlap relationship in the entire frame segment are classified by frame number. Sequences of the same target contact combination appearing in consecutive frames are identified and classified as target combinations with continuous contact features. Finally, the modal pairing coverage relationship group is obtained.

[0059] S103: Based on the modal pairing coverage relationship group, arrange the pairing relationship between image targets and point cloud targets in frame order, extract the target combination data with spatial contact features in each frame, and obtain the image point cloud target correspondence set;

[0060] First, frame numbers are extracted sequentially from smallest to largest in the frame sequence. Image target identifiers and point cloud target identifiers in each frame are then paired and extracted. Next, the status markers in spatial contact relationships are matched based on the identifier information. Combinations not marked as contact are filtered out, retaining only image-point cloud target combinations in contact. A spatial contact record column arranged in frame order is constructed. During processing, for example, if image target G2 and point cloud target C2 have a central line intersection in frame 48 and have been identified as a region contact combination, G2 and C2 are grouped into a valid target pair, and their correspondence in frame 48 is recorded. The process continues frame by frame, retrieving target identifiers and contact relationship markers, and recording the pairing of contact target pairs according to frame order. Simultaneously, the occurrence of the same target combination in different frames is marked for target consistency judgment in subsequent tracking modules. Throughout the process, the target shape or boundary dimensions are not interfered with; only the target identifier and frame order position combination are used to construct a corresponding set, ultimately obtaining the image-point cloud target correspondence set.

[0061] The specific steps of S2 are as follows:

[0062] S201: Based on the paired target entities in the target correspondence set of the image point cloud, obtain the target position sequence of the image mode and the point cloud mode in consecutive frames, extract the motion direction vector of each frame, and arrange the direction change order of the same target in the differentiated mode according to the frame order to obtain the direction order sequence group;

[0063] The target position coordinates of each paired target in consecutive frames are read in both the image modality and the point cloud modality. Spatial orientation information indicating directional changes is obtained through inter-frame differences between coordinate sets. The line connecting the center points of the target bounding boxes between adjacent frames in the image modality is determined as the motion direction in the image modality. The line connecting the centers of the reflection point clusters of the same target in consecutive frames in the point cloud modality is taken as the motion direction in the point cloud modality. During the generation of these two types of direction vectors, a one-to-one correspondence must be maintained between the data frames of the same target number in both modalities to ensure that the direction vectors have the prerequisite for comparison. The obtained... Image orientation sequences and point cloud orientation sequences are synchronously arranged according to frame numbers. The order of orientation changes of each target in consecutive frames is confirmed. For example, if the pairing of images and point clouds numbered G17-C17 continuously shows orientation reversal or switching between frames 45 and 50, the turning position of the orientation vector can be observed in the corresponding two modal orientation sequences. The orientation change results are converted into sequence data by arranging the frames, so that the image orientation changes and point cloud orientation changes form a sequential vector recording structure, resulting in an orientation sequence group.

[0064] S202: Based on the direction sequence group, calculate the proportion of direction switching in consecutive frames in image modality and point cloud modality, divide the rhythm difference between frame segments by comparing the rhythm of direction change, and obtain the motion direction rhythm offset segment.

[0065] The specific formula for calculating the proportion of orientation switching in consecutive frames in image modality and point cloud modality is as follows:

[0066]

[0067] Among them, P d The x represents the proportion of orientation switching in consecutive frames within the image modality and point cloud modality, where m represents the number of consecutive frames involved in the computation, and x represents the proportion of orientation switching within the image modality and point cloud modality. j y represents the motion direction encoding value of the image modality in the j-th frame. j w represents the motion direction encoding value of the point cloud mode in frame j. j δ represents the motion confidence weight of the j-th frame, δ represents the smoothing constant when the sum of the direction values ​​is zero, and λ represents the square root term adjustment value.

[0068] The operation logic is as follows:

[0069] 1. For each frame j, first calculate the difference in directional encoding values ​​between the image and the point cloud, |x|.j -y j |;

[0070] 2. After squaring the differences, divide by their sum and add a smoothing factor to obtain the intensity of change term;

[0071] 3. Simultaneously, take the absolute value of the difference and then the square root to form the amplitude adjustment term;

[0072] 4. Multiply each result by its corresponding weight term w. j The values ​​are summed with the control coefficient λ and then averaged to output the final normalized proportion value P. d ;

[0073] For example, the orientation of the first frame image is 45°, and the orientation of the point cloud is 60°;

[0074] but:

[0075]

[0076] If the angle between the two modal directions is less than 20°, the weight is taken as w. j =1, take w if the included angle is greater than 60° j =0.5;

[0077] For example, the angle between the first frame image and the point cloud is 15°, corresponding to a cosine value of approximately 0.96.

[0078] but:

[0079] w1=1.0, δ=0.01: λ=0.4;

[0080] Actual calculation example:

[0081] Take 3 consecutive frames of data (m=3):

[0082] x1=0.25, y1=0.33, w1=1.0;

[0083] x2=0.40, y2=0.42, w2=0.8;

[0084] x3=0.30, y3=0.60, w3=0.6;

[0085] Substitute into the formula to calculate:

[0086] Item 1:

[0087]

[0088]

[0089] =0.0108 + 0.1131 = 0.1239;

[0090] Item 2:

[0091]

[0092] =0.0004 + 0.0566;

[0093] =0.0570;

[0094] Item 3:

[0095]

[0096] =0.0593 + 0.2191;

[0097] =0.2784;

[0098] Final result:

[0099]

[0100] The results show that in the current 3 frames, the normalization ratio of the direction switching between the image modality and the point cloud modality is 0.1531, which is in the medium difference range. This can be used to determine whether it is a motion rhythm shift segment.

[0101] Explanation of the innovative aspects of the formula:

[0102] The advantage of this formula lies in its ability to enhance responsiveness to trends across different scales and directions by incorporating a weighted combination of a square root term and a magnitude term, thus integrating two methods of measuring difference. Furthermore, the introduction of a dynamic weight w further enhances its effectiveness. j The adjustment coefficient λ further enhances the sensitivity to local frame changes and the controllability of overall normalization.

[0103] S203: Based on the rhythm offset of motion direction segments, screen for target combinations in which the motion direction of the image modality and the point cloud modality are inconsistent within the same frame segment, and map the segments with continuously changing rhythm differences to the target entity position sequence to obtain the conflict segment identification book.

[0104] Based on the obtained direction sequence group, the motion direction vector sequence corresponding to the image mode and the point cloud mode is extracted from each frame segment. According to the target number consistency principle, frame alignment processing is performed on each pair of image and point cloud targets. The change of direction vectors in the two modes in each frame is extracted, and it is determined whether there is a spatial orientation difference. For example, if a target is oriented due east in the image mode but southwest in the point cloud mode, it is determined to be an inconsistent orientation. At the same time, it is checked whether the difference persists for more than three consecutive frames. If so, the frame segment to which the target belongs is marked as a rhythm difference segment. In this process, it is necessary to avoid interference from temporary orientation inconsistencies caused by frame omissions or target recognition offsets. Furthermore, the frame segments with rhythm differences are back-searched to the target entity position sequence in the image mode and the point cloud mode using the segment number as an index. It is confirmed that the target number with the conflicting orientation is aligned with the corresponding frame number, forming a rhythm abnormal segment based on the position sequence. Finally, a record entry is constructed according to the target number and the frame segment number and added to the record book column to obtain the conflict segment identification book.

[0105] The specific steps for S3 are as follows:

[0106] S301: Based on the frame segments recorded in the conflict segment identifier book, extract the bounding box edge coordinates of adjacent frames in the image modality, determine the extension direction of the edge line segments between consecutive frames according to the frame order, filter out the frame segments whose direction keeps changing continuously, and obtain the edge extension direction sequence.

[0107] First, select the bounding box data from the image modality output of each frame segment, extract the coordinate set of the four corner points of the bounding box, and read the edge position coordinates in the current frame and the previous frame respectively. By matching the corresponding points, identify the extension path of the boundary line segments of the same target edge in consecutive frames. Connect the top edge, bottom edge, left edge and right edge in each frame to generate line segments. Then compare the direction offset trend of the line segments at the corresponding positions in adjacent frames to determine whether they show a continuous extension or angular shift in space. If the direction trend between frames is consistent or the direction transition is smooth, the current frame segment is recorded as a directionally continuous frame segment. Otherwise, discontinuous points are removed and the merging of the current segment is stopped. During the operation, typical examples can be combined for identification. For example, when the bounding box of a vehicle target in the image modality advances to the upper right, right and lower right directions in frames 5, 6 and 7 respectively, it indicates that its direction extension is continuous. Frame segments 5 to 7 can be marked as a continuous sequence. Finally, collect the numbers of all frame segments with consistent directions to obtain the edge extension direction sequence.

[0108] S302: Based on the edge extension direction sequence, call the reflection point cluster position data of the corresponding frame segment in the point cloud modality, calculate the angle offset value of the point cluster arrangement direction between consecutive frames, and identify the frame segments where the angle offset variation range tends to be consistent within the continuous segment to obtain the arrangement trend consistent segment.

[0109] The formula for calculating the angular offset of the arrangement direction of consecutive inter-frame point clusters is as follows:

[0110]

[0111] in, S represents the angular offset between the cluster arrangement direction of frame v and the previous frame (v-1). v S represents the sum of the principal direction sequence projection values ​​of the reflection point cluster in frame v. v-1 α represents the sum of the principal direction sequence projection values ​​of the reflection point cluster in frame v-1, α represents the proportionality coefficient of the previous frame's order value in the current angle offset calculation, β represents the positive offset adjustment constant used to stabilize the denominator result, and γ represents the adjustment factor for the angular deflection dispersion of the reflection point cluster. This represents the deflection magnitude of the m-th cluster of reflection points in the v-th frame. N represents the average deflection amplitude of the reflection point cluster in frame v. v This represents the number of point clusters participating in the statistics in the v-th frame, where v represents the current lidar point cloud sequence frame number, and m represents the m-th reflection point cluster participating in the calculation in the current frame.

[0112] Parameter values ​​and explanations:

[0113] Assume that the sum of the principal direction scores of the reflection point clusters in the (v-1)th frame and the vth frame obtained by the lidar are S respectively. v-1 =150, S v =165, the score is obtained by weighting and adding three indicators: angular consistency of the main direction of the point cluster, point cluster density, and stability;

[0114] The value of coefficient α = 0.9 is obtained by statistically analyzing the standard deviation of the inter-frame sequence within a 5-frame sliding window. When the variance is lower than the set threshold of 100, it is set to 0.9.

[0115] β=5 is the minimum stability factor used for normalization obtained through multiple rounds of simulation experiments, thus avoiding the occurrence of offset amplification error;

[0116] γ = 1.2 is an adjustment factor set in the experiment to control the intensity of the influence of the permutation dispersion on the results;

[0117] The v-th frame has 4 point clusters, i.e., N v =4, deflection amplitude The angle between the main direction of each point cluster in the frame and the overall average direction is measured in degrees.

[0118] The mean deflection amplitude is:

[0119]

[0120] Substituting formulas into the calculation process

[0121] The first term (normalized square of order offset) is calculated as follows:

[0122]

[0123] The second term (squared dispersion of point cluster deflection) is calculated as follows:

[0124]

[0125]

[0126] The calculation result is:

[0127]

[0128] The result indicates that when If the value is less than the trend continuity identification threshold set by the system, it indicates that the v-th frame and the v-1-th frame have a high degree of consistency in the point cluster arrangement trend. The frame should be classified into the "arrangement trend matching segment". The driving step is used as the triggering basis for continuous frame segment trend identification, trajectory prediction or point cloud filtering strategy.

[0129] Explanation of the innovative aspects of the formula:

[0130] The advantage of the formula is that by introducing a coupling calculation mechanism between the inter-frame order difference and the internal deflection dispersion of the point cluster, combined with the state weight coefficient α and the deflection adjustment factor γ, it makes the identification of abrupt changes in the main direction and the consistency of inter-frame trends in the point cloud more sensitive, which facilitates the determination of the continuity of the frame segment and the control of the data flow processing path.

[0131] S303: Based on the consistent arrangement trend of the segment, extract the modal source identifier corresponding to each frame in the segment, classify and merge the modal identifiers according to the frame segment range correspondence, and obtain modal output groups;

[0132] First, select the frame numbers recorded in the image frames according to the segment sequence, and match the source labels of the current frame number in the image modality and point cloud modality to extract the corresponding modality identifier in the frame. Based on the obtained modality identifier of each frame, use the segment range formed by the start and end numbers of the frame segment as the grouping basis to classify the modality identifiers corresponding to the frame numbers in the same segment, and mark the modality affiliation of the classified modality. Determine whether there is a cross-modality identifier intersection. If there is a situation where the image modality and the point cloud modality alternate in the same segment, then integrate them into a group according to the frame number of the first appearance of the modality, and vice versa. For example, in a segment numbered F12 to F20, the image modality identifier appears continuously from F12 to F16, while the point cloud modality identifier appears from F17 to F20. Then mark this segment as belonging to the image modality start group, and so on. During the operation, multiple frame segments can be used as processing samples to confirm the consistency of the grouping logic. Finally, the affiliation results of the marked modalities in all segments are collected to obtain the modality output group.

[0133] The specific steps of S4 are as follows:

[0134] S401: Based on the target position of each frame sequence in the modal output group, read the linear markers and obstacle boundary layer data in the road image in the corresponding frame, extract the top, bottom, left and right edge position coordinates of the target border of the image, list the spatial position information of the border edge in each frame, and obtain the border edge position dataset.

[0135] First, in each frame image, the corresponding frame number within the assigned frame sequence range is located. The image target region corresponding to that frame is read, and the coordinates of the edge pixels on all four sides of the region are extracted, including the vertical coordinates of the top and bottom edges and the horizontal coordinates of the left and right edges. Then, these four types of edge information are integrated into the coordinate set of the frame image border edges. Next, the layer structure of the road image in each frame image is located, and the layer data of its linear markers and obstacle boundaries are extracted. This type of layer data can be obtained through image segmentation results or semantic partitioning processing results. In images with the same frame number, the position data of road components such as lane lines, dividing lines, and wall edges are marked respectively, and their spatial position ranges are recorded. Then, the edge coordinates of the image target border in the current frame are compared with the above layer position ranges to confirm the spatial region range of each edge. Furthermore, the spatial distance between the edge and the road component is recorded as the region attribute of the edge coordinates. In the process, the operation is repeated frame by frame until all image frames involved in the current modality output group are traversed. Finally, the spatial positions of the four sides of the image target border edges recorded in all frames are combined into a unified set to obtain the border edge position dataset.

[0136] S402: Based on the bounding box edge location dataset, obtain the coordinate range of lane markings and obstacle boundaries, read the positioning information of bounding box edge coordinates and component edge coordinates in each frame, calculate the coverage ratio of the intersection area of ​​bounding box edge and component edge in the image plane, and obtain the edge contact feature combination.

[0137] The formula for calculating the coverage ratio of the intersection area between the border edge and the component edge in the image plane is as follows:

[0138]

[0139] Where, γ ez p represents the proportion of the intersection area between the e-th border edge and the z-th component edge in the image plane. ek q represents the lateral coordinate value of the e-th border edge in the k-th frame image. zk p represents the vertical coordinate value of the z-th component edge in the k-th frame image. en q represents the horizontal coordinate value of the e-th border edge in the current frame. zn Δp represents the vertical coordinate value of the z-th component edge in the current frame. en Δq represents the lateral offset of the e-th border edge relative to the mean in the current frame. zn Δ represents the vertical offset of the z-th component's edge relative to the mean in the current frame. e zn represents the pixel coordinate difference between the border edge and the component edge in the current frame, ω is the coordinate difference adjustment factor, φ is the compensation constant, k represents the index variable from frame 1 to frame n, n is the current frame number, e represents the index number of the border edge, and z represents the index number of the component edge.

[0140] Assumption:

[0141] Take 3 frames of images and detect:

[0142] p e1 =120, p e2 =125, p e3 =130;

[0143] q z1 =200, q z2 =195, q z3 =190;

[0144] Calculate the average:

[0145]

[0146] Current frame n=3:

[0147] p e3 =130, q z3=190;

[0148] deviation:

[0149] Δp e3 =130-125=5;

[0150] Δq z3 =190-195=-5;

[0151] |Δ ez3 |=|130-190|=60;

[0152] Calculate ω:

[0153] Difference sequence: -80, -70, -60, average -70;

[0154] Standard deviation:

[0155]

[0156] The dynamic range is set to 255, and the normalized result is... Set φ = 13, based on the average pixel fluctuation amplitude sampled at the image boundary; substitute into the formula to calculate:

[0157] Molecular part:

[0158] |5 2 +(-5) 2 -0.032·60|=|25+25-1.92|=48.08; Denominator:

[0159] (p e1 +q z1 ) 2 +(p e2 +q z2 ) 2 +(p e3 +q z3 ) 2 ;

[0160] =320 2 +320 2 +320 2 ;

[0161] =307200;

[0162]

[0163] Final result:

[0164]

[0165] The result shows that the intersection coverage ratio of the edge of the e-th border and the edge of the z-th component in the current frame is 0.0847, indicating that there is a contact or proximity relationship between the two, which can be used as a basis for image judgment.

[0166] Explanation of the innovative aspects of the formula:

[0167] The advantage of the formula is that it controls the sensitivity of spatial contact by introducing a normalization factor ω based on frame difference statistics, and at the same time introduces φ as a stability constant to avoid error amplification caused by fluctuations at the edge of the image plane.

[0168] S403: Based on edge contact feature combination, mark the frame order of the intersection of the edge of the border and the edge of the road component, and associate the modal source identification information in each frame with the relationship between the spatial position and modal information in each frame to obtain the modal and scene structure mapping set;

[0169] First, determine the frame sequence information where the coordinates of the border edges intersect with the road component edge data. Based on the spatial coordinate overlap range, filter out records in each frame where the edge of the image target border is in the same area as the edges of road components such as lane markings, guardrail edges, and curb boundaries. Extract the frame numbers of these types and list them as a contact frame set. Then, search for the modal source identifier of each frame number in the contact frame set. In the stored modal output group, match the modal classification result according to the frame number and record the modal type corresponding to the frame. Combine the spatial position parameters of the target border in the image frame, extract the position interval of the corresponding image target area in the image coordinate plane, and associate this interval with the modal source identifier to establish a frame-level mapping relationship between the two. At the same time, for cases where multiple target border edges overlap with component edges in the same frame, list the intersection positions between the top, bottom, left, and right edges of each target border and the component edge line in each record, and mark them with the modal identifier to which the target border belongs. Finally, summarize the modal source, spatial position, and edge overlap information of all intersecting frames into a unified table to obtain the modal and scene structure mapping set.

[0170] The specific steps of S5 are as follows:

[0171] S501: Based on the contact positions recorded in the modality-scene structure mapping set, the modality identifiers of continuous frame segment handover phenomena are screened out, and the repeated handover information of the same modality in multiple frame sequences is classified into a single index item to obtain the modality repeated contact combination.

[0172] First, each record in the mapping set is examined frame-by-frame for the combination characteristics between modality identifiers and spatial locations. Contact records with consecutive frame numbers are grouped according to modality type, and modality identifiers that recur under consecutive frame order are listed. Next, a timeline sequence is established, and the positions of each modality within this sequence are marked to observe its repetitive behavior in different frame segments. Based on this, modality groups exhibiting consecutive frame segment handover phenomena are identified. Subsequently, multiple handover information of the same modality appearing in different time periods is categorized and organized, redundant indexes and duplicate frame segments are removed, and a unique identifier is assigned to each type of handover record. For example, modality A in frame segment 5... -9 and the boundary contact records occurring in frames 15-18 are grouped together and given a unified index number for later retrieval. In addition, during processing, it is necessary to determine whether the modality belongs to different spatial location types in the two frame sequences. For example, if modality A contacts lane markings in frames 5-9 but contacts obstacle edges in frames 15-18, this situation should still be included in the category of repeated contact. However, a specific component category field needs to be added to the final record for subsequent screening operations. Finally, all modal repeated contact records that meet the above rules are arranged according to modal type and merged with the corresponding frame sequence information to obtain the modal repeated contact combination.

[0173] S502: Based on the frame segment information of the mode appearing in the modal repetitive contact combination, track the corresponding scene handover features according to the frame segment arrangement order, index the segments in which the differentiated mode appears in consecutive frames and intersects with road components, and obtain the modal handover sequence index table;

[0174] First, the modal identifier and corresponding frame segment number associated with each group of records in the combination are read. All frame segment numbers are arranged in ascending order of numerical value to form a frame segment sorting index. The frame sequence of the same modality in different segments is then assembled into a complete timeline. The edge contact area information appearing in consecutive frames of the image or point cloud modality is tracked on this timeline. By referring to the scene component layer data attached to the road image, the edge contact state between the target boundary and lane lines, median strips, and obstacle edges in each frame segment is read. At the same time, by referring to the modal source identifier marked in each frame, it is searched for whether there are records with different modal types but simultaneously in a spatial contact state. If so, they are marked as differentiated modalities. In the handover process, the start and end frames of such frames are combined into a group of handover segment identifiers, and the handover type is also marked, such as image being converted into point cloud or point cloud being replaced by image after continuous contact. Then, the above handover segments and corresponding modalities are indexed and written into a table in chronological order, and fields are established to store information such as modal identifier, handover type, frame number, and scene structure category label. For example, if a modality contacts an obstacle boundary in frame segment 40-45, and then another modality completes the contact in frame segment 46-49, it is recorded as a continuous image-point cloud handover with the component type being obstacle boundary. Finally, all records are arranged and output according to modality and frame segment to obtain the modal handover sequence index table.

[0175] S503: Based on the segment identifier of the modality in the modality handover sequence index table, the handover performance of the corresponding modality in the differentiated frame segments is marked into the fusion sorting process node according to the scene order to obtain the fusion perception path sorting framework.

[0176] First, the modality type, corresponding frame segment number, handover status, and component type of each record in the index table are read. An overall timeline is constructed according to the order of frame segment numbers. All identified segments are mapped to their positions on this timeline, and the manifestation of each modality within the current frame segment is located frame by frame. For example, if the image modality continuously creates spatial coverage of the curb edge in frames 20 to 25, and the point cloud modality continues the contact features of the target component in frames 26 to 30, these two segments are connected sequentially and considered as a modality handover event. The start segment, end segment, and handover point of the image modality are marked in this handover event, and similarly, the start segment, end segment, and connection status of the point cloud modality are marked. Then, by determining whether the event's position on the timeline overlaps with other handover events, handover events with temporal overlap are rearranged according to time priority. The modal roles in each event are divided into dominant modality and continuing modality. The dominant mode is the mode that first occurs in the current event when a spatial coverage relationship occurs, while the continuing mode is the mode in which the contact state of the connecting components occurs in subsequent frames. For example, if the image mode has shown contact characteristics with the edge of the sidewalk in frames 10 to 20, and the point cloud mode continues this trend in frames 21 to 27, then the image mode is set as the dominant mode and the point cloud mode is marked as the continuing mode. Then, the priority of the image mode is set in the fusion sorting process node, and its corresponding frame segment is marked at the beginning of the process node. The point cloud mode is added sequentially in the process node, and the mode type, frame range, contact component type and handover label are recorded in each node. For example, node "C1" is set as image mode - frames 10-20, and the contact target is the road centerline. "C2" is point cloud mode - frames 21-27, indicating that it is sequentially connected with "C1". Finally, all nodes are connected in series to obtain the fusion perception path sorting framework.

[0177] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments that can be applied to other fields. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

Claims

1. A method for perceiving complex scenes in autonomous driving based on multimodal fusion, characterized in that, Includes the following steps: S1: Obtain the positions of the center of the image modal bounding box and the center of the point cloud modal reflection point cluster, identify spatially overlapping targets within the same frame, pair the image and point cloud targets, extract the intra-frame correlation, and obtain the corresponding set of image and point cloud targets; S2: Based on the paired targets in the target correspondence set of the image point cloud, analyze the order of directional changes in the continuous frames of the modality, identify the alternating frame segments in the trajectory movement, divide them into trajectory difference regions, and obtain the conflict segment identification book; S3: Based on the frame segments in the conflict segment identifier book, analyze the image edge trend and point cloud arrangement changes, identify output frame groups with consistent edge directions and no jump in point cluster arrangement, divide the modal source according to the frame segment position, and obtain modal output groups; S4: Based on the target position in the modal output group, read the position of the four sides of the image border, compare the spatial position with the road markings and obstacle edges, summarize the contact area and modal source, and obtain the modal and scene structure mapping set; S5: Based on the contact situation between the modal and the scene structure mapping set, analyze the continuous contact performance of the modal with road components in the differentiated frame segments, arrange the perception fusion order according to the spatial distribution relationship, and obtain the fusion perception path sorting framework.

2. The method for perceiving complex scenes in autonomous driving based on multimodal fusion according to claim 1, characterized in that, The image point cloud target correspondence set includes the center position of the bounding box, the center position of the reflection point cluster, the positional overlap combination, and the intra-frame pairing relationship. The conflict segment identifier book includes the direction change order, the directionally inconsistent segments, and the trajectory behavior markers. The modal output grouping includes the edge extension direction, the point cluster arrangement amplitude, the modal identifier, and the frame segment position range. The modal and scene structure mapping set includes the spatial contact area, the structural association relationship, the target edge position, and the road component edge. The fusion perception path sorting framework includes the contact frequency, the fusion priority order, the number of times the modality appears, and the number of boundary markers.

3. The method for perceiving complex scenes in autonomous driving based on multimodal fusion according to claim 1, characterized in that, The specific steps of S1 are as follows: S101: Obtain the center position of the vehicle bounding box in each frame of the image modality and the center position of the reflection point cluster in each frame of the point cloud modality. Pair the image target position with the point cloud target position frame by frame according to the frame number, identify the target point pairs that appear in the same spatial region, and obtain the modal spatial association dataset. S102: Based on the modal space association dataset, analyze the regional coverage relationship between image targets and point cloud targets, determine whether the line connecting the center points falls within the common area of ​​the image border and the edge of the point cloud distribution, filter target combinations with continuous contact features, and obtain modal pairing coverage relationship groups. S103: Based on the modal pairing coverage relationship group, arrange the pairing relationship between image targets and point cloud targets in frame order, extract the target combination data with spatial contact features in each frame, and obtain the image point cloud target correspondence set.

4. The method for perceiving complex scenes in autonomous driving based on multimodal fusion according to claim 1, characterized in that, The specific steps of S2 are as follows: S201: Based on the paired target entities in the target correspondence set of the image point cloud, obtain the target position sequence of the image mode and the point cloud mode in consecutive frames, extract the motion direction vector of each frame, and arrange the direction change order of the same target in the differentiated mode according to the frame order to obtain the direction order sequence group. S202: Based on the direction sequence group, calculate the proportion of direction switching in consecutive frames in the image modality and point cloud modality, divide the rhythm difference between frame segments by comparing the rhythm of direction change, and obtain the motion direction rhythm offset segment. S203: Based on the motion direction rhythm offset segment, screen the target combination where the image modality and point cloud modality have inconsistent motion directions in the same frame segment, and map the segments with continuously changing rhythm differences to the target entity position sequence to obtain the conflict segment identification book.

5. The method for perceiving complex scenes in autonomous driving based on multimodal fusion according to claim 4, characterized in that, The specific formula for calculating the proportion of orientation switching in consecutive frames in the image modality and point cloud modality is as follows: Among them, P d The x represents the proportion of orientation switching in consecutive frames within the image modality and point cloud modality, where m represents the number of consecutive frames involved in the computation, and x represents the proportion of orientation switching within the image modality and point cloud modality. j y represents the motion direction encoding value of the image modality in the j-th frame. j w represents the motion direction encoding value of the point cloud mode in frame j. j δ represents the motion confidence weight of the j-th frame, δ represents the smoothing constant when the direction value sum is zero, and λ represents the square root term adjustment value.

6. The method for perceiving complex scenes in autonomous driving based on multimodal fusion according to claim 1, characterized in that, The specific steps for S3 are as follows: S301: Based on the frame segments recorded in the conflict segment identifier book, extract the bounding box edge coordinates of adjacent frames in the image modality, determine the extension direction of the edge line segments between consecutive frames according to the frame order, filter out the frame segments whose direction keeps changing continuously, and obtain the edge extension direction sequence. S302: Based on the edge extension direction sequence, call the reflection point cluster position data of the corresponding frame segment in the point cloud modality, calculate the angle offset value of the point cluster arrangement direction between consecutive frames, and identify the frame segments where the angle offset variation range tends to be consistent within the continuous segment to obtain the arrangement trend consistent segment. S303: Based on the matching segments of the arrangement trend, extract the modal source identifier corresponding to each frame in the segment, classify and merge the modal identifiers according to the frame segment range correspondence, and obtain modal output groups.

7. The method for perceiving complex scenes in autonomous driving based on multimodal fusion according to claim 6, characterized in that, The formula for calculating the angular offset value of the arrangement direction of the continuous inter-frame point clusters is as follows: in, S represents the angular offset between the cluster arrangement direction of frame v and the previous frame (v-1). v S represents the sum of the principal direction sequence projection values ​​of the reflection point cluster in frame v. v-1 α represents the sum of the principal direction sequence projection values ​​of the reflection point cluster in frame v-1, α represents the proportionality coefficient of the previous frame's order value in the current angle offset calculation, β represents the positive offset adjustment constant used to stabilize the denominator result, and γ represents the adjustment factor for the angular deflection dispersion of the reflection point cluster. This represents the deflection magnitude of the m-th cluster of reflection points in the v-th frame. N represents the average deflection amplitude of the reflection point cluster in frame v. v This represents the number of point clusters participating in the statistics in the v-th frame, where v represents the current lidar point cloud sequence frame number, and m represents the m-th reflection point cluster participating in the calculation in the current frame.

8. The method for perceiving complex scenes in autonomous driving based on multimodal fusion according to claim 1, characterized in that, The specific steps of S4 are as follows: S401: Based on the target position of each frame sequence in the modal output group, read the linear markers and obstacle boundary layer data in the road image in the corresponding frame, extract the top, bottom, left and right edge position coordinates of the image target border, list the spatial position information of the border edge in each frame, and obtain the border edge position dataset. S402: Based on the bounding box edge position dataset, obtain the coordinate range of lane markings and obstacle boundaries, read the positioning information of bounding box edge coordinates and component edge coordinates in each frame, calculate the coverage ratio of the intersection area of ​​bounding box edge and component edge in the image plane, and obtain the edge contact feature combination. S403: Based on the edge contact feature combination, mark the frame order of the intersection of the border edge and the road component edge position, and associate the modal source identification information in each frame with the relationship between the spatial position and modal information in each frame to obtain the modal and scene structure mapping set.

9. The method for perceiving complex scenes in autonomous driving based on multimodal fusion according to claim 8, characterized in that, The formula for calculating the coverage ratio of the intersection area between the border edge and the component edge line in the image plane is as follows: Where, γ ez p represents the proportion of the intersection area between the e-th border edge and the z-th component edge in the image plane. ek q represents the lateral coordinate value of the e-th border edge in the k-th frame image. zk p represents the vertical coordinate value of the z-th component edge in the k-th frame image. en q represents the horizontal coordinate value of the e-th border edge in the current frame. zn Δp represents the vertical coordinate value of the z-th component edge in the current frame. en Δq represents the lateral offset of the e-th border edge relative to the mean in the current frame. zn Δ represents the vertical offset of the z-th component's edge relative to the mean in the current frame. e zn represents the pixel coordinate difference between the border edge and the component edge in the current frame, ω is the coordinate difference adjustment factor, φ is the compensation constant, k represents the index variable from frame 1 to frame n, n is the current frame number, e represents the index number of the border edge, and z represents the index number of the component edge.

10. The method for perceiving complex scenes in autonomous driving based on multimodal fusion according to claim 1, characterized in that, The specific steps of S5 are as follows: S501: Based on the contact positions recorded in the modality and scene structure mapping set, filter out the modality identifiers of continuous frame segment handover phenomena, and classify the repeated handover information of the same modality in multiple frame sequences into a single index item to obtain modality repeated contact combination. S502: Based on the frame segment information of the mode appearing in the modal repetitive contact combination, track the corresponding scene handover features according to the frame segment arrangement order, index the segments in which the differentiated mode appears in consecutive frames and intersects with road components, and obtain the modal handover sequence index table; S503: Based on the segment identifier of the modality in the modality handover sequence index table, the handover performance of the corresponding modality in the differentiated frame segments is marked into the fusion sorting process node according to the scene order to obtain the fusion perception path sorting framework.

Images