A method and system for cross-camera target tracking in autonomous driving based on graph neural networks

Abstract

This application provides a method and system for cross-camera target tracking in unmanned vehicles, incorporating graph neural networks. The method includes: generating local trajectory segments corresponding to each camera based on target observation data, and determining the trajectory quality representation and field-of-view boundary events of the local trajectory segments; constructing dynamic topological transfer relationships between cameras and generating candidate cross-camera association edges; constructing a heterogeneous spatiotemporal trajectory graph using local trajectory segments as trajectory nodes and candidate cross-camera association edges as graph edges; inputting the heterogeneous spatiotemporal trajectory graph into an edge-enhanced graph neural network to iteratively update the features of the trajectory nodes and graph edges, generating identity association confidence scores for candidate cross-camera association edges; and connecting the local trajectory segments across cameras based on identity association confidence scores and trajectory mutual exclusion constraints to generate a global tracking trajectory for targets around the unmanned vehicle or unmanned vessel. This application can improve the accuracy of cross-camera association, reduce the identity switching rate, and enhance trajectory continuity.

Description

Technical Field

[0001] This application relates to the field of intelligent driving perception technology, and in particular to an unmanned driving cross-camera target tracking method and system that integrates graph neural networks. Background Technology

[0002] As autonomous driving demands increasingly higher levels of continuous perception of surrounding traffic participants, multi-camera systems, such as those used in autonomous vehicles, are widely employed for detecting and tracking targets including vehicles, pedestrians, and cyclists. Because each camera has different installation positions, field of view angles, imaging scales, and shooting directions, the same target typically appears sequentially within the field of view of different cameras during the vehicle's circumferential movement. Therefore, cross-camera target tracking is necessary to form a continuous and stable global target trajectory, providing a data foundation for subsequent trajectory prediction, behavior assessment, and driving decisions.

[0003] Existing technologies typically perform object detection and single-camera tracking separately within each camera, and then perform cross-camera association based on the appearance similarity of trajectory segments, time intervals, or spatial distances. Some solutions introduce re-identification models or graph association models to improve the identity matching capability between different cameras. However, these methods mostly rely on static matching thresholds or ordinary trajectory similarity graphs, failing to fully integrate camera calibration relationships, vehicle self-motion, field-of-view boundary events, target motion continuity, and semantic constraints of the road scene.

[0004] In autonomous driving scenarios, targets often experience trajectory breaks or identity changes due to sudden changes in perspective, vehicle blind spots, non-overlapping fields of view, occlusion, and lighting variations. Existing methods struggle to dynamically correct the transfer relationships between cameras based on the vehicle's real-time motion and to continuously infer the target state in invisible areas. This makes cross-camera correlation results susceptible to local tracking errors, thereby affecting the integrity and reliability of the global target trajectory. Summary of the Invention

[0005] In view of this, embodiments of this application provide an unmanned vehicle cross-camera target tracking method and system that integrates graph neural networks to solve the problems of cross-camera trajectory breakage, identity switching, and low reliability of cross-field association in the prior art.

[0006] A first aspect of this application provides an unmanned vehicle cross-camera target tracking method fused with graph neural networks, comprising: acquiring multi-camera image data, camera calibration data, and vehicle motion data of an unmanned vehicle or unmanned vessel; converting target observation data from each camera to a unified spatiotemporal reference based on the camera calibration data and vehicle motion data; generating local trajectory segments corresponding to each camera based on the target observation data, and determining the trajectory quality representation and field-of-view boundary events of the local trajectory segments; constructing dynamic topology transfer relationships between cameras based on the camera calibration data, vehicle motion data, and field-of-view boundary events, and generating candidate cross-camera association edges between local trajectory segments; constructing a heterogeneous spatiotemporal trajectory graph that fuses target appearance representation, motion continuity representation, camera topology representation, and scene semantic constraint representation using local trajectory segments as trajectory nodes and candidate cross-camera association edges as graph edges; inputting the heterogeneous spatiotemporal trajectory graph into an edge-enhanced graph neural network, iteratively updating the features of the trajectory nodes and graph edges, and generating identity association confidence scores for candidate cross-camera association edges; and connecting local trajectory segments across cameras based on identity association confidence scores and trajectory mutual exclusion constraints to generate a global tracking trajectory for targets around the unmanned vehicle or unmanned vessel.

[0007] A second aspect of this application provides an unmanned vehicle cross-camera target tracking system fused with graph neural networks, comprising: an acquisition module, configured to acquire multi-camera image data, camera calibration data, and vehicle motion data of an unmanned vehicle or unmanned vessel, and convert target observation data from each camera to a unified spatiotemporal reference based on the camera calibration data and vehicle motion data; a determination module, configured to generate local trajectory segments corresponding to each camera based on the target observation data, and determine the trajectory quality characterization and field-of-view boundary events of the local trajectory segments; and a construction module, configured to construct a dynamic topology transfer between cameras based on the camera calibration data, vehicle motion data, and field-of-view boundary events. The system consists of three modules: a fusion module, a fusion module, and a generation module. The fusion module uses local trajectory segments as trajectory nodes and candidate cross-camera association edges as graph edges to construct a heterogeneous spatiotemporal trajectory graph that integrates target appearance representation, motion continuity representation, camera topology representation, and scene semantic constraint representation. The update module inputs the heterogeneous spatiotemporal trajectory graph into an edge-enhanced graph neural network to iteratively update the features of trajectory nodes and graph edges, generating identity association confidence scores for candidate cross-camera association edges. The generation module connects local trajectory segments across cameras based on identity association confidence scores and trajectory mutual exclusion constraints to generate global tracking trajectories for targets around the autonomous vehicle or unmanned vessel.

[0008] The above-described technical solutions adopted in the embodiments of this application can achieve the following beneficial effects: By acquiring multi-camera image data, camera calibration data, and vehicle motion data from unmanned vehicles or unmanned vessels, and based on the camera calibration data and vehicle motion data, the target observation data from each camera is transformed to a unified spatiotemporal reference. Local trajectory segments corresponding to each camera are generated based on the target observation data, and the trajectory quality representation and field-of-view boundary events of the local trajectory segments are determined. Based on the camera calibration data, vehicle motion data, and field-of-view boundary events, dynamic topological transfer relationships between cameras are constructed, and candidate cross-camera association edges between local trajectory segments are generated. Using local trajectory segments as trajectory nodes and candidate cross-camera association edges as graph edges, a heterogeneous spatiotemporal trajectory graph is constructed, fusing target appearance representation, motion continuity representation, camera topological representation, and scene semantic constraint representation. The heterogeneous spatiotemporal trajectory graph is input into an edge-enhanced graph neural network to iteratively update the features of trajectory nodes and graph edges, generating identity association confidence scores for candidate cross-camera association edges. Based on identity association confidence scores and trajectory mutual exclusion constraints, cross-camera connections are made to local trajectory segments to generate global tracking trajectories for targets around the unmanned vehicle or unmanned vessel. This application can improve the accuracy of cross-camera association, reduce the identity switching rate, and enhance trajectory continuity. Attached Figure Description

[0009] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0010] Figure 1 This is a flowchart illustrating the unmanned cross-camera target tracking method based on the fusion of graph neural networks provided in this application embodiment; Figure 2 This is a schematic diagram of the structure of the unmanned cross-camera target tracking system fused with graph neural networks provided in the embodiments of this application; Figure 3 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0011] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0012] Existing autonomous vehicles or unmanned boats typically employ onboard multi-camera systems to perceive surrounding traffic participants such as vehicles, pedestrians, and cyclists. Each camera performs object detection and single-camera tracking within its field of view to obtain local trajectory segments from that camera. When continuous circumferential perception results are needed, current technologies generally perform cross-camera association based on appearance similarity, time intervals, spatial distances, or preset matching thresholds between trajectory segments from different cameras. Some solutions also introduce object re-identification models or general graph association models to improve the ability to match the identity of the same object from different perspectives.

[0013] However, in autonomous driving cross-camera target tracking scenarios, there are differences in installation angles between different cameras, non-overlapping fields of view, blind spots on the vehicle body, changes in imaging scale, and changes in illumination. The same target is prone to sudden changes in appearance, short-term invisibility, occlusion recovery, and trajectory fragmentation during cross-camera transfer. Existing methods often rely on static thresholds or simple trajectory similarity, which makes it difficult to fully utilize camera calibration relationships, real-time vehicle motion status, field of view boundary events, target motion continuity, and semantic constraints of road scenes. This results in cross-camera trajectories being prone to breakage, target identities being prone to switching, and low reliability of cross-field of view association.

[0014] To address the aforementioned issues, this application provides a cross-camera target tracking method for autonomous vehicles that integrates graph neural networks. This method acquires multi-camera image data, camera calibration data, and vehicle motion data from an autonomous vehicle or unmanned vessel. Based on the camera calibration data and vehicle motion data, it transforms the target observation data from each camera to a unified spatiotemporal reference. It then generates local trajectory segments corresponding to each camera based on the target observation data under the unified spatiotemporal reference, and determines the trajectory quality characterization and field-of-view boundary events of these local trajectory segments. Finally, based on the camera calibration data, vehicle motion data, and field-of-view boundary events, it constructs dynamic topological transfer relationships between cameras and generates candidate cross-camera association edges between local trajectory segments.

[0015] Furthermore, this application constructs a heterogeneous spatiotemporal trajectory graph that integrates target appearance representation, motion continuity representation, camera topology representation, and scene semantic constraint representation, using local trajectory segments as trajectory nodes and candidate cross-camera association edges as graph edges. By inputting the heterogeneous spatiotemporal trajectory graph into an edge-enhanced graph neural network, iterative feature updates are performed on the trajectory nodes and graph edges. This enables the cross-camera association process to simultaneously integrate target identity features, camera transfer relationships, vehicle motion compensation relationships, and scene constraint relationships within the graph structure, thereby generating identity association confidence scores for candidate cross-camera association edges. Subsequently, based on the identity association confidence scores and trajectory mutual exclusion constraints, cross-camera connections are made to the local trajectory segments to generate global tracking trajectories for targets around the autonomous vehicle or unmanned vessel.

[0016] Through the above technical solution, this application no longer relies solely on single appearance similarity or fixed spatiotemporal thresholds for cross-camera trajectory matching. Instead, it incorporates camera dynamic topology, field-of-view boundary events, vehicle motion state, and scene semantic constraints into the graph neural network inference process. This can improve the accuracy of cross-camera association, reduce identity switching rate, enhance trajectory continuity, and provide more complete and reliable target tracking results for perception fusion, trajectory prediction, and driving decision-making of unmanned vehicles or unmanned ships.

[0017] It should be noted that the following embodiments of this application are described using a cross-camera target tracking method for autonomous vehicles as an application scenario. However, the technical solution of this application is not limited to autonomous vehicle scenarios, and is still applicable to other types of autonomous driving scenarios. For example, autonomous boats are used for garbage cleaning, weed cutting, and tourism in water areas such as inland rivers, reservoirs, and parks; and autonomous sweepers are used for cleaning closed roads in gardens, parks, and industrial parks. The application scenarios in the following embodiments do not constitute a limitation on the technical solution of this application.

[0018] The technical solution of this application will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0019] Figure 1 This is a flowchart illustrating the unmanned vehicle cross-camera target tracking method based on the fusion of graph neural networks provided in this application embodiment. Figure 1 As shown, the method may specifically include: S101: Acquire multi-camera image data, camera calibration data, and vehicle motion data of unmanned vehicles or unmanned ships, and based on the camera calibration data and vehicle motion data, convert the target observation data under each camera to a unified spatiotemporal reference. S102, Generate local trajectory segments corresponding to each camera based on target observation data, and determine the trajectory quality characterization and field of view boundary events of the local trajectory segments; S103, based on camera calibration data, vehicle motion data and field of view boundary events, constructs dynamic topological transfer relationships between cameras and generates candidate cross-camera association edges between local trajectory segments; S104: Using local trajectory segments as trajectory nodes and candidate cross-camera associated edges as graph edges, a heterogeneous spatiotemporal trajectory graph is constructed that integrates target appearance representation, motion continuity representation, camera topology representation, and scene semantic constraint representation. S105, input the heterogeneous spatiotemporal trajectory graph into the edge-enhanced graph neural network, perform iterative feature updates on trajectory nodes and graph edges, and generate identity association confidence of candidate cross-camera associated edges; S106, based on identity association confidence and trajectory mutual exclusion constraints, connects local trajectory segments across cameras to generate global tracking trajectories for targets around the unmanned vehicle or unmanned vessel.

[0020] In some embodiments, acquiring multi-camera image data, camera calibration data, and vehicle motion data of an unmanned vehicle or unmanned vessel includes: Image acquisition is triggered by multiple cameras distributed around the autonomous vehicle or unmanned boat based on the unified synchronous clock of the vehicle. The acquired frames are bound to the camera calibration data in the preset calibration parameter library according to the camera identification. Vehicle motion data is read synchronously within the time window corresponding to image acquisition, and the same time-series index is added to multi-camera image data, camera calibration data and vehicle motion data to generate multi-source perception data for target observation conversion.

[0021] Specifically, after an autonomous vehicle or unmanned vessel initiates a cross-camera target tracking task, the onboard perception controller first reads the multi-camera acquisition configuration and integrates the front-view camera, left front-side camera, right front-side camera, left-side camera, right-side camera, and rear-view camera into the same synchronous acquisition link. The unified onboard synchronization clock can be provided by the vehicle domain controller or distributed by the onboard time synchronization module based on the vehicle's communication bus. Upon receiving the same synchronization trigger pulse, each camera performs image acquisition according to preset exposure parameters and frame rates, ensuring that the acquired frames from different circumferential angles have a comparable time reference. For cameras with different frame rates or slight trigger delays, the perception controller records a hardware timestamp, frame number, and camera identifier when the acquired frame enters the buffer, and performs time merging of the acquired frames based on the synchronization clock.

[0022] In a specific example, an autonomous vehicle or unmanned boat travels at 30 km / h on an urban road, with the front-view camera, right-front-side camera, and right-side camera all capturing images at 25 frames per second. The current synchronization cycle corresponds to the time sequence index T1058. The front-view camera captures a cyclist located to the right front of the vehicle, the right-front-side camera captures the road boundary and adjacent vehicles in the same direction, and the right-side camera captures the nearby area to the right of the vehicle. The perception controller writes camera identifiers (e.g., front-view, right-front-side, right-side) to each captured frame and reads the corresponding intrinsic, extrinsic, distortion correction, and installation pose parameters from a pre-set calibration parameter library according to the camera identifier, establishing a binding relationship between the captured frames and the camera calibration data. This binding relationship is used not only for subsequent image correction but also for converting target detection boxes, target grounding points, or 3D detection centers to the vehicle's unified coordinate system.

[0023] The pre-set calibration parameter library can be stored in the local storage space of the perception controller and associated with the vehicle's factory calibration data, online calibration data, or maintenance calibration data. When reading camera calibration data, the perception controller first determines the corresponding calibration record based on the camera identifier, and then verifies the version number, validity status, and timestamp of the calibration record. When multiple calibration versions exist for the same camera, the calibration record that matches the current vehicle attitude configuration and installation status is prioritized. For minor installation deviations caused by temperature changes, vibration, or maintenance replacements, the external parameters can be corrected using online calibration results, and the corrected calibration data is bound to the current acquisition frame, thereby avoiding systematic shifts during subsequent target observation conversions.

[0024] Within the time window corresponding to image acquisition, the perception controller synchronously reads vehicle motion data. This vehicle motion data can come from the inertial measurement unit, wheel speed sensors, steering angle sensors, vehicle positioning module, and chassis controller, and includes data characterizing the vehicle's motion state such as vehicle speed, yaw rate, longitudinal acceleration, lateral acceleration, steering angle, and vehicle pose increment. Using the image acquisition time window as a reference, the perception controller performs time interpolation and state alignment on the vehicle motion data, ensuring that the vehicle motion data corresponds to the same temporal index as the multi-camera image data. For example, if the image acquisition time window corresponding to T1058 is 40 milliseconds, within this window the vehicle yaw rate is 3.2 degrees per second and the vehicle speed is 30 kilometers per hour. The perception controller uses this motion state as the unified motion compensation basis for the multi-camera acquisition frames under T1058.

[0025] To generate multi-source sensing data for target observation conversion, the sensing controller writes the same temporal index into multi-camera image data, camera calibration data, and vehicle motion data, and generates data association records according to data source, acquisition time, camera identifier, and synchronization status. This multi-source sensing data can include acquired frame data, frame-level time information, camera calibration data, vehicle motion status, synchronization quality identifier, and data integrity identifier. If a camera experiences frame loss or timestamp anomalies within the current synchronization cycle, the sensing controller marks the corresponding acquired frame as having low synchronization quality and reduces the reliability weight of this data during subsequent target observation conversion and trajectory segment generation.

[0026] Through the above implementation method, multi-camera image data, camera calibration data, and vehicle motion data can be collected, bound, and configured with temporal indexes under a unified synchronous clock, providing a data foundation for subsequently converting target observation data from different cameras to a unified spatiotemporal reference. This embodiment can improve the temporal consistency and spatial convertibility of multi-source sensing data, and reduce cross-camera target association deviations caused by asynchronous acquisition, incorrect calibration references, or mismatched vehicle motion states.

[0027] In some embodiments, based on camera calibration data and vehicle motion data, target observation data from each camera is converted to a unified spatiotemporal reference, including: The frames captured by each camera are time-aligned according to the same time sequence index, and motion compensation is performed on the target observation position at different acquisition times based on vehicle motion data. The camera calibration data bound to the camera identifier is called to transform the compensated target observation position from the corresponding camera coordinate system to the vehicle body unified coordinate system, generating target observation data under a unified spatiotemporal reference.

[0028] Specifically, after generating multi-source sensing data, the sensing controller merges the frames acquired by each camera according to the same time-series index. For frames acquired by the front-view camera, left front-side camera, right front-side camera, side-view camera, and rear-view camera under the same time-series index, the frame-level hardware timestamp is first read, and the time deviation of each acquired frame relative to the reference time is calculated using the reference time corresponding to the unified synchronization clock as the alignment target. When the time deviation is within the allowable synchronization range, the acquired frame is directly included in the current synchronized frame group; when the time deviation exceeds the allowable synchronization range but does not reach the frame loss threshold, time compensation is performed on the target observation position based on adjacent acquired frames and vehicle motion data; when the time deviation reaches the frame loss threshold, a low-confidence flag is added to the acquired frame, and the participation weight of the corresponding observation data is reduced in the subsequent target observation conversion process.

[0029] When performing motion compensation at the target observation position, the perception controller first extracts the target observation position from the target detection results. The target observation position can be the center of the bottom edge of the target detection box, the target ground point, the center of the 3D detection box, or a target spatial point obtained through depth estimation. Then, it reads the vehicle motion data within the same time window, calculates the vehicle's pose change from the actual time of acquisition to the reference time based on the vehicle speed, yaw rate, longitudinal acceleration, and vehicle pose increment, and applies this pose change to the target observation position so that the target observation position at different acquisition times is compensated to the same reference time.

[0030] For example, in a specific instance, when an autonomous vehicle or unmanned boat is making a right turn on an urban road, the frame captured by the right front-side camera is 8 milliseconds later than the frame captured by the front-view camera. The vehicle's yaw rate is 5 degrees per second. Based on the amount of vehicle rotation and displacement within 8 milliseconds, the perception controller performs reverse compensation on the target ground point of the cyclist under the right front-side camera, so that the target ground point has a consistent vehicle attitude reference with the observation position under the front-view camera at the same reference moment.

[0031] After completing time alignment and motion compensation, the perception controller calls the camera calibration data bound to the camera identifier to perform coordinate transformation on the compensated target observation position. For each camera, distortion correction is first performed on the target observation position based on intrinsic parameters and distortion parameters to obtain normalized image coordinates; then, based on extrinsic parameters and installation pose parameters, the spatial observation point corresponding to the normalized image coordinates is transformed from the camera coordinate system to the vehicle body unified coordinate system.

[0032] For cases where the target observation position is at the center of the bottom edge of a two-dimensional detection frame, the target grounding point can be determined by combining the ground constraint plane or road surface height model. For cases where the target observation position is at the center of a three-dimensional detection frame, it can be directly converted into three-dimensional coordinates in a unified vehicle coordinate system based on the extrinsic parameter matrix. Thus, the same target or adjacent targets acquired by different cameras can all express their longitudinal position, lateral position, height position, and corresponding time index in the same vehicle coordinate system.

[0033] In a specific example, the forward-looking camera, under time sequence index T1058, detects a target vehicle located 18.6 meters in front and 1.4 meters to the right in the vehicle coordinate system. The right-front camera, under the same time sequence index, detects the same target vehicle's rear-side region. The perception controller first performs 6 millisecond time compensation on the frames acquired by the right-front camera, then uses the calibration data from the right-front camera to transform the target ground point to the unified vehicle coordinate system, obtaining a position of 18.3 meters in front and 1.6 meters to the right. Because the two observation positions have spatial proximity and temporal consistency under a unified spatiotemporal reference, the subsequent trajectory segment generation process can use them as the basis for multi-view observation of the same target, avoiding false separation caused by camera acquisition delays or coordinate system differences.

[0034] Through the above implementation method, target observation data from each camera can undergo spatiotemporal normalization processing within a unified temporal index, vehicle motion compensation, and a unified vehicle coordinate system. This provides a consistent data foundation for subsequent local trajectory segment generation, dynamic topology transfer relationship construction, and cross-camera association edge generation. This embodiment can improve the spatiotemporal consistency of multi-camera target observation and reduce the correlation deviation caused by vehicle self-motion and camera coordinate differences in cross-camera target tracking.

[0035] In some embodiments, local trajectory segments corresponding to each camera are generated based on target observation data, and trajectory quality characterization and field-of-view boundary events of the local trajectory segments are determined, including: The target observation data is correlated with the target within the camera and the temporal state is estimated to generate local trajectory segments with camera identifier, target category identifier and spatiotemporal state sequence; Based on the observation continuity, state stability, and occlusion association state of local trajectory segments, a trajectory quality characterization is generated. Based on the changes in the start and end positions of local trajectory segments within the corresponding camera's field of view and the proximity status of the boundaries, field of view boundary events are identified.

[0036] Specifically, after obtaining target observation data under a unified spatiotemporal reference, the perception controller performs segmentation processing on the target observation data according to the camera identifier, so that the target detection results of consecutive moments within the same camera enter the corresponding local tracking queue. For any camera, the target observation data of adjacent frames are first matched based on the target category identifier, position coordinates in the unified vehicle coordinate system, image region features, and detection confidence; then, the temporarily missing target observations are compensated by combining motion prediction results to form a continuous target state within the same camera. Target association can adopt a matching method constrained by appearance similarity, spatial distance, velocity direction consistency, and time interval. When multiple candidate observations simultaneously meet the matching conditions, the observation with the smallest residual of the predicted state and consistent with the target category is selected as the update input for the current trajectory.

[0037] In a specific example, the forward-looking camera continuously detects a target vehicle from time index T1058 to T1072. This target vehicle moves from 22.4 meters in front and 0.8 meters to the right to 16.1 meters in front and 1.5 meters to the right in the vehicle's unified coordinate system. Based on the target observation position and vehicle motion compensation results for each frame, the perception controller performs state estimation for the target vehicle, obtaining a spatiotemporal state sequence including position, velocity, direction of motion, time index, and detection confidence. The forward-looking camera identifier, vehicle category identifier, and this spatiotemporal state sequence are then written into the same local trajectory segment. If the detection box confidence decreases at time T1065 due to occlusion by the vehicle in front, the perception controller performs short-time interpolation based on the target states at T1064 and T1066, marking this moment as an occlusion-related state, without directly interrupting the local trajectory segment.

[0038] After generating local trajectory segments, the perception controller further calculates trajectory quality representations. Observation continuity can be determined based on the number of effective observation frames, the interval between missing frames, and the proportion of interpolated frames; state stability can be determined based on position residuals, velocity fluctuations, changes in motion direction, and changes in the scale of the detection box; occlusion association states can be determined based on the duration of occlusion markers, consistency of states before and after occlusion, and matching residuals after observation recovery. For local trajectory segments with longer durations, higher proportions of effective observations, and smoother state changes, a higher confidence level is generated; for local trajectory segments with shorter observation frames, higher proportions of occlusion, or abnormal motion residuals, a lower confidence level is generated, and lower node confidence weights are configured in subsequent heterogeneous spatiotemporal trajectory maps.

[0039] When identifying field-of-view boundary events, the perception controller pre-determines the effective field-of-view area for each camera based on camera calibration data and establishes a boundary proximity region in both the image coordinate system and the vehicle body unified coordinate system. For each local trajectory segment, the positional relationship between the initial and final states and the boundary proximity region is analyzed, as well as the change in the trajectory segment's direction of motion over consecutive time intervals. When the ending position of a local trajectory segment continuously approaches the right field-of-view boundary of the forward-looking camera, and the lateral displacement direction points towards the coverage area of ​​the right front-side camera, it is identified as a right boundary disappearance event. When a local trajectory segment in the right front-side camera appears from its corresponding entry area, and its starting time falls within the dynamic transition time window of the adjacent camera, it is identified as a right front-side boundary entry event. For situations where a target is simultaneously observed by two cameras within an overlapping field-of-view region, overlapping region crossing events can also be identified based on the spatial overlap and temporal overlap relationship of the two local trajectory segments.

[0040] In another specific example, the target cyclist first exhibits continuous lateral movement in the lower right boundary area of ​​the forward-looking camera. Subsequently, a local trajectory segment in the forward-looking camera ends at T2036, while a local trajectory segment in the right front-side camera begins from the left entry area at T2038. Based on the changes in the start and end positions of the two trajectory segments, the boundary proximity state, and the consistency of the movement direction, the perception controller adds a boundary disappearance event to the forward-looking trajectory segment and a boundary entry event to the right front-side trajectory segment. This event information serves as an important basis for the subsequent generation of candidate cross-camera associated edges.

[0041] Through the above implementation method, target observation data can be organized into local trajectory segments with camera identifiers, target category identifiers, and spatiotemporal state sequences. Simultaneously, trajectory quality representations and field-of-view boundary events are generated for each local trajectory segment. This embodiment improves the state integrity and boundary transition identifiability of local trajectory segments, providing reliable input for subsequent dynamic topology transition relationship construction, heterogeneous spatiotemporal trajectory map generation, and cross-camera identity association reasoning.

[0042] In some embodiments, based on camera calibration data, vehicle motion data, and field-of-view boundary events, a dynamic topology transfer relationship between cameras is constructed, and candidate cross-camera association edges between local trajectory segments are generated, including: Based on the camera calibration data, the spatial adjacency relationship between the field of view areas of each camera is determined, and the transfer direction and transfer time window corresponding to the spatial adjacency relationship are dynamically corrected by combining vehicle motion data to generate dynamic topology transfer relationship; Based on dynamic topological transfer relationships and field of view boundary events, spatiotemporal matching is performed on local trajectory segments with associated reachability to generate candidate cross-camera associated edges.

[0043] Specifically, after obtaining the local trajectory segments, trajectory quality representations, and field-of-view boundary events corresponding to each camera, the perception controller first establishes a camera field-of-view model in the vehicle's unified coordinate system based on the camera calibration data. For each camera, based on intrinsic parameters, extrinsic parameters, mounting pose, and effective sensing distance, the visible boundary, entry area, exit area, and overlapping coverage area of ​​the camera in the vehicle's unified coordinate system are calculated. The visible areas of different cameras are then spatially superimposed to determine the spatial adjacency relationships between the front-view camera and the front-side camera, the front-side camera and the side-view camera, and the side-view camera and the rear-side camera. This spatial adjacency relationship is not a fixed, simple adjacent label, but a topological description that includes the exit boundary direction, the entry area range, the overlapping area range, and the non-overlapping blind zone range.

[0044] After determining the spatial adjacency relationship, the perception controller dynamically corrects the transfer direction and transfer time window between cameras based on vehicle motion data. Vehicle motion data includes vehicle speed, yaw rate, steering angle, and vehicle pose increments, reflecting the vehicle's motion state. When the vehicle is traveling in a straight line, the transfer direction between the exit area on the right side of the forward-looking camera and the entrance area on the right front-side camera is primarily determined along a combination of the vehicle's lateral and longitudinal directions. When the vehicle turns right, the coverage relationship between the entrance area of ​​the right front-side camera and the boundary of the forward-looking camera changes. The perception controller corrects this topological transfer direction based on the yaw rate and vehicle pose increments, and adjusts the transfer time window accordingly. If the vehicle speed is high and the target is relatively stationary, the cross-camera transfer time window is compressed; if the target moves in the same direction as the vehicle or is obstructed, the transfer time window is appropriately expanded.

[0045] In a specific example, an autonomous vehicle or unmanned boat is traveling at 32 km / h along the right lane of a road, while a cyclist gradually moves away from the right boundary of the forward-facing camera. The perception controller, based on camera calibration data, determines that the exit area to the right of the forward-facing camera and the entry area to the left of the right-front camera are spatially adjacent. Then, considering the vehicle's current yaw rate of 2.6 degrees per second, its speed of 32 km / h, and the cyclist's relatively low longitudinal speed, the basic transfer time window from the forward-facing camera to the right-front camera is adjusted from 0.12-0.50 seconds to 0.08-0.36 seconds, and the transfer direction is corrected from the front-right area to the right-front near-field area. The resulting dynamic topology transfer relationship reflects the actual field-of-view transfer process that the target may undergo under the current vehicle motion state.

[0046] After generating dynamic topology transition relationships, the perception controller filters locally accessible trajectory segments based on field-of-view boundary events. For locally accessible trajectory segments with boundary disappearance events, the end time, end position, target category, motion direction, and trajectory quality characteristics are first extracted. Then, along the dynamic topology transition relationships, possible successive target cameras and target entry areas are searched. Candidate segments whose start time falls within the transition time window, whose start position is located in the target entry area, whose target category matches, and whose motion direction satisfies the transition constraints are retrieved from the local trajectory segment set of the corresponding camera. For locally accessible trajectory segments with boundary entry events, possible source cameras and exit areas are searched in reverse to form a bidirectionally consistent candidate recall result.

[0047] During the generation of candidate cross-camera association edges, the perception controller calculates edge attributes for each pair of locally accessible trajectory segments. Edge attributes include time interval matching degree, boundary region matching degree, spatial position continuity, motion direction consistency, target category consistency, trajectory quality combination weight, and topology transfer reliability. For the cyclist trajectory segment ending at T2036 in the forward-facing camera and the cyclist trajectory segment starting at T2038 in the right front-facing camera, if the time interval between the two is 0.08 seconds, the ending and starting positions fall into the corresponding exit and entry areas respectively, and the motion direction is pointing towards the right front of the vehicle, then the perception controller generates a candidate cross-camera association edge between the two locally accessible trajectory segments and writes this association edge into the subsequent heterogeneous spatiotemporal trajectory map. For trajectory segments that do not meet the dynamic transfer time window or cross the field of view of unreachable cameras, no candidate cross-camera association edges are generated to reduce irrelevant matching relationships.

[0048] Through the above implementation, the transfer relationships between cameras can be dynamically constructed based on the vehicle's motion state, camera calibration relationships, and field-of-view boundary events. Based on this, candidate cross-camera association edges are generated by filtering locally accessible trajectory segments. This embodiment can reduce the probability of false associations caused by full trajectory matching, improve the consistency between candidate association relationships and the actual field-of-view transfer process, and provide a more reliable graph edge foundation for subsequent cross-camera identity association reasoning using graph neural networks.

[0049] In some embodiments, a heterogeneous spatiotemporal trajectory graph is constructed, which integrates target appearance representation, motion continuity representation, camera topology representation, and scene semantic constraint representation, using local trajectory segments as trajectory nodes and candidate cross-camera associated edges as graph edges. Multidimensional feature encoding is performed on local trajectory segments to generate node attributes corresponding to trajectory nodes, and the credibility weights of trajectory nodes are configured according to the trajectory quality representation. Transition constraint encoding is performed on candidate cross-camera associated edges to generate edge attributes that include associated reachability and scene constraint relationships; A heterogeneous spatiotemporal trajectory graph is constructed based on node attributes, trusted weights, and edge attributes.

[0050] Specifically, after candidate cross-camera association edges are generated, the perception controller uses each local trajectory segment within the sliding time window as trajectory nodes and candidate cross-camera association edges with associative reachability as graph edges to construct a heterogeneous spatiotemporal trajectory graph for cross-camera identity reasoning. For each local trajectory segment, the target appearance representation, spatiotemporal state sequence, target category identifier, camera identifier, field of view boundary events, and trajectory quality representation are extracted first.

[0051] The target appearance representation can be generated by the image region corresponding to the target detection box through a feature extraction network, and the appearance features at multiple times within the same trajectory segment are fused with quality weighting; the spatiotemporal state sequence can include position changes, velocity changes, motion direction changes and time span under the unified coordinate system of the vehicle body; the camera identifier is embedded and encoded to represent the source of the target observation; the field of view boundary event is used to represent the state of the target entering, leaving or crossing the field of view of the corresponding camera.

[0052] During node attribute generation, the perception controller normalizes and encodes the aforementioned multi-dimensional features to form the node attributes of the trajectory nodes. For stable and continuous local trajectory segments, appearance representation and motion continuity representation can be assigned higher encoding weights; for short-term occlusion recovery segments or fragmented segments, the weight of appearance representation can be reduced, while the participation of boundary events, motion prediction, and camera topology-related features can be increased. The reliability weight of the trajectory node is determined based on the trajectory quality representation, specifically by mapping observation continuity, state stability, and occlusion-related states.

[0053] For example, in a specific example, if the target vehicle is continuously observed for 18 frames in the forward-looking camera, with smooth speed changes and a low occlusion ratio, then the trajectory node is configured with a high confidence weight; if the same vehicle only appears for 4 frames in the right front-facing camera and there is partial occlusion, then the trajectory node is configured with a medium confidence weight, and the interference of low-quality observations on global identity judgment is reduced in subsequent graph message transmission.

[0054] For candidate cross-camera associated edges, the perception controller performs transition constraint encoding to generate edge attributes that include associated reachability and scene constraint relationships. Transition constraint encoding can be obtained based on dynamic topological transition relationships, time interval matching degree, spatial location continuity, motion direction consistency, target category consistency, and boundary region matching degree. Scene constraint relationships can be encoded by combining lane regions, drivable areas, roadside areas, and target category motion boundaries.

[0055] For example, in some examples, the cyclist trajectory segment in the front-view camera ends in the exit area on the right, and the cyclist trajectory segment in the right front-view camera starts in the entry area on the left. The time interval between the two is 0.08 seconds. The spatial transfer direction is consistent with the topological relationship of the right front side of the vehicle and is located near the non-motorized vehicle lane. In this case, the corresponding candidate cross-camera associated edge is assigned a higher association reachability code and a higher scene semantic consistency code.

[0056] During the construction of the heterogeneous spatiotemporal trajectory graph, the perception controller organizes trajectory nodes, node attributes, trusted weights, graph edges, and edge attributes into a graph data structure according to time windows. Different edge attributes in the graph can be labeled as appearance similarity relationships, motion continuity relationships, camera topology relationships, and scene semantic constraints, enabling subsequent edge-enhanced graph neural networks to perform differentiated message passing according to different relationship types. For scenarios with multiple similar targets within the same time window, mutually exclusive constraint edges can also be configured between trajectory nodes with spatial or class conflicts to restrict erroneous connections that cannot belong to the same target.

[0057] Through the above implementation methods, local trajectory fragments can be converted into trajectory nodes with multi-dimensional node attributes and reliable weights, and candidate cross-camera association edges can be converted into graph edges with transfer constraints and scene constraints, thereby forming a heterogeneous spatiotemporal trajectory graph that integrates target appearance, motion continuity, camera topology, and scene semantics. This embodiment can improve the graph structure's ability to express real cross-camera transfer relationships, enhance the accuracy of subsequent graph neural network inferences on target identity associations, and reduce the interference of low-quality trajectory fragments on the global tracking trajectory generation process.

[0058] In some embodiments, the process of constructing a heterogeneous spatiotemporal trajectory map further includes: When there is a non-overlapping field of view transfer state between adjacent local trajectory segments corresponding to the field of view boundary event, a virtual node in the blind zone between adjacent trajectory nodes is generated based on the dynamic topology transfer relationship, vehicle motion data and the spatiotemporal state sequence of the local trajectory segments. Based on the predicted motion state and transfer confidence of the virtual nodes in the blind zone, a compensation association edge connecting the virtual nodes in the blind zone and the adjacent trajectory nodes is generated, and the virtual nodes in the blind zone and the compensation association edge are added to the heterogeneous spatiotemporal trajectory graph.

[0059] Specifically, during the construction of the heterogeneous spatiotemporal trajectory map, the perception controller further determines whether there is a non-overlapping field-of-view transition state between adjacent local trajectory segments. A non-overlapping field-of-view transition state can be understood as the target leaving the exit area of ​​one camera and not immediately entering the overlapping observation area of ​​another camera, but being observed again by another camera only after passing through the vehicle structure occlusion area, camera coverage gap, near-vehicle blind spot, or temporary occlusion area.

[0060] The perception controller determines whether there is a transfer interval that cannot be directly observed between two local trajectory segments based on the exit direction, entry direction, start and end time difference, camera identifier, and dynamic topological transfer relationship in the field of view boundary event. If the transfer path is located in a non-overlapping area of ​​the camera's field of view, and the time interval, target category, and motion direction meet the association reachability conditions, then a blind zone virtual node is generated between the two adjacent trajectory nodes.

[0061] When generating virtual nodes in the blind zone, the perception controller first extracts the spatiotemporal state sequence of several moments before the end of the previous local trajectory segment, including the target's position, velocity, direction of motion, detection confidence, and boundary approach state in the vehicle's unified coordinate system; then it extracts the spatiotemporal state sequence of several moments after the start of the next local trajectory segment, including the target's entry position, initial velocity, entry direction, and observation recovery confidence.

[0062] Subsequently, vehicle motion data is used to compensate for the target's relative motion during the invisible transfer, calculating the impact of vehicle pose changes on the target's relative position. Based on dynamic topological transfer relationships, the possible blind zone channels, transfer directions, and time ranges the target may traverse are defined. The node attributes of the blind zone virtual nodes are formed by the predicted position sequence, predicted velocity sequence, blind zone path identifier, invisible duration, and prediction confidence, used to represent the potential motion state of the target during the period when it is not directly observed by the camera in the graph structure.

[0063] In a specific example, when an autonomous vehicle or unmanned boat makes a low-speed right turn, the target cyclist disappears at the right boundary of the forward-looking camera at time T3012 and reappears in the near-vehicle area of ​​the right-side camera at time T3018. Because there is a non-overlapping field of view near the right front fender between the forward-looking and right-side cameras, the perception controller predicts that the cyclist will move along the edge of the vehicle's right front blind spot between T3013 and T3017 based on the lateral movement trend before the end of the forward-looking trajectory segment, the yaw rate during the vehicle's right turn, the position of the right-side camera's entry area, and the dynamic topology transition relationship. This predicts that the cyclist will move along the edge of the vehicle's right front blind spot between T3013 and T3017, and generates a virtual blind spot node. This virtual blind spot node does not correspond to the detected target in the actual captured frame, but rather to an intermediate motion state obtained based on the spatiotemporal state sequence and vehicle motion compensation.

[0064] After generating the virtual node in the blind zone, the perception controller generates compensating association edges based on the predicted motion state and transfer confidence. These compensating association edges include a first compensating association edge from the previous trajectory node to the virtual node in the blind zone, and a second compensating association edge from the virtual node in the blind zone to the next trajectory node. The first compensating association edge is encoded based on the exit position, exit direction, predicted initial velocity, and blind zone entry matching degree; the second compensating association edge is encoded based on the predicted end position, entry position, observation recovery state, and blind zone exit matching degree. If the residual between the predicted motion state and the entry state of the next local trajectory segment is small, the transfer confidence of the corresponding compensating association edge is increased; if the predicted path is inconsistent with the drivable area, roadside area, or target category motion boundary, the participation weight of the compensating association edge is decreased.

[0065] After the perception controller adds blind zone virtual nodes and compensated association edges to the heterogeneous spatiotemporal trajectory graph, the candidate cross-camera association relationship, which originally directly connects two adjacent trajectory nodes, is expanded into a segmented connection relationship through blind zone virtual nodes. During message passing, the edge-enhanced graph neural network can receive motion continuation information of the previous trajectory node through blind zone virtual nodes and pass predicted transfer information in the invisible area to the next trajectory node. This allows cross-camera identity reasoning to no longer rely solely on the appearance similarity between the exit and entry segments, but to make a joint judgment based on motion reachability within the blind zone.

[0066] Through the above implementation, when the target is temporarily invisible due to non-overlapping camera fields of view or vehicle blind spots, the heterogeneous spatiotemporal trajectory map can use virtual nodes in the blind spots to express the potential motion process of the target, and use compensated associative edges to maintain the reasonable connection between adjacent local trajectory segments. This embodiment can enhance the continuity of target association across blind spots, improve the accuracy of identity recovery under non-overlapping fields of view, and reduce trajectory breaks and identity switching caused by the target's temporary invisibility.

[0067] In some embodiments, a heterogeneous spatiotemporal trajectory graph is input into an edge-enhanced graph neural network to iteratively update the features of trajectory nodes and graph edges, generating identity association confidence scores for candidate cross-camera associated edges, including: Latent features are initialized for trajectory nodes and graph edges respectively, generating node latent representations and edge latent representations; The message passing intensity between adjacent trajectory nodes is modulated based on edge implicit representation, and the node implicit representation and edge implicit representation are updated synchronously during the multi-round graph message aggregation process. Based on the updated node implicit representation, edge implicit representation, and corresponding transition constraint encoding, calculate the identity association confidence of candidate cross-camera associated edges.

[0068] Specifically, after the heterogeneous spatiotemporal trajectory graph is constructed, the perception controller encodes and organizes the trajectory nodes, graph edges, node attributes, edge attributes, trust weights, and transition constraints into graph neural network input data. For each trajectory node, the target appearance representation, motion continuity representation, camera identifier embedding, target category identifier, field of view boundary events, and trajectory quality representation are first concatenated and encoded, and then mapped to a fixed-dimensional node implicit representation through a node encoding network.

[0069] In some examples, for each graph edge, the time interval matching degree, spatial location continuity, dynamic topological reachability, boundary region matching degree, scene semantic consistency, and mutual exclusion conflict state are first fused and encoded, and then mapped to an edge latent representation through an edge encoding network. For graph structures containing blind zone virtual nodes and compensated associated edges, the predicted motion state, invisible duration, and transition confidence are also added to the corresponding latent feature initialization process, so that the inferred motion information in the blind zone can participate in subsequent graph reasoning.

[0070] In the multi-round graph message aggregation process, the edge-enhanced graph neural network does not simply perform average aggregation based on the adjacency relationship between trajectory nodes, but rather modulates the message transmission intensity between adjacent trajectory nodes based on the edge implicit representation. Specifically, for any candidate cross-camera association edge, the network calculates a gating weight based on the edge implicit representation and uses this gating weight to control the appearance identity information, motion state information, and boundary transition information transmitted from the previous trajectory node to the next trajectory node. When the candidate edge corresponds to a high degree of matching in the transition time window, strong spatial continuity, and camera topology reachability, the gating weight is increased, and the corresponding node message is enhanced during the aggregation process; when the candidate edge has class conflict, spatial unreachability, or scene semantic inconsistency, the gating weight is decreased, and the corresponding node message is suppressed. Thus, the graph neural network can distinguish between truly reachable associations and weakly correlated interfering associations in each iteration.

[0071] In a specific example, the target vehicle trajectory segment in the forward-looking camera continuously moves to the right front during T1058 to T1072 and disappears at the right boundary; the candidate trajectory segment in the right front-view camera appears at T1074, with its entry position located in the entry area to the left of the right front-view camera. There is a candidate cross-camera association edge between the two trajectory nodes. This edge has high dynamic topological reachability, a time interval of 0.08 seconds, and consistent movement direction, but the appearance representation reliability of the right front-view trajectory segment is low due to partial occlusion.

[0072] In the first round of aggregation, the edge-enhanced graph neural network transmits stable motion continuity representations and boundary vanishing events from the forward-looking trajectory nodes to the right-front trajectory nodes, while reducing the impact of low-quality appearance features. In the second round of aggregation, it combines the mutual exclusion relationships of neighboring target trajectory nodes to suppress candidate edges that form incorrect connections with another adjacent vehicle. In the third round of aggregation, it further strengthens the candidate edges corresponding to the real transfer paths based on the updated node hidden representations and edge hidden representations.

[0073] Node implicit representations and edge implicit representations are updated synchronously after each round of message aggregation. When updating a node implicit representation, weighted messages from adjacent trajectory nodes, blind zone virtual nodes, and mutually exclusive nodes are received, and residual fusion is performed by combining the original trajectory quality weights of the node. When updating an edge implicit representation, the differences in hidden states, motion state residuals, and identity appearances of the two endpoints are reread and fused with the original transition constraint encoding, so that the graph edges can reflect the latest node association states during the iteration process.

[0074] For example, if after multiple rounds of aggregation, the consistency between the forward trajectory node and the right front trajectory node in terms of motion direction, target category, and boundary transfer relationship is further enhanced, then the same target association feature in the corresponding edge implicit representation is enhanced; if another candidate edge is not possible to be true at the same time in the mutual exclusion conflict verification, then the edge implicit representation of the candidate edge is weakened.

[0075] Furthermore, after completing a preset number of graph message aggregation rounds, the identity association output layer calculates the identity association confidence score for each candidate cross-camera association edge based on the updated node hidden representation, edge hidden representation, and transition constraint encoding. During calculation, the hidden state differences between the two trajectory nodes, edge hidden states, dynamic topology reachability parameters, and scene semantic constraint parameters can be input into the association discrimination network, and the output is the confidence score indicating that two local trajectory segments belong to the same target. For segmented connection relationships containing blind zone virtual nodes, the segmented confidence scores from the previous trajectory node to the blind zone virtual node and from the blind zone virtual node to the next trajectory node can be calculated separately, and combined with the blind zone transition confidence score to generate the overall identity association confidence score.

[0076] Through the above implementation, the edge-enhanced graph neural network can simultaneously learn the trajectory node states and cross-camera associated edge states in a heterogeneous spatiotemporal trajectory graph, and dynamically modulate the message passing process using edge implicit representation. This embodiment can improve the accuracy of identity association judgment in scenarios with complex viewpoint changes, occlusion recovery, and non-overlapping field of view transfer, reduce the probability of false connections between adjacent similar targets, and enhance the stability of the global tracking trajectory generation process.

[0077] In some embodiments, based on identity association confidence and trajectory mutual exclusion constraints, local trajectory segments are connected across cameras to generate a global tracking trajectory for targets around the autonomous vehicle or unmanned vessel, including: Candidate cross-camera association edges are filtered based on identity association confidence, and target association edges that meet the association threshold are generated. Conflict checks are performed on the target-related edges based on trajectory mutual exclusion constraints, and cross-camera connection relationships are determined according to trajectory continuity constraints; By merging corresponding local trajectory segments based on cross-camera connection relationships, a global tracking trajectory with a global identity identifier is generated.

[0078] Specifically, after obtaining the identity association confidence scores of each candidate cross-camera association edge, the perception controller first sorts and filters the candidate cross-camera association edges according to a sliding time window. For any candidate cross-camera association edge, if the identity association confidence score is higher than the association threshold matching the target category, trajectory quality level, and camera transfer type, then the candidate cross-camera association edge is determined as a target association edge; if the identity association confidence score is lower than the association threshold, but there are blind spot virtual nodes, boundary event continuity, and high topological reachability between the corresponding trajectory segments, then the candidate cross-camera association edge is treated as an association edge to be confirmed, awaiting further confirmation from subsequent observation data. The association threshold can be set separately for vehicle targets, pedestrian targets, and cyclist targets, or it can be dynamically adjusted according to the transfer difficulty between cameras and the trajectory quality characterization.

[0079] In a specific example, the vehicle trajectory segment numbered F23 in the front-view camera disappears from the right boundary at time T1072, while the vehicle trajectory segment numbered RF11 in the right front-view camera appears from the left entry area at time T1074. The edge-enhancing graph neural network outputs an identity association confidence score of 0.91, which is higher than the vehicle target association threshold of 0.78 in adjacent camera transfer scenarios. Therefore, the candidate cross-camera association edge from F23 to RF11 is identified as the target association edge. At the same time, another vehicle trajectory segment numbered RF12 in the right front-view camera also has an appearance similarity to F23, but its entry position is not within the entry area defined by the dynamic topology transfer relationship. Its identity association confidence score is 0.62, and it is not identified as a target association edge.

[0080] After filtering target association edges, the perception controller performs conflict verification based on trajectory mutual exclusion constraints. Trajectory mutual exclusion constraints limit the number of local trajectory segments that can belong to only one global identity within the same time frame. The same global identity cannot connect to multiple spatially conflicting trajectory segments within the same time frame, and cross-camera connections must satisfy target category consistency, physical motion boundaries, and dynamic topological reachability. When a local trajectory segment connects to multiple target association edges simultaneously, the perception controller compares the identity association confidence, trajectory quality combination weight, motion state residual, and scene semantic consistency of each target association edge. It retains the connection with the highest overall evaluation and marks the remaining target association edges as conflict edges. For cases where confidence levels are close and conflict relationships are not yet clear, multiple hypothetical connection states can be retained, and delayed decisions can be made based on newly generated local trajectory segments in subsequent sliding time windows.

[0081] During the determination of trajectory continuity constraints, the perception controller organizes the target association edges in chronological order, ensuring that the connection directions between local trajectory segments are consistent with the dynamic topology transfer relationship of the camera, the field of view boundary events, and the sequence of motion states. If adjacent local trajectory segments pass through a blind zone virtual node, the predicted motion state corresponding to the blind zone virtual node is added as an intermediate state to the trajectory continuity check to determine whether the exit state, the blind zone predicted state, and the entry state form a continuous motion link.

[0082] For example, when a cyclist moves from the front-view camera to the right-side camera, if the end position of the front-view trajectory segment, the predicted path of the blind spot virtual node, and the start position of the right-side trajectory segment form a continuous lateral movement trend in the vehicle's unified coordinate system, then the cross-camera connection is established; if the intermediate predicted path is inconsistent with the target category's movement boundary, then the priority of this connection is reduced.

[0083] Furthermore, after determining the cross-camera connection relationship, the perception controller merges the corresponding local trajectory segments according to the time index and assigns a global identity identifier to the merged trajectory. During the merging process, the original camera identifier, spatiotemporal state sequence, trajectory quality characterization, and correlation confidence of each local trajectory segment are retained, and cross-camera connection markers, blind spot compensation status, and conflict verification results are added to the global trajectory.

[0084] In some examples, for the global vehicle trajectory formed by F23 and RF11, the system integrates the observation sequence from the front-view camera, the observation sequence from the right front-side camera, and the cross-camera connection relationship between the two into a single global identity, forming a continuous position sequence, velocity sequence, and motion direction sequence of the vehicle in a unified vehicle coordinate system, and provides this global tracking trajectory to the perception fusion, trajectory prediction, and driving decision-making processes.

[0085] Through the above implementation method, candidate cross-camera association edges can be filtered based on identity association confidence, and multi-target conflicts can be resolved through trajectory mutual exclusion constraints and trajectory continuity constraints, thereby generating a global tracking trajectory with a global identity identifier. This embodiment can improve the reliability of cross-camera connection relationships, reduce false associations and identity switching between similar targets, and enhance the continuity of target trajectories around unmanned vehicles or unmanned vessels across the fields of view of multiple cameras.

[0086] The following are system embodiments of this application, which can be used to execute the method embodiments of this application. For details not disclosed in the system embodiments of this application, please refer to the method embodiments of this application.

[0087] Figure 2 This is a schematic diagram of the structure of the unmanned cross-camera target tracking system fused with graph neural networks provided in an embodiment of this application. Figure 2 As shown, the system includes: The acquisition module 201 is used to acquire multi-camera image data, camera calibration data and vehicle motion data of unmanned vehicles or unmanned ships, and convert the target observation data under each camera to a unified spatiotemporal reference based on the camera calibration data and vehicle motion data. The determination module 202 is used to generate local trajectory segments corresponding to each camera based on the target observation data, and to determine the trajectory quality characterization and field of view boundary events of the local trajectory segments; Module 203 is used to construct dynamic topological transfer relationships between cameras based on camera calibration data, vehicle motion data, and field of view boundary events, and to generate candidate cross-camera association edges between local trajectory segments; The fusion module 204 is used to construct a heterogeneous spatiotemporal trajectory graph that integrates target appearance representation, motion continuity representation, camera topology representation, and scene semantic constraint representation, using local trajectory segments as trajectory nodes and candidate cross-camera association edges as graph edges. Update module 205 is used to input the heterogeneous spatiotemporal trajectory graph into the edge-enhanced graph neural network, perform iterative feature updates on trajectory nodes and graph edges, and generate identity association confidence scores for candidate cross-camera associated edges; The generation module 206 is used to connect local trajectory segments across cameras based on identity association confidence and trajectory mutual exclusion constraints to generate a global tracking trajectory of targets around the unmanned vehicle or unmanned vessel.

[0088] In some embodiments, Figure 2The acquisition module 201 triggers multiple cameras distributed around the autonomous vehicle or autonomous vessel to acquire images based on the vehicle's unified synchronization clock. It binds the acquired frames with the camera calibration data in the preset calibration parameter library according to the camera identifier. It synchronously reads vehicle motion data within the time window corresponding to the image acquisition and adds the same time-series index to the multi-camera image data, camera calibration data and vehicle motion data to generate multi-source perception data for target observation conversion.

[0089] In some embodiments, Figure 2 The acquisition module 201 performs time alignment on the frames acquired by each camera according to the same time sequence index, and performs motion compensation on the target observation position at different acquisition times based on vehicle motion data; it calls the camera calibration data bound to the camera identifier, and transforms the compensated target observation position from the corresponding camera coordinate system to the vehicle body unified coordinate system to generate target observation data under a unified spatiotemporal reference.

[0090] In some embodiments, Figure 2 The determination module 202 performs target association and temporal state estimation on the target observation data with the camera, and generates local trajectory segments with camera identifier, target category identifier and spatiotemporal state sequence; based on the observation continuity, state stability and occlusion association state of the local trajectory segments, it generates trajectory quality characterization; and based on the changes in the start and end positions of the local trajectory segments in the corresponding camera field of view and the boundary proximity state, it identifies field of view boundary events.

[0091] In some embodiments, Figure 2 The construction module 203 determines the spatial adjacency relationship between the field of view areas of each camera based on the camera calibration data, and dynamically corrects the transfer direction and transfer time window corresponding to the spatial adjacency relationship by combining vehicle motion data, thereby generating a dynamic topological transfer relationship; based on the dynamic topological transfer relationship and the field of view boundary events, it performs spatiotemporal matching on local trajectory segments with associated reachability, and generates candidate cross-camera associated edges.

[0092] In some embodiments, Figure 2 The fusion module 204 performs multi-dimensional feature encoding on local trajectory segments, generates node attributes corresponding to trajectory nodes, and configures the credibility weights of trajectory nodes according to the trajectory quality characterization; it performs transfer constraint encoding on candidate cross-camera associated edges, generating edge attributes that include associated reachability and scene constraint relationships; and it constructs a heterogeneous spatiotemporal trajectory graph based on node attributes, credibility weights, and edge attributes.

[0093] In some embodiments, Figure 2When there is a non-overlapping field of view transfer state between adjacent local trajectory segments corresponding to the field of view boundary event, the fusion module 204 generates a blind zone virtual node located between adjacent trajectory nodes based on the dynamic topological transfer relationship, vehicle motion data and spatiotemporal state sequence of local trajectory segments; according to the predicted motion state and transfer confidence of the blind zone virtual node, it generates a compensation association edge connecting the blind zone virtual node and the adjacent trajectory node, and adds the blind zone virtual node and the compensation association edge to the heterogeneous spatiotemporal trajectory graph.

[0094] In some embodiments, Figure 2 The update module 205 initializes the latent features of the trajectory nodes and graph edges respectively, generating node latent representations and edge latent representations; modulates the message passing intensity between adjacent trajectory nodes based on the edge latent representations, and synchronously updates the node latent representations and edge latent representations during multi-round graph message aggregation; and calculates the identity association confidence of candidate cross-camera associated edges based on the updated node latent representations, edge latent representations and corresponding transition constraint codes.

[0095] In some embodiments, Figure 2 The generation module 206 filters candidate cross-camera association edges based on identity association confidence and generates target association edges that meet the association threshold; it performs conflict verification on the target association edges based on trajectory mutual exclusion constraints and determines the cross-camera connection relationship according to trajectory continuity constraints; it merges the corresponding local trajectory segments according to the cross-camera connection relationship and generates a global tracking trajectory with a global identity identifier.

[0096] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0097] Figure 3 This is a schematic diagram of the electronic device 3 provided in an embodiment of this application. Figure 3 As shown, the electronic device 3 of this embodiment includes: a processor 301, a memory 302, and a computer program 303 stored in the memory 302 and executable on the processor 301. When the processor 301 executes the computer program 303, it implements the steps in the various method embodiments described above. Alternatively, when the processor 301 executes the computer program 303, it implements the functions of each module / unit in the various system embodiments described above.

[0098] Electronic device 3 can be a desktop computer, laptop, handheld computer, cloud server, or other electronic device. Electronic device 3 may include, but is not limited to, processor 301 and memory 302. Those skilled in the art will understand that... Figure 3This is merely an example of electronic device 3 and does not constitute a limitation on electronic device 3. It may include more or fewer components than shown, or different components.

[0099] The processor 301 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0100] The memory 302 can be an internal storage unit of the electronic device 3, such as a hard disk or memory of the electronic device 3. The memory 302 can also be an external storage device of the electronic device 3, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the electronic device 3. The memory 302 can also include both internal and external storage units of the electronic device 3. The memory 302 is used to store computer programs and other programs and data required by the electronic device.

[0101] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the system can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0102] If integrated modules / units are implemented as software functional units and sold or used as independent products, they can be stored in a readable storage medium (e.g., a computer-readable storage medium). Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program may include computer program code, which may be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable storage medium may include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc.

[0103] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A cross-camera target tracking method for autonomous driving that integrates graph neural networks, characterized in that, include: Acquire multi-camera image data, camera calibration data, and vehicle motion data of unmanned vehicles or unmanned ships, and based on the camera calibration data and vehicle motion data, convert the target observation data under each camera to a unified spatiotemporal reference; Based on the target observation data, local trajectory segments corresponding to each camera are generated, and the trajectory quality characterization and field of view boundary events of the local trajectory segments are determined. Based on the camera calibration data, the vehicle motion data, and the field of view boundary events, a dynamic topological transfer relationship between cameras is constructed, and candidate cross-camera association edges between the local trajectory segments are generated; Using the local trajectory segments as trajectory nodes and the candidate cross-camera associated edges as graph edges, a heterogeneous spatiotemporal trajectory graph is constructed that integrates target appearance representation, motion continuity representation, camera topology representation, and scene semantic constraint representation. The heterogeneous spatiotemporal trajectory graph is input into the edge-enhanced graph neural network, and the trajectory nodes and graph edges are iteratively updated to generate the identity association confidence of the candidate cross-camera associated edges. Based on the identity association confidence and trajectory mutual exclusion constraints, the local trajectory segments are connected across cameras to generate a global tracking trajectory for targets around the unmanned vehicle or unmanned vessel.

2. The method according to claim 1, characterized in that, The acquisition of multi-camera image data, camera calibration data, and vehicle motion data of the unmanned vehicle or unmanned vessel includes: Image acquisition is triggered by multiple cameras distributed around the autonomous vehicle or unmanned boat based on the unified synchronous clock of the vehicle. The acquired frames are bound to the camera calibration data in the preset calibration parameter library according to the camera identification. Vehicle motion data is read synchronously within the time window corresponding to image acquisition, and the same time-series index is added to the multi-camera image data, the camera calibration data, and the vehicle motion data to generate multi-source perception data for target observation conversion.

3. The method according to claim 2, characterized in that, The step of converting target observation data from each camera to a unified spatiotemporal reference based on the camera calibration data and the vehicle motion data includes: The frames captured by each camera are time-aligned according to the same time sequence index, and motion compensation is performed on the target observation position at different acquisition times based on the vehicle motion data. The camera calibration data bound to the camera identifier is invoked to transform the compensated target observation position from the corresponding camera coordinate system to the vehicle body unified coordinate system, thereby generating target observation data under the unified spatiotemporal reference.

4. The method according to claim 1, characterized in that, The step of generating local trajectory segments corresponding to each camera based on the target observation data, and determining the trajectory quality characterization and field-of-view boundary events of the local trajectory segments, includes: The target observation data is correlated with the target within the camera and the temporal state is estimated to generate a local trajectory segment with camera identifier, target category identifier and spatiotemporal state sequence; The trajectory quality characterization is generated based on the observation continuity, state stability, and occlusion association state of the local trajectory segment. The field of view boundary event is identified based on the changes in the start and end positions of the local trajectory segment within the corresponding camera's field of view and the boundary proximity status.

5. The method according to claim 1, characterized in that, The process of constructing dynamic topological transfer relationships between cameras based on the camera calibration data, vehicle motion data, and field-of-view boundary events, and generating candidate cross-camera association edges between the local trajectory segments, includes: The spatial adjacency relationship between the field of view areas of each camera is determined based on the camera calibration data, and the transfer direction and transfer time window corresponding to the spatial adjacency relationship are dynamically corrected by combining the vehicle motion data to generate the dynamic topology transfer relationship. Based on the dynamic topology transfer relationship and the field of view boundary event, spatiotemporal matching is performed on the local trajectory segments with associated reachability to generate the candidate cross-camera associated edges.

6. The method according to claim 1, characterized in that, The construction of a heterogeneous spatiotemporal trajectory graph, which integrates target appearance representation, motion continuity representation, camera topology representation, and scene semantic constraint representation, using the local trajectory segments as trajectory nodes and the candidate cross-camera association edges as graph edges, includes: Multidimensional feature encoding is performed on the local trajectory segment to generate node attributes corresponding to the trajectory node, and the credibility weight of the trajectory node is configured according to the trajectory quality characterization; The candidate cross-camera associated edges are subjected to transfer constraint encoding to generate edge attributes that include associated reachability and scene constraint relationships; The heterogeneous spatiotemporal trajectory graph is constructed based on the node attributes, the trusted weights, and the edge attributes.

7. The method according to claim 6, characterized in that, The construction of the heterogeneous spatiotemporal trajectory map also includes: When there is a non-overlapping field of view transfer state between adjacent local trajectory segments corresponding to the field of view boundary event, a blind zone virtual node is generated between adjacent trajectory nodes based on the dynamic topology transfer relationship, the vehicle motion data and the spatiotemporal state sequence of the local trajectory segment. Based on the predicted motion state and transfer confidence of the virtual node in the blind zone, a compensation association edge connecting the virtual node in the blind zone and the adjacent trajectory node is generated, and the virtual node in the blind zone and the compensation association edge are added to the heterogeneous spatiotemporal trajectory graph.

8. The method according to claim 1, characterized in that, The step of inputting the heterogeneous spatiotemporal trajectory graph into an edge-enhanced graph neural network, iteratively updating the features of the trajectory nodes and the graph edges, and generating the identity association confidence of the candidate cross-camera associated edges includes: The trajectory nodes and the graph edges are initialized with latent features to generate node latent representations and edge latent representations; The message passing intensity between adjacent trajectory nodes is modulated based on the edge implicit representation, and the node implicit representation and the edge implicit representation are updated synchronously during the multi-round graph message aggregation process. Based on the updated node implicit representation, edge implicit representation, and corresponding transition constraint encoding, calculate the identity association confidence of the candidate cross-camera associated edge.

9. The method according to claim 1, characterized in that, The step of connecting the local trajectory segments across cameras based on the identity association confidence and trajectory mutual exclusion constraints to generate a global tracking trajectory for targets around the unmanned vehicle or unmanned vessel includes: The candidate cross-camera association edges are filtered based on the identity association confidence level to generate target association edges that meet the association threshold. Based on the trajectory mutual exclusion constraint, conflict verification is performed on the target associated edges, and cross-camera connection relationships are determined according to the trajectory continuity constraint; Based on the cross-camera connection relationship, the corresponding local trajectory segments are merged to generate the global tracking trajectory with a global identity identifier.

10. An unmanned cross-camera target tracking system integrating graph neural networks, characterized in that, include: The acquisition module is used to acquire multi-camera image data, camera calibration data and vehicle motion data of unmanned vehicles or unmanned ships, and based on the camera calibration data and the vehicle motion data, convert the target observation data under each camera to a unified spatiotemporal reference. The determination module is used to generate local trajectory segments corresponding to each camera based on the target observation data, and to determine the trajectory quality characterization and field of view boundary events of the local trajectory segments; The construction module is used to construct dynamic topological transfer relationships between cameras based on the camera calibration data, the vehicle motion data, and the field of view boundary events, and to generate candidate cross-camera association edges between the local trajectory segments; The fusion module is used to construct a heterogeneous spatiotemporal trajectory graph that integrates target appearance representation, motion continuity representation, camera topology representation, and scene semantic constraint representation, using the local trajectory segments as trajectory nodes and the candidate cross-camera association edges as graph edges. The update module is used to input the heterogeneous spatiotemporal trajectory graph into the edge-enhanced graph neural network, perform iterative feature updates on the trajectory nodes and the graph edges, and generate the identity association confidence of the candidate cross-camera association edges; The generation module is used to connect the local trajectory segments across cameras based on the identity association confidence and trajectory mutual exclusion constraints to generate a global tracking trajectory for targets around the unmanned vehicle or unmanned vessel.

Images