A vehicle conflict early warning method and system from the perspective of a drone

Abstract

The application provides a vehicle conflict early warning method and system from the perspective of a UAV, and relates to the technical field of intelligent transportation. By obtaining historical video images of a UAV in a road area, a future occupancy map and a vehicle center response map are predicted through multi-scale feature and time sequence fusion; a two-dimensional occupancy flow field with direction information is estimated only from the occupancy map; candidate vehicles are repositioned frame by frame, and identity-level continuous trajectory association is not established; the occupancy map and the occupancy flow field are transformed to the ego vehicle coordinate system, and a space-time co-moving body containing occupancy, forward and lateral relative speed channels is constructed; after risk pre-screening, the co-moving body of the selected vehicle is encoded as a continuous occupancy flow token, which is input into a language model together with a task prompt token, and a conflict determination is output by the decision layer, and a conflict explanation conditioned on the determination is output by the explanation layer, to generate early warning information. The application reduces the influence of trajectory fragmentation and identity switching, and improves the early warning advance, physical verifiability and explainability.

Description

Technical Field

[0001] This application relates to the field of intelligent transportation technology, and in particular to a method and system for early warning of vehicle conflicts from the perspective of unmanned aerial vehicles. Background Technology

[0002] Road traffic accidents are characterized by low frequency and high loss, and safety assessments relying solely on accident records require long observation periods. Traffic conflict analysis based on UAV video can simultaneously observe the interaction processes of multiple vehicles over a large area, thus it can be used for road risk assessment, traffic organization evaluation, and proactive safety management. Existing traffic conflict identification methods typically detect and track vehicles first, and then use vehicle trajectories to calculate alternative safety indicators such as collision time, intrusion time, or generalized collision time.

[0003] The aforementioned solutions heavily rely on stable multi-target tracking and identity association. In dense traffic, when vehicles obstruct each other, targets are small, cameras shake slightly, or vehicles rapidly change position, vehicle trajectories are prone to fragmentation, identity switching, and positional instability. Since collision time-based metrics require differentiation or extrapolation of relative speed and vehicle spacing, small positional errors can be amplified into significant speed errors, leading to distorted collision metrics. Improving traditional tracking metrics does not necessarily improve downstream collision detection; therefore, simply replacing the tracker cannot fundamentally solve this problem.

[0004] Another type of approach uses occupancy heatmaps or probabilistic occupancy maps to describe the possible locations of traffic participants. Scalar occupancy representations can reduce the reliance on identity associations, but they typically only preserve spatial amplitude and lose direction of movement, making it difficult to distinguish between approaching and moving vehicles. Some approaches directly use visual language models to analyze traffic images and output natural language descriptions, but distant vehicles in the original images are small, and the models lack independently verifiable kinematic evidence, easily leading to judgments lacking physical grounding.

[0005] Therefore, a technical solution is needed that can preserve the relative motion direction of vehicles from drone images, does not rely on the identity-level association of surrounding vehicles, can identify vehicle conflicts in advance, and output verifiable explanations. Summary of the Invention

[0006] To address the problems of existing UAV traffic conflict recognition, such as strong reliance on continuous vehicle trajectories and identity associations, lack of directional information in scalar occupancy, lack of physical grounding in visual language models, and difficulty in auditing conflict outputs, this invention provides a UAV-based method and system for early warning of vehicle conflicts. The overall process includes video image acquisition, high-resolution future occupancy prediction, occupancy flow estimation, frame-by-frame relocation of candidate vehicles, construction of co-moving bodies from the perspective of the UAV, risk pre-screening, continuous token grounding language inference, and early warning output.

[0007] To achieve the above objectives, the present invention adopts the following technical solution: In a first aspect, the present invention provides a method for early warning of vehicle conflicts from the perspective of an unmanned aerial vehicle (UAV), comprising: S1, acquire the sequence of historical video images of the target road area before the current decision time; S2, perform multi-scale feature extraction and temporal fusion on the historical video image sequence, predict the occupancy map of multiple future moments after the current decision moment while maintaining the spatial resolution of the vehicle target, and obtain the vehicle center response map. S3, input the occupancy map into the occupancy flow estimation network to obtain the two-dimensional velocity vector corresponding to each occupancy grid, forming an occupancy flow field containing motion direction information; S4. Based on the vehicle center response map, candidate vehicles are determined, and the candidate vehicles are locally relocated at each sampling time of the historical video image sequence to obtain the spatial position, heading and speed of the candidate vehicles at the current decision time. However, no identity-level continuous trajectory association is established between the candidate vehicles and surrounding vehicles. S5. For each candidate vehicle, the occupancy map and the occupancy flow field are transformed into a vehicle coordinate system centered on the candidate vehicle and based on the candidate vehicle's heading, and a vehicle-view spatiotemporal co-movement body is constructed, which includes at least the occupancy channel, the forward relative velocity channel and the lateral relative velocity channel. S6, calculate the risk score of the candidate vehicle based on the spatiotemporal co-movement body from the perspective of the vehicle, and select a predetermined number of candidate vehicles based on the risk score; S7, the spatiotemporal co-movement of the selected candidate vehicle from the vehicle's perspective is encoded into multiple continuous occupancy stream tokens, the continuous occupancy stream tokens and the task prompt tokens are input into the trained language model, the decision layer of the language model outputs the vehicle conflict determination, and the interpretation layer based on the vehicle conflict determination outputs the conflict interpretation information. S8, Generate early warning information for vehicle conflict based on the vehicle conflict determination and the conflict interpretation information.

[0008] Optionally, the multi-scale feature extraction and temporal fusion includes: extracting at least four features with different downsampling ratios for each sampled image; unifying the number of channels for the multi-scale features including the feature with the lowest downsampling ratio and upsampling them to the target occupancy resolution; concatenating and convolving the upsampled multi-scale features to obtain a bird's-eye view feature consistent with the target occupancy grid resolution; concatenating the bird's-eye view features from multiple historical moments along the channel dimension, inputting them into a fully convolutional decoder with downsampling paths, skip connections, and upsampling paths, and outputting the occupancy map in parallel through multiple output channels corresponding to different future moments.

[0009] Optionally, the occupancy flow estimation network is a three-dimensional convolutional network that maintains temporal and spatial resolution; the occupancy flow estimation network uses the occupancy grid as a mask and is trained using the endpoint error of the two-dimensional velocity vector, so that each occupancy grid outputs forward velocity components and lateral velocity components; in the online inference stage, the input of the occupancy flow estimation network consists entirely of the occupancy map, without the need to input vehicle identity or vehicle trajectory.

[0010] Optionally, the determination and relocation of candidate vehicles includes: performing local maximum detection on the vehicle center response map; performing subgrid offset correction on the location of the local maximum; selecting the vehicle center closest to the current candidate vehicle position within a predetermined spatial threshold of adjacent sampling times as the relocation position at that sampling time; estimating the heading and vehicle speed of the candidate vehicles based on multiple relocation positions, wherein, within a local time window before and after the current decision time, robust slope estimation is performed on the candidate vehicle position sequence to reduce the impact of single-frame detection position noise on the vehicle speed.

[0011] Optionally, constructing a spatiotemporal co-movement body from the vehicle's perspective includes: translating the occupancy map and the occupancy flow field with the spatial position of the candidate vehicle as the origin; rotating the occupancy map and the occupancy flow field according to the heading of the candidate vehicle, so that the forward direction of the candidate vehicle corresponds to a preset axis of the vehicle coordinate system; subtracting the vehicle's velocity from the occupancy flow field to obtain a relative velocity field; projecting the relative velocity field onto the forward and lateral axes of the vehicle coordinate system; stacking the occupancy channel, the forward relative velocity channel, and the lateral relative velocity channel at multiple time steps within a predetermined spatial range around the candidate vehicle; wherein the forward relative velocity channel and the lateral relative velocity channel are multiplied with the occupancy mask and normalized and limited according to a predetermined velocity scale.

[0012] Optionally, the risk score is output by a convolutional risk scoring network based on the spatiotemporal coherence from the vehicle's perspective; K candidate vehicles are selected in descending order of risk score and input into the language model, and unselected candidate vehicles are maintained and monitored by a real-time risk screening layer and reordered in the next sliding window; where K is set according to the computational budget and the recall rate of the target event.

[0013] Optionally, encoding the spatiotemporal co-movement body from the vehicle's perspective into multiple consecutive occupancy stream tokens includes: extracting the spatiotemporal features of the spatiotemporal co-movement body from the vehicle's perspective using a 3D convolutional encoder; adaptively pooling the spatiotemporal features into a predetermined 3D mesh; flattening each mesh feature of the 3D mesh into a token sequence; mapping each mesh feature to the embedding dimension of the language model through a projection network; and placing the resulting consecutive occupancy stream tokens before the task prompt token.

[0014] Optionally, the language model includes a decision layer and an explanation layer that are separate from each other; the decision layer outputs at least a binary classification judgment of danger and safety or a multi-level risk judgment based on the continuous occupancy stream token; the input prompt of the explanation layer includes the vehicle conflict judgment output by the decision layer, and generates one or more conflict explanation information including at least relative orientation, conflict type, closing speed, expected conflict time, risk cause, handling suggestion, and confidence level; training the language model includes: freezing the basic parameters of the language model, training the low-rank adaptation parameters of at least one attention layer or feedforward layer of the three-dimensional convolutional encoder, projection network, and insertion language model; first training the decision ability with binary risk labels, and then training the explanation ability with rich labels containing multiple explanation fields; when training the explanation layer, the risk label or the output of the decision layer is used as a condition, and the autoregressive language loss is calculated only for the target answer part.

[0015] Optionally, the conflict supervision labels used in training the occupancy prediction network, occupancy flow estimation network, risk scoring network, or language model are obtained in the following ways: for the target vehicle and surrounding vehicles, the directional closure velocity is calculated based on the relative position vector and relative velocity vector; when the directional closure velocity is positive, the generalized collision time is calculated based on the distance after deducting vehicle size and the directional closure velocity; the minimum value of the generalized collision time is taken within the future prediction window, and a conflict label is formed based on one or more thresholds; wherein the conflict supervision labels are only used for training or evaluation, and all inputs in the online inference stage are obtained from the historical video image sequence of the UAV; the vehicle conflict early warning information includes at least one of the following: target vehicle identification, target vehicle position in the image or map, risk level, risk score, relative orientation, conflict type, closure velocity, expected conflict time, risk cause, handling suggestions, and confidence level, and is output through a monitoring terminal, traffic management platform, or data interface.

[0016] Secondly, this application also provides a vehicle conflict early warning system from an unmanned aerial vehicle (UAV) perspective, comprising: The image acquisition module is used to acquire historical video image sequences of the target road area from drones; The occupancy prediction module is used to predict occupancy maps and vehicle center response maps for multiple future moments from the historical video image sequence. The occupancy prediction module includes a multi-scale image backbone network, a feature pyramid network, a high-resolution bird's-eye view feature fusion network, and a fully convolutional future occupancy decoder. Occupancy flow estimation module, used to obtain occupancy flow field with directional information based solely on the occupancy map; The candidate vehicle anchoring module is used to determine candidate vehicles based on the vehicle center response map and relocate them frame by frame to obtain the position, heading and speed of the candidate vehicles, without establishing identity-level continuous trajectory associations between the candidate vehicles and surrounding vehicles. The co-movement body construction module is used to transform the occupancy map and the occupancy flow field to the vehicle coordinate system of the candidate vehicle, forming a spatiotemporal co-movement body from the vehicle's perspective that includes occupancy, forward relative velocity, and lateral relative velocity; The risk pre-screening module is used to sort and select candidate vehicles based on the spatiotemporal coherence from the perspective of the vehicle. The token grounding inference module is used to encode the spatiotemporal co-movement of the selected candidate vehicles from the perspective of the vehicle into multiple consecutive occupancy stream tokens, and generate vehicle conflict determination and conflict interpretation information sequentially through the decision layer of the language model and the interpretation layer based on the decision result. The token grounding inference module includes a three-dimensional convolutional encoder, a projection network, a language model decision layer with low-rank adaptation, and a language model interpretation layer. The early warning output module is used to generate and output early warning information for vehicle conflicts.

[0017] Compared with the prior art, the present invention has the following significant advantages: (1) By saving the velocity direction of each grid through the occupancy stream, the relative closed motion is directly read from the field representation, reducing the impact of the fragmentation of the surrounding vehicle trajectory and identity switching on the conflict judgment; (2) By using high-resolution multi-scale fusion including low downsampling rate features and fully convolutional future occupancy decoding, the positioning information of sparse small vehicles in the UAV image is preserved; (3) By relocating candidate vehicles frame by frame and estimating local robust velocity, only the target vehicle is anchored in real time. There is no need to maintain the long-term identity of multiple targets in the field, which reduces the vulnerability of data association in the structure. (4) By using the three channels of occupancy, forward relative velocity and lateral relative velocity in the vehicle coordinate system, conflict evidence under different road directions and geometric shapes can have a unified meaning; (5) By conducting risk pre-screening, continuous convolutional screening and on-demand language interpretation are implemented in a layered manner, taking into account both the efficiency of continuous monitoring of the entire road and the ability to provide interpretable reports for each vehicle; (6) By directly preserving the spatial and directional structure of the dense field through continuous tokens, and by adopting a method of separating the decision layer and the condition interpretation layer, the interference of rich text generation on risk judgment is reduced, and the consistency between decision and interpretation is improved. Attached Figure Description

[0018] Figure 1 This is a flowchart illustrating the first embodiment of the vehicle conflict early warning method from the perspective of an unmanned aerial vehicle (UAV) in this application; Figure 2 This is a schematic diagram of the vehicle-view co-moving body and continuous token grounding from the perspective of the first embodiment of the vehicle conflict early warning method from the perspective of the UAV in this application; Figure 3 This is a structural block diagram of the first embodiment of the vehicle conflict early warning system from the perspective of an unmanned aerial vehicle (UAV) in this application. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0020] Traditional traffic conflict identification schemes heavily rely on stable multi-target tracking and identity association. In dense traffic, when vehicles occlude each other, or when vehicles rapidly change position, vehicle trajectories are prone to fragmentation, identity switching, and positional jitter, leading to distortion of conflict indicators. Schemes using occupancy heatmaps typically only preserve spatial amplitude while losing motion direction, making it difficult to distinguish between approaching and receding vehicles. When directly using visual language models to analyze traffic images, the small size of distant vehicles in the original images results in a lack of independently verifiable kinematic evidence, easily leading to judgments lacking physical grounding.

[0021] To address this, this application provides a method for early warning of vehicle conflicts from an unmanned aerial vehicle (UAV) perspective, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the vehicle conflict early warning method from the perspective of an unmanned aerial vehicle (UAV) in this application.

[0022] In this embodiment, the vehicle conflict early warning method from the perspective of an unmanned aerial vehicle includes the following steps: Step S1: Obtain the sequence of historical video images of the target road area before the current decision time.

[0023] For ease of understanding, the following explains some key terms in this embodiment: A drone historical video image sequence refers to a series of consecutive image frames continuously captured and recorded by a drone over a target road area, which is used to provide temporal information about traffic scenes.

[0024] An occupancy map is a two-dimensional grid map that represents the probability or state of each grid cell being occupied by traffic participants after a road area has been divided into grids. This map is usually presented from a bird's-eye view and reflects the spatial distribution of vehicles.

[0025] A vehicle center response map is a response intensity map that further highlights the center position of a vehicle, building upon the occupancy map. This map helps to accurately identify and locate individual vehicles within a scene.

[0026] Occupancy flow estimation networks are deep learning network models that take an occupancy map as input and output a two-dimensional velocity vector for each occupancy grid. This network aims to estimate the direction and velocity of movement in each occupancy area within a traffic scene.

[0027] The occupancy flow field refers to the field jointly constructed by the occupancy map and the two-dimensional velocity vector output by the occupancy flow estimation network. This field not only contains spatial occupancy information of vehicles, but also includes the motion direction and velocity information of each occupancy region.

[0028] Candidate vehicles refer to individual vehicles that may pose a conflict risk, initially identified through analysis of vehicle center response maps. These vehicles will be the focus of subsequent risk assessments.

[0029] A vehicle coordinate system is a local coordinate system established with a specific candidate vehicle as the origin and its heading as the reference. Within this coordinate system, the relative motion between the candidate vehicle and its surrounding environment can be analyzed more intuitively.

[0030] The vehicle-centric spatiotemporal dynamic volume refers to a three-dimensional data volume formed by stacking the occupancy map, forward relative velocity channel, and lateral relative velocity channel across multiple time steps in the vehicle coordinate system. This data volume comprehensively reflects the spatiotemporal dynamic information of the environment surrounding the candidate vehicle.

[0031] The risk score is a numerical value calculated by analyzing the spatiotemporal interaction from the perspective of the vehicle, used to quantify the potential risk of a conflict between candidate vehicles. The higher the score, the greater the risk of conflict.

[0032] Continuous occupancy flow tokens refer to the process by which the spatiotemporal dynamics from the vehicle's perspective are processed by an encoder and converted into a series of continuous digital representations with semantic information. These tokens serve as input to the language model, carrying dynamic information about the traffic scene.

[0033] A task hint token is text or encoded information provided along with a continuous occupancy stream token when inputting into a language model, used to guide the language model to perform a specific task.

[0034] A language model is a deep learning model trained on large-scale text data that can understand and generate natural language. In this embodiment, the model is used to analyze traffic scene information and output conflict determination and explanation.

[0035] The decision-making layer refers to the part of the language model responsible for classifying vehicle conflicts into binary categories or determining multi-level risks based on the input information.

[0036] The explanation layer refers to the part of the language model responsible for generating detailed conflict explanation information based on the decision-making layer's judgment. This layer aims to provide understandable descriptions of the specific causes, types, and locations of the conflict.

[0037] Vehicle conflict determination refers to the judgment result of the decision layer of the language model regarding whether a vehicle conflict exists.

[0038] Conflict explanation information refers to the textual information output by the explanation layer of the language model that provides a detailed description of the vehicle conflict determination, such as the relative location, type, and estimated time of the conflict.

[0039] Vehicle conflict early warning information refers to a notification generated based on vehicle conflict determination and interpretation information, used to issue warnings to relevant parties. This information aims to remind drivers or traffic management personnel to take preventive measures.

[0040] An alternative implementation uses a 1280×720 pixel video at 30 frames per second, selecting 10 frames at fixed steps within a 3-second historical interval. The video can first undergo image stabilization, road area cropping, scale calibration, and pixel-to-ground coordinate mapping; scale calibration can be obtained from ground control points, road marking dimensions, UAV pose, or known map information.

[0041] Step S2: Perform multi-scale feature extraction and temporal fusion on the historical video image sequence, predict the occupancy map of multiple future moments after the current decision moment while maintaining the spatial resolution of the vehicle target, and obtain the vehicle center response map.

[0042] In the aforementioned UAV-based vehicle conflict early warning method, multi-scale feature extraction and temporal fusion of historical UAV video image sequences are crucial steps in predicting future occupancy maps. However, effectively integrating visual information at different scales while ensuring no loss of fine spatial resolution of vehicle targets and efficiently generating accurate future occupancy maps presents a key challenge for achieving high-precision early warning.

[0043] To address this, this embodiment further proposes the following multi-scale feature extraction and temporal fusion: extracting at least four features with different downsampling ratios for each sampled image; unifying the number of channels for the multi-scale features, including those with the lowest downsampling ratio, and upsampling them to the target occupancy resolution; concatenating and convolving the upsampled multi-scale features to obtain a bird's-eye view feature consistent with the target occupancy grid resolution; concatenating the bird's-eye view features from multiple historical moments along the channel dimension, inputting them into a fully convolutional decoder with downsampling paths, skip connections, and upsampling paths, and outputting the occupancy map in parallel through multiple output channels corresponding to different future moments.

[0044] Specifically, features at least four different downsampling magnifications are extracted from each sampled image to capture visual information of varying granularity from the original image. For example, features at lower downsampling magnifications (close to the original resolution) preserve the fine structure and edge information of the vehicle, which is crucial for identifying small vehicles or the precise location of vehicles; while features at higher downsampling magnifications capture broader contextual information and scene layout, helping to understand the overall movement trends of the vehicle and its environment. By extracting features at least four different magnifications, sufficiently rich and comprehensive information can be ensured for subsequent processing to handle vehicle targets of different sizes and distances.

[0045] Subsequently, the multi-scale features, including those at the lowest downsampling rate, are unified in number of channels and upsampled to the target occupied resolution. This step aims to adjust feature maps with different downsampling rates to the same dimension and spatial resolution for effective fusion. Unifying the number of channels can be achieved through 1x1 convolution, while upsampling can be performed using methods such as bilinear interpolation, deconvolution, or nearest neighbor interpolation to restore the spatial dimensions of the feature maps to match the target occupied resolution. Figure 1 This ensures consistent resolution. This approach guarantees that features at all scales are comparable before fusion and lays the foundation for subsequent generation of high-resolution occupancy maps.

[0046] Building upon this, the upsampled multi-scale features are concatenated and fused via convolution to obtain a bird's-eye view feature consistent with the grid resolution occupied by the target. The concatenation operation stacks feature maps from different scales along the channel dimension, forming a comprehensive feature map rich in information. Subsequently, convolutional operations fuse the concatenated features, enabling the network to learn how to effectively combine this multi-scale information, thereby generating a bird's-eye view feature representation that includes both local details and global context. This fusion approach helps improve the overall understanding of the scene while maintaining the spatial resolution of the vehicle target.

[0047] Furthermore, the bird's-eye view features from multiple historical moments are concatenated along the channel dimension and input into a fully convolutional decoder with downsampling paths, skip connections, and upsampling paths. By concatenating bird's-eye view features from multiple historical moments along the channel dimension, the network can acquire temporal dynamic information about the vehicle and its environment, thereby better predicting future motion states. The fully convolutional decoder used, such as the U-Net architecture, employs a downsampling path to progressively extract high-level semantic features and expand the receptive field; an upsampling path to progressively restore the spatial resolution of the feature map; and skip connections to directly pass detailed information from the downsampling path to the upsampling path, effectively avoiding the loss of detailed information during downsampling, which is crucial for maintaining the fine spatial resolution of the vehicle target.

[0048] Finally, the occupancy map is output in parallel through multiple output channels corresponding to different future time points. This parallel output mechanism enables the decoder to predict occupancy maps for multiple future time points at once, greatly improving prediction efficiency. Each output channel corresponds to a specific future time point, and the network can learn to generate independent, high-precision occupancy predictions for each future time point, thereby providing comprehensive temporal information for subsequent conflict warning.

[0049] Through the above technical solution, this embodiment effectively integrates multi-scale and temporal information from historical UAV video image sequences. By employing refined multi-scale feature extraction and fusion, combined with a fully convolutional decoder with skip connections, it ensures the preservation of fine spatial resolution of vehicle targets when predicting future occupancy maps, avoiding the detail loss problems that may occur in traditional methods. Simultaneously, by outputting occupancy maps for multiple future timeframes in parallel, prediction efficiency is significantly improved, enabling the system to respond quickly and provide timely conflict warnings. This method not only improves the prediction accuracy of occupancy maps, especially when handling vehicle targets of different sizes and distances, but also provides a more accurate and richer spatiotemporal information foundation for subsequent vehicle conflict determination and interpretation, thereby enhancing the reliability and practicality of the entire warning system.

[0050] Preferably, the image backbone network outputs four levels of features with downsampling ratios of 4, 8, 16, and 32. Features with downsampling ratios of 8, 16, and 32 are fused using a feature pyramid. The high-resolution features with a downsampling ratio of 4 are then subjected to lateral convolution to unify the number of channels. Finally, all levels of features are upsampled to the target resolution and stitched together to obtain high-resolution bird's-eye view features. This processing allows the small vehicle to occupy more grid space in the feature domain, reducing localization collapse caused by coarse attention grids.

[0051] High-resolution bird's-eye view features from multiple historical moments are stitched together along the channel dimension, and the occupancy probabilities for multiple future moments are output in parallel through a fully convolutional encoder-decoder structure. In one optional implementation, the target grid is 180×320, predicting 30 future time steps of 0.1 seconds each; the decoder employs two levels of downsampling and two levels of upsampling, with each level using residual convolutional blocks and skip connections, and finally, multiple output channels correspond to each future moment. The occupancy loss can be the sum of the binary cross-entropy loss and the Dice loss, and spatial summation can be calculated under sufficient floating-point precision.

[0052] Step S3: Input the occupancy map into the occupancy flow estimation network to obtain the two-dimensional velocity vector corresponding to each occupancy grid, forming an occupancy flow field containing motion direction information.

[0053] In some of the embodiments described above, an occupancy map is input into an occupancy flow estimation network to obtain a two-dimensional velocity vector corresponding to each occupancy grid, thereby forming an occupancy flow field containing motion direction information. However, in practical applications, designing an efficient and accurate occupancy flow estimation network that can accurately output fine velocity components for each occupancy grid while maintaining temporal and spatial resolution, and avoiding dependence on complex vehicle identity or trajectory information, is a key challenge for achieving robust early warning.

[0054] To address this, this embodiment further proposes that the occupancy flow estimation network is designed as a three-dimensional convolutional network that maintains temporal and spatial resolution. This network uses an occupancy grid as a mask and is trained using the endpoint errors of the two-dimensional velocity vector, causing each occupancy grid to output forward and lateral velocity components. During the online inference phase, the input to the occupancy flow estimation network consists entirely of the occupancy map, eliminating the need for inputting vehicle identity or vehicle trajectory.

[0055] Specifically, the occupancy flow estimation network is a deep learning model whose core function is to infer the motion vector, or "occupancy flow," of each spatial grid cell from a series of consecutive occupancy maps. This network employs a three-dimensional convolutional structure, meaning it can extract features simultaneously in both spatial (e.g., the width and height of an image) and temporal (e.g., consecutive video frames). By performing convolutional operations in three-dimensional space, the network effectively captures the motion patterns and spatiotemporal relationships of objects across consecutive time steps, rather than just processing spatial information from a single frame. To maintain temporal and spatial resolution, the network is designed to avoid or minimize downsampling operations that could lead to information loss, or employs mechanisms such as upsampling layers and skip connections to recover and fuse features from different levels, ensuring that the output occupancy flow field has the same spatial and temporal granularity as the input occupancy map, thereby guaranteeing the fineness of the motion information. For example, a U-Net or similar encoder-decoder architecture can be used, where the encoder path is responsible for extracting multi-scale spatiotemporal features, while the decoder path is responsible for upsampling these features and restoring them to their original resolution, and using skip connections to directly pass high-resolution features from the encoder path to the decoder path to preserve detail.

[0056] During training, the occupancy flow estimation network uses occupancy grids as masks. This means that when calculating the loss function, the error is only calculated for grids that are actually occupied (i.e., grids with vehicles or other obstacles), while ignoring empty grids. This strategy allows the network to focus its learning on meaningful motion regions, improving training efficiency and model performance. During training, the network uses the endpoint error of the 2D velocity vector as the loss function. Endpoint error is a commonly used optical flow loss function that measures the Euclidean distance between the endpoint of the predicted 2D velocity vector and the endpoint of the ground truth 2D velocity vector. By minimizing this error, the network can accurately learn the motion direction and magnitude of each occupancy grid. Furthermore, the network is designed to output a forward velocity component and a lateral velocity component for each occupancy grid. This means that the network's output layer provides two independent channels for each occupancy grid, representing the forward velocity component and the lateral velocity component perpendicular to the forward direction in a preset reference coordinate system (e.g., aligned with the drone or road direction). This structured velocity component output provides a richer and more easily understood description of the motion state for subsequent conflict risk assessment.

[0057] During the online inference phase, the input to the occupancy flow estimation network consists entirely of the occupancy graph, requiring no input of vehicle identity or trajectory. This means that in actual deployment and operation, the system only needs the occupancy graph generated from the UAV's historical video image sequence as input to directly perform occupancy flow estimation. The network does not rely on any modules that require identifying specific vehicles or tracking their complete trajectories. This design greatly simplifies the system architecture and operation, avoiding reliance on costly and high-latency vehicle identification and trajectory tracking modules, thereby improving the system's real-time performance and robustness.

[0058] By employing the aforementioned technical solution and maintaining temporal and spatial resolution through a 3D convolutional network, the spatiotemporal motion features in the occupancy map sequence can be effectively captured. This ensures that the output occupancy flow field has the same high resolution as the input occupancy map, providing detailed motion information and avoiding information loss due to downsampling, thus improving the accuracy of motion estimation. Combining a training strategy using occupancy grids as masks with endpoint error loss for 2D velocity vectors allows the network to focus on motion estimation of meaningful occupancy regions and accurately learn the forward and lateral velocity components of each grid. This refined velocity component output provides a richer and more accurate description of the motion state for subsequent conflict risk assessment. More importantly, the online inference stage relies solely on the occupancy map as input, completely avoiding reliance on complex vehicle identification or trajectory tracking modules. This not only significantly simplifies the system architecture, reduces computational complexity and latency, but also improves the system's robustness and real-time performance, enabling it to generate occupancy flow fields containing motion direction information more quickly and stably, providing a solid foundation for early warning of vehicle conflicts. In summary, the above technical solutions make the occupancy flow estimation process more efficient, accurate and robust, providing high-quality motion information for subsequent construction of spatiotemporal co-movement bodies from the perspective of the vehicle and risk assessment, and significantly improving the accuracy and real-time performance of early warning of vehicle conflicts from the perspective of UAVs.

[0059] One or both of the historical and predicted occupancy components are input into a 3D convolutional network that maintains temporal and spatial resolution. The output is an occupancy flow field with two channels, representing the first and second velocity components in a predetermined global coordinate system, respectively. During training, mask endpoint errors are calculated only on the occupancy mesh to avoid empty background dominating the loss. The occupancy flow loss function is: (1) In the formula, M represents the occupied mask. F hat Indicates the estimated flow. Indicates training supervision flow, ε To prevent constants with zero denominators, vehicle identity or trajectory is not required during online inference.

[0060] Step S4: Determine candidate vehicles based on the vehicle center response map, and perform local relocation of candidate vehicles at each sampling time of the historical video image sequence to obtain the spatial position, heading and speed of candidate vehicles at the current decision time, without establishing identity-level continuous trajectory association between candidate vehicles and surrounding vehicles.

[0061] In some of the embodiments described above, candidate vehicles are determined based on the vehicle center response map, and local relocalization is performed on the candidate vehicles at each sampling time of the historical video image sequence to obtain their spatial position, heading, and vehicle speed. However, in practical applications, the detection results of the vehicle center response map may contain certain noise or instability. If the heading and vehicle speed are directly estimated based on these raw detection results, the estimation accuracy may decrease, affecting the subsequent conflict warning judgment.

[0062] To address this, this embodiment further proposes a specific method for determining and relocating candidate vehicles. First, local maximum detection is performed on the vehicle center response map to identify the possible center location of the vehicle in the image. This can be achieved by applying the non-maximum suppression (NMS) algorithm to the response map, i.e., within a local region, only the point with the largest response value is retained as the candidate vehicle center. Subsequently, sub-grid offset correction is performed on the locations of the local maxima. Since the response map is typically a discrete grid representation, local maxima may not fall precisely at the grid center. Interpolation methods (e.g., quadratic interpolation or Gaussian fitting) can refine the detected local maximum locations, thereby obtaining sub-grid level precision and improving localization accuracy.

[0063] Within a predetermined spatial threshold between adjacent sampling times, the center of the vehicle closest to the current candidate vehicle's location is selected as the relocation location for that sampling time. This step aims to associate candidate vehicles across different time frames while avoiding complex identity-level trajectory tracking. Specifically, for a candidate vehicle detected at the current time, in the image at the next sampling time, a search is conducted only within a preset spatial range around its current location, and the nearest vehicle center point within that range is used as the candidate vehicle's relocation location at the next time. This local association method effectively handles transient vehicle occlusion or detection instability while avoiding the computational complexity of global trajectory association.

[0064] The heading and vehicle speed of candidate vehicles are estimated based on multiple relocation locations. Once the relocation position sequence of candidate vehicles at multiple consecutive sampling times is obtained, this positional information can be used to infer their motion state. The heading can be obtained by calculating the direction vector between consecutive position points, while the vehicle speed can be estimated by the ratio of position change to time interval. To further improve the robustness of the estimation, robust slope estimation is performed on the candidate vehicle position sequence within a local time window before and after the current decision time. Robust slope estimation methods, such as RANSAC (Random Sample Consensus) or Huber regression, can effectively identify and eliminate outliers or noise points in the position sequence, thus obtaining a more accurate and stable vehicle speed estimate even in the presence of single-frame detection noise, ensuring the reliability of subsequent conflict warnings.

[0065] Through the above technical solutions, this embodiment significantly improves the accuracy and robustness of the vehicle conflict early warning method from the perspective of UAVs. By performing local maximum detection and sub-mesh offset correction on the vehicle center response map, more accurate initial positions of candidate vehicles can be obtained, laying a good foundation for subsequent motion state estimation. Without establishing identity-level continuous trajectory association, local relocalization within a predetermined spatial threshold effectively solves the problem of vehicle association between different frames, while avoiding the computational burden and potential errors caused by complex trajectory tracking. Crucially, by performing robust slope estimation on the candidate vehicle position sequence within a local time window, the influence of single-frame detection noise on the vehicle's speed estimation can be effectively suppressed, ensuring the accuracy of heading and vehicle speed. This precise motion state information serves as a key input for constructing the spatiotemporal co-movement body from the vehicle's perspective, making subsequent risk score calculation and language model conflict determination more reliable, thereby improving the timeliness and accuracy of vehicle conflict early warning and reducing the risk of false alarms and missed alarms.

[0066] Preferably, Theil-Sen slope estimation, Huber regression, median differencing, or other robust estimators are employed within a local window of 0.7 to 1.5 seconds to reduce single-frame position noise. The vehicle heading can be determined a priori by the velocity direction, the major axis of the detection box, or the road direction.

[0067] Step S5: For each candidate vehicle, transform the occupancy map and the occupancy flow field to a vehicle coordinate system centered on the candidate vehicle and with the candidate vehicle's heading as the reference, and construct a vehicle-view spatiotemporal co-movement body that includes at least the occupancy channel, the forward relative velocity channel, and the lateral relative velocity channel.

[0068] In some of the embodiments described above, a method is proposed to provide basic information for vehicle conflict early warning by predicting occupancy maps and occupancy flow fields, and determining candidate vehicles and their motion states. However, directly analyzing this information in the global coordinate system may not intuitively reflect the relative motion relationship between a candidate vehicle and its surrounding environment when assessing the conflict risk of a specific candidate vehicle, thus affecting the accuracy and efficiency of risk assessment.

[0069] To address this, this embodiment further proposes a method for constructing a spatiotemporal co-moving body from a vehicle's perspective, comprising: translating the occupancy map and the occupancy flow field with the spatial position of the candidate vehicle as the origin; rotating the occupancy map and the occupancy flow field according to the heading of the candidate vehicle, so that the forward direction of the candidate vehicle corresponds to a preset axis of the vehicle coordinate system; subtracting the vehicle's velocity from the occupancy flow field to obtain a relative velocity field; projecting the relative velocity field onto the forward and lateral axes of the vehicle coordinate system; stacking the occupancy channel, the forward relative velocity channel, and the lateral relative velocity channel at multiple time steps within a predetermined spatial range around the candidate vehicle; wherein the forward relative velocity channel and the lateral relative velocity channel are multiplied with an occupancy mask and normalized and limited according to a predetermined velocity scale.

[0070] Specifically, translating the occupancy map and the occupancy flow field with the spatial location of the candidate vehicle as the origin aims to transform the environmental information (including the occupancy map and occupancy flow field) in the global coordinate system to a local coordinate system centered on the specific candidate vehicle. This translation operation positions the candidate vehicle itself at the origin of the new coordinate system, thereby unfolding the environmental information "around" the candidate vehicle, facilitating subsequent analysis centered on that vehicle.

[0071] Based on this, the occupancy map and the occupancy flow field are rotated according to the candidate vehicle's heading, so that the candidate vehicle's forward direction corresponds to a preset axis in the vehicle coordinate system. This step, following translation, further rotates the local coordinate system to align it with the candidate vehicle's heading. Typically, the vehicle coordinate system defines a preset axis (e.g., a positive X-axis or positive Y-axis) as the vehicle's forward direction. This rotation allows all environmental information to be located relative to the candidate vehicle's "front," "rear," "left," and "right" positions. This is crucial for vehicle behavior prediction and risk assessment, as vehicle perception and decision-making are typically based on its own orientation.

[0072] To more accurately assess conflict risk, the candidate vehicle's own velocity is subtracted from the occupancy flow field to obtain a relative velocity field. The occupancy flow field represents the absolute velocity of all occupancy grids in the environment. By subtracting the candidate vehicle's own velocity from the absolute velocity of each grid, a relative velocity field is obtained, which visually represents how other objects in the environment "approach" or "move away" from the candidate vehicle.

[0073] The relative velocity field is projected onto the forward and lateral axes of the vehicle coordinate system. The relative velocity field is a two-dimensional vector field. To construct the "forward relative velocity channel" and the "lateral relative velocity channel," each relative velocity vector needs to be decomposed into a component along the forward axis of the vehicle coordinate system (usually the vehicle's forward direction) and a component along the lateral axis (perpendicular to the forward direction).

[0074] Within a predetermined spatial range around the candidate vehicle, the occupancy channel, the forward relative velocity channel, and the lateral relative velocity channel are stacked at multiple time steps to construct a spatiotemporal co-movement body from the vehicle's perspective. This step integrates different types of information (occupancy, forward relative velocity, and lateral relative velocity) in both spatial and temporal dimensions. By extracting information within the "predetermined spatial range" around the candidate vehicle, the focus can be placed on the local environment directly related to the vehicle. By "stacking at multiple time steps," a temporal dimension is introduced, allowing the co-movement body to contain not only information from the current moment but also predicted information from multiple future moments, thus forming a three-dimensional (spatial X, spatial Y, temporal) data volume, where each "pixel" or "voxel" contains occupancy status and relative velocity information. A rectangular or circular region centered on the candidate vehicle can be defined as the predetermined spatial range. For each time step (future moment), the occupancy map, forward relative velocity map, and lateral relative velocity map within this range are stacked as independent channels. This ultimately forms a shape with (spatial X dimension, spatial Y dimension, number of time steps). The tensor of the number of channels is the spatiotemporal co-movement body from the perspective of the vehicle.

[0075] The forward and lateral relative velocity channels are multiplied by the occupancy mask and then normalized and clipped according to a predetermined velocity scale. Multiplying by the occupancy mask ensures that velocity information exists only in occupied grids. If a grid is empty (unoccupied), its velocity information is meaningless. By multiplying by the occupancy mask, velocity information in empty regions can be effectively "zeroed out," allowing the model to focus only on the velocity in regions where objects exist. The occupancy mask can be a binary map where occupied grids are 1 and empty grids are 0. Each element of the forward and lateral relative velocity channels is multiplied by the corresponding element of the occupancy mask. Normalization adjusts the velocity values ​​to a specific range (e.g., 0 to 1 or -1 to 1) to suit the input requirements of the neural network and prevent some velocity values ​​from being too large or too small, leading to training instability. Clipping restricts the velocity values ​​to a reasonable physical range, removing outliers and further improving the robustness of the data. Normalization can be achieved through linear mapping or the Sigmoid / Tanh function, mapping velocity values ​​to a preset range. Limiting, on the other hand, forces values ​​that exceed the maximum / minimum speed threshold to be set to the threshold before or after normalization.

[0076] The aforementioned technical solution accurately transforms the occupancy map and occupancy flow field in the global coordinate system to a vehicle-centric coordinate system centered on the candidate vehicle and based on its heading, and further calculates the relative velocity field. This transformation allows subsequent risk assessment to be performed directly based on the candidate vehicle's own perspective and relative motion state, greatly improving the intuitiveness and accuracy of risk assessment. In particular, by constructing a spatiotemporal co-movement body from the vehicle's perspective that includes occupancy channels, forward relative velocity channels, and lateral relative velocity channels, and by performing occupancy mask multiplication, normalization, and amplitude limiting on the velocity channels, the effectiveness and robustness of the input information are ensured. This enables the model to more accurately capture potential conflict situations, thus laying a solid foundation for generating reliable early warning information for vehicle conflicts.

[0077] like Figure 2 As shown, for candidate vehicle i, the global occupancy map and occupancy flow field are translated to the candidate vehicle's position and rotated according to the candidate vehicle's heading so that the vehicle's forward direction corresponds to the longitudinal axis of its own coordinate system. The flow field is then subtracted by the candidate vehicle's own velocity v. i Then project it onto the forward unit vector e of the vehicle. f and the rightward unit vector e r The forward relative velocity v is obtained. f and lateral relative velocity v r : (2) A fixed spatial range is cropped around the candidate vehicle, and the data are stacked over time to obtain the spatiotemporal co-movement body of the candidate vehicle from its own perspective. Three channels are defined: an occupancy channel, a forward relative velocity channel, and a lateral relative velocity channel; the relative velocity channel is preferably multiplied by the occupancy mask. An alternative implementation uses 24 time steps, an 80×80 grid, and a spatial radius of approximately 36 meters, and normalizes and limits the velocity according to a predetermined velocity scale.

[0078] Step S6: Calculate the risk score of the candidate vehicle based on the spatiotemporal coherence from the perspective of the vehicle, and select a predetermined number of candidate vehicles based on the risk score.

[0079] In UAV-based vehicle conflict early warning methods, occupancy maps and vehicle center response maps for multiple future moments can be obtained by processing historical video image sequences, and a spatiotemporal co-movement model from the vehicle's perspective can be constructed for each candidate vehicle. However, in real-world applications, there may be a large number of candidate vehicles. Performing detailed analysis on all candidate vehicles (e.g., conflict determination and interpretation using language models) would pose significant computational resource consumption and real-time challenges. Therefore, efficiently selecting the vehicles with the highest potential conflict risk from numerous candidate vehicles and prioritizing them is crucial to ensuring the real-time performance and effectiveness of the early warning system.

[0080] In response, this embodiment further proposes that in step S6, the risk score is output by the convolutional risk scoring network based on the spatiotemporal coherence from the vehicle's perspective; K candidate vehicles are selected in descending order of risk score and input into the language model; unselected candidate vehicles are maintained and monitored by the real-time risk screening layer and reordered in the next sliding window; where K is set according to the computational budget and the target event recall rate.

[0081] Specifically, the risk score is output by a convolutional risk scoring network based on the spatiotemporal coherence from the vehicle's perspective. This convolutional risk scoring network is a specially designed neural network model, typically employing a convolutional neural network (CNN) architecture. Its function is to receive complex spatiotemporal data as input and output a single, quantified risk value. This network can automatically learn and extract key risk patterns in the spatiotemporal coherence from the vehicle's perspective through its convolutional and pooling layers, such as vehicle density, rapid changes in relative speed, and potential collision paths. Through multi-layer convolutional operations, the network can capture risk features at different scales and levels of abstraction, ultimately mapping these features into a continuous risk score through fully connected layers or global average pooling layers. This score intuitively reflects the probability or severity of a collision involving the current candidate vehicle. The spatiotemporal coherence from the vehicle's perspective is a multi-channel 3D data structure constructed in step S5. Centered on the candidate vehicle and based on its heading, it includes occupancy channels, forward relative speed channels, and lateral relative speed channels. This representation integrates the geometric information (occupancy) and dynamic information (relative speed) of the vehicle's surrounding environment into a unified, vehicle-centric framework that also includes a temporal dimension. Therefore, it provides rich and structured input to convolutional risk scoring networks, enabling them to comprehensively assess the potential conflict risks faced by candidate vehicles.

[0082] After obtaining the risk scores of all candidate vehicles, the system selects K candidate vehicles in descending order of risk score and inputs them into the language model. Specifically, the system sorts these scores from highest to lowest. Then, the K candidate vehicles with the highest risk scores are selected as the objects requiring further detailed analysis, and their corresponding self-vehicle perspective spatiotemporal co-movement bodies are input into the subsequent language model. This selection mechanism ensures that computational resources are preferentially allocated to the vehicles most likely to cause conflict, thereby improving the efficiency of the entire early warning system and the response speed to high-risk events. K is a configurable parameter, and its value directly affects the system's computational burden and the coverage of potential conflicts.

[0083] Meanwhile, unselected candidate vehicles are continuously monitored by the real-time risk screening layer and reordered in the next sliding window. This real-time risk screening layer is a lightweight monitoring mechanism that continuously tracks candidate vehicles with low risk scores that are not selected for detailed analysis in the language model. This screening layer may employ simpler models or heuristic rules than the language model to update the risk status of these vehicles in real time with lower computational cost. Its purpose is to prevent potentially risky vehicles from being missed due to limitations in the initial risk assessment or dynamic environmental changes. The early warning system typically operates in a sliding window manner, meaning that at each new decision point, data is reacquired and the entire early warning process is executed. For candidate vehicles not selected in the current window, the real-time risk screening layer continuously monitors their status, and these vehicles will participate again in the risk score calculation and sorting process when the next sliding window arrives. This means that even if a vehicle has a low risk at a certain moment, its risk score may rise over time and with changes in the environment, thus being selected for detailed analysis in subsequent windows, ensuring the system's dynamic adaptability and comprehensiveness.

[0084] Here, K is set based on the computational budget and the target event recall rate. Determining the parameter K is a trade-off process. The computational budget refers to the available computing resources of the system, such as the processing power of CPU / GPU, memory size, etc. The larger the value of K, the more vehicles need to be input into the language model, and the greater the computational load. The target event recall rate refers to the proportion of potentially risky vehicles that the system can successfully detect. The larger the value of K, the more potentially risky vehicles the system covers, and theoretically the higher the recall rate. Therefore, the setting of K needs to be optimized based on the hardware limitations of the actual deployment environment and the requirements for early warning performance (especially avoiding false negatives) to maximize the capture of potential conflict events with limited computing resources.

[0085] By introducing a convolutional risk scoring network through the aforementioned technical solution, this method can accurately quantify the potential conflict risk of each candidate vehicle based on the rich spatiotemporal information of the vehicle's spatiotemporal co-movement, overcoming the limitations of simple threshold or rule-based judgments. Based on this risk score, the system can select the K most dangerous candidate vehicles in descending order and prioritize their input into a computationally resource-intensive language model for detailed conflict determination and interpretation. This significantly optimizes the allocation of computational resources, avoids the enormous computational burden caused by performing equally complex analysis on all vehicles, and effectively improves the real-time performance of the early warning system. Simultaneously, for candidate vehicles that are not selected, continuous monitoring is performed through a real-time risk screening layer, and they are re-sorted in the next sliding window. This ensures that even in dynamically changing environments, potentially risky vehicles are not overlooked for extended periods, thereby maximizing the recall rate and comprehensiveness of early warnings for various conflict events while maintaining system efficiency. This mechanism, with a limited computational budget, achieves precise focusing on high-risk events and continuous attention to low-risk events, providing an efficient and robust solution for early warning of vehicle conflicts from a UAV perspective.

[0086] In practice, the co-movement data of each candidate vehicle is input into a convolutional risk scoring network to obtain continuous risk scores. The top K candidate vehicles are selected in descending order of risk scores to enter the language inference layer. K can be 1 to 20, preferably 5 to 10, and can also be dynamically adjusted according to GPU load, road traffic, warning and recall targets, or risk thresholds. For vehicles that do not enter the language inference layer, the convolutional risk scoring network can still continuously output real-time scores and reorder them in the next sliding window.

[0087] Step S7: The spatiotemporal co-movement of the selected candidate vehicles from the vehicle's perspective is encoded into multiple continuous occupancy stream tokens. The continuous occupancy stream tokens and the task prompt tokens are input into the trained language model. The decision layer of the language model outputs the vehicle conflict determination, and the interpretation layer based on the vehicle conflict determination outputs the conflict interpretation information.

[0088] In some of the embodiments described above, a method for constructing a spatiotemporal coherent entity from a vehicle's perspective for vehicle conflict early warning was proposed. However, there are certain technical challenges in effectively inputting this complex, multi-dimensional spatiotemporal coherent entity data and having it understood and processed by a language model. Language models typically use discrete token sequences as input; directly inputting the raw spatiotemporal coherent entity data may lead to information distortion, loss of key spatiotemporal correlations, or failure to fully utilize the powerful reasoning and interpretive capabilities of the language model, thereby affecting the accuracy of conflict determination and the richness of interpretive information.

[0089] To address this, this embodiment further proposes a method for encoding a vehicle-view spatiotemporal co-movement body into multiple consecutive occupancy stream tokens. The method includes: extracting the spatiotemporal features of the vehicle-view spatiotemporal co-movement body using a 3D convolutional encoder; adaptively pooling the spatiotemporal features into a predetermined 3D mesh; flattening each mesh feature of the 3D mesh into a token sequence; mapping each mesh feature to the embedding dimension of the language model through a projection network; and prepending the obtained consecutive occupancy stream tokens to the task prompt token.

[0090] Specifically, a 3D convolutional encoder is employed to extract the spatiotemporal features of the vehicle-view spatiotemporal co-movement body. A 3D convolutional encoder is a deep learning model specifically designed to process data with spatial and temporal dimensions. It captures local features by sliding convolutional kernels across three-dimensional space (e.g., x, y, and time axes) and stacking multiple convolutional and pooling layers to extract higher-level abstract features. For the vehicle-view spatiotemporal co-movement body, this encoder effectively captures spatiotemporal correlation information such as changes in occupancy of the vehicle's surrounding environment and the dynamic evolution of the relative velocity field, thereby compressing the raw, high-dimensional co-movement body data into a more representative and semantically meaningful spatiotemporal feature representation. Its implementation may include a series of 3D convolutional layers, activation functions (such as ReLU), batch normalization layers, and 3D pooling layers (such as max pooling or average pooling) to progressively reduce the spatial and temporal resolution of the feature maps while increasing the number of feature channels, thereby extracting rich spatiotemporal features.

[0091] Subsequently, the spatiotemporal features are adaptively pooled into a predetermined 3D mesh. Adaptive pooling is a flexible pooling operation that does not rely on a fixed pooling window size but dynamically adjusts the pooling region according to the desired output size, thereby converting input feature maps of arbitrary size into fixed-size outputs. Here, the spatiotemporal features extracted by the 3D convolutional encoder are adaptively pooled into a predetermined 3D mesh, aiming to unify spatiotemporal features generated by different input sizes or different feature extraction stages into a standardized 3D mesh representation. This standardization is crucial for the subsequent conversion of features into token sequences, as it ensures the consistency of the length and structure of the token sequences received by the language model, regardless of the specific size of the original vehicle-view spatiotemporal co-moving body.

[0092] Based on this, the features of each grid in the 3D mesh are flattened into a token sequence. The flattening operation is the process of converting a multidimensional data structure (such as a 3D mesh) into a one-dimensional sequence. Here, the feature vector represented by each grid in the predetermined 3D mesh is flattened and arranged in a specific order (e.g., from left to right, from bottom to top, from front to back) to form a continuous token sequence. Each "token" can be understood as a spatiotemporal feature representation of a specific region in the 3D mesh. This transformation is necessary because language models typically use serialized tokens as input; by flattening the grid features, complex spatiotemporal information can be transformed into a linear structure that the language model can process.

[0093] Furthermore, a projection network maps each grid feature to the embedding dimension of the language model. The projection network typically consists of one or more fully connected layers (linear layers), whose function is to transform the dimension of the input features to the target dimension. Here, the projection network is used to map the flattened grid features (i.e., the feature vector of each token) to a dimension compatible with the embedding layers within the language model. When processing input tokens, the language model first converts each token into a fixed-dimensional embedding vector, which is the semantic representation learned by the model. Dimension matching through the projection network ensures that tokens generated by the spatiotemporal co-movement from the vehicle's perspective can be correctly recognized and processed by the language model, thus integrating them into the semantic space of the language model and providing effective input for subsequent attention mechanisms and inference processes.

[0094] Simultaneously, the obtained continuous occupancy flow token is placed before the task prompt token. The task prompt token is a pre-defined sequence of text or special symbols used to guide the language model to perform a specific task (such as conflict determination). The continuous occupancy flow token is a sequence of tokens generated by the above encoding process, representing spatiotemporal co-movement information from the vehicle's perspective. Placing the continuous occupancy flow token before the task prompt token means that when inputting to the language model, visual-spatiotemporal information about environmental dynamics and vehicle interaction is provided first, followed by specific task instructions. This pre-positioning method helps the language model fully understand the context and key features of the current scene before receiving task instructions, thereby better focusing on reasoning related to conflict warning and improving the accuracy of decision-making and the relevance of interpretation.

[0095] Through the above technical solution, this embodiment provides an efficient and semantically rich mechanism to transform complex spatiotemporal dynamic data from the vehicle's perspective into an input form that the language model can understand. The 3D convolutional encoder can deeply mine the dynamic interaction features inherent in the spatiotemporal dynamic data, adaptive pooling ensures the standardization of feature representations, and the flattening and projection networks seamlessly bridge these high-dimensional features to the embedding space of the language model. In particular, by placing the continuous occupancy stream token before the task prompt token, the language model fully perceives and understands the spatiotemporal dynamics of the vehicle's surrounding environment before receiving task instructions, thereby enabling more accurate vehicle conflict determination and generating more insightful and targeted conflict explanation information. This encoding method effectively solves the problems of information loss and low processing efficiency faced by directly inputting multidimensional data into the language model, significantly improving the language model's ability to reason and make decisions in complex traffic scenarios.

[0096] An alternative implementation pools the features into a 4×2×2 grid, forming 16 consecutive tokens.

[0097] The continuous occupancy stream token is placed before the task prompt token. The language model first outputs a binary classification of hazard / safety, a multi-class classification of hazard / warning / attention / safety, or a continuous risk confidence level at the decision layer; then, this decision is written into the explanation prompt, and the explanation layer generates a structured explanation under this decision condition. Explanation fields may include target location, type of conflict in the same lane or adjacent lane, approach speed, estimated conflict time, risk attribution, handling suggestions, and confidence level. Since the explanation layer no longer simultaneously undertakes the main decision optimization objective, it can reduce the phenomenon of training the model biased towards the safety category due to complex text labels, and structurally ensure the consistency between decision fields and explanation fields.

[0098] Step S8: Generate early warning information for vehicle conflicts based on the vehicle conflict determination and the conflict interpretation information.

[0099] The server associates the risk assessment and interpretation fields with the candidate vehicle's position in the original map, overlays warning markers on the monitoring interface, or outputs them through message queues, HTTP interfaces, or traffic management platform interfaces. Warning information can be used for manual review, road conflict statistics, on-site guidance, inspection report generation, or traffic organization optimization, but does not directly replace vehicle control decisions.

[0100] Furthermore, this embodiment illustrates the conflict supervision labeling and training method. During the training phase, conflict supervision labels can be generated using data containing real vehicle trajectories, but during the online inference phase, real trajectories or real future states are not input. For the target vehicle i and surrounding vehicles j, let their relative positions be... Δp equal p j reduce p i The relative speed is Δvequal v j reduce v i The directional closure velocity is defined as: (3) When the directional closing velocity is greater than zero, the two vehicles are approaching each other along the line connecting them; the generalized collision time, after deducting the equivalent dimensions of the two vehicles, can be expressed as: (4) In the formula, r i , r j , respectively, are the equivalent size radii of the target vehicle i and the surrounding vehicles j. Vehicle pairs that are unlikely to have spatial interaction can be screened out under the constraints of lateral distance threshold or road reachability area, and the generalized collision time of all effective vehicle pairs is minimized within the future prediction window; the samples are divided into binary or multi-level risk labels according to a threshold, which can be calibrated by management regulations, historical conflict data or manual risk judgment.

[0101] The occupancy prediction network, occupancy flow estimation network, risk scoring network, and language model can be trained in stages. The language model preferably has its basic parameters and visual tower frozen, training only the 3D convolutional encoder, projection network, and low-rank fitting parameters. The preferred training order is to first train the binary risk classification for short answers, followed by training the explanatory fields; during explanatory training, the language loss is calculated only for the target answer token, conditioned on the true risk judgment or decision-making layer prediction. A warm start can be performed using a converged occupancy flow encoder or model parameters with a similar input distribution.

[0102] This embodiment uses a near-stationary UAV video verification scheme for highways. The input video is 1280×720 pixels, with a historical period of approximately 3 seconds. Ten sampling moments are extracted from the historical frames, and the occupancy map is output for the next 30 steps of 0.1 seconds. The high-resolution bird's-eye view grid is 180×320, with vehicles occupying approximately 1.3% of the total grid. The occupancy flow estimation network recovers the two-dimensional velocity field solely based on occupancy. In the experimental data, a mean absolute error of 0.34 m / s, a rank correlation of 0.973, and a 98.5% approximation / departure sign accuracy are obtained for the teacher-defined directional closure velocity, indicating that occupancy flow can serve as a physical risk carrier independent of the identity association of surrounding vehicles. Under the same visual perception front-end and the same evaluation pool, the occupancy flow continuous token scheme of this embodiment achieved a balance accuracy of 0.765, a ROC-AUC of 0.821, and a PR-AUC of 0.804 on the validation set, and a balance accuracy of 0.732 and a ROC-AUC of 0.780 on the one-time test evaluation. The balance accuracy of the rule-based scheme using video trajectory to calculate the directional generalized collision time was approximately 0.580, and that of the trajectory sequence network was approximately 0.687. In the event-level experiment, with a 10% false alarm budget for safe vehicles, the proportion of at least one warning within 3 seconds before a conflict event occurred was 69%, and the median earliest warning time for the warned event was 2.28 seconds.

[0103] The present invention also provides the following alternative solutions: (1) The UAV can be replaced by a fixed high-position camera, a tethered UAV or other aerial platform that can provide a road top view image; when the viewpoint changes, the unified road coordinates can be obtained first through homography, three-dimensional projection or learning-based bird's-eye view transformation. (2) The image backbone network can adopt a convolutional network, a visual Transformer or a hybrid network; the future occupancy decoder can adopt two-dimensional convolution, three-dimensional convolution, circular convolution or diffusion prediction, as long as the output maintains the resolution required for vehicle spatial positioning. (3) The occupancy flow can be estimated from historical occupancy, future predicted occupancy or a combination of both; the speed supervision can come from real trajectory, simulation data, optical flow, adjacent occupancy matching or self-supervised consistency constraints. (4) The candidate vehicle center can be obtained by an independent detector, an instance segmenter, an occupancy center response head or a query-based detection head; frame-by-frame relocalization can adopt nearest neighbor gating, local correlation matching or short-term template matching, but it is not required to establish a stable long-term identity for all road users. (5) The number of consecutive occupancy stream tokens can be from 4 to 256; the encoder can use 3D convolution, spatiotemporal Transformer, state space network or a combination thereof; the language model can be a plain text generation model or a multimodal model with visual input capability, and the consecutive tokens are directly entered into its embedding sequence. (6) The decision layer can output binary classification, multi-class classification, regression risk scores or event probabilities; the interpretation layer can output natural language, JSON, key-value pairs or constrained syntax, and the field range and decision consistency can be verified by the rule validator.

[0104] This embodiment preserves vehicle motion direction information by introducing an occupancy flow field and employs a candidate vehicle local relocation mechanism that does not rely on identity-level continuous trajectory association. This effectively avoids the conflict judgment distortion caused by trajectory fragmentation and identity switching in complex traffic scenarios, which is common in traditional solutions. By constructing a spatiotemporal co-movement body from the vehicle's perspective and combining it with a language model for conflict determination and interpretation, potential vehicle conflicts can be identified in advance, and verifiable conflict explanation information can be provided. This provides more reliable and interpretable early warnings for road risk assessment, traffic organization evaluation, and proactive safety management.

[0105] Reference Figure 3 , Figure 3 This is a structural block diagram of the first embodiment of the vehicle conflict early warning system from the perspective of an unmanned aerial vehicle (UAV) in this application.

[0106] like Figure 3 As shown, the vehicle conflict early warning system from the perspective of an unmanned aerial vehicle (UAV) proposed in this application includes: Image acquisition module 10 is used to acquire historical video image sequences of the target road area from drones; Occupancy prediction module 20 is used to predict occupancy maps and vehicle center response maps for multiple future moments from the historical video image sequence. The occupancy prediction module includes a multi-scale image backbone network, a feature pyramid network, a high-resolution bird's-eye view feature fusion network, and a fully convolutional future occupancy decoder. Occupancy flow estimation module 30 is used to obtain an occupancy flow field with directional information based solely on the occupancy map; The candidate vehicle anchoring module 40 is used to determine the candidate vehicle based on the vehicle center response map and relocate frame by frame to obtain the position, heading and speed of the candidate vehicle, without establishing an identity-level continuous trajectory association between the candidate vehicle and surrounding vehicles. The co-movement body construction module 50 is used to transform the occupancy map and the occupancy flow field to the vehicle coordinate system of the candidate vehicle, forming a spatiotemporal co-movement body from the vehicle's perspective that includes occupancy, forward relative velocity, and lateral relative velocity; The risk pre-screening module 60 is used to sort and select candidate vehicles based on the spatiotemporal coherence from the perspective of the vehicle. The token grounding inference module 70 is used to encode the spatiotemporal co-movement of the selected candidate vehicle from the perspective of the vehicle into multiple consecutive occupancy stream tokens, and generate vehicle conflict determination and conflict interpretation information sequentially through the decision layer of the language model and the interpretation layer based on the decision result. The token grounding inference module includes a three-dimensional convolutional encoder, a projection network, a language model decision layer with low-rank adaptation, and a language model interpretation layer. The early warning output module 80 is used to generate and output early warning information for vehicle conflicts.

[0107] It should be understood that the above are merely illustrative examples and do not constitute any limitation on the technical solution of this application. In specific applications, those skilled in the art can make settings as needed, and this application does not impose any restrictions on this.

[0108] This embodiment preserves vehicle motion direction information by introducing an occupancy flow field and employs a candidate vehicle local relocation mechanism that does not rely on identity-level continuous trajectory association. This effectively avoids the conflict judgment distortion caused by trajectory fragmentation and identity switching in complex traffic scenarios, which is common in traditional solutions. By constructing a spatiotemporal co-movement body from the vehicle's perspective and combining it with a language model for conflict determination and interpretation, potential vehicle conflicts can be identified in advance, and verifiable conflict explanation information can be provided. This provides more reliable and interpretable early warnings for road risk assessment, traffic organization evaluation, and proactive safety management.

[0109] It should be noted that the workflow described above is merely illustrative and does not limit the scope of protection of this application. In practical applications, those skilled in the art can select some or all of it to achieve the purpose of this embodiment according to actual needs, and no restrictions are imposed here.

[0110] In addition, for technical details not described in detail in this embodiment, please refer to the method for early warning of vehicle conflicts from the perspective of drones provided in any embodiment of this application, which will not be repeated here.

[0111] Furthermore, it should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0112] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0113] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as read-only memory (ROM) / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of this application. The above are only preferred embodiments of this application and do not limit the patent scope of this application. All equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A method for early warning of vehicle conflicts from the perspective of unmanned aerial vehicles (UAVs), characterized in that, include: S1, acquire the sequence of historical video images of the target road area before the current decision time; S2, perform multi-scale feature extraction and temporal fusion on the historical video image sequence, predict the occupancy map of multiple future moments after the current decision moment while maintaining the spatial resolution of the vehicle target, and obtain the vehicle center response map. S3, input the occupancy map into the occupancy flow estimation network to obtain the two-dimensional velocity vector corresponding to each occupancy grid, forming an occupancy flow field containing motion direction information; S4. Based on the vehicle center response map, candidate vehicles are determined, and the candidate vehicles are locally relocated at each sampling time of the historical video image sequence to obtain the spatial position, heading and speed of the candidate vehicles at the current decision time. However, no identity-level continuous trajectory association is established between the candidate vehicles and surrounding vehicles. S5. For each candidate vehicle, the occupancy map and the occupancy flow field are transformed into a vehicle coordinate system centered on the candidate vehicle and based on the candidate vehicle's heading, and a vehicle-view spatiotemporal co-movement body is constructed, which includes at least the occupancy channel, the forward relative velocity channel and the lateral relative velocity channel. S6, calculate the risk score of the candidate vehicle based on the spatiotemporal co-movement body from the perspective of the vehicle, and select a predetermined number of candidate vehicles based on the risk score; S7, the spatiotemporal co-movement of the selected candidate vehicle from the vehicle's perspective is encoded into multiple continuous occupancy stream tokens, the continuous occupancy stream tokens and the task prompt tokens are input into the trained language model, the decision layer of the language model outputs the vehicle conflict determination, and the interpretation layer based on the vehicle conflict determination outputs the conflict interpretation information. S8, Generate early warning information for vehicle conflict based on the vehicle conflict determination and the conflict interpretation information.

2. The vehicle conflict early warning method according to claim 1, characterized in that: In step S2, the multi-scale feature extraction and temporal fusion include: extracting at least four features with different downsampling ratios for each sampled image; unifying the number of channels for the multi-scale features including the feature with the lowest downsampling ratio and upsampling them to the target occupancy resolution; concatenating and convolving the upsampled multi-scale features to obtain a bird's-eye view feature consistent with the target occupancy grid resolution; concatenating the bird's-eye view features from multiple historical moments along the channel dimension, inputting them into a fully convolutional decoder with downsampling paths, skip connections, and upsampling paths, and outputting the occupancy map in parallel through multiple output channels corresponding to different future moments.

3. The vehicle conflict early warning method according to claim 1, characterized in that: In step S3, the occupancy flow estimation network is a three-dimensional convolutional network that preserves temporal and spatial resolution; The occupancy flow estimation network uses occupancy grids as masks and is trained using the endpoint errors of two-dimensional velocity vectors, so that each occupancy grid outputs forward velocity components and lateral velocity components. During the online inference phase, the input to the occupancy flow estimation network consists entirely of the occupancy graph, without requiring input of vehicle identity or vehicle trajectory.

4. The vehicle conflict early warning method according to claim 1, characterized in that: In step S4, determining and relocating candidate vehicles includes: performing local maximum detection on the vehicle center response map; correcting the subgrid offset of the local maximum location; selecting the vehicle center closest to the current candidate vehicle location within a predetermined spatial threshold of adjacent sampling times as the relocation location for that sampling time; estimating the heading and vehicle speed of the candidate vehicles based on multiple relocation locations, wherein, within a local time window before and after the current decision time, robust slope estimation is performed on the candidate vehicle location sequence to reduce the impact of single-frame detection location noise on the vehicle speed.

5. The vehicle conflict early warning method according to claim 1, characterized in that: In step S5, constructing the spatiotemporal co-movement body from the perspective of the vehicle includes: translating the occupancy map and the occupancy flow field with the spatial position of the candidate vehicle as the origin; rotating the occupancy map and the occupancy flow field according to the heading of the candidate vehicle, so that the forward direction of the candidate vehicle corresponds to a preset axis of the vehicle coordinate system; subtracting the vehicle velocity of the candidate vehicle from the occupancy flow field to obtain a relative velocity field; projecting the relative velocity field onto the forward axis and the lateral axis of the vehicle coordinate system; stacking the occupancy channel, the forward relative velocity channel, and the lateral relative velocity channel at multiple time steps within a predetermined spatial range around the candidate vehicle; wherein the forward relative velocity channel and the lateral relative velocity channel are multiplied with the occupancy mask and normalized and limited according to a predetermined velocity scale.

6. The vehicle conflict early warning method according to claim 1, characterized in that: In step S6, the risk score is output by the convolutional risk scoring network based on the spatiotemporal co-movement of the vehicle's perspective; K candidate vehicles are selected in descending order of risk score and input into the language model; unselected candidate vehicles are maintained and monitored by the real-time risk screening layer and reordered in the next sliding window. K is set based on the computational budget and the target event recall rate.

7. The vehicle conflict early warning method according to claim 1, characterized in that: In step S7, encoding the spatiotemporal co-movement body from the vehicle's perspective into multiple consecutive occupancy stream tokens includes: extracting the spatiotemporal features of the spatiotemporal co-movement body from the vehicle's perspective using a 3D convolutional encoder; adaptively pooling the spatiotemporal features into a predetermined 3D mesh; flattening each mesh feature of the 3D mesh into a token sequence; mapping each mesh feature to the embedding dimension of the language model through a projection network; and placing the obtained consecutive occupancy stream tokens before the task prompt token.

8. The vehicle conflict early warning method according to claim 1 or 7, characterized in that: The language model includes a decision layer and an interpretation layer that are separate from each other; the decision layer outputs at least a binary classification judgment of danger and safety or a multi-level risk judgment based on the continuous occupancy stream token; The input prompts of the interpretation layer include the vehicle conflict determination output by the decision layer, and generate at least one or more conflict interpretation information including relative orientation, conflict type, closing speed, expected conflict time, risk cause, handling suggestion and confidence level; Training the language model includes: freezing the basic parameters of the language model; training the low-rank adaptation parameters of at least one attention layer or feedforward layer of the 3D convolutional encoder, projection network, and insertion language model; first training the decision-making ability using binary risk labels, and then training the explanatory ability using rich labels containing multiple explanatory fields; when training the explanatory layer, the risk label or the output of the decision layer is used as a condition, and the autoregressive language loss is calculated only for the target answer part.

9. The vehicle conflict early warning method according to claim 1, characterized in that: The conflict supervision labels used in training the occupancy prediction network, occupancy flow estimation network, risk scoring network, or language model are obtained in the following way: for the target vehicle and surrounding vehicles, the directional closure velocity is calculated based on the relative position vector and the relative velocity vector; when the directional closure velocity is positive, the generalized collision time is calculated based on the distance after deducting the vehicle size and the directional closure velocity; the minimum value of the generalized collision time is taken within the future prediction window, and a conflict label is formed based on one or more thresholds. The conflict supervision label is used only for training or evaluation, and the input for the online inference stage is obtained entirely from the historical video image sequence of the UAV. The vehicle conflict early warning information includes at least one of the following: target vehicle identification, target vehicle location in the image or map, risk level, risk score, relative orientation, conflict type, closing speed, expected conflict time, risk cause, handling suggestions, and confidence level, and is output through a monitoring terminal, traffic management platform, or data interface.

10. A vehicle conflict early warning system from an unmanned aerial vehicle (UAV) perspective, characterized in that, include: The image acquisition module is used to acquire historical video image sequences of the target road area from drones; The occupancy prediction module is used to predict occupancy maps and vehicle center response maps for multiple future moments from the historical video image sequence. The occupancy prediction module includes a multi-scale image backbone network, a feature pyramid network, a high-resolution bird's-eye view feature fusion network, and a fully convolutional future occupancy decoder. Occupancy flow estimation module, used to obtain occupancy flow field with directional information based solely on the occupancy map; The candidate vehicle anchoring module is used to determine candidate vehicles based on the vehicle center response map and relocate them frame by frame to obtain the position, heading and speed of the candidate vehicles, without establishing identity-level continuous trajectory associations between the candidate vehicles and surrounding vehicles. The co-movement body construction module is used to transform the occupancy map and the occupancy flow field to the vehicle coordinate system of the candidate vehicle, forming a spatiotemporal co-movement body from the vehicle's perspective that includes occupancy, forward relative velocity, and lateral relative velocity; The risk pre-screening module is used to sort and select candidate vehicles based on the spatiotemporal coherence from the perspective of the vehicle. The token grounding inference module is used to encode the spatiotemporal co-movement of the selected candidate vehicles from the perspective of the vehicle into multiple consecutive occupancy stream tokens, and generate vehicle conflict determination and conflict interpretation information sequentially through the decision layer of the language model and the interpretation layer based on the decision result. The token grounding inference module includes a three-dimensional convolutional encoder, a projection network, a language model decision layer with low-rank adaptation, and a language model interpretation layer. The early warning output module is used to generate and output early warning information for vehicle conflicts.

Images