Multimodal traffic data set construction method and device based on cross-modal constraint

Abstract

The embodiment of the application provides a multi-modal traffic dataset construction method and device based on cross-modal constraints. Through feature extraction and scene classification, effective data cleaning is realized. A generation mechanism is constructed, physical constraints and adversarial training are combined, and a reliable synthesis strategy is established. Feature fusion is introduced, and the quality of the dataset is ensured through importance evaluation and data screening. This method effectively solves the shortcomings of traditional technologies in data processing, constraint generation, and feature fusion, providing technical support for traffic dataset construction.

Description

Technical Field

[0001] This application relates to the field of data processing, specifically to a method and apparatus for constructing a multimodal traffic dataset based on cross-modal constraints. Background Technology

[0002] Existing methods for constructing traffic datasets have significant shortcomings. Traditional systems perform poorly in data acquisition and preprocessing, failing to effectively integrate multimodal features and thus affecting data quality.

[0003] Furthermore, existing technologies suffer from bottlenecks in data generation and constraint embedding. Most systems lack robust physical constraint mechanisms and generation strategies, resulting in synthetic data that is not realistic enough.

[0004] Existing systems have technical shortcomings in feature fusion. They lack in-depth analysis of modal importance, making it difficult to achieve efficient feature selection through attention mechanisms, thus impacting model performance. Solving these problems is crucial for improving the ability to construct traffic datasets. Summary of the Invention

[0005] To address the problems in existing technologies, this application provides a method and apparatus for constructing multimodal traffic datasets based on cross-modal constraints. This method effectively solves the shortcomings of traditional technologies in data processing, constraint generation, and feature fusion, and provides technical support for the construction of traffic datasets.

[0006] To solve at least one of the above problems, this application provides the following technical solution: Firstly, this application provides a method for constructing a multimodal traffic dataset based on cross-modal constraints, including: Multimodal traffic data is collected to generate a raw dataset. The raw dataset is preprocessed to generate a multimodal feature package, which includes video features, image features, and trajectory features. Metadata information is extracted based on the multimodal feature package to generate a scene description set. The scene description set is input into a scene classifier to generate scene labels. Data verification rules are selected from a dynamic rule base according to the scene labels. The data verification rules are applied to the multimodal feature package to generate data cleaning results. The data cleaning results are used to construct a generative network model branch set according to modality type. The video features are input into a 3D convolutional layer to extract temporal features. Based on the temporal features, scene physical constraints are embedded to generate a video generation branch. The image features are input into a target detection network to extract target features. Based on the target features, illumination attenuation laws are embedded to generate an image generation branch. The trajectory features are input into a temporal encoder to extract motion features. Based on the motion features, vehicle dynamics constraints are embedded to generate a trajectory generation branch. A discriminant network is constructed based on the video generation branch, the image generation branch, and the trajectory generation branch. The generation parameters are updated through adversarial training. Based on the generation parameters, a synthetic dataset is expanded and generated. An attention fusion model is trained on the synthetic dataset according to scene type to generate a feature weight matrix. Multimodal feature importance is calculated based on the feature weight matrix. The synthetic dataset is then filtered and combined according to the multimodal feature importance to generate a traffic dataset. The traffic dataset is used to train a traffic scene recognition model. Based on the traffic scene recognition model, real-time traffic data is analyzed and processed to generate traffic control instructions.

[0007] Furthermore, it also includes: acquiring multiple video streams from traffic monitoring equipment, acquiring image sequences from vehicle-mounted cameras, and acquiring trajectory point sets from vehicle-mounted positioning equipment to generate a data acquisition stream; dividing the acquisition time window according to the data acquisition stream; establishing a data structure mapping table based on preset format specifications; converting the data within the acquisition time window according to the data structure mapping table to generate a raw data packet containing timestamps; Based on the original data packet, data preprocessing is performed to generate a multimodal dataset. The multimodal data is time-aligned according to the timestamp marker. The aligned data is downsampled according to a preset sampling interval. The downsampled data is normalized to generate standardized features. A feature index table is constructed based on the standardized features to generate a multimodal feature package containing video features, image features, and trajectory features.

[0008] Furthermore, it also includes: performing feature vectorization processing on the scene description set to generate a feature matrix, constructing a scene classification training set based on the scene annotation data, training a multilayer perceptron classifier based on the scene classification training set, inputting the feature matrix into the multilayer perceptron classifier to generate a scene confidence distribution, filtering the scene confidence distribution according to a preset confidence threshold, and generating a scene label set containing scene type and confidence. Based on the scene tag set, a dynamic rule base is queried to generate a verification rule set. According to the scene type, the corresponding physical constraint rule is selected. The physical constraint rule is combined with the confidence level to construct a data cleaning template. The data cleaning template is applied to the multimodal feature package to perform anomaly detection. The data cleaning result is generated based on the detection result.

[0009] Furthermore, it also includes: performing modal separation on the data cleaning results to generate a modal feature set; constructing a feature extraction network based on the modal feature set; inputting video features into a three-dimensional convolutional layer to extract spatiotemporal features; inputting image features into a target detection network to extract region features; inputting trajectory features into a temporal encoder to extract motion features; and normalizing the spatiotemporal features, the region features, and the motion features to generate a feature vector group containing multimodal coding representation. A physical constraint layer is constructed based on the feature vector group, and the scene physical rules are mapped into a constraint parameter matrix. Based on the constraint parameter matrix, the spatiotemporal features are embedded with scene constraints to generate a video branch, the region features are embedded with illumination constraints to generate an image branch, and the motion features are embedded with dynamic constraints to generate a trajectory branch. The video branch, the image branch, and the trajectory branch are combined to generate a model branch set.

[0010] Furthermore, it also includes: performing feature fusion on the video generation branch, image generation branch and trajectory generation branch to generate discriminative input data, inputting the discriminative input data into a multimodal discriminator to extract intermodal correlation features, constructing a discriminative scoring function based on the intermodal correlation features, establishing an adversarial loss calculation module according to the discriminative scoring function, and combining the adversarial loss calculation module with the physical constraint loss to generate a training objective function; Generative adversarial training is performed based on the training objective function. Multimodal data is augmented and sampled according to the generation parameters. The augmented sampling results are input into the multimodal discriminator to calculate the realism score. High-quality samples are selected based on the realism score. The high-quality samples are labeled and organized to generate a synthetic dataset.

[0011] Furthermore, it also includes: classifying the synthetic dataset into scene sample groups, dividing the scene sample groups into training data according to a preset batch, constructing a multi-head attention module based on the training data, establishing an attention mask matrix according to scene labels, using the attention mask matrix to calculate inter-modal attention scores, and normalizing the inter-modal attention scores to generate a feature weight matrix. Modal feature importance scores are calculated based on the feature weight matrix. The modal feature importance scores are compared with preset screening thresholds. The samples in the synthetic dataset are ranked by importance according to the comparison results. High-quality samples are selected and combined according to the importance ranking results to generate a traffic dataset containing cross-modal constraint relationships.

[0012] Furthermore, it also includes: dividing the traffic dataset into samples according to scene type to generate a training sample library; constructing a multimodal feature extractor based on the training sample library; using the multimodal feature extractor to extract key scene features; establishing a feature learning module based on the key scene features; and training and optimizing the feature learning module to generate a scene recognition model. Real-time traffic data is processed based on the scene recognition model. The input data is processed by the multimodal feature extractor to generate feature representations. The scene type probability distribution is calculated based on the feature representations. Decision analysis is performed on the scene type probability distribution. Traffic control instructions are generated based on preset control rules.

[0013] Secondly, this application provides a device for constructing a multimodal traffic dataset based on cross-modal constraints, comprising: The data processing module is used to collect multimodal traffic data to generate a raw dataset, preprocess the raw dataset to generate a multimodal feature package, wherein the multimodal feature package includes video features, image features and trajectory features, extract metadata information based on the multimodal feature package to generate a scene description set, input the scene description set into a scene classifier to generate scene labels, select data verification rules from a dynamic rule base according to the scene labels, and apply the data verification rules to the multimodal feature package to generate data cleaning results. A multidimensional constraint module is used to construct a generative network generation model branch set according to modality type based on the data cleaning results. The video features are input into a three-dimensional convolutional layer to extract temporal features. Based on the temporal features, scene physical constraints are embedded to generate a video generation branch. The image features are input into a target detection network to extract target features. Based on the target features, illumination attenuation laws are embedded to generate an image generation branch. The trajectory features are input into a temporal encoder to extract motion features. Based on the motion features, vehicle dynamics constraints are embedded to generate a trajectory generation branch. A discriminant network is constructed based on the video generation branch, the image generation branch, and the trajectory generation branch. The generation parameters are updated through adversarial training. Based on the generation parameters, a synthetic dataset is expanded and generated. The dataset application module is used to train an attention fusion model to generate a feature weight matrix for the synthetic dataset according to scene type, calculate the importance of multimodal features based on the feature weight matrix, filter and combine the synthetic dataset according to the importance of multimodal features to generate a traffic dataset, use the traffic dataset to train a traffic scene recognition model, and analyze and process real-time traffic data based on the traffic scene recognition model to generate traffic control instructions.

[0014] Thirdly, this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the method for constructing a multimodal traffic dataset based on cross-modal constraints.

[0015] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method for constructing a multimodal traffic dataset based on cross-modal constraints.

[0016] Fifthly, this application provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the method for constructing a multimodal traffic dataset based on cross-modal constraints.

[0017] As described above, this application provides a method and apparatus for constructing a multimodal traffic dataset based on cross-modal constraints. Through feature extraction and scene classification, it achieves effective data cleaning. A generation mechanism is constructed, combining physical constraints and adversarial training to establish a reliable synthesis strategy. Feature fusion is introduced, and through importance assessment and data filtering, the quality of the dataset is ensured. This method effectively addresses the shortcomings of traditional techniques in data processing, constraint generation, and feature fusion, providing technical support for the construction of traffic datasets. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating the method for constructing a multimodal traffic dataset based on cross-modal constraints in an embodiment of this application. Figure 2 This is a structural diagram of the device for constructing a multimodal traffic dataset based on cross-modal constraints in an embodiment of this application; Figure 3 This is a schematic diagram of the structure of the electronic device in the embodiments of this application.

[0020] Figure label: Electronic device 9600, central processing unit 9100, memory 9140, communication module 9110, input unit 9120, audio processor 9130, display 9160, power supply 9170, buffer memory 9141, application / function storage unit 9142, data storage unit 9143, driver storage unit 9144, antenna 9111, speaker 9131, microphone 9132. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0022] The acquisition, storage, use, and processing of data in this application comply with relevant laws and regulations.

[0023] To address the problems existing in current technologies, this application provides a method and apparatus for constructing a multimodal traffic dataset based on cross-modal constraints. Through feature extraction and scene classification, effective data cleaning is achieved. A generation mechanism is constructed, combining physical constraints and adversarial training to establish a reliable synthesis strategy. Feature fusion is introduced, and the quality of the dataset is ensured through importance assessment and data filtering. This method effectively solves the shortcomings of traditional techniques in data processing, constraint generation, and feature fusion, providing technical support for the construction of traffic datasets.

[0024] To effectively address the shortcomings of traditional technologies in data processing, constraint generation, and feature fusion, and to provide technical support for the construction of traffic datasets, this application provides an embodiment of a method for constructing multimodal traffic datasets based on cross-modal constraints. See [link to embodiment]. Figure 1 The method for constructing a multimodal traffic dataset based on cross-modal constraints specifically includes the following: Step S101: Collect multimodal traffic data to generate a raw dataset, preprocess the raw dataset to generate a multimodal feature package, wherein the multimodal feature package includes video features, image features and trajectory features, extract metadata information based on the multimodal feature package to generate a scene description set, input the scene description set into a scene classifier to generate scene labels, select data verification rules from a dynamic rule base according to the scene labels, and apply the data verification rules to the multimodal feature package to generate data cleaning results; This embodiment integrates multiple video streams, image sequences, and trajectory point sets, generating a raw dataset according to the acquisition time window. Within the same window, keyframes are extracted from the video and their brightness and color are normalized. Geometric distortion correction and bounding box standardization are performed on the images. Trajectories are sorted by time, and missing measurement points are filled in using interpolation, while drift is corrected. The processed data is encoded into a multimodal feature package under a unified index, containing video features, image features, and trajectory features, while also registering the window number and device identifier.

[0025] This embodiment extracts metadata information based on the multimodal feature package, reading time period identifiers, geographic coordinates, device numbers, and target quantity annotations, mapping them into structured fields to form a scene description set. This embodiment binds the scene description set and the feature package using the same index key to ensure the uniqueness of subsequent retrieval, backfilling, and comparison.

[0026] In this embodiment, the scene description set is input into a multilayer perceptron classifier (hereinafter referred to as the "scene classifier"). The classifier uses a concatenated vector of description fields as input and outputs the scene type and confidence distribution. This embodiment sets channel masks for sensitive fields such as nighttime, rainy days, and construction, and adjusts the opening and closing of input channels in conjunction with time period identifiers. The classification output is backfilled as scene tags and attached to the corresponding index key, serving as trigger items for subsequent rule scheduling.

[0027] This embodiment retrieves data verification rules from a dynamic rule base based on the scene tags. The rules use scene type as the primary key and return two types of entries: physical consistency constraints and cross-modal consistency constraints, such as speed range, target quantity and scene matching, and illumination attenuation and exposure rationality. This embodiment instantiates the search results into a cleaning template, ensuring a one-to-one correspondence between template entries and feature fields, and records the rule version number and effective time.

[0028] This embodiment verifies each multimodal feature package according to the cleaning template. On the trajectory side, instantaneous velocity is derived based on the distance between adjacent points and the time difference, and then compared against the compliance range according to the road type field. On the video side, the brightness distribution and spatial attenuation trend of keyframes are statistically analyzed to verify consistency with the physical scene. On the image side, the number and size distribution of vehicles and pedestrians are statistically analyzed and matched with scene labels. This embodiment writes the pass and reason for each verification into the sample-level audit record, forming a traceable verification trajectory.

[0029] This embodiment sets a unified decision value for multiple constraints to trigger repair and removal: Φ = φ1·Ψv + φ2·Ψi φ3·Ξt.

[0030] In the formula, Φ is the sample cleaning score; Ψv is the video-side consistency measure, derived from the comparison of brightness attenuation and motion continuity; Ψi is the image-side consistency measure, derived from the comparison of target quantity and size matching; Ξt is the trajectory-side penalty term, reflecting the cumulative intensity of instantaneous velocity exceeding the limit and abnormal turning; φ1, φ2, and φ3 are non-negative weights, given by the rule base according to the scene type. In this embodiment, samples whose scores fall into the abnormal range are sent to the review queue, and Φ is used as one of the inputs for sampling weights in subsequent steps.

[0031] If the trajectory field points to a highway while the video label is a residential road segment within the same time window, a metadata conflict is marked and the conflicting field is recorded. For samples that can be repaired through interpolation, recalculation, or threshold condition rollback, this embodiment backfills the updated features and metadata; samples that cannot be repaired are noted for removal and removed from the current window to avoid polluting the training sample pool.

[0032] This embodiment generates data cleaning results after the cleaning process is completed. The results are indexed by window number and include sample retention status, scene label, cleaning score Φ, conflict marker, rule version, and timestamp. This embodiment also retains read-only links to the original features, facilitating direct reading of video, image, and trajectory features when constructing generative network branches according to modality type, enabling input integration with the next step.

[0033] Step S102: Construct a generative network generation model branch set according to modality type for the data cleaning results; input the video features into a 3D convolutional layer to extract temporal features; embed scene physical constraints based on the temporal features to generate a video generation branch; input the image features into a target detection network to extract target features; embed illumination attenuation laws based on the target features to generate an image generation branch; input the trajectory features into a temporal encoder to extract motion features; embed vehicle dynamics constraints based on the motion features to generate a trajectory generation branch; construct a discriminant network based on the video generation branch, the image generation branch, and the trajectory generation branch; update the generation parameters through adversarial training; and expand and generate a synthetic dataset based on the generation parameters. After splitting the cleaning results from step S101 according to modal type, the video feature sequence, image feature vector, and trajectory feature point column are directly read, while retaining the window number and scene label. To avoid cross-modal misalignment, the three types of inputs are first aligned in terms of sampling step size and clipped in length on a unified time axis, and then organized into batch units and placed into the input buffer of the generator network.

[0034] On the video side, the aligned sequence is fed into 3D convolution and temporal pooling to obtain temporal feature representation; based on scene labels, parameters such as illumination range, raindrop direction and velocity range are extracted from the physical parameter library, mapped as constraint masks, and applied to the channels and spatial positions of the convolution output to suppress structures and motion patterns that do not conform to the scene, outputting intermediate features and reconstructed frame sequences, which are denoted as the video generation branch.

[0035] On the image side, an object detection network is first used to extract region-level target features. Then, the illumination attenuation law is loaded according to the scene label, and the mapping between attenuation and distance is used to generate a pixel-level weight map. The target features are then spatially weighted to balance the contributions of high-contrast regions and background regions. The intermediate features and the reconstructed image are output, which is denoted as the image generation branch. The weight map of this branch is bound to the scene label for easy backtracking later.

[0036] The trajectory side inputs the time-sorted feature point sequence into the temporal encoder to obtain the motion trend vector; vehicle dynamics constraints (including turning radius, acceleration, and deceleration feasible zones) are converted into time step masks and penalty markers to restrict unreasonable rapid acceleration and sharp turns, and output intermediate features and reconstructed point sequences, which are recorded as the trajectory generation branch. The dynamics mask and window number are registered together to support subsequent consistency comparison.

[0037] The discriminant network reads the intermediate features and reconstruction results of the three branches, which are then aggregated into a joint representation by time slices through the alignment layer. Finally, the relation layer provides the true / false score and the cross-modal consistency score.

[0038] To ensure consistent training signal accuracy, a round target is introduced to guide the alternating updates of the generator and discriminator: Ω = ω1·Γadv + ω2·Γphy ω3·Γcons.

[0039] In the formula, Ω represents the single-round training objective; Γadv is the adversarial term, derived from the discriminator's score of the reconstructed samples; Γphy is the constraint term, aggregating the violation rates of video physical constraints, image illumination attenuation, and trajectory dynamics; Γcons is the consistency penalty, quantifying the scene deviation on the three branches for the same time slice; ω1, ω2, and ω3 are non-negative weights, determined by scene label retrieval and fixed within a training session. In subsequent paragraphs, Ω is directly read by the training loop to update the parameters.

[0040] Alternating training is performed with a fixed stride: first, the discriminator network is frozen, and the generation parameters of the three branches are updated to reduce Ω; then, the generation branch is frozen, and the weights of the discriminator network are updated to improve the ability to distinguish between true and false data and consistency. The input buffer continuously carries the window index to prevent parameters from interfering with each other in different scenarios.

[0041] During the augmentation sampling phase, each of the three branches generates segments, and the network output is evaluated for authenticity and consistency scores. Segments that meet the defined intervals are combined into synthetic data units, and parameter snapshots and scene labels are generated simultaneously. Segments that do not meet the intervals are registered as difficult samples and added to the retraining list for the next round. The entire sampling process remains consistent with the time slice of the input buffer to avoid splicing across segments.

[0042] The final synthesized dataset uses window numbers as the primary index, along with scene labels, cross-modal alignment indices, and snapshots of generated parameters. This serves as the direct input for subsequent steps of training the attention fusion model and re-discrimination. The reconstructed video, image, and trajectory results can be read downstream using the same indexes, eliminating the need for repeated preprocessing and maintaining the same referencing relationship as in step S101.

[0043] Step S103: Train an attention fusion model on the synthetic dataset according to scene type to generate a feature weight matrix, calculate the multimodal feature importance based on the feature weight matrix, filter and combine the synthetic dataset according to the multimodal feature importance to generate a traffic dataset, use the traffic dataset to train a traffic scene recognition model, and analyze and process real-time traffic data based on the traffic scene recognition model to generate traffic control instructions.

[0044] After grouping the synthetic dataset by window number and scene label, it is divided into training batches and validation batches. In this embodiment, the three types of inputs for each sample are organized into time-slice aligned tensors, corresponding to video reconstruction segments, image reconstruction results, and trajectory point sequences, respectively. The snapshot of the generated parameters and the cross-modal alignment index from step S102 are retained as a traceable reference during training.

[0045] Based on the aforementioned grouping, an attention fusion model is constructed. The model comprises a modal self-attention layer and a cross-modal attention layer, both using time slices as the smallest unit. In this embodiment, self-attention is first calculated within each modality to extract stable components. Then, in the cross-modal layer, the segment representations of video, images, and trajectories are interacted to obtain unnormalized relevance scores. To avoid scene bias, scene masks are introduced during training, limiting the attention channels that can participate to labels such as nighttime, rainy days, and construction.

[0046] The relevance scores are normalized to form a feature weight matrix, with rows corresponding to time slices and columns corresponding to the three modalities. In this embodiment, this matrix is ​​archived together with the scene labels for subsequent importance calculation and audit backtracking. To improve the ability to identify abnormal segments, difficult sample replay is added during training. The difficult samples are from the set of segments near the scoring boundary of the discriminant network.

[0047] Multimodal feature importance is calculated based on the feature weight matrix. Importance is aggregated at the time slice level and then weighted across samples to obtain the modal importance vector for each sample. In this embodiment, the importance is compared with a preset screening threshold interval, retaining samples with high importance and good cross-modal consistency, while retaining a small proportion of samples with low importance but covering scarce scenarios to maintain diversity, thus producing a candidate set.

[0048] The candidate set is further combined according to scene type and time window to form a traffic dataset. During combination, the cross-modal alignment index remains unchanged to ensure that all three modalities can be located simultaneously when the same time slice is read downstream. This embodiment also records the sample source label and the corresponding feature weight snapshot, facilitating the location of source segments with abnormal weights during subsequent training.

[0049] Traffic datasets are used to train a traffic scene recognition model. The model includes a multimodal feature extractor and a feature learning module. The former reuses the encoding structure validated in the generation phase, while the latter is jointly trained using classification loss and consistency regularization. In this embodiment, the feature weight matrix is ​​injected as an attention prior during training, enabling the model to focus more on the modal channels that are most discriminative for the current scene during the learning process.

[0050] After training, the traffic scene recognition model is deployed in the real-time data processing link. The input real-time video, images, and trajectories are first aligned according to the preprocessing method in step S101, and then feature representations are generated by a multimodal feature extractor. The model outputs the probability distribution and confidence score of the scene type. In this embodiment, the scene type and confidence score are transcribed into traffic control instructions based on preset control rules, such as triggering speed limit and traffic light timing adjustment strategies under the tag combination of rainy day and night.

[0051] The aforementioned traffic control commands, along with scene recognition results and key intermediate quantities, are recorded as an operation log. The log includes scene labels, importance vector snapshots, and time slice indexes. This embodiment uses this log as the input for subsequent retraining and online evaluation, achieving a closed-loop connection from synthetic datasets and weight learning to online decision-making.

[0052] As described above, the multimodal traffic dataset construction method based on cross-modal constraints provided in this application can achieve effective data cleaning through feature extraction and scene classification. A generation mechanism is constructed, combining physical constraints and adversarial training to establish a reliable synthesis strategy. Feature fusion is introduced, and the quality of the dataset is ensured through importance assessment and data filtering. This method effectively addresses the shortcomings of traditional techniques in data processing, constraint generation, and feature fusion, providing technical support for traffic dataset construction.

[0053] In one embodiment of the method for constructing a multimodal traffic dataset based on cross-modal constraints in this application, the following may also be included: Step S201: Obtain multiple video streams from traffic monitoring equipment, image sequences from vehicle-mounted cameras, and trajectory point sets from vehicle-mounted positioning devices to generate a data acquisition stream. Divide the acquisition time window according to the data acquisition stream, establish a data structure mapping table based on preset format specifications, and convert the data within the acquisition time window according to the data structure mapping table to generate a raw data packet containing timestamps. Step S202: Based on the original data packet, perform data preprocessing to generate a multimodal dataset, perform time-series alignment of the multimodal data according to the timestamp marker, downsample the aligned data according to the preset sampling interval, normalize the downsampled data to generate standardized features, construct a feature index table based on the standardized features, and generate a multimodal feature package containing video features, image features, and trajectory features.

[0054] This embodiment integrates video streams from traffic monitoring equipment, image sequences from vehicle-mounted cameras, and trajectory point sets from vehicle-mounted positioning devices to generate a data acquisition stream. To ensure the locationability of subsequent processing, the device number and geographic coordinates are first read, an acquisition time window is established, and multiple inputs within the same window are cached in a temporary queue at the acquisition end, recording the start and end times and data source identifiers as window-level metadata.

[0055] A data structure mapping table is established on the data acquisition stream. The mapping table defines the field layout of video frame sequences, image frames, and trajectory points according to a preset format specification, including timestamp format, spatial resolution, channel order, and latitude / longitude precision bits. In this embodiment, the data within the acquisition time window is format-converted according to this mapping table: the frame rate and pixel format are unified on the video side, the color space is unified on the image side, and the coordinate system is unified on the trajectory side, with timestamps standardized to the same reference. The converted content is packaged together with window-level metadata to generate a raw data package containing timestamps.

[0056] The raw data packets are fed into the preprocessing pipeline, where time alignment is first performed based on timestamps. The alignment process uses the window start time as zero, mapping video keyframes, image frames, and trajectory points to a unified timeline, retaining misaligned edge sample markers. After alignment, downsampling is performed at preset intervals to ensure consistent time steps for the three types of data, preventing misalignment during subsequent cross-modal fusion.

[0057] Normalization is performed on the downsampling results. For video and image samples, luminance and color channel normalization is used, while for trajectory samples, scale normalization is performed using coordinate and temporal differences. The normalized output is encoded as a normalized feature and associated with the window number and device number. In this embodiment, samples lacking key fields are removed at this stage, and the reasons for removal are recorded in the window-level audit entry for easy backtracking.

[0058] A feature index table is constructed based on the standardized features. The index table uses the window number as the primary key and the time slice as the secondary key. Each entry records the access pointers, timestamps, and data source identifiers for the three types of features. To support random access during subsequent training and inference, each index entry includes a cross-modal alignment indicator to determine whether the current time slice simultaneously possesses video, image, and trajectory features. Missing modalities are indicated by placeholder markers.

[0059] Organized by the index table, a multimodal feature package is generated. The feature package comprises three parts: video features, image features, and trajectory features, corresponding to aligned and normalized tensor or vector representations, respectively, and carrying an index key, window-level metadata, and references to discarded records. The feature package is written to a persistent cache, serving as the direct input for metadata extraction and scene classification in step S101, and also as the foundational data source for subsequent network branch generation.

[0060] To enhance upstream and downstream connectivity, a read-only link and version marker to the original data package are retained within the feature package. Step S101 directly reads this index and link when extracting the scene description set to reduce redundant parsing. Step S102, when constructing the generative model branch, reads the three types of features in batches by index key and processes them in parallel using aligned time slices, avoiding further time synchronization and format conversion. Through this organization, the intermediate quantities formed by acquisition, alignment, downsampling, and normalization are read by key in subsequent processes, resulting in a clear and traceable calling path.

[0061] In one embodiment of the method for constructing a multimodal traffic dataset based on cross-modal constraints in this application, the following may also be included: Step S301: Perform feature vectorization processing on the scene description set to generate a feature matrix, construct a scene classification training set based on the scene annotation data, train a multilayer perceptron classifier based on the scene classification training set, input the feature matrix into the multilayer perceptron classifier to generate a scene confidence distribution, filter the scene confidence distribution according to a preset confidence threshold, and generate a scene label set containing scene type and confidence. Step S302: Generate a verification rule set by querying the dynamic rule base based on the scene tag set, select the corresponding physical constraint rule according to the scene type, combine the physical constraint rule with the confidence level to construct a data cleaning template, apply the data cleaning template to the multimodal feature package to perform anomaly detection, and generate data cleaning results based on the detection results.

[0062] In this embodiment, based on the multimodal feature package and window-level metadata generated in steps S201 and S202, the structured fields of the scene description set are read, including time period identifier, geographic coordinates, device number, target quantity, and scene prompts. Each field is encoded and mapped into a numerical vector, where categorical fields use one-hot encoding and frequency smoothing, and numerical fields use interval standardization and missing placeholders, which are then concatenated to form a feature vector; the feature matrix is ​​obtained by stacking by time slices, and the index key is retained for subsequent backfilling.

[0063] The feature matrix and existing scene annotation data are organized into a scene classification training set. The training set uses time slices as the sample unit, with labels derived from the intersection of historical manual annotations and trusted models, and source confidence markers are recorded. To reduce the impact of scene distribution offsets, stratified sampling by window and location is used during training to ensure coverage of different road types and time periods. The training set is divided into training and validation parts, with the index key remaining unchanged throughout the data stream.

[0064] This embodiment uses a multilayer perceptron classifier as the "scene classifier". The input is the aforementioned feature vector, and the output is the confidence distribution for each scene type. The network depth and width follow non-fixed values ​​but satisfy constraints on expressiveness and inference latency. The activation function adopts a continuously differentiable form for end-to-end training. Sample weights are introduced during the training phase, and the weights are obtained by mapping source confidence labels to class frequencies. The loss function maintains stable convergence in imbalanced scenarios. After training, the feature matrix is ​​fed into the "scene classifier" in batches, outputting the scene confidence distribution, which is cached along with the index key.

[0065] The scene confidence distribution is filtered based on a preset threshold interval to generate a scene label set containing scene type and confidence level. The threshold interval varies with road type and time period; samples below the lower limit are marked as pending verification, while samples above the upper limit directly confirm the scene type. The scene label set is then populated into the corresponding index key, carrying the source of the threshold interval and time slice identifier, providing triggering conditions for downstream rule retrieval.

[0066] Based on the scene tag set, a dynamic rule base is queried line by line to generate a validation rule set. The rule base uses scene type as the primary key and returns two types of rule entries: one is physical consistency constraints, covering feasible zones for speed range, acceleration, and turning radius; the other is cross-modal consistency constraints, covering target quantity and scene matching, illumination attenuation, and exposure rationality. A mapping relationship is established between the query results and scene confidence levels for subsequent weight allocation.

[0067] This embodiment combines physical constraint rules with confidence levels to construct a data cleaning template for the current time slice. Template entries correspond one-to-one with multimodal feature fields, including execution order, judgment conditions, and backfilling strategies. Specifically, the trajectory-side template reads the distance between adjacent points and the time difference in trajectory features to deduce instantaneous velocity; the video-side template reads the brightness distribution of keyframes to compare spatial decay trends; and the image-side template reads the number and size distribution of detected targets to verify consistency with the scene type. Each entry generates a pass flag and a reason field after execution.

[0068] The data cleaning template is applied to each time slice of the multimodal feature package, outputting anomaly detection records. Anomaly types include velocity exceeding limits, lighting mismatch, target quantity mismatch with scene, and cross-modal conflicts. For repairable items (e.g., isolated drift points on the trajectory), the template triggers interpolation and recalculation strategies and backfills the update; for irreparable items, the reason for removal is recorded and the data is removed from the current window's data view, while a read-only link is retained for backtracking.

[0069] After anomaly detection is completed, data cleaning results are generated. These results use the window number and index key as a combined primary key, and include sample retention status, scenario type and confidence level, triggered rule entries and execution logs, along with rule version and timestamps. The data cleaning results, as the data source interface used in steps S101 and S102, remain unchanged and can be directly read by subsequent network branches, avoiding redundant processing and ensuring consistency in upstream and downstream constraints.

[0070] In one embodiment of the method for constructing a multimodal traffic dataset based on cross-modal constraints in this application, the following may also be included: Step S401: Modal separation is performed on the data cleaning results to generate a modal feature set. A feature extraction network is constructed based on the modal feature set. Video features are input into a 3D convolutional layer to extract spatiotemporal features. Image features are input into a target detection network to extract region features. Trajectory features are input into a temporal encoder to extract motion features. The spatiotemporal features, region features, and motion features are normalized to generate a feature vector group containing multimodal coding representation. Step S402: Construct a physical constraint layer based on the feature vector group, map the scene physical rules into a constraint parameter matrix, embed the spatiotemporal features into scene constraints to generate a video branch according to the constraint parameter matrix, embed the region features into illumination constraints to generate an image branch, embed the motion features into dynamic constraints to generate a trajectory branch, and combine the video branch, the image branch and the trajectory branch to generate a model branch set.

[0071] In this embodiment, the data cleaning result generated in step S302 is read, and modal separation is performed according to the index key to obtain three subsets: video feature sequence, image feature vector, and trajectory feature point sequence. To ensure uniform processing in the future, the three subsets are aligned and truncated on the same time axis. Missing segments are marked with placeholders and saved together with window number and scene label to form a modal feature set that can be loaded in parallel.

[0072] A feature extraction network is constructed based on the aforementioned modal feature set. On the video side, the aligned sequence is input into 3D convolution and temporal pooling to extract spatiotemporal features; on the image side, a target detection network extracts region-level target features and aggregates them into an image representation; on the trajectory side, time-ordered point sequences are input into a temporal encoder, outputting motion features and trend vectors. To eliminate scale differences, channel normalization and amplitude constraints are applied to the three types of features respectively, and the temporal dimension is padded to a uniform length, resulting in a feature vector set containing multimodal encoded representations. The vector set maintains a one-to-one correspondence between time slices and index keys.

[0073] The feature vector group is used to construct the physical constraint layer. First, scene physical rules and vehicle dynamics rules are retrieved based on scene tags. The rule terms are discretized into constraint parameter matrices. The matrices are organized by time slices, with rows corresponding to time slices and columns corresponding to different constraint channels, and include enableable mask bits. The matrices and vector groups are associated through index keys to ensure that a consistent constraint view is shared across the three modalities for the same time slice.

[0074] Branch embedding is completed under the drive of the constraint parameter matrix. On the video side, scene constraints are applied to the spatiotemporal feature channels, mapping the illumination range, occlusion probability, and motion continuity into channel weights and spatial masks to obtain the intermediate representation and reconstructed frame of the video branch; on the image side, illumination constraints are embedded in the region features, converting the attenuation and distance relationship into a pixel-level weight map, suppressing highlight and shadow structures that do not match the scene, to obtain the image branch; on the trajectory side, dynamic constraints are embedded in the motion features, converting the feasible regions of acceleration, deceleration, and turning radius into time step masks and penalty markers, restricting abnormal acceleration, deceleration, and sharp turns, to obtain the trajectory branch.

[0075] Based on the outputs of the aforementioned three types of branches, a model branch set is generated. During the combination, the intermediate representations of video, image, and trajectory are aligned on a time-slice basis, keeping the cross-modal alignment index unchanged. At the same time, the snapshot of the constraint parameters and the mask state used by the branches are recorded as the entry point for subsequent adversarial training and consistency discrimination. The model branch set and feature vector group are written back to the cache for direct use by the discriminative network in step S502, avoiding repeated extraction and constraint mapping.

[0076] In one embodiment of the method for constructing a multimodal traffic dataset based on cross-modal constraints in this application, the following may also be included: Step S501: Perform feature fusion on the video generation branch, image generation branch and trajectory generation branch to generate discriminative input data, input the discriminative input data into a multimodal discriminator to extract intermodal correlation features, construct a discriminative scoring function based on the intermodal correlation features, establish an adversarial loss calculation module according to the discriminative scoring function, and combine the adversarial loss calculation module with the physical constraint loss to generate a training objective function; Step S502: Perform generative adversarial training based on the training objective function, augment the multimodal data according to the generation parameters, input the augmented sampling results into the multimodal discriminator to calculate the realism score, select high-quality samples according to the realism score, and label and organize the high-quality samples to generate a synthetic dataset.

[0077] After aligning the model branch set obtained in step S402 by time slice, the intermediate representations and reconstructed frames of the video branch, the intermediate representations and reconstructed maps of the image branch, and the intermediate representations and reconstructed point columns of the trajectory branch are read and merged to form the discriminative input data. During merging, the cross-modal alignment index and constraint parameter snapshots are preserved, and the channel scale is unified through a lightweight mapping layer, so that the segments of the three modalities can participate in the correlation modeling in the same metric space.

[0078] The input data is fed into a multimodal discriminator. This discriminator consists of an alignment layer and a relation layer: the alignment layer generates a joint representation in units of time slices, and the relation layer calculates intermodal correlation features on the joint representation, outputting two types of score streams, one reflecting true / false judgments and the other reflecting cross-modal consistency. To ensure the discriminator has clear readability, the score streams and corresponding time slice indices are written back to the cache for subsequent training.

[0079] Based on the intermodal correlation features, a discriminative scoring function is defined, and an adversarial loss calculation module is constructed. The scoring function generates true / false scores and consistency scores at the fragment level. The adversarial loss module reads the scores and forms inverse signals between the generator and the discriminator. Simultaneously, the physical consistency violation degree is read from the constraint parameter snapshot retained in step S402 to form the physical constraint loss. The two types of losses will be invoked jointly in the same round.

[0080] To facilitate the standardization of training signals, this embodiment introduces only one round-based objective function to drive the alternating optimization of generation and discrimination: Π = π1·Λadv + π2·Λphy π3·Λcon.

[0081] In the formula, Π represents the single-round training objective; Λadv is the adversarial term, derived from the discriminator's aggregation of true / false scores for reconstructed samples; Λphy is the physics term, derived from the aggregation of violations of video scene constraints, image lighting constraints, and trajectory dynamics constraints; Λcon is the consistency penalty, quantifying the scene deviation of the three branches in the same time slice; π1, π2, and π3 are non-negative weights, adaptively selected according to scene type and fixed within the current training session. Subsequent training loops directly read Π to update parameters, and the weights switch with the scene but do not drift within the round.

[0082] Generative adversarial training is conducted based on the aforementioned training objective function. A single training round is executed in two steps: first, the discriminator is fixed, and the parameters of the three generation branches are updated to decrease Π; then, the generation branches are fixed, and the multimodal discriminator is updated to improve the distinction between true / false and consistent data. Training samples are batch-fed according to window and scene groups, and cross-modal alignment indexes are used throughout to avoid gradient interference between different scenes.

[0083] Augmented sampling is performed within the training convergence interval. The generation side samples according to scene labels and window numbers, outputting video clips, image clips, and trajectory clips respectively. These are then merged and input into a multimodal discriminator to obtain realism and consistency scores. Results that simultaneously satisfy the set intervals are marked as high-quality samples, and snapshots of generation parameters and constraints are recorded. Boundary samples are registered as the hard set for focused learning in subsequent rounds.

[0084] High-quality samples are compiled into a synthetic dataset. The compilation process maintains the time slice and cross-modal alignment indexes unchanged, packaging video reconstruction frames, image reconstruction results, and trajectory reconstruction point sequences into unified sample units, along with scene labels, generation parameter snapshots, and discrimination scores. This synthetic dataset is read as direct input in steps S102 and S103, used for attention fusion training and subsequent scene recognition model sample supplementation, avoiding redundant preprocessing and maintaining a consistent index caliber with the upstream dataset.

[0085] In one embodiment of the method for constructing a multimodal traffic dataset based on cross-modal constraints in this application, the following may also be included: Step S601: Perform scene classification on the synthetic dataset to generate scene sample groups, divide the scene sample groups into training data according to a preset batch, construct a multi-head attention module based on the training data, establish an attention mask matrix according to the scene labels, use the attention mask matrix to calculate the inter-modal attention score, and normalize the inter-modal attention score to generate a feature weight matrix. Step S602: Calculate the modal feature importance score based on the feature weight matrix, compare the modal feature importance score with a preset screening threshold, sort the samples in the synthetic dataset by importance according to the comparison result, select high-quality samples according to the importance sorting result and combine them to generate a traffic dataset containing cross-modal constraint relationships.

[0086] After reading the window number and scene label from the synthetic dataset output in step S502, scene classification is performed to form scene sample groups. Classification is based on existing labels and necessary rapid verification to ensure that the three modalities of each sample unit are aligned on the same time slice. The scene sample groups are then divided into training and validation data according to preset batches. Within each batch, cross-modal alignment indices and generation parameter snapshots are retained to facilitate tracing back to specific segments during training.

[0087] A multi-head attention module is constructed on the training data. The module uses time slices as the smallest unit, receiving embedded representations from video reconstruction frames, image reconstruction results, and trajectory reconstruction point sequences. To control the participation range of channels, an attention mask matrix is ​​established based on scene labels. The matrix is ​​organized along a "head × modality" dimension, indicating the visibility of different heads on video, images, and trajectories in a specific scene. The mask matrix is ​​bound to a batch index to ensure consistency in the training signal profile within the same scene.

[0088] The multi-head attention module calculates inter-modal attention scores. Specifically, it first performs self-attention within a modality to extract stable components, then completes the interaction across modal layers using query, key, and value mappings, resulting in an unnormalized score tensor. This score tensor is then multiplied element-wise by an attention mask matrix to mask channel contributions irrelevant to the current scene. After normalization along the modal dimension, a feature weight matrix is ​​obtained, where each row corresponds to a time slice and each column corresponds to a modality.

[0089] The feature weight matrix is ​​archived along with the scene labels and used as input for subsequent calculation of modal feature importance. To improve the ability to distinguish boundary segments, a difficult sample replay strategy is introduced, in which samples whose scores fall within the boundary interval during the discrimination stage appear more frequently in the training batch, thereby making the weight learning more focused on easily confused situations.

[0090] Modal feature importance scores are calculated based on the aforementioned feature weight matrix. Importance scores are first aggregated along the time slice dimension, then weighted and summarized along the sample dimension to obtain a sample-level trimodal importance vector. This vector is compared modally with a preset screening threshold to generate retention tags and description fields. Samples with good cross-modal consistency and high importance are given priority, while samples with lower importance but covering scarce scenarios are retained in a small proportion to maintain diversity.

[0091] The synthetic dataset is ranked according to the above priorities, taking into account both scene coverage and time window distribution of the samples to avoid concentration in a single scene or time period. The ranking results are used for stratified extraction of high-quality samples, while maintaining the cross-modal alignment index during extraction to ensure that videos, images, and trajectories stored in the same sample unit strictly correspond in time slices.

[0092] The final output is a traffic dataset containing cross-modal constraints. Each sample unit in the dataset carries a scene label, a snapshot of the feature weight matrix, and an importance vector, and records the source window and generation parameter snapshots, facilitating downstream training to reproduce the weight environment at that time. This traffic dataset serves as the training input for subsequent scene recognition models. The model can directly read the trimodal representation and weight priors of the sample units without repeatedly calculating attention masks and alignment relationships.

[0093] In one embodiment of the method for constructing a multimodal traffic dataset based on cross-modal constraints in this application, the following may also be included: Step S701: Divide the traffic dataset into samples according to scene type to generate a training sample library, construct a multimodal feature extractor based on the training sample library, use the multimodal feature extractor to extract key scene features, establish a feature learning module based on the key scene features, and train and optimize the feature learning module to generate a scene recognition model. Step S702: Process real-time traffic data based on the scene recognition model, generate feature representations from the input data through the multimodal feature extractor, calculate the scene type probability distribution based on the feature representations, perform decision analysis on the scene type probability distributions, and generate traffic control instructions based on preset control rules.

[0094] This embodiment uses the traffic dataset output in step S602 to divide samples according to scene type and generate a training sample library. During the division, sub-libraries are established for labels such as nighttime, rainy days, construction, intersections, and highways. Within each sub-library, the hierarchical ratio of time windows and locations is maintained to prevent scene and region bias in the training set. Sample entries use cross-modal aligned indexes to ensure that videos, images, and trajectories from the same time slice can be read simultaneously.

[0095] A multimodal feature extractor is constructed based on the training sample library. The video branch uses 3D convolutional coding for spatiotemporal cues, the image branch uses object detection backbone to extract region-level semantics, and the trajectory branch uses a temporal encoder to extract motion trends. The outputs of the three branches are aligned along the time slice dimension and then a joint representation is generated through a lightweight fusion layer. In this embodiment, the attention prior corresponding to the scene label is used as an optional input to limit the influence of irrelevant channels, making the extractor more focused on the effective modalities of the current scene.

[0096] The joint representation is fed into a feature learning module, which consists of a classification head and a consistency constraint head. The classification head predicts the scene type using the joint representation; the consistency constraint head calculates similarity between modality pairs to suppress cross-modal bias on the same time slice. A hierarchical sampling strategy is used during training to ensure that each sub-database participates in each training round; during optimization, the loss component and sub-database hit rate are recorded to monitor the learning sufficiency of different scenes. After training is complete, a scene recognition model is obtained, and a snapshot of the parameters of the feature extractor and classification head is retained.

[0097] After the scene recognition model is ready, real-time traffic data is used for inference. The real-time input follows the preprocessing and alignment methods of steps S201 and S202 to generate standardized video, image, and trajectory representations, each carrying a time-slice index. A multimodal feature extractor generates a joint representation on a time-slice basis, the classification head outputs the scene type probability distribution, and the consistency constraint head produces a cross-modal consistency score for confidence correction and auditing records.

[0098] Decision analysis is performed based on the aforementioned probability distribution and consistency score. First, the probability distribution is judged for threshold intervals. Time slices below the lower limit are marked as pending review and trigger a mitigation strategy; time slices falling into the stable interval are entered into rule mapping. The rule mapping table uses scenario type as the primary key and associates control items such as speed limit, signal timing, lane guidance, and warning broadcasts. It allows for weighted combinations based on confidence level and consistency score to generate traffic control instructions for the current time slice.

[0099] The generated traffic control commands, along with the scene prediction results, confidence scores, and consistency scores, are written to the operation log. The log uses a time-slice index as the primary key and links back to the scene sub-library and extractor parameter snapshots in the training sample library, facilitating retraining and parameter backtracking in subsequent maintenance cycles. This embodiment retains an alarm entry point for abnormal channels on the deployment side. When the consistency score consistently falls below a threshold range, the corresponding segment is automatically reported and added to the offline self-check queue, forming a closed loop from training to online inference and then backtracking for revision.

[0100] To effectively address the shortcomings of traditional technologies in data processing, constraint generation, and feature fusion, and to provide technical support for the construction of traffic datasets, this application provides an embodiment of a device for constructing multimodal traffic datasets based on cross-modal constraints, which implements all or part of the aforementioned method for constructing multimodal traffic datasets based on cross-modal constraints. See [link to embodiment]. Figure 2 The device for constructing a multimodal traffic dataset based on cross-modal constraints specifically includes the following components: Data processing module 10 is used to collect multimodal traffic data to generate a raw dataset, preprocess the raw dataset to generate a multimodal feature package, wherein the multimodal feature package includes video features, image features and trajectory features, extract metadata information based on the multimodal feature package to generate a scene description set, input the scene description set into a scene classifier to generate scene labels, select data verification rules from a dynamic rule base according to the scene labels, and apply the data verification rules to the multimodal feature package to generate data cleaning results. The multidimensional constraint module 20 is used to construct a generative network generation model branch set according to modal type based on the data cleaning results, input the video features into a three-dimensional convolutional layer to extract temporal features, embed scene physical constraints based on the temporal features to generate a video generation branch, input the image features into a target detection network to extract target features, embed illumination attenuation laws based on the target features to generate an image generation branch, input the trajectory features into a temporal encoder to extract motion features, embed vehicle dynamics constraints based on the motion features to generate a trajectory generation branch, construct a discriminant network based on the video generation branch, the image generation branch, and the trajectory generation branch, update the generation parameters through adversarial training, and expand and generate a synthetic dataset based on the generation parameters; The dataset application module 30 is used to train an attention fusion model to generate a feature weight matrix for the synthetic dataset according to scene type, calculate the importance of multimodal features based on the feature weight matrix, filter and combine the synthetic dataset according to the importance of multimodal features to generate a traffic dataset, use the traffic dataset to train a traffic scene recognition model, and analyze and process real-time traffic data based on the traffic scene recognition model to generate traffic control instructions.

[0101] As described above, the multimodal traffic dataset construction device based on cross-modal constraints provided in this application can achieve effective data cleaning through feature extraction and scene classification. A generation mechanism is constructed, combining physical constraints and adversarial training to establish a reliable synthesis strategy. Feature fusion is introduced, and the quality of the dataset is ensured through importance assessment and data filtering. This method effectively solves the shortcomings of traditional technologies in data processing, constraint generation, and feature fusion, providing technical support for the construction of traffic datasets.

[0102] From a hardware perspective, in order to effectively address the shortcomings of traditional technologies in data processing, constraint generation, and feature fusion, and to provide technical support for the construction of traffic datasets, this application provides an embodiment of an electronic device for implementing all or part of the aforementioned method for constructing multimodal traffic datasets based on cross-modal constraints. The electronic device specifically includes the following components: The system comprises a processor, memory, a communication interface, and a bus; wherein the processor, memory, and communication interface communicate with each other via the bus; the communication interface is used to realize information transmission between the multimodal traffic dataset construction device based on cross-modal constraints and core business systems, user terminals, and related databases and other related devices; the logic controller can be a desktop computer, tablet computer, or mobile terminal, etc., and this embodiment is not limited to these. In this embodiment, the logic controller can be implemented with reference to the embodiments of the multimodal traffic dataset construction method based on cross-modal constraints and the embodiments of the multimodal traffic dataset construction device based on cross-modal constraints in the embodiments, the contents of which are incorporated herein, and repeated details will not be described again.

[0103] It is understood that the user terminal may include smartphones, tablet computers, network set-top boxes, portable computers, desktop computers, personal digital assistants (PDAs), in-vehicle devices, smart wearable devices, etc. Among these, the smart wearable devices may include smart glasses, smartwatches, smart bracelets, etc.

[0104] In practical applications, parts of the method for constructing multimodal traffic datasets based on cross-modal constraints can be executed on the electronic device side as described above, or all operations can be completed in the client device. The choice can be made based on the processing power of the client device and the limitations of the user's usage scenario. This application does not impose any limitations on this. If all operations are completed in the client device, the client device may further include a processor.

[0105] The aforementioned client device may have a communication module (i.e., a communication unit) that can communicate with a remote server to achieve data transmission with the server. The server may include a server on the task scheduling center side; in other implementation scenarios, it may also include a server on an intermediate platform, such as a server on a third-party server platform that has a communication link with the task scheduling center server. The server may include a single computer device, a server cluster consisting of multiple servers, or a distributed server structure.

[0106] Figure 3This is a schematic block diagram illustrating the system configuration of the electronic device 9600 according to an embodiment of this application. Figure 3 As shown, the electronic device 9600 may include a central processing unit 9100 and a memory 9140; the memory 9140 is coupled to the central processing unit 9100. It is worth noting that... Figure 3 This is an example; other types of structures can also be used to supplement or replace this structure to achieve telecommunications functions or other functions.

[0107] In one embodiment, the functionality of the method for constructing a multimodal traffic dataset based on cross-modal constraints can be integrated into the central processing unit 9100. The central processing unit 9100 can be configured to perform the following controls: Step S101: Collect multimodal traffic data to generate a raw dataset, preprocess the raw dataset to generate a multimodal feature package, wherein the multimodal feature package includes video features, image features and trajectory features, extract metadata information based on the multimodal feature package to generate a scene description set, input the scene description set into a scene classifier to generate scene labels, select data verification rules from a dynamic rule base according to the scene labels, and apply the data verification rules to the multimodal feature package to generate data cleaning results; Step S102: Construct a generative network generation model branch set according to modality type for the data cleaning results; input the video features into a 3D convolutional layer to extract temporal features; embed scene physical constraints based on the temporal features to generate a video generation branch; input the image features into a target detection network to extract target features; embed illumination attenuation laws based on the target features to generate an image generation branch; input the trajectory features into a temporal encoder to extract motion features; embed vehicle dynamics constraints based on the motion features to generate a trajectory generation branch; construct a discriminant network based on the video generation branch, the image generation branch, and the trajectory generation branch; update the generation parameters through adversarial training; and expand and generate a synthetic dataset based on the generation parameters. Step S103: Train an attention fusion model on the synthetic dataset according to scene type to generate a feature weight matrix, calculate the multimodal feature importance based on the feature weight matrix, filter and combine the synthetic dataset according to the multimodal feature importance to generate a traffic dataset, use the traffic dataset to train a traffic scene recognition model, and analyze and process real-time traffic data based on the traffic scene recognition model to generate traffic control instructions.

[0108] As described above, the electronic device provided in this application embodiment achieves effective data cleaning through feature extraction and scene classification. A generation mechanism is constructed, combining physical constraints and adversarial training to establish a reliable synthesis strategy. Feature fusion is introduced, and the quality of the dataset is ensured through importance assessment and data filtering. This method effectively addresses the shortcomings of traditional techniques in data processing, constraint generation, and feature fusion, providing technical support for the construction of traffic datasets.

[0109] In another embodiment, the multimodal traffic dataset construction device based on cross-modal constraints can be configured separately from the central processing unit 9100. For example, the multimodal traffic dataset construction device based on cross-modal constraints can be configured as a chip connected to the central processing unit 9100, and the function of the multimodal traffic dataset construction method based on cross-modal constraints can be realized through the control of the central processing unit.

[0110] like Figure 3 As shown, the electronic device 9600 may further include: a communication module 9110, an input unit 9120, an audio processor 9130, a display 9160, and a power supply 9170. It is worth noting that the electronic device 9600 does not necessarily need to include these components. Figure 3 All components shown; in addition, the electronic device 9600 may also include Figure 3 For components not shown, please refer to existing technologies.

[0111] like Figure 3 As shown, the central processing unit 9100, sometimes also referred to as a controller or operating control, may include a microprocessor or other processor device and / or logic device, which receives inputs and controls the operation of various components of the electronic device 9600.

[0112] The memory 9140 may be, for example, one or more of a cache, flash memory, hard drive, removable media, volatile memory, non-volatile memory, or other suitable devices. It may store the aforementioned failure-related information, and also store a program for executing that information. The central processing unit 9100 may execute the program stored in the memory 9140 to perform information storage or processing, etc.

[0113] Input unit 9120 provides input to central processing unit 9100. Input unit 9120 may be, for example, a keypad or touch input device. Power supply 9170 provides power to electronic device 9600. Display 9160 displays images and text. Display may be, for example, an LCD display, but is not limited thereto.

[0114] The memory 9140 can be a solid-state memory, such as a read-only memory (ROM), random access memory (RAM), a SIM card, etc. It can also be a memory that retains information even when power is off, can be selectively erased, and contains more data; examples of this type of memory are sometimes referred to as EPROMs. The memory 9140 can also be some other type of device. The memory 9140 includes a buffer memory 9141 (sometimes referred to as a buffer). The memory 9140 may include an application / function storage unit 9142 for storing application programs and function programs or processes for executing the operation of the electronic device 9600 via the central processing unit 9100.

[0115] The memory 9140 may also include a data storage unit 9143 for storing data, such as contacts, digital data, pictures, sounds, and / or any other data used by the electronic device. The driver storage unit 9144 of the memory 9140 may include various drivers for the electronic device for communication functions and / or for performing other functions of the electronic device (such as messaging applications, address book applications, etc.).

[0116] The communication module 9110 is a transmitter / receiver that sends and receives signals via the antenna 9111. The communication module 9110 (transmitter / receiver) is coupled to the central processing unit 9100 to provide input signals and receive output signals, which is the same as in a conventional mobile communication terminal.

[0117] Based on different communication technologies, multiple communication modules 9110 can be configured in the same electronic device, such as cellular network modules, Bluetooth modules, and / or wireless LAN modules. The communication module 9110 (transmitter / receiver) is also coupled to a speaker 9131 and a microphone 9132 via an audio processor 9130 to provide audio output via the speaker 9131 and receive audio input from the microphone 9132, thereby realizing typical telecommunications functions. The audio processor 9130 may include any suitable buffer, decoder, amplifier, etc. Additionally, the audio processor 9130 is coupled to a central processing unit 9100, enabling on-device recording via the microphone 9132 and on-device playback of stored audio via the speaker 9131.

[0118] Embodiments of this application also provide a computer-readable storage medium capable of implementing all steps of the multimodal traffic dataset construction method based on cross-modal constraints, where the execution subject is a server or client, as described in the above embodiments. The computer-readable storage medium stores a computer program that, when executed by a processor, implements all steps of the multimodal traffic dataset construction method based on cross-modal constraints, where the execution subject is a server or client, as described in the above embodiments. For example, when the processor executes the computer program, it implements the following steps: Step S101: Collect multimodal traffic data to generate a raw dataset, preprocess the raw dataset to generate a multimodal feature package, wherein the multimodal feature package includes video features, image features and trajectory features, extract metadata information based on the multimodal feature package to generate a scene description set, input the scene description set into a scene classifier to generate scene labels, select data verification rules from a dynamic rule base according to the scene labels, and apply the data verification rules to the multimodal feature package to generate data cleaning results; Step S102: Construct a generative network generation model branch set according to modality type for the data cleaning results; input the video features into a 3D convolutional layer to extract temporal features; embed scene physical constraints based on the temporal features to generate a video generation branch; input the image features into a target detection network to extract target features; embed illumination attenuation laws based on the target features to generate an image generation branch; input the trajectory features into a temporal encoder to extract motion features; embed vehicle dynamics constraints based on the motion features to generate a trajectory generation branch; construct a discriminant network based on the video generation branch, the image generation branch, and the trajectory generation branch; update the generation parameters through adversarial training; and expand and generate a synthetic dataset based on the generation parameters. Step S103: Train an attention fusion model on the synthetic dataset according to scene type to generate a feature weight matrix, calculate the multimodal feature importance based on the feature weight matrix, filter and combine the synthetic dataset according to the multimodal feature importance to generate a traffic dataset, use the traffic dataset to train a traffic scene recognition model, and analyze and process real-time traffic data based on the traffic scene recognition model to generate traffic control instructions.

[0119] As described above, the computer-readable storage medium provided in this application embodiment achieves effective data cleaning through feature extraction and scene classification. A generation mechanism is constructed, combining physical constraints and adversarial training to establish a reliable synthesis strategy. Feature fusion is introduced, and the quality of the dataset is ensured through importance assessment and data filtering. This method effectively addresses the shortcomings of traditional techniques in data processing, constraint generation, and feature fusion, providing technical support for the construction of traffic datasets.

[0120] Embodiments of this application also provide a computer program product capable of implementing all steps in the method for constructing a multimodal traffic dataset based on cross-modal constraints, where the execution subject is a server or client, as described in the above embodiments. When executed by a processor, this computer program / instruction implements the steps of the method for constructing a multimodal traffic dataset based on cross-modal constraints. For example, the computer program / instruction implements the following steps: Step S101: Collect multimodal traffic data to generate a raw dataset, preprocess the raw dataset to generate a multimodal feature package, wherein the multimodal feature package includes video features, image features and trajectory features, extract metadata information based on the multimodal feature package to generate a scene description set, input the scene description set into a scene classifier to generate scene labels, select data verification rules from a dynamic rule base according to the scene labels, and apply the data verification rules to the multimodal feature package to generate data cleaning results; Step S102: Construct a generative network generation model branch set according to modality type for the data cleaning results; input the video features into a 3D convolutional layer to extract temporal features; embed scene physical constraints based on the temporal features to generate a video generation branch; input the image features into a target detection network to extract target features; embed illumination attenuation laws based on the target features to generate an image generation branch; input the trajectory features into a temporal encoder to extract motion features; embed vehicle dynamics constraints based on the motion features to generate a trajectory generation branch; construct a discriminant network based on the video generation branch, the image generation branch, and the trajectory generation branch; update the generation parameters through adversarial training; and expand and generate a synthetic dataset based on the generation parameters. Step S103: Train an attention fusion model on the synthetic dataset according to scene type to generate a feature weight matrix, calculate the multimodal feature importance based on the feature weight matrix, filter and combine the synthetic dataset according to the multimodal feature importance to generate a traffic dataset, use the traffic dataset to train a traffic scene recognition model, and analyze and process real-time traffic data based on the traffic scene recognition model to generate traffic control instructions.

[0121] As described above, the computer program product provided in this application achieves effective data cleaning through feature extraction and scene classification. A generation mechanism is constructed, combining physical constraints and adversarial training to establish a reliable synthesis strategy. Feature fusion is introduced, and the quality of the dataset is ensured through importance assessment and data filtering. This method effectively addresses the shortcomings of traditional techniques in data processing, constraint generation, and feature fusion, providing technical support for the construction of traffic datasets.

[0122] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, apparatus, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0123] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (devices), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0124] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0125] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0126] Specific embodiments have been used to illustrate the principles and implementation methods of this invention. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of this invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this invention. Therefore, the content of this specification should not be construed as a limitation of this invention.

Claims

1. A method for constructing a multimodal traffic dataset based on cross-modal constraints, characterized in that, The method includes: Multimodal traffic data is collected to generate a raw dataset. The raw dataset is preprocessed to generate a multimodal feature package, which includes video features, image features, and trajectory features. Metadata information is extracted based on the multimodal feature package to generate a scene description set. The scene description set is input into a scene classifier to generate scene labels. Data verification rules are selected from a dynamic rule base according to the scene labels. The data verification rules are applied to the multimodal feature package to generate data cleaning results. The data cleaning results are used to construct a generative network model branch set according to modality type. The video features are input into a 3D convolutional layer to extract temporal features. Based on the temporal features, scene physical constraints are embedded to generate a video generation branch. The image features are input into a target detection network to extract target features. Based on the target features, illumination attenuation laws are embedded to generate an image generation branch. The trajectory features are input into a temporal encoder to extract motion features. Based on the motion features, vehicle dynamics constraints are embedded to generate a trajectory generation branch. A discriminant network is constructed based on the video generation branch, the image generation branch, and the trajectory generation branch. The generation parameters are updated through adversarial training. Based on the generation parameters, a synthetic dataset is expanded and generated. An attention fusion model is trained on the synthetic dataset according to scene type to generate a feature weight matrix. Multimodal feature importance is calculated based on the feature weight matrix. The synthetic dataset is then filtered and combined according to the multimodal feature importance to generate a traffic dataset. The traffic dataset is used to train a traffic scene recognition model. Based on the traffic scene recognition model, real-time traffic data is analyzed and processed to generate traffic control instructions.

2. The method for constructing a multimodal traffic dataset based on cross-modal constraints according to claim 1, characterized in that, The process involves collecting multimodal traffic data to generate a raw dataset, preprocessing the raw dataset to generate a multimodal feature package, wherein the multimodal feature package includes video features, image features, and trajectory features, and extracting metadata information based on the multimodal feature package to generate a scene description set, including: Data acquisition streams are generated by acquiring multiple video streams from traffic monitoring equipment, image sequences from vehicle-mounted cameras, and trajectory point sets from vehicle-mounted positioning equipment. Acquisition time windows are divided according to the data acquisition streams. A data structure mapping table is established based on preset format specifications. Data within the acquisition time windows is converted according to the data structure mapping table to generate raw data packets containing timestamps. Based on the original data packet, data preprocessing is performed to generate a multimodal dataset. The multimodal data is time-aligned according to the timestamp marker. The aligned data is downsampled according to a preset sampling interval. The downsampled data is normalized to generate standardized features. A feature index table is constructed based on the standardized features to generate a multimodal feature package containing video features, image features, and trajectory features.

3. The method for constructing a multimodal traffic dataset based on cross-modal constraints according to claim 1, characterized in that, The step of inputting the scene description set into a scene classifier to generate scene labels, selecting data verification rules from a dynamic rule base based on the scene labels, and applying the data verification rules to the multimodal feature package to generate data cleaning results includes: The scene description set is processed by feature vectorization to generate a feature matrix. A scene classification training set is constructed based on the scene annotation data. A multilayer perceptron classifier is trained based on the scene classification training set. The feature matrix is ​​input into the multilayer perceptron classifier to generate a scene confidence distribution. The scene confidence distribution is filtered according to a preset confidence threshold to generate a scene label set containing scene type and confidence. Based on the scene tag set, a dynamic rule base is queried to generate a verification rule set. According to the scene type, the corresponding physical constraint rule is selected. The physical constraint rule is combined with the confidence level to construct a data cleaning template. The data cleaning template is applied to the multimodal feature package to perform anomaly detection. The data cleaning result is generated based on the detection result.

4. The method for constructing a multimodal traffic dataset based on cross-modal constraints according to claim 1, characterized in that, The process of constructing a generative network model branch set based on modality type for the data cleaning results, inputting the video features into a 3D convolutional layer to extract temporal features, embedding scene physical constraints based on the temporal features to generate a video generation branch, inputting the image features into a target detection network to extract target features, embedding illumination attenuation laws based on the target features to generate an image generation branch, inputting the trajectory features into a temporal encoder to extract motion features, and embedding vehicle dynamics constraints based on the motion features to generate a trajectory generation branch, includes: Modal separation is performed on the data cleaning results to generate a modal feature set. A feature extraction network is constructed based on the modal feature set. Video features are input into a 3D convolutional layer to extract spatiotemporal features, image features are input into a target detection network to extract region features, and trajectory features are input into a temporal encoder to extract motion features. The spatiotemporal features, region features, and motion features are normalized to generate a feature vector group containing multimodal coding representation. A physical constraint layer is constructed based on the feature vector group, and the scene physical rules are mapped into a constraint parameter matrix. Based on the constraint parameter matrix, the spatiotemporal features are embedded with scene constraints to generate a video branch, the region features are embedded with illumination constraints to generate an image branch, and the motion features are embedded with dynamic constraints to generate a trajectory branch. The video branch, the image branch, and the trajectory branch are combined to generate a model branch set.

5. The method for constructing a multimodal traffic dataset based on cross-modal constraints according to claim 1, characterized in that, The step of constructing a discriminative network based on the video generation branch, the image generation branch, and the trajectory generation branch, updating the generation parameters through adversarial training, and expanding and generating a synthetic dataset based on the generation parameters includes: Feature fusion is performed on the video generation branch, image generation branch, and trajectory generation branch to generate discriminative input data. The discriminative input data is input into a multimodal discriminator to extract intermodal correlation features. A discriminative scoring function is constructed based on the intermodal correlation features. An adversarial loss calculation module is established according to the discriminative scoring function. The adversarial loss calculation module is combined with physical constraint loss to generate a training objective function. Generative adversarial training is performed based on the training objective function. Multimodal data is augmented and sampled according to the generation parameters. The augmented sampling results are input into the multimodal discriminator to calculate the realism score. High-quality samples are selected based on the realism score. The high-quality samples are labeled and organized to generate a synthetic dataset.

6. The method for constructing a multimodal traffic dataset based on cross-modal constraints according to claim 1, characterized in that, The process involves training an attention fusion model on the synthetic dataset according to scene type to generate a feature weight matrix, calculating multimodal feature importance based on the feature weight matrix, and filtering and combining the synthetic dataset according to the multimodal feature importance to generate a traffic dataset, including: Scene classification is performed on the synthetic dataset to generate scene sample groups. The scene sample groups are divided into training data according to a preset batch. A multi-head attention module is constructed based on the training data. An attention mask matrix is ​​established according to the scene labels. The attention mask matrix is ​​used to calculate the inter-modal attention score. The inter-modal attention score is normalized to generate a feature weight matrix. Modal feature importance scores are calculated based on the feature weight matrix. The modal feature importance scores are compared with preset screening thresholds. The samples in the synthetic dataset are ranked by importance according to the comparison results. High-quality samples are selected and combined according to the importance ranking results to generate a traffic dataset containing cross-modal constraint relationships.

7. The method for constructing a multimodal traffic dataset based on cross-modal constraints according to claim 1, characterized in that, The step of using the traffic dataset to train a traffic scene recognition model, and then analyzing and processing real-time traffic data based on the traffic scene recognition model to generate traffic control instructions includes: The traffic dataset is divided into samples according to scene type to generate a training sample library. A multimodal feature extractor is constructed based on the training sample library. The multimodal feature extractor is used to extract key scene features. A feature learning module is established based on the key scene features. The feature learning module is trained and optimized to generate a scene recognition model. Real-time traffic data is processed based on the scene recognition model. The input data is processed by the multimodal feature extractor to generate feature representations. The scene type probability distribution is calculated based on the feature representations. Decision analysis is performed on the scene type probability distribution. Traffic control instructions are generated based on preset control rules.

8. A device for constructing a multimodal traffic dataset based on cross-modal constraints, characterized in that, The device includes: The data processing module is used to collect multimodal traffic data to generate a raw dataset, preprocess the raw dataset to generate a multimodal feature package, wherein the multimodal feature package includes video features, image features and trajectory features, extract metadata information based on the multimodal feature package to generate a scene description set, input the scene description set into a scene classifier to generate scene labels, select data verification rules from a dynamic rule base according to the scene labels, and apply the data verification rules to the multimodal feature package to generate data cleaning results. A multidimensional constraint module is used to construct a generative network generation model branch set according to modality type based on the data cleaning results. The video features are input into a three-dimensional convolutional layer to extract temporal features. Based on the temporal features, scene physical constraints are embedded to generate a video generation branch. The image features are input into a target detection network to extract target features. Based on the target features, illumination attenuation laws are embedded to generate an image generation branch. The trajectory features are input into a temporal encoder to extract motion features. Based on the motion features, vehicle dynamics constraints are embedded to generate a trajectory generation branch. A discriminant network is constructed based on the video generation branch, the image generation branch, and the trajectory generation branch. The generation parameters are updated through adversarial training. Based on the generation parameters, a synthetic dataset is expanded and generated. The dataset application module is used to train an attention fusion model to generate a feature weight matrix for the synthetic dataset according to scene type, calculate the importance of multimodal features based on the feature weight matrix, filter and combine the synthetic dataset according to the importance of multimodal features to generate a traffic dataset, use the traffic dataset to train a traffic scene recognition model, and analyze and process real-time traffic data based on the traffic scene recognition model to generate traffic control instructions.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method for constructing a multimodal traffic dataset based on cross-modal constraints as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the method for constructing a multimodal traffic dataset based on cross-modal constraints as described in any one of claims 1 to 7.

Images