Bus dynamic scheduling method and system based on visual perception and multi-modal large model

Abstract

The invention discloses a bus dynamic scheduling method based on visual perception and a multi-modal large model. The method comprises the following steps: acquiring basic information of buses and visual perception data and real-time dynamic data of vehicle operation; performing feature extraction on the obtained data to obtain visual perception features and real-time dynamic features; carrying out association alignment on the bus basic information, the visual perception features and the real-time dynamic features to generate a multi-modal feature matrix, generating a fusion weight matrix through cross-modal association, and generating multi-modal fusion features through the fusion weight matrix; and on the basis of the multi-modal fusion features, generating an abnormal traffic event identification result and future traffic condition prediction data through a preset multi-modal scheduling model, and then generating a bus scheduling scheme through a preset scheduling objective function. According to the invention, through multi-mode perception prediction, conversion from passive response to active prediction is realized, and the efficiency and reliability of bus scheduling are significantly improved.

Description

Technical Field

[0001] This application relates to the field of public transport scheduling technology, and in particular to a method and system for dynamic public transport scheduling based on visual perception and a multimodal large model. Background Technology

[0002] Bus dispatching refers to the process by which bus operators, based on urban passenger demand, road conditions, vehicle and personnel resources, scientifically plan and dynamically adjust the operation plan, departure intervals, routes, stops, and driver and conductor schedules of buses.

[0003] Although current bus dispatching systems incorporate visual perception, using cameras and other visual devices deployed on buses and at stops, combined with deep learning and big data technologies, they can obtain real-time information on passenger flow, vehicle location, and operational status, enabling dynamic dispatching and optimization. However, when faced with situations such as road congestion or temporary road closures, the existing dispatching mechanism still suffers from slow response and untimely dispatching. For example, in the event of a sudden traffic jam, the dispatch center may not be aware of the situation and may not be able to effectively dispatch buses, leading to delays and an inability to quickly and effectively adjust routes and supplement capacity, which can easily cause large-scale delays. Summary of the Invention

[0004] The purpose of this application is to provide a method and system for dynamic bus scheduling based on visual perception and multimodal large models. By using multimodal data perception and model prediction, the system can improve the timely response and effective scheduling capabilities of the entire bus scheduling system in the face of emergencies such as road congestion and temporary traffic control.

[0005] Firstly, this application provides a dynamic bus scheduling method based on visual perception and a multimodal large model, employing the following technical solution: Acquire basic information about public transport vehicles and visual perception data and real-time dynamic data of vehicle operation; the visual perception data includes passenger flow image data at stations, passenger flow image data inside the vehicle, and road condition image data outside the vehicle; the real-time dynamic data includes vehicle operation data and road traffic control data. Feature extraction is performed on visual perception data and real-time dynamic data respectively to obtain visual perception features and real-time dynamic features; The basic information of public transport vehicles is associated and aligned with visual perception features and real-time dynamic features to generate a multimodal feature matrix; Based on the multimodal feature matrix, a fusion weight matrix is ​​generated through cross-modal correlation, and multimodal fusion features are generated through the fusion weight matrix; Based on multimodal fusion features, abnormal traffic event identification results and future traffic condition prediction data are generated by pre-setting a multimodal scheduling model; Based on the results of abnormal traffic event identification and future traffic condition prediction data, a bus dispatching scheme is generated by pre-setting a dispatching objective function.

[0006] Through the above technical solutions, visual perception data can firstly capture visual features such as road congestion and accidents in real time. Combined with real-time traffic control data, abnormal events can be quickly identified through multimodal models. In addition, multimodal models can predict future traffic conditions, enabling public transport scheduling to shift from passive response to proactive prevention, thereby effectively alleviating the problem of untimely response to emergencies such as road congestion and temporary traffic control.

[0007] Optionally, the step of extracting features from visual perception data and real-time dynamic data respectively to obtain visual perception features and real-time dynamic features includes: Visual perception data is preprocessed, and the preprocessed image is subjected to target detection to obtain visual perception features, including in-vehicle passenger flow features, out-of-vehicle road condition features, and station passenger flow features. Real-time dynamic features are generated by extracting time-series and text features from real-time dynamic data. These real-time dynamic features include vehicle motion features and traffic control features.

[0008] Optionally, the step of associating and aligning the basic information of the bus with visual perception features and real-time dynamic features to generate a multimodal feature matrix includes: Set a baseline time granularity and uniformly convert visual perception features and real-time dynamic features into time slices at that baseline time granularity. By pre-setting a city geographic information system, all data is mapped to a unified latitude and longitude coordinate system; Visual perception features and real-time dynamic features within the same time slice and geographical location range are bound to basic information of public transport vehicles to generate a set of unit features; Based on a single-source feature set, the dimensions of features from different modalities are unified to generate a structured multimodal feature matrix.

[0009] Optionally, after associating and aligning the basic information of the bus with visual perception features and real-time dynamic features to generate a multimodal feature matrix, the method further includes: Based on environmental data, the influencing features and influence coefficients are obtained through a pre-set environment-feature mapping library; Based on the impact characteristics and impact coefficients, fine-tuning factors are obtained through preset calculation rules; Based on the fine-tuning factor, the multimodal feature matrix is ​​fine-tuned to obtain the adjusted multimodal feature matrix.

[0010] Optionally, the step of fine-tuning the multimodal feature matrix based on a fine-tuning factor to obtain the adjusted multimodal feature matrix includes: Based on environmental data, the corresponding environmental error rate is obtained through a preset environment-error mapping rule; Based on the multimodal feature matrix, the feature values ​​corresponding to the influencing features are determined, and the restored feature values ​​are calculated and obtained through the environmental error rate. The restored eigenvalues ​​are adjusted using a fine-tuning factor to obtain the fine-tuned eigenvalues. Based on the fine-tuned eigenvalues, the multimodal feature matrix is ​​adjusted to obtain the adjusted multimodal feature matrix.

[0011] Optionally, the step of generating a fusion weight matrix based on the multimodal feature matrix through cross-modal correlation, and generating multimodal fusion features through the fusion weight matrix, includes: Based on the multimodal feature matrix, the first association weight matrix is ​​obtained through preset association rules; Based on the multimodal feature matrix, a second association weight matrix is ​​obtained through an attention mechanism network; The first correlation weight matrix and the second correlation weight matrix are merged to generate a fused weight matrix; Based on the multimodal feature matrix, multimodal fusion features are generated by fusing the weight matrix.

[0012] Optionally, the step of generating a bus dispatching scheme based on the abnormal traffic event identification results and future traffic condition prediction data, through a preset dispatching objective function, includes: Based on the results of abnormal traffic event identification and future traffic condition prediction data, determine the type of dispatching demand; Based on the type of scheduling requirement, multiple candidate scheduling strategies are obtained through a pre-set scheduling strategy library, and a candidate scheduling strategy set is formed. Traverse the candidate scheduling strategy set, and for each candidate scheduling strategy, obtain a strategy score through a preset scheduling objective function; After the traversal is complete, the candidate scheduling policy with the highest policy score is recorded as the target scheduling policy; Based on the target scheduling strategy, an executable bus scheduling scheme is generated.

[0013] Optionally, after generating a bus dispatching scheme based on the abnormal traffic event identification results and future traffic condition prediction data through a preset dispatching objective function, the process includes: By pre-setting monitoring dimensions, the execution status of the bus dispatching plan can be obtained in real time; Based on the execution status of the bus dispatching plan, an evaluation report is obtained through a pre-set evaluation method.

[0014] Secondly, this application provides a public transport dynamic scheduling system based on visual perception and a multimodal large model, comprising: The data acquisition module 101 is used to acquire basic information of the bus and visual perception data and real-time dynamic data of vehicle operation; the visual perception data includes station passenger flow image data, in-vehicle passenger flow image data and out-of-vehicle road condition image data, and the real-time dynamic data includes vehicle operation data and road traffic control data. The feature extraction module 102 is used to extract features from visual perception data and real-time dynamic data respectively, obtain visual perception features and real-time dynamic features, and associate and align the basic information of the bus with the visual perception features and real-time dynamic features to generate a multimodal feature matrix. The feature fusion module 103 is used to generate a fusion weight matrix based on the multimodal feature matrix through cross-modal correlation, and to generate multimodal fusion features through the fusion weight matrix; The model prediction module 104 is used to generate abnormal traffic event identification results and future traffic condition prediction data based on multimodal fusion features and a preset multimodal scheduling model. The bus dispatch generation module 105 is used to generate a bus dispatch plan based on the results of abnormal traffic event identification and future traffic condition prediction data, through a preset dispatch objective function.

[0015] Thirdly, this application provides a computer-readable storage medium storing a computer program that can be loaded by a processor and executed as described above, a method for dynamic bus scheduling based on visual perception and a multimodal large model.

[0016] In summary, this application first achieves a shift from passive response to proactive prediction through multimodal data perception and model prediction, effectively alleviating the problem of untimely response to emergencies such as road congestion and temporary traffic control, thereby improving the efficiency and reliability of bus scheduling. Furthermore, when fusing multimodal features, it employs a fusion method combining explicit association weight matrices and attention association weights to achieve dual protection. The resulting multimodal fusion features not only conform to objective logic but also capture the implicit associations in complex scenarios, improving the prediction accuracy and decision reliability of the multimodal scheduling model. In addition, considering the differences in the actual contribution of features under different environments, the application fine-tunes the multimodal fusion features through dynamic environment adaptation, improving the matching degree between the multimodal fusion features and the current scenario, further enhancing the prediction accuracy of the subsequent multimodal scheduling model. Attached Figure Description

[0017] Figure 1 This is a flowchart of a dynamic bus scheduling method based on visual perception and multimodal large model provided in an embodiment of this application; Figure 2 This is a flowchart of the process for generating a multimodal feature matrix provided in an embodiment of this application; Figure 3This is a flowchart illustrating the generation of multimodal fusion features provided in an embodiment of this application; Figure 4 This is a flowchart provided in this application embodiment that generates a bus dispatching scheme based on abnormal traffic event identification results and future traffic condition prediction data, through a preset dispatching objective function; Figure 5 This is a schematic diagram of a public transport dynamic scheduling system based on visual perception and multimodal large model provided in an embodiment of this application. Detailed Implementation

[0018] The following is in conjunction with the appendix Figure 1 - Appendix Figure 5 This application will be described in further detail below.

[0019] This application provides a dynamic bus scheduling method based on visual perception and a multimodal large model. See [link to relevant documentation]. Figure 1 This includes the following steps: S100: Acquire basic information about public transport vehicles and visual perception data and real-time dynamic data of vehicle operation.

[0020] S200: Extract features from visual perception data and real-time dynamic data respectively to obtain visual perception features and real-time dynamic features.

[0021] S300: Associate and align the basic information of public transport vehicles with visual perception features and real-time dynamic features to generate a multimodal feature matrix.

[0022] S400: Based on the multimodal feature matrix, a fusion weight matrix is ​​generated through cross-modal correlation, and multimodal fusion features are generated through the fusion weight matrix.

[0023] S500, based on multimodal fusion features, generates abnormal traffic event identification results and future traffic condition prediction data through a preset multimodal scheduling model.

[0024] S600 generates a bus dispatching scheme based on the results of abnormal traffic event identification and future traffic condition prediction data through a preset dispatching objective function.

[0025] Among them, the basic information of public transport vehicles mainly refers to the vehicle's number, route, specifications, and historical dispatch information.

[0026] Visual perception data includes station passenger flow image data, in-vehicle passenger flow image data, and external road condition image data. Station passenger flow image data is acquired through visual acquisition equipment deployed at bus stations, including images of the station area and the surrounding road surface. In-vehicle passenger flow image data is acquired through image acquisition equipment deployed inside buses, including images of the inside of the bus. External road condition image data is acquired through data sharing with transportation departments and can be captured by road surveillance cameras, including real-time images of the sections covered by bus routes and real-time images of intersections or key road sections.

[0027] Real-time dynamic data includes vehicle operation data and road traffic control data; vehicle operation data includes real-time GPS location and speed; road traffic control information mainly refers to road event information synchronized with temporary or sudden traffic control measures (such as traffic accidents, road collapses, and temporary emergency tasks). Both the acquired visual perception data and real-time dynamic data are timestamped.

[0028] After acquiring visual perception data and real-time dynamic data, feature extraction will be performed on the visual perception data and real-time dynamic data respectively to obtain visual perception features and real-time dynamic features.

[0029] Specifically, key features are extracted from visual perception data and real-time dynamic data respectively to generate visual perception features and real-time dynamic features, including the following steps: S210. Perform image preprocessing on the visual perception data, and obtain visual perception features by target detection on the preprocessed image.

[0030] S220. Real-time dynamic features are generated by extracting time-series and text features from real-time dynamic data.

[0031] Among them, visual perception features include station passenger flow features, in-vehicle passenger flow features, and external road condition features. Station passenger flow features are used to characterize the passenger flow status at the station, including the number of people waiting and the density; in-vehicle passenger flow features are used to characterize the passenger flow status inside the bus, including the number of people inside and the crowding level; external road condition features are used to characterize the traffic status of the road outside the vehicle, including road segment traffic density, congestion level, road events, etc.

[0032] Real-time dynamic features include vehicle motion features and traffic control features. Vehicle motion features are used to characterize the operating status of buses, including speed, acceleration, real-time location, and estimated arrival time. Traffic control features are used to characterize temporary control notices or descriptions of traffic accidents issued by traffic management departments.

[0033] In addition, considering that environmental factors can also affect the scheduling of public transport vehicles, the real-time dynamic data also includes environmental data, mainly referring to weather and time-of-day conditions.

[0034] First, the visual perception data, namely images or visual modal data, is preprocessed. For example, images are denoised using Gaussian filtering, and images are cropped by extracting the Region of Interest (ROI) to eliminate background interference.

[0035] Then, the preprocessed image data is used to identify passengers in the image through object detection models, such as YOLOv8 or Faster R-CNN, and statistical data is collected to obtain features such as passenger data or the number of people waiting for the bus. Furthermore, through density clustering algorithms, such as BSCAN, the spatial distribution of passengers can be analyzed to calculate the cluster density and obtain features such as density and crowding.

[0036] By analyzing vehicle-view images and vehicle location data, features such as traffic density and congestion levels can be determined. By using object detection and OCR recognition, road event characteristics can be identified if warning signs such as construction or accidents are detected.

[0037] Real-time dynamic data is divided into two categories: one is time-series numerical data, which corresponds to vehicle operation data. First, the vehicle's speed and current location can be determined. By combining the station coordinates of the preset route, the estimated time to the next station or the waiting time at the current station can be determined.

[0038] Another type is text-based, which corresponds to road traffic control data and road events identified through parking space and road condition data. All of these are text descriptions. First, the text is preprocessed, for example, by cleaning the text and removing punctuation and redundant words. Then, text features are extracted, such as by using named entity recognition models (like BERT-NER) to extract key entities, including location entities, time entities, and event type entities. These entities are then integrated into structured traffic control features.

[0039] After obtaining the visual perception features and real-time dynamic features, it is also necessary to associate and align the basic information of the bus with the visual perception features and real-time dynamic features to generate a multimodal feature matrix.

[0040] The core of association alignment is to break down the information silos of multi-source data and construct a complete description of the spatiotemporal state of the public transportation system. The multimodal feature matrix, as the structured result after association alignment, is essentially the structured carrier of this complete description. The multimodal feature matrix is ​​characterized as a multi-dimensional state set that integrates static attributes (basic information of public transportation vehicles), spatial state (visual perception features), and temporal state (real-time dynamic features) with vehicle-time-space as three-dimensional coordinates.

[0041] Specifically, see Figure 2 The process involves associating and aligning basic bus vehicle information with visual perception features and real-time dynamic features to generate a multimodal feature matrix, including the following steps: S310. Set a reference time granularity and convert visual perception features and real-time dynamic features into time slices of this reference time granularity.

[0042] S320: By pre-setting a city geographic information system, all data is mapped to a unified latitude and longitude coordinate system.

[0043] S330. Bind the visual perception features and real-time dynamic features within the same time slice and geographical location range with the basic information of the bus to generate a set of unit features.

[0044] S340. Based on a single-source feature set, the features of different modalities are unified in dimensionality to generate a structured multimodal feature matrix.

[0045] Since the acquisition frequency of different modal features varies, for example, visual perception features may be acquired at 20 seconds per frame, while GPS real-time dynamic features may be acquired at 10 seconds per time, time alignment needs to be achieved by unifying the time granularity.

[0046] Therefore, in this embodiment, a reference time granularity is first set, and visual perception features and real-time dynamic features are uniformly converted into time slices of the reference time granularity.

[0047] For example, if the baseline time granularity is set to 1 minute, a visual sensing device (such as a camera) can collect one frame of image every 30 seconds, and two consecutive frames can be merged into an image sequence feature with a 1-minute granularity; GPS positioning data is updated every 10 seconds, and a location trajectory with a 1-minute granularity can be generated by interpolation.

[0048] In addition, since the locations where multimodal features are collected may be based on different coordinate systems, such as the local coordinate system used by cameras and the WGS84 latitude and longitude used by GPS, it is necessary to convert them to a unified latitude and longitude coordinate system through a city geographic information system (GIS) to achieve spatial alignment.

[0049] Using the same time slice and spatial range as the basis for judgment, visual perception features, real-time dynamic features, and basic information of public transport vehicles can be bound together to form a spatiotemporal unit-multi-feature mapping relationship, denoted as the unit feature set. The same time slice means judging whether the visual perception features and real-time dynamic features belong to the same time period, that is, the same time slice. For example, the passenger flow inside the vehicle collected at 00:01:20 and the vehicle speed collected at 00:01:45 both belong to the 00:01-00:02 slice.

[0050] The "same spatial range" refers to setting a spatial location range. Only data belonging to the same spatial location range are considered related. For example, a distance threshold can be set. Station passenger flow data has corresponding location information, and vehicle operation data also has corresponding location information. The distance between the locations is calculated, and if it is less than a preset distance, they are considered to belong to the same spatial range. For example, the number of people waiting at a station must be matched with the vehicle operation data that passes through that station. If the passenger flow data of that station is associated with vehicles that do not pass through that station, it is of no value for vehicle scheduling.

[0051] In addition, the basic information of public transport vehicles includes vehicle ID (vehicle ID adopts the format of route number + vehicle serial number, such as "Route 3 B1023"). Therefore, the vehicle ID can be used as an association identifier to bind visual perception features and real-time dynamic features with the basic information of public transport vehicles in order to obtain a set of unit features.

[0052] For example, for bus route B1023, the unit feature set generated within 50 meters of stop S028 between 08:30 and 08:31 is: {"Basic information of the bus":{"Vehicle number":"B1023","Route":"Route 3","Passenger capacity":80},"Visual perception features":{"Passenger flow characteristics at the stop":{"Waiting passengers":30,"Density (people / m²)} 2 )":2.8},"Passenger Flow Characteristics":{"Number of Passengers":42,"Crowding Degree":0.525},"Parking Space and Road Condition Characteristics":{"Road Segment Traffic Density (vehicles / minute)":120,"Congestion Level (Levels 1-5)":3,"Road Events":{"Event Type":"None","Event Identification Confidence":0.1}}},"Real-Time Dynamic Characteristics":{"Vehicle Motion Characteristics":{"Speed ​​(km / h)":28,"Acceleration (m / s²) 2 )":-0.5,"Real-time location":39.9147、116.4038,Estimated arrival time (seconds):87},"Traffic control characteristics":{"Type":"None"}}}}.

[0053] Because different features have different dimensions—some features are numerical and others are categorical—standardization is still needed to achieve uniformity.

[0054] The core of standardization is to eliminate the dimensional differences between different features, mapping all feature values ​​to a uniform interval (usually [0,1] or [-1,1]), which facilitates the model's fair learning of the importance of each feature. Different types of features will have corresponding standardization methods. For example, for numerical features, min-max standardization is used, where "yes" = 1 and "no" = 0 for data of type "yes".

[0055] The effectiveness of features depends on environmental conditions, as the actual contribution of the same feature may differ significantly under different environments.

[0056] Therefore, in this embodiment of the application, after obtaining the multimodal feature matrix, the impact of the current environment on the multimodal feature matrix is ​​measured based on the current environmental data, and corresponding fine-tuning is performed.

[0057] Specifically, after obtaining the multimodal feature matrix, the following steps are also included: S350: Based on environmental data, obtain influencing features and influence coefficients through a preset environment-feature mapping library.

[0058] S360: Based on the influence characteristics and influence coefficients, fine-tuning factors are obtained through preset calculation rules.

[0059] S370. Based on the fine-tuning factor, the multimodal fusion features are fine-tuned to obtain the adjusted modal fusion features.

[0060] Among them, the preset environment-feature mapping library is a database of mapping relationships between environments and features built based on historical data and domain experience. It stores corresponding influencing features and influence coefficients for different environmental data.

[0061] Environmental data is divided into weather environment and time period environment. For example, the weather environment has visual perception features, which mainly affect the detection of image data. The impact dimensions are precipitation level and visibility. The higher the precipitation level, the lower the effectiveness of visual perception features; the lower the visibility, the lower the effectiveness of visual perception features.

[0062] The time period environment has an impact on real-time passenger flow characteristics (number of people waiting at stations, congestion). If it is during peak hours, the passenger flow characteristics will be enhanced, and the impact dimension is the core peak duration.

[0063] The impact coefficient is set according to the impact dimension. For example, if the precipitation level is level 5 (heavy rain) and the visibility is <0.5km, the impact coefficient range is [0.7, 0.9]; if it is the morning rush hour (7:30-9:00), the impact coefficient range is [1.2, 1.5].

[0064] First, based on the current environmental data, the influencing features and influence coefficients can be obtained through a pre-set environment-feature mapping library.

[0065] Then, based on the influencing features and influence coefficients, fine-tuning factors are obtained through preset calculation rules. The fine-tuning factor is used to characterize the effectiveness of features in the current environment. By multiplying the affected features by the corresponding fine-tuning factor, useful features can be strengthened and useless features can be suppressed, ultimately making the adjusted fused features more in line with the needs of the current scheduling scenario. The preset calculation rules will set corresponding calculation rules for different environmental dimensions.

[0066] For example, regarding weather conditions, the fine-tuning factor is adjusted according to the corresponding calculation rules. It can be represented as: =max(lower limit of influence coefficient, upper limit of influence coefficient - m1 × precipitation level - m2 × (1 / visibility)).

[0067] Where m1 and m2 are weights, such as m1=0.05 and m2=0.1; (1 / visibility) indicates that the lower the visibility, the greater the impact. For example, if the precipitation level is 5 and the visibility is 0.5km, the result can be calculated by substituting these values. =max(0.7, 0.35)=0.7.

[0068] For time-based environments, fine-tuning factors are applied according to the corresponding calculation rules. It can be represented as: = Lower limit of impact coefficient + n × (remaining minutes of core peak / 30).

[0069] Here, (remaining minutes of the core peak / 30) indicates that the total duration of the core peak is set at 30 minutes; the more time remaining, the higher the importance. n represents the weight, such as n=0.3. For example, if the remaining time of the core peak is 15 minutes, the weight can be calculated by substituting the values. =1.35.

[0070] Once the fine-tuning factor is determined, the multimodal fusion features can be fine-tuned. Fine-tuning can be divided into two steps: first, the data error is corrected by restoring the true value of the data through environmental data and historical patterns; then, the restored feature values ​​are adjusted. Since the visual data still has uncertainty due to environmental influences, the weight of the feature can be reduced.

[0071] Specifically, the multimodal feature matrix is ​​fine-tuned based on a fine-tuning factor to obtain the adjusted multimodal feature matrix, including the following steps: S371. Based on environmental data, obtain the corresponding environmental error rate through a preset environment-error mapping rule.

[0072] S372. Based on the multimodal feature matrix, determine the feature values ​​corresponding to the influencing features, and calculate and obtain the restored feature values ​​through the environmental error rate.

[0073] S373. Adjust the restored eigenvalues ​​using a fine-tuning factor to obtain the fine-tuned eigenvalues.

[0074] S374. Based on the fine-tuned eigenvalues, adjust the multimodal feature matrix to obtain the adjusted multimodal feature matrix.

[0075] The preset environment-error mapping rule is generated based on historical data and historical rules, and it stores the environment error rate corresponding to different environment data.

[0076] First, based on environmental data, the corresponding environmental error rate can be obtained by pre-setting environmental-error mapping rules.

[0077] Then, based on the multimodal feature matrix, the feature values ​​corresponding to the influencing features are determined, and the restored feature values ​​are calculated and obtained through the environmental error rate.

[0078] For example, if the environmental data is heavy rain and the influencing feature is the number of people waiting on the platform, and the visual passenger flow count is determined to be 40% lower on average by a preset environment-error mapping rule, then the error rate is 0.4. According to the multimodal feature matrix, the feature value (the value before normalization) corresponding to the original number of people waiting on the platform is 30. Through the environmental error rate, the restored feature value can be calculated as: original feature value / (1 - environmental error rate) = 30 / (1 - 0.4) = 50.

[0079] The restored eigenvalues ​​are adjusted by a fine-tuning factor to obtain the fine-tuned eigenvalues. In the example above, the fine-tuning factor is 0.7. Therefore, by calculation, the fine-tuned eigenvalue can be obtained as 50 × 0.7 = 35.

[0080] By replacing the corresponding eigenvalues ​​of the multimodal feature matrix with the fine-tuned eigenvalues, the adjusted multimodal feature matrix can be obtained.

[0081] After obtaining the multimodal feature matrix, it is necessary to fuse the features of each modality to generate multimodal fused features, which can then be used as input for the subsequent multimodal large model. Considering that there are still correlations between the features of each modality, a fusion weight matrix is ​​generated based on the multimodal feature matrix through cross-modal correlation. The fusion weight matrix is ​​then used to weight and fuse the features of each modality to generate multimodal fused features.

[0082] Specifically, see Figure 3 Based on the multimodal feature matrix, a fusion weight matrix is ​​generated through cross-modal correlation, and multimodal fusion features are generated through the fusion weight matrix, including the following steps: S410. Based on the multimodal feature matrix, obtain the first association weight matrix through preset association rules.

[0083] S420. Based on the multimodal feature matrix, the second association weight matrix is ​​obtained through an attention mechanism network.

[0084] S430. Merge the first correlation weight matrix and the second correlation weight matrix to generate a merged weight matrix.

[0085] S440. Based on the multimodal feature matrix, multimodal fusion features are generated by fusing the weight matrix.

[0086] Since the correlations between the modalities can be divided into two categories, one is explicit correlation, such as the road speed characteristics will be affected within 5 minutes after a construction event occurs; the other is implicit correlation, such as the correlation between passenger flow at a station and the speed of a road section 3 kilometers away. This type of correlation is difficult to define with explicit rules.

[0087] For explicit associations, the first association weight matrix can be obtained by pre-setting association rules. The so-called pre-setting association rules are rules that conform to objective laws, based on domain common sense and interpretable rules, and quantify the spatial, temporal and causal relationships between features. The first association weight matrix is ​​used to represent the explicit association strength between any two features, and takes a value of 0 to 1.

[0088] For spatial association, the closer the geographical locations corresponding to the features, the higher the association weight. The association degree can be quantized using a preset distance quantization method, such as using a Gaussian function. Features With features The spatial weights between them, then It can be represented as: in, Representation of features With features Geographical distance between them This is a spatial scale parameter (e.g., 500 meters for urban roads).

[0089] For example, the passenger flow characteristics of a station and the vehicle speed characteristics of adjacent road segments, at a distance of 100 meters, can be calculated. =0.92 (strong correlation); Based on vehicle speed characteristics at a distance of 3000 meters from the road segment 3 km away, the correlation was calculated. =0.15 (weak association).

[0090] For time-related features, the more the lag / leader relationship between features conforms to known patterns, the higher the association weight. This can be achieved by predefining time thresholds; for example, if the change in road speed lags by 3-10 minutes after a construction event, then the time weight within this time window can be defined. =0.8, outside of this time window, =0.2.

[0091] For causal relationships, if one feature is a known cause or effect of another feature, the association weight is significantly increased. This can be achieved through causal chains, where core associations and weights are predefined. For example, following the causal chain "construction event → road congestion → speed decrease," the causal weights of the construction event and the congestion level can be determined. =0.9, causal weight of congestion level and vehicle speed =0.85.

[0092] The three weights are summed according to their importance to obtain the final association weight matrix, which is also known as the first association weight matrix. The importance can be set according to the actual situation. For example, spatial association accounts for 0.3, temporal association for 0.3, and causal association for 0.4. Then the first association weight matrix A can be expressed as: For implicit associations, a data-driven approach is used, namely through an attention mechanism network, to obtain a second association weight matrix B. The second association weight matrix is ​​used to represent the implicit association strength between any two features, and its value ranges from 0 to 1.

[0093] Attention mechanisms in networks will focus on any two features With features By calculating the similarity through cross-attention, and then normalizing it with Softmax, the attention weights can be obtained. , Characteristic features With features The implicit connections between them.

[0094] After determining the first and second correlation weight matrices, they are then merged. This involves weighted summation of the first and second correlation weight matrices A and B to generate a merged weight matrix W, which can be expressed as: in, Dynamic weighting coefficients (0≤ ≤1), which can be adaptively adjusted according to the complexity of the scenario, for example, for scenarios with clear rules (such as construction events). =0.7 (prioritize trusting the first association weight matrix A); for complex and unknown scenarios (such as sudden large-scale events), then... =0.3 (Preferred trust second association weight matrix B).

[0095] Once the fusion weight matrix is ​​determined, the multimodal feature matrix can be fused using the fusion weight matrix to generate multimodal fused features.

[0096] Specifically, the fusion of multimodal feature matrices will be carried out using correlation propagation weighting and residual fusion. The so-called correlation propagation weighting not only considers the weight of individual features, but also transmits the correlation influence between features through the fusion weight matrix.

[0097] Let F be the multimodal feature matrix. For any feature... The corresponding fusion weights can be determined through the fusion weight matrix. It can also obtain other related features. and the corresponding fusion weights , Based on the degree of correlation between features, only the top K correlated features are selected.

[0098] Therefore, characteristics Final weighted value It can be represented as: After determining the weighting values, residual fusion, i.e., feature fusion, is then performed. Final fusion features ,in, It is the retention coefficient of the original features, usually taken as 0.1-0.3, to ensure that the original information is not completely covered.

[0099] This fusion method strengthens important features through element-level weighting, incorporates the relationships between features through association propagation weighting, and finally combines the residuals of the original features to ensure information integrity. Compared with simple feature stacking, it can enrich the expression of features and provide better input for subsequent anomaly identification and traffic condition prediction.

[0100] All features are weighted and fused, and the final fusion result is the multimodal fusion feature.

[0101] Once the multimodal fusion features are determined, they can be used as input to generate abnormal traffic event identification results and future traffic condition prediction data through a preset multimodal scheduling model.

[0102] The preset multimodal scheduling model is based on a shared encoder and a dual-task decoder architecture. The multimodal large model generated through training can simultaneously support "abnormal event recognition" (classification task) and "traffic condition prediction" (temporal prediction task).

[0103] The pre-defined multimodal scheduling model, after acquiring the multimodal fusion features, will first perform temporal processing, sort the features by timestamp, and take N consecutive time slices (e.g., N=10, i.e., 10 minutes of data) to form a time series sample (shape [N, total feature dimension]), which will be used as the input for the prediction task.

[0104] Then, feature encoding and feature interaction are performed through a shared encoder. Finally, a dual-task decoder is used to simultaneously perform "abnormal event recognition" and "traffic condition prediction" based on the fused features, ultimately generating abnormal traffic event recognition results and future traffic condition prediction data.

[0105] The results of abnormal traffic event identification include the type of abnormal event, the location of the event, and the confidence level; the future traffic condition prediction data includes road conditions: the congestion level (level 1-5) and traffic density for each time period in the future; and passenger flow conditions: the number of people in vehicles and the number of people waiting at stations for each time period in the future.

[0106] After obtaining the results of abnormal traffic event identification and the prediction data of future traffic conditions, a bus dispatching scheme can be generated based on the results of abnormal traffic event identification and the prediction data of future traffic conditions by using a preset dispatching objective function.

[0107] Specifically, see Figure 4 Based on the results of abnormal traffic event identification and future traffic condition prediction data, a bus dispatching scheme is generated by pre-setting a dispatching objective function, including the following steps: S610. Based on the results of abnormal traffic event identification and future traffic condition prediction data, determine the type of dispatching demand.

[0108] S620. Based on the scheduling requirement type, multiple candidate scheduling strategies are obtained through a preset scheduling strategy library, and a candidate scheduling strategy set is formed.

[0109] S630. Traverse the candidate scheduling strategy set. For each candidate scheduling strategy, obtain the strategy score through the preset scheduling objective function.

[0110] S640. After the traversal is completed, the candidate scheduling strategy with the highest strategy score is recorded as the target scheduling strategy.

[0111] S650. Based on the target scheduling strategy, generate an executable bus scheduling scheme.

[0112] First, based on the results of abnormal traffic event identification and the prediction data of future traffic conditions, the type of dispatching demand can be determined. This step is mainly obtained by matching through a pre-set dispatching demand library, which is configured with corresponding dispatching demands for various scenarios based on domain experience.

[0113] For example, if the abnormal event is construction, the dispatch demand type can be determined as: detour (route adjustment); if the future traffic situation is predicted to be that the number of people waiting at the station will reach 60 in the next 30 minutes (overcapacity of 50), the dispatch demand type can be determined as: increase the frequency of departures and dispatch additional backup vehicles (departure adjustment + capacity supplement).

[0114] Then, based on the type of scheduling requirement, multiple candidate scheduling strategies can be obtained through a preset scheduling strategy library, and a candidate scheduling strategy set can be formed.

[0115] The preset scheduling strategy library is built and generated based on historical cases and domain experience. It stores various scheduling strategies, and the strategies correspond to the types of requirements. For example, for the route adjustment category, the matching candidate scheduling strategies are: "Change straight to detour via XX road" and "Temporarily shorten the route to XX station"; for the departure adjustment category, the matching candidate scheduling strategies are: "Reduce the departure interval from 5 minutes to 3 minutes during peak hours" and "Extend the interval from 10 minutes to 12 minutes during off-peak hours". For capacity replenishment, the matching candidate scheduling strategies are: "transfer 2 vehicles from the backup depot to the main line of XX route" and "call on XX route shuttle buses to support XX route high-passenger-flow stations".

[0116] After obtaining the candidate scheduling strategy set, the system will iterate through the candidate scheduling strategy set and obtain a strategy score for each candidate scheduling strategy by using a preset scheduling objective function.

[0117] The preset scheduling objective function is generated by combining three sub-objectives: total passenger waiting time, vehicle punctuality rate, and operating cost. The weights of these sub-objectives are set, for example, to 0.5, 0.2, and 0.1 respectively. Therefore, the preset scheduling objective function F can be expressed as: F = 0.5 × passenger waiting time + 0.2 × vehicle punctuality rate + 0.1 × operating cost.

[0118] For each candidate scheduling strategy, a strategy score, or comprehensive benefit score, can be obtained by pre-setting a scheduling objective function.

[0119] After the traversal is complete, under the premise of satisfying the preset constraints, the candidate scheduling strategy with the highest strategy score can be recorded as the target scheduling strategy. The constraints here mainly refer to satisfying the physical and operational constraints of the public transportation system, including resource constraints, such as the upper limit of reserve vehicles and vehicle range limits; route constraints, such as road physical constraints (such as narrow roads that small cars can detour but large buses cannot pass through) and the lower limit of departure intervals; and safety constraints, such as road speed limits and vehicle addition restrictions under severe weather conditions.

[0120] Finally, based on the target scheduling strategy, an executable bus scheduling plan can be generated, which is the specific vehicle scheduling instruction. That is, the overall strategy is transformed into local operable instructions. First, the target scheduling strategy is broken down into elements, including routes, vehicle resources, route adjustments, time parameters, etc.

[0121] Then, based on the decomposed elements, the real-time resource database is queried, such as vehicle status, driver schedule, and station conditions, to verify resource availability and perform resource matching. Finally, using "vehicle (driver)-time-operation" as the execution template, the elements are transformed into structured execution instructions.

[0122] For example, the generated vehicle dispatch instructions include: specific vehicle IDs (such as "standby vehicles B005 and B006 shall arrive at the starting station of Route 3 before 08:30"); route changes (such as "all vehicles on Route 3 shall detour via Route XX from 08:35, skipping the section from XX to XX"); and departure time adjustments (such as "Route 3 vehicle B1023, originally scheduled to depart at 08:40, will depart at 08:38").

[0123] After generating vehicle dispatch instructions, the instructions will be pushed to the corresponding vehicle terminals and relevant personnel to begin vehicle dispatching. After the dispatch plan is implemented, follow-up execution tracking and comprehensive evaluation of the plan are required to iteratively optimize the multimodal dispatch model.

[0124] Therefore, in this embodiment of the application, after generating a bus dispatching scheme based on the abnormal traffic event identification results and future traffic condition prediction data through a preset dispatching objective function, the following steps are also included: S710: By pre-setting monitoring dimensions, the execution status of the bus dispatching plan can be obtained in real time.

[0125] S720: Based on the execution status of the bus dispatching scheme, an evaluation report is obtained through a preset evaluation method.

[0126] The preset monitoring dimensions include vehicle execution status, passenger flow matching degree, and passenger experience. Each monitoring dimension includes corresponding monitoring indicators. For example, vehicle execution status includes real-time vehicle location, on-time rate, and compliance of detour routes; passenger flow matching degree includes real-time passenger flow at stations and vehicle congestion; and passenger experience includes average waiting time and number of complaints. By using these preset monitoring dimensions, the execution status of the bus dispatching plan can be obtained in real time, i.e., the specific situation of the monitoring indicators. Then, based on the execution status of the bus dispatching plan, an evaluation report can be generated through a preset evaluation scheme. The preset method mainly sets corresponding evaluation indicators and evaluation logic to evaluate the execution effect of the bus dispatching plan from multiple dimensions of "passengers, operations, and costs".

[0127] After obtaining the evaluation report, the optimization direction can be determined based on the evaluation, which can then be fed back into the preceding stages of generating the bus dispatch plan, forming a closed loop of "execution-evaluation-optimization" to make subsequent bus dispatch more accurate and better in line with actual needs.

[0128] This application also provides a public transport dynamic scheduling system based on visual perception and a multimodal large model. See [link to relevant documentation]. Figure 5 The system includes: a data acquisition module 101, a feature extraction module 102, a feature fusion module 103, a model prediction module 104, and a bus dispatch generation module 105.

[0129] Among them, the data acquisition module 101 is used to acquire basic information of buses and visual perception data and real-time dynamic data of vehicle operation.

[0130] The feature extraction module 102 is used to extract features from visual perception data and real-time dynamic data respectively, obtain visual perception features and real-time dynamic features, and associate and align the basic information of the bus with the visual perception features and real-time dynamic features to generate a multimodal feature matrix.

[0131] The feature fusion module 103 is used to generate a fusion weight matrix based on the multimodal feature matrix through cross-modal correlation, and to generate multimodal fusion features through the fusion weight matrix.

[0132] The model prediction module 104 is used to generate abnormal traffic event identification results and future traffic condition prediction data based on multimodal fusion features and a preset multimodal scheduling model.

[0133] The bus dispatch generation module 105 is used to generate a bus dispatch plan based on the results of abnormal traffic event identification and future traffic condition prediction data, through a preset dispatch objective function.

[0134] In this embodiment of the application, the data acquisition module 101 is specifically used to acquire basic information of the bus and visual perception data and real-time dynamic data of vehicle operation. The visual perception data includes station passenger flow image data, in-vehicle passenger flow image data and out-of-vehicle road condition image data. The real-time dynamic data includes vehicle operation data and road traffic control data.

[0135] The feature extraction module 102 is specifically used to extract features from the visual perception data and real-time dynamic data acquired by the data acquisition module 101, respectively, to obtain visual perception features and real-time dynamic features, and to associate and align the basic information of the bus with the visual perception features and real-time dynamic features to generate a multimodal feature matrix.

[0136] The feature fusion module 103 is specifically used to generate a fusion weight matrix based on the multimodal feature matrix generated by the feature extraction module 102 through cross-modal correlation, and generate multimodal fusion features through the fusion weight matrix.

[0137] The model prediction module 104 is specifically used to generate abnormal traffic event identification results and future traffic condition prediction data based on the multimodal fusion features generated by the feature fusion module 103 and a preset multimodal scheduling model.

[0138] The bus dispatch generation module 105 is specifically used to generate a bus dispatch plan based on the abnormal traffic event identification results and future traffic condition prediction data generated by the model prediction module 104 and by setting a dispatch objective function.

[0139] This application also provides a computer-readable storage medium storing a computer program that can be loaded by a processor and executed by any of the above-described methods for dynamic bus scheduling based on visual perception and multimodal large models.

[0140] The embodiments described in this application are preferred embodiments of this application and are not intended to limit the scope of protection of this application. Therefore, all equivalent changes made in accordance with the principles of this application should be included within the scope of protection of this application.

Claims

1. A dynamic bus scheduling method based on visual perception and a multimodal large model, characterized in that, include: Acquire basic information about public transport vehicles and visual perception data and real-time dynamic data on vehicle operation; The visual perception data includes station passenger flow image data, in-vehicle passenger flow image data, and out-of-vehicle road condition image data; the real-time dynamic data includes vehicle operation data and road traffic control data. Feature extraction is performed on visual perception data and real-time dynamic data respectively to obtain visual perception features and real-time dynamic features; The basic information of public transport vehicles is associated and aligned with visual perception features and real-time dynamic features to generate a multimodal feature matrix; Based on the multimodal feature matrix, a fusion weight matrix is ​​generated through cross-modal correlation, and multimodal fusion features are generated through the fusion weight matrix; Based on multimodal fusion features, abnormal traffic event identification results and future traffic condition prediction data are generated by pre-setting a multimodal scheduling model; Based on the results of abnormal traffic event identification and future traffic condition prediction data, a bus dispatching scheme is generated by pre-setting a dispatching objective function.

2. The method for dynamic bus scheduling based on visual perception and a multimodal large model according to claim 1, characterized in that, The step of extracting features from visual perception data and real-time dynamic data to obtain visual perception features and real-time dynamic features includes: Visual perception data is preprocessed, and the preprocessed image is subjected to target detection to obtain visual perception features, including in-vehicle passenger flow features, out-of-vehicle road condition features, and station passenger flow features. Real-time dynamic features are generated by extracting time-series and text features from real-time dynamic data. These real-time dynamic features include vehicle motion features and traffic control features.

3. The method for dynamic bus scheduling based on visual perception and a multimodal large model according to claim 1, characterized in that, The step of associating and aligning basic bus information with visual perception features and real-time dynamic features to generate a multimodal feature matrix includes: Set a baseline time granularity and uniformly convert visual perception features and real-time dynamic features into time slices at that baseline time granularity. By pre-setting a city geographic information system, all data is mapped to a unified latitude and longitude coordinate system; Visual perception features and real-time dynamic features within the same time slice and geographical location range are bound to basic information of public transport vehicles to generate a set of unit features; Based on a single-source feature set, the dimensions of features from different modalities are unified to generate a structured multimodal feature matrix.

4. The method for dynamic bus scheduling based on visual perception and a multimodal large model according to claim 1, characterized in that, After associating and aligning the basic information of the bus with visual perception features and real-time dynamic features to generate a multimodal feature matrix, the method further includes: Based on environmental data, the influencing features and influence coefficients are obtained through a pre-set environment-feature mapping library; Based on the impact characteristics and impact coefficients, fine-tuning factors are obtained through preset calculation rules; Based on the fine-tuning factor, the multimodal feature matrix is ​​fine-tuned to obtain the adjusted multimodal feature matrix.

5. The method for dynamic bus scheduling based on visual perception and a multimodal large model according to claim 4, characterized in that, The step of fine-tuning the multimodal feature matrix based on a fine-tuning factor to obtain the adjusted multimodal feature matrix includes: Based on environmental data, the corresponding environmental error rate is obtained through a preset environment-error mapping rule; Based on the multimodal feature matrix, the feature values ​​corresponding to the influencing features are determined, and the restored feature values ​​are calculated and obtained through the environmental error rate. The restored eigenvalues ​​are adjusted using a fine-tuning factor to obtain the fine-tuned eigenvalues. Based on the fine-tuned eigenvalues, the multimodal feature matrix is ​​adjusted to obtain the adjusted multimodal feature matrix.

6. The method for dynamic bus scheduling based on visual perception and a multimodal large model according to claim 1, characterized in that, The process of generating a fusion weight matrix based on a multimodal feature matrix through cross-modal correlation, and then generating multimodal fusion features using the fusion weight matrix, includes: Based on the multimodal feature matrix, the first association weight matrix is ​​obtained through preset association rules; Based on the multimodal feature matrix, a second association weight matrix is ​​obtained through an attention mechanism network; The first correlation weight matrix and the second correlation weight matrix are merged to generate a fused weight matrix; Based on the multimodal feature matrix, multimodal fusion features are generated by fusing the weight matrix.

7. The method for dynamic bus scheduling based on visual perception and a multimodal large model according to claim 1, characterized in that, The process of generating a bus dispatching scheme based on abnormal traffic event identification results and future traffic condition prediction data, through a preset dispatching objective function, includes: Based on the results of abnormal traffic event identification and future traffic condition prediction data, determine the type of dispatching demand; Based on the type of scheduling requirement, multiple candidate scheduling strategies are obtained through a pre-set scheduling strategy library, and a candidate scheduling strategy set is formed. Traverse the candidate scheduling strategy set, and for each candidate scheduling strategy, obtain a strategy score through a preset scheduling objective function; After the traversal is complete, the candidate scheduling policy with the highest policy score is recorded as the target scheduling policy; Based on the target scheduling strategy, an executable bus scheduling scheme is generated.

8. The method for dynamic bus scheduling based on visual perception and a multimodal large model according to claim 1, characterized in that, The process of generating a bus dispatching plan based on abnormal traffic event identification results and future traffic condition prediction data, through a preset dispatching objective function, includes: By pre-setting monitoring dimensions, the execution status of the bus dispatching plan can be obtained in real time; Based on the execution status of the bus dispatching plan, an evaluation report is obtained through a pre-set evaluation method.

9. A dynamic bus dispatching system based on visual perception and a multimodal large model, characterized in that, include: The data acquisition module (101) is used to acquire basic information of buses and visual perception data and real-time dynamic data of vehicle operation; The visual perception data includes station passenger flow image data, in-vehicle passenger flow image data, and out-of-vehicle road condition image data; the real-time dynamic data includes vehicle operation data and road traffic control data. The feature extraction module (102) is used to extract features from visual perception data and real-time dynamic data respectively, obtain visual perception features and real-time dynamic features, and associate and align the basic information of the bus with the visual perception features and real-time dynamic features to generate a multimodal feature matrix. The feature fusion module (103) is used to generate a fusion weight matrix based on the multimodal feature matrix through cross-modal correlation, and to generate multimodal fusion features through the fusion weight matrix; The model prediction module (104) is used to generate abnormal traffic event identification results and future traffic condition prediction data based on multimodal fusion features and a preset multimodal scheduling model; The bus dispatch generation module (105) is used to generate a bus dispatch plan based on the abnormal traffic event identification results and future traffic condition prediction data, by using a preset dispatch objective function.

10. A computer-readable storage medium storing a computer program capable of being loaded by a processor and executed as described in any one of claims 1 to 8, which is a method for dynamic bus scheduling based on visual perception and multimodal large models.

Images