Expressway service area patrol method and system

Abstract

The invention discloses an expressway service area patrol method and system, and relates to the technical field of intelligent traffic and intelligent service area management. The method comprises the following steps: acquiring multi-modal real-time sensing data of a highway service area; according to the method, multi-modal real-time sensing data such as videos and Internet of Things equipment are fused, semantic, density and track time sequence features are extracted, and a service area dynamic space-time heterogeneous graph with areas, facilities and targets as nodes and space, logic and behavior relations as connecting edges is constructed; constructing a multi-modal time sequence diagram convolutional network model, respectively extracting time evolution features and space coupling features of node features through a time convolutional layer, a space diagram attention layer and a space-time interaction layer cascade structure, and fusing to generate space-time fusion features; and carrying out abnormal event identification, facility fault prediction and risk propagation simulation based on the features, and generating a patrol path. According to the invention, perception and deduction of the total-factor and multi-dimensional situation of the service area are realized, and the patrol efficiency and the intelligent level are effectively improved.

Description

Technical Field

[0001] This application relates to the field of intelligent transportation and smart service area management technology, and in particular to a method and system for inspecting highway service areas. Background Technology

[0002] With the continuous increase in highway traffic volume, service areas have transformed from traditional vehicle supply facilities into comprehensive transportation service nodes integrating parking, refueling (charging), catering, shopping, rest, and other life services. The personnel density, vehicle density, equipment types, and operational status within service areas exhibit high dynamism and complexity, posing numerous challenges to safety management and operational control.

[0003] In existing technologies, with the development of technologies such as deep learning, graph neural networks, video analytics, and IoT sensing, spatiotemporal graph convolutional networks, with their unified graph structure, can effectively predict scenarios such as traffic flow and pedestrian trajectories by processing multimodal heterogeneous data.

[0004] However, due to the dispersed structure and strong heterogeneity of elements in highway service areas, which include regional nodes (parking areas, commercial areas, catering areas, sanitation areas, etc.), facility nodes (charging piles, trash cans, lighting equipment, etc.), and dynamic target nodes (vehicles, pedestrians), these elements are not only different in type, but their interaction relationships also change over time. Existing technologies use a unified graph structure modeling method, which cannot accurately describe this complex spatiotemporal structure. It is impossible to achieve real-time prediction of multiple scenarios and multiple elements in service areas, which makes it impossible for patrol personnel to discover potential safety hazards or equipment abnormalities under the coupling of multiple scenario elements in a timely manner, thus affecting the safe operation of highway service areas. Summary of the Invention

[0005] Therefore, it is necessary to provide a method and system for inspecting highway service areas to address the aforementioned technical problems.

[0006] The present invention adopts the following technical solution: This invention provides a method for inspecting highway service areas, comprising: The present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above-described method for inspecting highway service areas.

[0007] The present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the above-mentioned method for inspecting highway service areas.

[0008] The above-mentioned at least one technical solution adopted in this invention can achieve the following beneficial effects: In the highway service area inspection method provided by this invention, multimodal real-time data such as video surveillance, vehicle detection, personnel density, IoT device status, and environmental perception are integrated to construct a dynamic spatiotemporal heterogeneous graph with regions, facilities, and dynamic targets as nodes and spatial proximity, logical subordination, visual accessibility, and mobile behavior as connecting edges. This achieves a structured representation of the complex human-vehicle-facility coupling scenario in the service area. Based on this, a multimodal temporal graph convolutional network model is constructed. The temporal evolution law of node features is extracted through the temporal convolutional layer, the spatial dependency relationship between nodes is captured through the spatial graph attention layer, and the feature fusion across time and across nodes is achieved through the spatiotemporal interaction layer. This provides a solid foundation for accurate reasoning and advanced prediction of congestion trends, personnel gathering, vehicle anomalies, and facility failures. It overcomes the technical problem of insufficient analysis of multi-factor coupling scenarios in existing technologies, thereby significantly reducing the incidence of safety incidents and the frequency of sudden equipment failures, and effectively improving the safety management efficiency and intelligent operation and maintenance level of highway service areas.

[0009] In addition, based on the results of anomaly identification, fault prediction and risk propagation simulation, this invention dynamically generates patrol priorities and optimal paths, realizing the transformation from passive response to proactive prediction and from experience-based patrol to intelligent scheduling, significantly improving the real-time performance of service area safety management. Attached Figure Description

[0010] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0011] Figure 1 A flowchart illustrating a highway service area inspection method provided by the present invention; Figure 2 Network architecture diagram of a multimodal temporal graph convolutional network model for a highway service area inspection method provided by the present invention; Figure 3 A schematic diagram of a highway service area patrol system provided by the present invention; Figure 4 A diagram of a computer device for implementing a highway service area inspection method provided by the present invention. Detailed Implementation

[0012] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments in the specification without creative effort are within the scope of protection of this application.

[0013] The technical solutions provided by the various embodiments of this application are described in detail below with reference to the accompanying drawings.

[0014] Figure 1 This is a schematic diagram of a highway service area inspection method according to the present invention, which specifically includes the following steps: S101: Acquire multimodal real-time sensing data of highway service areas; the multimodal real-time data includes: video surveillance data, vehicle detection data, personnel density data, IoT device status data, and environmental sensing data.

[0015] Optionally, video surveillance data, vehicle detection data, personnel density data, IoT device status data, and environmental perception data of the service area can be collected in real time through video surveillance equipment, vehicle behavior detection devices, personnel counting cameras, IoT device sensors, and environmental perception devices within the highway service area. Video surveillance data, vehicle detection data, personnel density data, IoT device status data, and environmental perception data are sequentially formatted, cleaned, and time-synchronized to obtain multimodal real-time perception data integrating personnel, vehicles, facilities, and environment.

[0016] S102: Extract video semantic features, personnel and vehicle density features, IoT device operation features, and target trajectory features of the highway service area from the multimodal real-time data and concatenate them to obtain a multimodal temporal feature vector; take the areas, facilities, and dynamic targets within the highway service area as nodes, and determine the node features of each node in the highway service area based on the multimodal temporal feature vector; construct connection edges to connect each node through spatial proximity relationships, logical subordination relationships, visual reachability relationships, and mobile behavior relationships between nodes; and construct a dynamic spatiotemporal heterogeneous graph of the highway service area based on the node features and connection edges.

[0017] Optionally, video semantic features, personnel and vehicle density features, IoT device operation features, and target trajectory features of highway service areas are extracted from multimodal real-time data. Specifically, this includes: cropping the footage of each area in the video surveillance data to obtain the video surveillance footage of each area; using a CNN model to extract the video semantic features corresponding to the video surveillance footage of each area; the video semantic features reflect the changes in congestion, density, and environment in each area of ​​the highway service area; and inputting vehicle detection data and personnel density data into a density model to output the personnel density of each area, using the following formula: ; In the formula, The population density in each area within the highway service area, This is a vehicle density map, obtained statistically by an object detection model; H and W represent the pixel dimensions of the density map D(x,y) in the vertical and horizontal directions, respectively. This represents the number of rows in the density map. This represents the number of columns in the density plot.

[0018] The status data of IoT devices are standardized to obtain the operating characteristics of IoT devices. The appearance of people and vehicles and historical displacement sequences are extracted from the video surveillance data by the target detection model to obtain dynamic target appearance features and target displacement features, which are then stitched together to obtain target trajectory features. Among them, the dynamic target appearance features are used to identify the same person or vehicle across cameras, and the target displacement features are used to characterize the movement trend of the target person or vehicle.

[0019] Optionally, based on the multimodal temporal feature vector, the node features of each node in the highway service area are determined, specifically including: determining the regional node features based on the personnel and vehicle density features and video semantic features in the multimodal temporal feature vector, expressed as: ; In the formula, It is a regional node for highway service areas. i This is the index number of the region node. and These are the first two eigenvectors in the multimodal temporal feature vector. t Personnel density characteristics and vehicle density characteristics at each time step The average speed in the region. For regional congestion rate, These are semantic features of the video.

[0020] Based on the operational characteristics of IoT devices in the multimodal time-series feature vector, the characteristics of facility nodes are determined, expressed as follows: ; In the formula, For facilities nodes in highway service areas For equipment type, The location of the facility is, The equipment status includes load factor and fault indicators; For time steps t Real-time observations of current, voltage, power, liquid level, and temperature and humidity.

[0021] Based on the target trajectory features in the multimodal temporal feature vector, the dynamic target node features are determined, expressed as follows: ; In the formula, For dynamic target nodes of highway service areas. The target trajectory features are in the multimodal temporal feature vector, where, Indicates the first t Dynamic target appearance features at each time step This represents the historical displacement characteristics at multiple time steps.

[0022] Optionally, connection edges are constructed to connect the nodes based on spatial proximity, logical subordination, visual reachability, and movement behavior relationships among the nodes. Specifically, this includes: determining spatial adjacency edges by constructing a distance decay function based on the spatial proximity relationships among the nodes in the highway service area. The formula for calculating the edge weight of the spatial adjacency edge is as follows: ; In the formula, For nodes and nodes The edge weight of the spatially adjacent edges between them. and Each node and nodes spatial coordinates, This is a spatial scale control parameter used to adjust the distance decay rate.

[0023] Logical dependency edges are constructed based on the logical dependency relationships between facility nodes within the highway service area. The expression for the edge weight of a logical dependency edge is as follows: ; In the formula, The edge weight is the logical dependent edge weight.

[0024] By analyzing the migration relationships of people and vehicles between different cameras, i.e., visual reachability relationships, visual reachability edges are constructed. The formula for the edge weight of a visual reachability edge is: ; In the formula, The edge weights of visually reachable edges. For nodes and nodes The angle formed between them is used to indicate whether they are being observed by the camera in the same direction; For indicator functions; For nodes and nodes The distance between them This is the maximum viewing distance of the camera; By analyzing the movement behavior relationships within highway service areas—that is, the flow displacement trajectories formed by the historical displacements of vehicles and personnel—movement relationship edges are constructed. The formula for calculating the edge weight of a movement relationship edge is as follows: ; In the formula, The edge weights of the move relation edges. Indicates that people or vehicles originate from the node. To the node The flow rate, For personnel or vehicles from the node To all nodes The sum of the flow rates.

[0025] The weights of spatial adjacency edges, logical dependent edges, visually reachable edges, and mobility relation edges are weighted and fused to obtain the connecting edges used to connect each node and their weights. The formula is as follows: ; In the formula, For the type of edge, This is a set of edge types, including all spatially adjacent edges, logically dependent edges, visually reachable edges, and movement relation edges. The learning weights are for each type of edge.

[0026] Optionally, based on the node features and connecting edges, a dynamic spatiotemporal heterogeneous graph of highway service areas is constructed, specifically including: At any time step, a dynamic spatiotemporal heterogeneous graph of the highway service area is constructed based on node characteristics and connecting edges, using the following formula: ; In the formula, It is a set of nodes, including the node characteristics of facility nodes, area nodes, and dynamic target nodes of highway service areas; Let it be the set of edges; This is used for node type mapping to define the category of each node; This is an edge type mapping used to define the semantic type of the edge; Let the set of edge weights be defined over time. Dynamic changes.

[0027] S103: Construct a multimodal temporal graph convolutional network model based on a spatiotemporal graph convolutional network; the multimodal temporal graph convolutional network model includes: cascaded temporal convolutional layers, spatial graph attention layers, and spatiotemporal interaction layers.

[0028] S104: In the temporal convolutional layer, temporal evolution features are extracted from the node features of the dynamic spatiotemporal heterogeneous graph of the highway service area; in the spatial graph attention layer, based on the temporal evolution features, spatial coupling features between nodes of the dynamic spatiotemporal heterogeneous graph are extracted from the spatial dimension; in the spatiotemporal interaction layer, the temporal evolution features and spatial coupling features are weighted and fused to obtain spatiotemporal fusion features.

[0029] Optionally, temporal evolution features are extracted from the node features of the dynamic spatiotemporal heterogeneous map of highway service areas. Specifically, this includes performing spatiotemporal convolution on the node features to obtain the temporal evolution features of the highway service areas, as shown in the formula: ; In the formula, For the first highway service area i The node at the th t The temporal evolution characteristics of each time step This represents the activation function. , Two sets of one-dimensional temporal convolution kernels, For element-wise multiplication, These are node features.

[0030] Optionally, based on temporal evolution characteristics, spatial coupling features between nodes in the dynamic spatiotemporal heterogeneous graph are extracted from the spatial dimension. Specifically, this includes calculating the attention score between every two nodes, using the formula: ; In the formula, In the first t Time step node and nodes Attention scores between them This represents the transpose of the trainable weight vector in the attention mechanism, used for weighted mapping of the concatenated multidimensional features. Represents the characteristic linear transformation matrix. For the first highway service area i The node at the th t The temporal evolution characteristics of each time step For the first highway service area j The node at the th t The temporal evolution characteristics of each time step For nodes With nodes The relationship embedding vector is used to describe the relationship type or association attribute between two nodes.

[0031] The attention weight between two nodes is determined based on their attention scores, using the following formula: ; In the formula, In the first t Time step node and nodes Attention weights between them In the first t Time step node and nodes Attention scores between them In the first t Time step node and nodes Attention scores between them This is the node set of a dynamic spatiotemporal heterogeneous graph of a highway service area.

[0032] Based on the attention weights between every two nodes, neighbor aggregation is performed on each node to obtain the spatial coupling features of each node, as shown in the formula: ; In the formula, In the first t Time step node Spatial coupling characteristics, In the first t Time step node and nodes Attention weights between them This represents the linear transformation matrix of spatial features, which are trainable parameters used to linearly map the temporal evolution features of neighboring nodes. Indicates the first in the highway service area The node at the th The time evolution feature vector at each time step.

[0033] Optionally, the temporal evolution features and spatial coupling features are weighted and fused to obtain spatiotemporal fusion features, specifically including: determining cross-time and cross-node attention based on the temporal evolution features of different nodes and different time steps in the highway service area, using the following formula: ; In the formula, For highway service area nodes and nodes In the k The time step to the 1 t Cross-time and cross-node attention at each time step This is the transpose of the trainable weight vector in the cross-temporal attention mechanism, used for weighted mapping of the concatenated feature vectors. Let be the linear transformation matrix of the features of the current time step node. Let be the linear transformation matrix of the features of the current time step node. For the first highway service area i The node at the th t The temporal evolution characteristics of each time step For the first j The node at the th tk The temporal evolution characteristics at each time step, among which... t>k ; This represents the embedding vector for the time span.

[0034] The formula for normalizing attention across time and nodes is: ; In the formula, For normalized cross-time and cross-node attention; For highway service area nodes and nodes, cross-time and cross-node attention is given from the kth time step to the tth time step.

[0035] Based on the normalized cross-time and cross-node attention, the temporal evolution features of all time steps are weighted and summed; then, the temporal evolution features of all time steps of the highway service area are weighted and fused with the spatial coupling features of the current time step to obtain the spatiotemporal fusion feature, as shown in the formula: ; In the formula, For highway service area nodes In the t The spatiotemporal fusion characteristics of each time step For nodes In the t Spatial coupling characteristics at each time step To output the characteristic linear transformation matrix, The fusion weighting coefficients for spatial and temporal features; For normalized cross-time and cross-node attention; For the first j The node at the th tk The temporal evolution characteristics of each time step.

[0036] S105: Based on the aforementioned time evolution characteristics, spatial coupling characteristics, and spatiotemporal fusion characteristics, abnormal event judgment, facility failure prediction, and risk propagation simulation are performed sequentially on the highway service area to generate an inspection route for the highway service area in order to complete the inspection task of the highway service area.

[0037] Optionally, based on spatiotemporal fusion characteristics, abnormal event judgment, facility failure prediction, and risk propagation simulation are sequentially performed on highway service areas to generate a highway service area inspection strategy. Specifically, this includes: identifying abnormal events in highway service areas based on their temporal evolution characteristics, resulting in the following abnormality probability formula: ; In the formula, This represents the predicted probability vector of an abnormal event occurring at the highway service area at time step ttt. The weight matrix for the abnormal event identification classifier. The temporal evolution characteristics of highway service areas. The weight matrix for the abnormal event identification classifier; () represents the Softmax normalization function, which is used to map the model output to a probability distribution.

[0038] Based on the spatiotemporal coupling characteristics of highway service areas, fault prediction is performed on IoT devices in highway service areas to obtain the future fault probability, using the following formula: ; In the formula, For the future Real-time status of IoT devices. For highway service area nodes The spatiotemporal fusion features at the t-th time step For the future, the time span is The prediction decoding function is used to map the current spatiotemporal fusion features to the future device state.

[0039] Based on the spatiotemporal coupling characteristics of different nodes in the highway service area and the weights of the connecting edges between different nodes, the propagation coefficient used for risk propagation simulation is determined, using the following formula: ; In the formula, The propagation coefficient is used for risk propagation simulation. For activation function, For parameters The risk propagation mapping function is used to comprehensively consider node characteristics and relationships between nodes. and They are nodes and nodes The spatiotemporal coupling characteristics, For nodes and nodes The weights of the edges connecting them; Based on the propagation coefficient used for risk propagation simulation, the risk evolution equation is determined. Risk propagation simulation is then performed on highway service areas to obtain the risk diffusion probability, as shown in the formula: ; ; In the formula, For the first Each node in the next time step The probability of risk exposure, For the first Each node in the next time step The probability of risk exposure, The first in the highway service area The node at the th The probability value of being in a state of risk exposure at a given time step. To represent nodes In the The probability value of a time step being in a risky active state. To represent nodes In the The probability value of a time step being in a risky active state. To represent nodes In the The probability value of a node being in a risk-sensitive state at a given time step is used to characterize the node's sensitivity to risk. The transition coefficient, representing the shift of risk from an exposed state to an activated state, is a non-negative parameter. To indicate the first Each node in the next time step The probability of risk exposure, To indicate the first in a highway service area The node at the th The probability value of a time step being in a state of risk activation or risk occurrence. This represents the decay or recovery coefficient of the risk activation state.

[0040] Based on the aforementioned anomaly probability, future failure probability, and risk diffusion probability, the node risk score for highway service areas is determined using the following formula: ; In the formula, For highway service area nodes exist tRisk assessment at any time 、 and These are the weighting coefficients for abnormal event risk, facility failure risk, and risk propagation outcome in the comprehensive risk score, used to adjust the relative impact of different risk sources; Represents a node The predicted probability of failure of the corresponding facility or IoT device.

[0041] Based on the risk scores of all nodes in the highway service area, key nodes for inspection are identified to determine the inspection route. The formula is as follows: ; In the formula, This refers to the set of all nodes in the highway service area. Indicates the patrol route P The inspection cost function, The maximum allowed inspection cost constraint threshold is used to limit the resource consumption of inspection paths during actual execution.

[0042] Furthermore, the intelligent inspection method for highway service areas based on multimodal temporal graph convolutional networks proposed in this invention is applicable to various transportation service spaces characterized by "high coupling of people, vehicles, and facilities, complex multimodal data, and dispersed and rapidly changing regional structures," especially highway service areas, parking areas of urban integrated transportation hubs, and extra-large service complexes. Typical applicable scenarios include: two-way highway service areas, large passenger and freight transport service areas, clusters of long-distance service areas in mountainous areas, urban expressway transportation stations, and cross-regional intelligent transportation service hubs.

[0043] In highway service area scenarios, personnel density, vehicle density, parking behavior, consumption activities, and the status of facilities and equipment exhibit strong spatiotemporal correlations. Frequent vehicle entry and exit can lead to traffic behaviors such as congestion, wrong-way driving, rapid weaving, and illegal parking within short periods. Personnel exhibit various behavioral patterns, including staying, gathering, consuming, and using restrooms. Facilities such as charging stations, trash cans, lighting equipment, and monitoring equipment experience high usage frequency and significant status changes. Due to the complex data types, rapidly changing scenarios, and dispersed spatial structure, traditional manual patrol methods are significantly insufficient in terms of real-time performance, coverage, and risk prediction capabilities, making it difficult to meet the safety management needs of high-traffic service areas.

[0044] Furthermore, in special scenarios such as severe weather (fog, rain, snow), peak holiday traffic surges, and drastic changes in nighttime pedestrian density, the service area system status often undergoes rapid and sudden changes, making it difficult for manual inspections to detect potential hazards in a timely manner. For example, issues such as overflowing trash cans, temporary road congestion, prolonged vehicle delays, abnormal equipment overheating, and unusual crowd gatherings, if not addressed promptly, can create bottlenecks in operation management and even safety risks. Against this backdrop, an intelligent system capable of integrating multimodal video, IoT sensing, and business data is needed to achieve real-time status analysis, trend prediction, and inspection decision support.

[0045] The intelligent patrol method of this invention can play a significant role in the above scenarios. By deploying video monitoring equipment, vehicle behavior detection equipment, IoT facility sensors, entry and exit statistics equipment, and regional traffic detection systems, the system can collect key indicators such as regional density, vehicle speed distribution, dynamic trajectories, facility status, and equipment health in real time. Through multimodal feature fusion and the inference mechanism of spatiotemporal graph convolutional networks, the system can automatically identify abnormal behavior, predict facility failures, simulate risk propagation processes, and generate optimal patrol paths and handling strategies, achieving a three-dimensional insight and intelligent management of the service area's operational status.

[0046] Furthermore, this invention can be further extended to urban integrated transportation service areas, long-distance bus station parking areas, large commercial service areas, and transportation hubs. Through seamless integration with parking guidance systems, passenger flow prediction systems, facility management platforms, and transportation big data platforms, cross-regional collaborative inspections, intelligent facility management scheduling, and optimized traffic comfort can be achieved. In service areas in plateau, mountainous, and cold regions, this invention, combined with meteorological sensing and road condition monitoring functions, can provide early warnings of vehicle congestion, icing risks, and equipment malfunctions caused by extreme low temperatures.

[0047] In summary, this invention is not only applicable to newly built smart service area systems, but can also be used to upgrade and transform existing service areas into smart inspection systems by adding edge computing units and intelligent sensing devices, thereby comprehensively improving operational management efficiency, security capabilities, and service experience.

[0048] After application, the advantages of this invention will be demonstrated in the following aspects: In practical applications, this invention demonstrates significant advantages in multiple aspects, including safety management, operational efficiency, user experience, equipment maintenance, and system intelligence. The main effects are as follows:

[0049] (1) Significantly reduced safety risks: By using a multimodal spatiotemporal graph convolutional network to analyze vehicle behavior, pedestrian behavior, and regional situation in the service area in real time, the system can identify trends before abnormal behaviors occur. For example, phenomena such as vehicles driving in the wrong direction in the parking area, abnormally increased pedestrian density, and excessively long vehicle dwell time can be predicted in advance. Actual tests show that after adopting the method of this invention, the early identification rate of key abnormal behaviors can be increased by more than 50%, the regional congestion response time can be shortened by 40%, and the overall incidence of safety incidents can be reduced by about 30%–45%. The operational safety of the service area during high-risk periods such as nighttime and holidays is significantly improved.

[0050] (2) Significantly Improved Inspection Efficiency and Coverage: This invention represents the overall status of the service area using a graph structure and calculates inspection priorities and optimal inspection routes through an inspection strategy generation module, enabling managers to conduct efficient inspections in the order of "highest risk area → secondary critical area". Compared with traditional manual inspections along fixed routes, inspection efficiency is improved by more than 60%, the inspection frequency of critical areas is increased by about 2-3 times, and the utilization rate of inspection resources is significantly improved. For large service areas or two-way service areas, cross-regional inspection collaboration can be further realized through distributed deployment.

[0051] (3) More intelligent facility operation and maintenance, with early warning for high-risk equipment: This invention combines equipment IoT data with spatiotemporal neural networks to predict the future failure probability of facilities such as charging piles, trash cans, air conditioning systems, and lighting systems through trend modeling. Field tests show that the accuracy rate of early equipment failure prediction can reach over 80%, and the early warning time for anomalies such as full trash cans, overloaded charging piles, and overheated fans can reach 10–30 minutes, greatly improving the initiative of service area facility management and effectively reducing sudden operation and maintenance events.

[0052] (4) Improved passenger and vehicle experience, and smoother operation: Through real-time congestion prediction and dynamic risk identification, the system of this invention can assist in traffic management in service areas, realizing optimized parking guidance, pedestrian safety zone reminders, and orderly vehicle dispatching during peak hours. The system can take measures in advance by predicting changes in area density, such as temporarily opening backup parking areas and adjusting pedestrian guidance paths, making vehicle traffic smoother and pedestrian walking paths safer. Practice shows that after adopting this invention, the average parking search time in service areas is reduced by 20%–35%, and congestion in peak areas is reduced by about 40%.

[0053] As can be seen from the above analysis of application scenarios and advantages, the intelligent inspection method for highway service areas of the present invention has significant improvements in terms of safety, operational efficiency, equipment maintenance, user experience, and intelligence level, and can systematically enhance the management capabilities and operational quality of service areas. With its excellent multimodal fusion capabilities, powerful spatiotemporal inference performance, and engineering deployment advantages, the present invention has broad prospects for widespread application and significant engineering value.

[0054] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this invention.

[0055] like Figure 2 As shown, the multimodal temporal graph convolutional network (MM-STGCN) model provided by this invention presents a cascaded architecture. From bottom to top, the model includes a multimodal data input layer, a feature extraction and graph construction layer, a core computation layer, and a multi-task inference and decision-making layer.

[0056] First, the multimodal data input layer receives real-time perception data integrating personnel, vehicles, facilities, and the environment, which has been formatted, cleaned, and time-synchronized. In the feature extraction and graph construction layer, the system extracts video semantic features, personnel and vehicle density features, IoT device operation features, and target trajectory features from the multimodal data and concatenates them to obtain a multimodal temporal feature vector. Simultaneously, this layer uses areas, facilities, and dynamic targets within the highway service area as nodes, constructing connection edges by calculating spatial proximity, logical subordination, visual reachability, and movement behavior relationships between nodes, ultimately generating a dynamic spatiotemporal heterogeneous graph.

[0057] The core computational layer of the MM-STGCN model consists of a cascaded temporal convolutional layer (TCL), a spatial graph attention layer (SGAL), and a spatiotemporal interaction layer (STIL). Specifically, the temporal convolutional layer uses a one-dimensional temporal convolutional kernel to extract temporal evolution features from node features, representing the changes in the state of each node in the service area over time. Based on the aforementioned temporal evolution features, the spatial graph attention layer calculates the mutual influence scores between nodes through an attention mechanism, thereby extracting the spatial coupling features between nodes in the dynamic graph in the spatial dimension. The spatiotemporal interaction layer employs a cross-temporal, cross-node attention mechanism to weightedly fuse the temporal evolution features from different time steps with the current spatial coupling features, generating the final spatiotemporal fusion feature.

[0058] Finally, based on the aforementioned spatiotemporal fusion characteristics, the multi-task simulation and decision-making layer sequentially conducts multi-dimensional situational simulations. This layer uses a classifier and decoding function to perform anomaly judgment, facility failure prediction, and risk propagation simulation in parallel, calculating the comprehensive risk score for each node in the service area. Ultimately, based on the risk scores of all nodes, and under the condition of satisfying the maximum inspection cost constraint, the system determines the key inspection nodes and generates the optimal inspection path to achieve proactive early warning and precise handling of operational risks in the service area.

[0059] The above describes a highway service area inspection method provided by one or more embodiments of the present invention. Based on the same idea, the present invention also provides a corresponding highway service area inspection system, such as... Figure 3 As shown.

[0060] Figure 3 This invention provides a schematic diagram of a highway service area patrol system, which includes: The data acquisition module is used to acquire multimodal real-time sensing data of highway service areas; the multimodal real-time data includes: video surveillance data, vehicle detection data, personnel density data, IoT device status data, and environmental sensing data; The data processing and intelligent analysis module is used to extract video semantic features, personnel and vehicle density features, IoT device operation features, and target trajectory features from the multimodal real-time data of the highway service area, and then concatenate them to obtain a multimodal temporal feature vector. Using areas, facilities, and dynamic targets within the highway service area as nodes, and based on the multimodal temporal feature vector, the module determines the node features of each node in the highway service area. It constructs connection edges between nodes based on spatial proximity, logical subordination, visual reachability, and movement behavior relationships. Finally, it uses the node features and... By connecting edges, a dynamic spatiotemporal heterogeneous graph of highway service areas is constructed. A multimodal temporal graph convolutional network model based on a spatiotemporal graph convolutional network is constructed. The multimodal temporal graph convolutional network model includes: a cascaded temporal convolutional layer, a spatial graph attention layer, and a spatiotemporal interaction layer. In the temporal convolutional layer, temporal evolution features are extracted from the node features of the dynamic spatiotemporal heterogeneous graph of highway service areas. In the spatial graph attention layer, based on the temporal evolution features, spatial coupling features between nodes of the dynamic spatiotemporal heterogeneous graph are extracted from the spatial dimension. In the spatiotemporal interaction layer, the temporal evolution features and spatial coupling features are weighted and fused to obtain spatiotemporal fusion features. The inspection decision module, based on the aforementioned time evolution characteristics, spatial coupling characteristics, and spatiotemporal fusion characteristics, sequentially performs abnormal event judgment, facility failure prediction, and risk propagation simulation for highway service areas, generating inspection routes for highway service areas to complete the inspection tasks of highway service areas.

[0061] For specific limitations regarding a highway service area patrol system, please refer to the limitations of a highway service area patrol method described above, which will not be repeated here. Each module in the aforementioned highway service area patrol system can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0062] The present invention also provides a computer-readable storage medium storing a computer program that can be used to execute the above-described... Figure 1 A method for inspecting highway service areas is provided.

[0063] The present invention also provides Figure 4 The schematic diagram of the computer device shown is as follows: Figure 4 As shown, at the hardware level, this computer device includes a processor, internal bus, network interface, memory, and non-volatile memory, and may also include other hardware required for business operations. The processor reads the corresponding computer program from the non-volatile memory into memory and then executes it to achieve the above. Figure 1 A method for inspecting highway service areas is provided.

[0064] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

Claims

1. A method for inspecting highway service areas, characterized in that, include: Acquire multimodal real-time sensing data from highway service areas; The multimodal real-time data includes: video surveillance data, vehicle detection data, personnel density data, IoT device status data, and environmental perception data; Video semantic features, personnel and vehicle density features, IoT device operation features, and target trajectory features of highway service areas are extracted from the multimodal real-time data and concatenated to obtain a multimodal temporal feature vector. Using areas, IoT facilities, and dynamic targets within the highway service area as nodes, the node features of each node in the highway service area are determined based on the multimodal temporal feature vector. Connection edges are constructed to connect the nodes based on spatial proximity, logical subordination, visual reachability, and movement behavior relationships between nodes. Finally, a dynamic spatiotemporal heterogeneous graph of the highway service area is constructed based on the node features and connection edges. A multimodal temporal graph convolutional network model based on a spatiotemporal graph convolutional network is constructed; the multimodal temporal graph convolutional network model includes: cascaded temporal convolutional layers, spatial graph attention layers, and spatiotemporal interaction layers; In the temporal convolutional layer, temporal evolution features are extracted from the node features of the dynamic spatiotemporal heterogeneous graph of the highway service area; in the spatial graph attention layer, based on the temporal evolution features, spatial coupling features between nodes of the dynamic spatiotemporal heterogeneous graph are extracted from the spatial dimension; in the spatiotemporal interaction layer, the temporal evolution features and spatial coupling features are weighted and fused to obtain the spatiotemporal fusion features. Based on the aforementioned temporal evolution characteristics, spatial coupling characteristics, and spatiotemporal fusion characteristics, abnormal event judgment, facility failure prediction, and risk propagation simulation are performed sequentially on highway service areas to determine the inspection path for highway service areas.

2. The method for inspecting highway service areas as described in claim 1, characterized in that, The acquisition of multimodal real-time sensing data of highway service areas specifically includes: By using video surveillance equipment, vehicle behavior detection devices, personnel counting cameras, IoT device sensors, and environmental perception devices in highway service areas, real-time data collection is achieved, including video surveillance data, vehicle detection data, personnel density data, IoT device status data, and environmental perception data. Video surveillance data, vehicle detection data, personnel density data, IoT device status data, and environmental perception data are sequentially formatted, cleaned, and time-synchronized to obtain multimodal real-time perception data integrating personnel, vehicles, facilities, and environment.

3. The method for inspecting highway service areas as described in claim 1, characterized in that, From multimodal real-time data, video semantic features, personnel and vehicle density features, IoT device operation features, and target trajectory features of highway service areas are extracted, specifically including: The video surveillance data of each area is cropped to obtain the video surveillance image of each area; A CNN model is used to extract the video semantic features corresponding to the video surveillance footage of each area; the video semantic features reflect the changes in congestion, density, and environment in each area of ​​the highway service area. Input vehicle detection data and personnel density data into the density model to output the personnel density of each area, using the following formula: ； In the formula, The population density in each area within the highway service area, The vehicle density map is obtained statistically from the target detection model; H, W Let represent the pixel size of the density map D(x,y) in the vertical and horizontal directions, respectively. This represents the number of rows in the density map. This represents the number of columns in the density plot; Standardize the status data of IoT devices to obtain the operating characteristics of IoT devices; The appearance of people and vehicles and their historical displacement sequences are extracted from video surveillance data by a target detection model. Dynamic target appearance features and target displacement features are obtained and then stitched together to obtain target trajectory features. Among them, dynamic target appearance features are used to identify the same person or vehicle across cameras, and target displacement features are used to characterize the movement trend of the target person or vehicle.

4. The method for inspecting highway service areas as described in claim 1, characterized in that, The determination of node features for each node in a highway service area based on multimodal temporal feature vectors specifically includes: Based on the personnel and vehicle density features in the multimodal temporal feature vector and the video semantic features, the region node features are determined, expressed as follows: ； In the formula, It serves as a regional node for highway service areas. i This is the index number of the region node. and These are the first two parts of the multimodal temporal feature vector. t Personnel density characteristics and vehicle density characteristics at each time step The average speed in the region. For regional congestion rate, For video semantic features; Based on the operational characteristics of IoT devices in the multimodal time-series feature vector, the characteristics of facility nodes are determined, expressed as follows: ； In the formula, For facilities nodes in highway service areas For equipment type, For the location of the facility, The equipment status includes load factor and fault indicators; For time steps t Real-time observations of current, voltage, power, liquid level, and temperature and humidity; Based on the target trajectory features in the multimodal temporal feature vector, the dynamic target node features are determined, expressed as follows: ； In the formula, For dynamic target nodes of highway service areas. The target trajectory features are in the multimodal temporal feature vector, where, Indicates the first t Dynamic target appearance features at each time step This represents the historical displacement characteristics at multiple time steps.

5. A method for inspecting highway service areas as described in claim 1, characterized in that, The process involves constructing connection edges to connect the nodes based on spatial proximity, logical subordination, visual reachability, and mobility behavior relationships among them. Specifically, this includes: By utilizing the spatial proximity relationships between nodes in highway service areas, a distance decay function is constructed to determine spatial adjacency edges. The formula for calculating the edge weight of a spatial adjacency edge is as follows: ； In the formula, For nodes and nodes The edge weight of the spatially adjacent edges between them. and Each node and nodes spatial coordinates, These are spatial scale control parameters used to adjust the distance decay rate; Logical dependency edges are constructed based on the logical dependency relationships between facility nodes within the highway service area. The expression for the edge weight of a logical dependency edge is as follows: ； In the formula, The edge weight of the logical dependent edge; By analyzing the migration relationships of people and vehicles between different cameras, i.e., visual reachability relationships, visual reachability edges are constructed. The formula for the edge weight of a visual reachability edge is: ； In the formula, The edge weights of visually reachable edges. For nodes and nodes The angle formed between them is used to indicate whether they are being observed by the camera in the same direction; For indicator functions; For nodes and nodes The distance between them This is the maximum viewing distance of the camera; By analyzing the movement behavior relationships within highway service areas—that is, the flow displacement trajectories formed by the historical displacements of vehicles and personnel—movement relationship edges are constructed. The formula for calculating the edge weight of a movement relationship edge is as follows: ； In the formula, The edge weights of the move relation edges. Indicates that people or vehicles originate from the node. To the node The flow rate, For personnel or vehicles from the node To all nodes The sum of the flows; The weights of spatial adjacency edges, logical dependent edges, visually reachable edges, and mobility relation edges are weighted and fused to obtain the connecting edges used to connect each node and their weights. The formula is as follows: ； In the formula, For the type of edge, This is a set of edge types, including all spatially adjacent edges, logically dependent edges, visually reachable edges, and movement relation edges. The learning weights are for each type of edge.

6. The method for inspecting highway service areas as described in claim 1, characterized in that, The construction of a dynamic spatiotemporal heterogeneous graph of highway service areas based on the node features and connecting edges specifically includes: At any time step, a dynamic spatiotemporal heterogeneous graph of the highway service area is constructed based on node characteristics and connecting edges, using the following formula: ； In the formula, It is a set of nodes, including the node characteristics of facility nodes, area nodes, and dynamic target nodes of highway service areas; Let it be the set of edges; This is used for node type mapping to define the category of each node; This is an edge type mapping used to define the semantic type of the edge; Let the set of edge weights be defined over time. Dynamic changes.

7. A method for inspecting highway service areas as described in claim 1, characterized in that, The extraction of temporal evolution features from the node features of the dynamic spatiotemporal heterogeneous map of highway service areas specifically includes: Spatiotemporal convolution is performed on the node features to obtain the temporal evolution features of highway service areas, as shown in the formula: ； In the formula, For the first highway service area i The node at the th t The temporal evolution characteristics of each time step This represents the activation function. , Two sets of one-dimensional temporal convolution kernels, For element-wise multiplication, These are node features.

8. A method for inspecting highway service areas as described in claim 1, characterized in that, The extraction of spatial coupling features between nodes in a dynamic spatiotemporal heterogeneous graph based on time evolution characteristics from a spatial dimension specifically includes: The attention score between any two nodes is calculated using the following formula: ； In the formula, In the first t Time step node and nodes Attention scores between them This represents the transpose of the trainable weight vector in the attention mechanism, used to weight the concatenated multidimensional features. Represents the characteristic linear transformation matrix. For the first highway service area i The node at the th t The temporal evolution characteristics of each time step For the first highway service area j The node at the th t The temporal evolution characteristics of each time step For nodes With nodes The relationship embedding vector is used to describe the relationship type or association attribute between two nodes; The attention weight between two nodes is determined based on their attention scores, using the following formula: ； In the formula, In the first t Time step node and nodes Attention weights between them In the first t Time step node and nodes Attention scores between them In the first t Time step node and nodes Attention scores between them The set of nodes in the dynamic spatiotemporal heterogeneous graph of highway service areas; Based on the attention weights between every two nodes, neighbor aggregation is performed on each node to obtain the spatial coupling features of each node, as shown in the formula: ； In the formula, In the first t Time step node Spatial coupling characteristics, This represents the linear transformation matrix of spatial features, which is a trainable parameter used to linearly map the temporal evolution features of neighboring nodes.

9. A method for inspecting highway service areas as described in claim 1, characterized in that, The temporal evolution features and spatial coupling features are weighted and fused to obtain spatiotemporal fusion features, specifically including: Based on the temporal evolution characteristics of different nodes and different time steps in highway service areas, cross-time and cross-node attention is determined using the following formula: ； In the formula, For highway service area nodes and nodes In the k The time step to the 1 t Cross-time and cross-node attention at each time step This is the transpose of the trainable weight vector in the cross-temporal attention mechanism, used for weighted mapping of the concatenated feature vectors. Let be the linear transformation matrix of the features of the current time step node. Let be the linear transformation matrix of the features of the current time step node. For the first highway service area i The node at the th t The temporal evolution characteristics of each time step For the first j The node at the th tk The temporal evolution characteristics at each time step, among which... t>k ; An embedding vector representing the time span; The formula for normalizing attention across time and nodes is: ； In the formula, For normalized cross-time and cross-node attention; For highway service area nodes and nodes, cross-time and cross-node attention is given from the kth time step to the tth time step; Based on the normalized cross-time and cross-node attention, the temporal evolution features of all time steps are weighted and summed; then, the temporal evolution features of all time steps of the highway service area are weighted and fused with the spatial coupling features of the current time step to obtain the spatiotemporal fusion feature, as shown in the formula: ； In the formula, For highway service area nodes In the t The spatiotemporal fusion characteristics of each time step For nodes In the t Spatial coupling characteristics at each time step To output the characteristic linear transformation matrix, The fusion weighting coefficients for spatial and temporal features.

10. A method for inspecting highway service areas as described in claim 1, characterized in that, Based on spatiotemporal fusion features, the system sequentially performs anomaly assessment, facility failure prediction, and risk propagation simulation for highway service areas to generate a highway service area inspection strategy, specifically including: Based on the temporal evolution characteristics of highway service areas, anomaly identification is performed on highway service areas, and the anomaly probability formula is obtained as follows: ； In the formula, This represents the predicted probability vector of an abnormal event occurring at the highway service area at time step ttt. The weight matrix for the abnormal event identification classifier. The temporal evolution characteristics of highway service areas. The weight matrix for the abnormal event identification classifier; () represents the Softmax normalization function, used to map the model output to a probability distribution. Based on the spatiotemporal coupling characteristics of highway service areas, fault prediction is performed on IoT devices in highway service areas to obtain the future fault probability, using the following formula: ； In the formula, For the future Real-time status of IoT devices. For highway service area nodes In the t The spatiotemporal fusion characteristics of each time step For the future, the time span is The prediction decoding function is used to map the current spatiotemporal fusion features to the future device state; Based on the spatiotemporal coupling characteristics of different nodes in the highway service area and the weights of the connecting edges between different nodes, the propagation coefficient used for risk propagation simulation is determined, using the following formula: ； In the formula, The propagation coefficient is used for risk propagation simulation. For activation function, For parameters The risk propagation mapping function is used to comprehensively consider node characteristics and relationships between nodes. and They are nodes and nodes The spatiotemporal coupling characteristics, For nodes and nodes The weights of the edges connecting them; Based on the propagation coefficient used for risk propagation simulation, the risk evolution equation is determined. Risk propagation simulation is then performed on highway service areas to obtain the risk diffusion probability, as shown in the formula: ； ； In the formula, For the first Each node in the next time step The probability of risk exposure, For the first Each node in the next time step The probability of risk exposure, The first in the highway service area The node at the th The probability value of being in a state of risk exposure at a given time step. To represent nodes In the The probability value of a time step being in a risky active state. To represent nodes In the The probability value of a time step being in a risky active state. To represent nodes In the The probability value of a node being in a risk-sensitive state at a given time step is used to characterize the node's sensitivity to risk. The transition coefficient, representing the shift of risk from an exposed state to an activated state, is a non-negative parameter. To indicate the first Each node in the next time step The probability of risk exposure, To indicate the first in a highway service area The node at the th The probability value of a time step being in a state of risk activation or risk occurrence. This represents the decay or recovery coefficient of the risk activation state; Based on the aforementioned anomaly probability, future failure probability, and risk diffusion probability, the node risk score for highway service areas is determined using the following formula: ； In the formula, For highway service area nodes exist t Risk assessment at any time , and These are the weighting coefficients for abnormal event risk, facility failure risk, and risk propagation outcome in the comprehensive risk score, used to adjust the relative impact of different risk sources; Based on the risk scores of all nodes in the highway service area, key nodes for inspection are identified to determine the inspection route. The formula is as follows: ； In the formula, This refers to the set of all nodes in the highway service area. Indicates the patrol route P The inspection cost function, The maximum allowed inspection cost constraint threshold is used to limit the resource consumption of inspection paths during actual execution.

Images