Visual perception method and system based on multi-source data and ai decision, and medium

Abstract

The application discloses a visual perception method and system based on multi-source data and AI decision, and a medium, relates to the technical field of artificial intelligence, and the method comprises the following steps: spatiotemporal alignment is carried out on multi-source heterogeneous perception data, and multi-modal fusion data flow is constructed; environmental semantic features and target dynamic behavior features are extracted, style constraint features of each sensor are analyzed, and a multi-modal perception feature library is constructed; the multi-modal fusion data flow is subjected to feature enhancement, and a multi-modal joint representation vector is obtained; a target perception feature vector is acquired, the multi-modal joint representation vector and the target perception feature vector are processed, and an initial visual perception scheme is obtained; feature space transformation is carried out to obtain a candidate perception scheme; the candidate perception scheme is subjected to closed-loop verification, if the verification fails, secondary reasoning is triggered, and the target visual perception scheme is determined until; and resource allocation and perception strategy optimization are carried out. The application meets the high-precision perception and multi-source data efficient collaboration demand in a complex environment.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a visual perception method, system, and medium based on multi-source data and AI decision-making. Background Technology

[0002] With the continuous expansion of the network scale of core infrastructure equipment such as railways and power, the demand for safety operation and maintenance and inspection is becoming increasingly urgent. The traditional manual inspection mode has problems such as low efficiency, limited coverage, and prominent safety risks to workers in high-risk scenarios, and can no longer meet the requirements of large-scale and refined safety management of modern infrastructure.

[0003] Current mainstream visual perception technologies suffer from numerous pain points, failing to fully meet the practical application needs of various industries. Traditional systems often rely on single visual image data, exhibiting weak anti-interference capabilities and poor scene adaptability. Target recognition accuracy drops significantly in complex environments such as strong light, rain, fog, and occlusion, making it difficult to handle the complex working conditions of infrastructure inspection. Most systems can only achieve target detection and alarm functions, lacking a closed-loop mechanism of "perception-reasoning-decision-feedback," and are unable to generate differentiated control solutions based on the business needs of different scenarios (security monitoring, power construction, disaster early warning, etc.), resulting in insufficient practicality and accuracy in decision-making. Multimodal data, such as visual images, infrared video, LiDAR point clouds, and environmental sensor data, suffer from significant domain differences and difficulty in mining feature complementarity. Existing fusion technologies struggle to balance recognition accuracy and processing efficiency, failing to fully leverage the collaborative value of multi-source data.

[0004] In summary, existing technologies are insufficient to simultaneously meet the demands for high-precision sensing and efficient collaboration of multi-source data in complex environments, leaving room for improvement. Summary of the Invention

[0005] To overcome the shortcomings of existing technologies, meet the needs of high-precision perception and efficient collaboration of multi-source data in complex environments, and improve the efficiency of multi-source data collaboration and scenario adaptability, this application provides a visual perception method, system and medium based on multi-source data and AI decision-making.

[0006] Firstly, the objective of this invention is achieved through the following technical solution: Visual perception methods based on multi-source data and AI decision-making include: Collect multi-source heterogeneous sensing data and perform spatiotemporal alignment on the multi-source heterogeneous sensing data to construct a multimodal fusion data stream; Based on historical high-precision labeled samples, environmental semantic features and target dynamic behavior features are extracted, style constraint features of each sensor are analyzed, and a multimodal perception feature library is constructed. The multimodal fused data stream is enhanced with features based on a cross-modal attention module to obtain a multimodal joint representation vector; Based on the task instructions, the target perception feature vector is obtained from the multimodal perception feature library. The multimodal joint representation vector and the target perception feature vector are processed using a pre-trained multimodal fusion model to obtain an initial visual perception scheme. Based on the preset platform deployment constraints and task scenario tone parameters, the initial visual perception scheme is transformed in feature space to obtain candidate perception schemes. The candidate perception schemes are verified in a closed loop based on security, logical consistency and perception robustness indicators. If the verification fails, a feedback signal is sent back to trigger secondary reasoning until the target visual perception scheme is determined. Based on the target visual perception scheme, a dynamic scene map is constructed, and a graph neural network is used to aggregate entity relationships. Combined with an intelligent scheduling algorithm, resource allocation and perception strategy optimization are performed.

[0007] By adopting the above technical solutions, this application provides a closed-loop visual perception system with high scene adaptability that combines multi-source data and AI decision-making. Specifically, by spatiotemporal alignment of multi-source heterogeneous data and construction of a multimodal perception feature library, the problems of large differences in multimodal data domains and difficulty in mining feature complementarity in traditional inspection are effectively solved. By extracting environmental semantic features, target dynamic behavior features, and sensor style constraint features and constructing a feature library with a unified index, semantic-level association and adaptation of multi-source data are achieved, significantly improving perception accuracy and environmental adaptability in complex scenarios such as strong light, rain, fog, and occlusion. At the same time, based on feature enhancement of cross-modal attention modules and feature collaborative processing of multimodal fusion models, combined with dynamic invocation of target perception feature vectors by task instructions, the limitation of weak anti-interference ability of single-modal data is overcome. Through semantic alignment and feature fusion, the scene understanding depth of the initial perception scheme is enhanced, making the collaborative output of target detection, semantic segmentation, and behavior prediction more in line with actual business needs. By leveraging platform deployment constraints and scene-specific parameters to drive feature space transformation, dynamic adaptation of the perception scheme in terms of output granularity, response latency, and confidence thresholds is achieved, resolving the challenge of balancing computing resources and accuracy during edge deployment. The closed-loop verification mechanism effectively eliminates abnormal perception results through iterative feedback of security, logical consistency, and robustness indicators, avoiding decision-making biases caused by error accumulation in traditional open-loop systems and improving system reliability in complex dynamic environments. The combination of dynamic scene graphs and graph neural networks enables deep aggregation of entity relationships and real-time updates of scene states. Coupled with intelligent scheduling algorithms for dynamic allocation of perception resources, this significantly improves resource utilization and system response efficiency in multi-task concurrent scenarios, meeting the needs of large-scale infrastructure inspection applications.

[0008] In a preferred example, the construction of the multimodal sensing feature library includes: A multimodal pre-trained model is used to extract high-level semantic entities of the environment from the raw observation data of each sensor. Based on the co-occurrence frequency of the high-level semantic entities of the environment in historical high-precision labeled samples, an environmental semantic feature vector with semantic confidence weight is generated. A spatiotemporal interaction graph is constructed based on the target's historical trajectory sequence. A graph neural network is used to aggregate the state information of neighboring targets and extract the target's explicit behavioral features. The environmental semantic feature vector is used as a query key to retrieve semantically aligned potential intent features in the behavioral state space and fuse them to generate a target dynamic behavioral feature vector. The high-quality perception output of each sensor in typical scenarios is analyzed, and the point cloud density distribution, thermal imaging contrast curve and image texture complexity features are extracted. The sensor style prototype vector is generated by clustering. The perception compatibility matrix between the sensor style prototype vector and the environmental semantic feature vector is calculated, and style constraint parameters with compatibility higher than a preset threshold are selected as sensor style constraint features. The target dynamic behavior feature vector and the sensor style constraint feature are associated and mapped to the same feature space index to construct the multimodal perception feature library.

[0009] By adopting the above technical solutions, the multimodal pre-trained model extracts high-level semantic entities of the environment and generates environmental semantic feature vectors with semantic confidence weights based on co-occurrence frequency. This solves the problems of ambiguous environmental semantics and unclear co-occurrence relationships in multi-source data, and enhances the feature library's ability to understand scene semantics. Based on the spatiotemporal interaction graph, the model extracts target dynamic behavior features and retrieves semantically aligned potential intentions, realizing the semantic space mapping between target behavior and the environment, and improving the rationality of behavior prediction. By clustering sensor style prototype vectors and calculating the perception compatibility matrix, sensor style constraint features that are suitable for the current scene are selected, solving the feature conflict problem caused by differences in sensor characteristics in multimodal data. By mapping target dynamic behavior features and sensor style constraint features to the same feature space index, a structured and scalable multimodal perception feature library is constructed.

[0010] In a preferred embodiment, this application describes the process of using a pre-trained multimodal fusion model to process the joint multimodal representation vector and the target perception feature vector to obtain an initial visual perception scheme, specifically including: Based on the Transformer architecture, scene semantic token sequences are generated using the target-aware feature vectors. Based on the improved multi-scale U-Net structure, the semantic embedding of the scene semantic token sequence and the multimodal joint representation vector are fused across modally to generate a semantic segmentation mask; Based on a spatiotemporal graph convolutional network, the behavior prediction trajectory is generated by autoregression using the scene semantic token sequence and target historical trajectory features. If the confidence level of the generated result is lower than the preset confidence threshold, the inference space is resampled based on the deterministic results of other modalities to determine the initial visual perception scheme.

[0011] By adopting the above technical solutions, the Transformer architecture generates scene semantic token sequences based on target perception feature vectors, achieving precise binding between task instructions and scene semantics, and improving the target orientation of perception tasks; based on the improved multi-scale U-Net structure, it fuses scene semantic tokens and multimodal joint representation vectors to generate semantic segmentation masks under dual constraints on semantic boundaries and geometric structures, solving the problem of semantic and geometric information being disconnected in traditional segmentation; by using a spatiotemporal graph convolutional network to autoregressively generate behavior prediction trajectories, and combining target historical trajectories with environmental context features, it improves the dynamic adaptability of trajectory prediction; for low-confidence results, it uses deterministic results from other modalities to constrain the inference space and resamples.

[0012] In a preferred embodiment of this application, the feature space transformation of the initial visual perception scheme includes: Construct a perceptual adaptation loss function that includes an output granularity deviation term, a time delay exceedance penalty term, and a confidence stability term; The lightweight discriminator module is used to perform condition-adaptive discrimination on the target detection box density, semantic segmentation region smoothness, and behavior prediction trajectory continuity of the initial visual perception scheme, and the environmental adaptation gradient is obtained. Based on the environment adaptation gradient and the task scene tone parameters, the attention weights and feature selection gating of the multimodal fusion model are adjusted through gradient backpropagation to minimize the perceptual adaptation loss function while maintaining the integrity of the original semantics. When deployed at the edge, a computing resource constraint is introduced to limit inference energy consumption and memory usage; the above transformation is iteratively executed until the environment adaptation gradient converges and meets the real-time requirements.

[0013] By adopting the above technical solutions, a perception adaptation loss function that includes output granularity deviation, latency exceedance penalty, and confidence stability is constructed. This quantifies the difference between the initial perception scheme and the platform deployment requirements, providing a clear optimization target for feature space transformation. A lightweight discriminator module is used to perform condition adaptability discrimination on the initial scheme and generate an environment adaptation gradient, realizing dynamic perception of the complexity of the current scene. Based on the environment adaptation gradient and task scene tone parameters, the attention weights and feature selection gating of the multimodal fusion model are adjusted to minimize the loss function while preserving the original semantic integrity, thus balancing perception accuracy and computational efficiency.

[0014] In a preferred example, this application performs closed-loop verification on candidate sensing schemes, including: The candidate perception schemes are scanned using a pre-trained anomaly detection model to identify and mark abnormal segments, and schemes with potential security risks are eliminated. Calculate the distance between the center point of the target detection box and the centroid of the semantic segmentation mask, and quantify the consistency score between the behavior prediction trajectory and the historical motion pattern. If it exceeds the preset tolerance range, it is judged as a multimodal logic mismatch. The stability index of the candidate sensing scheme under noise and occlusion disturbances is calculated. If it is lower than the preset baseline, it is determined to be a fragile scheme. The mismatch type or performance gap value is converted into a negative reward signal, and the loss function and model parameters of the multimodal fusion model are updated.

[0015] By adopting the above technical solutions, the pre-trained anomaly detection model scans the target categories, trajectory abrupt changes, and occlusion regions in candidate schemes, identifies and marks anomalies that pose safety risks or are physically infeasible, and eliminates perception results that may cause safety accidents from the source. By calculating the Euclidean distance between the center point of the target detection box and the centroid of the semantic segmentation mask, and quantifying the consistency score between the predicted behavior trajectory and the historical motion pattern, the multimodal logic mismatch problem is accurately located and feedback labels are generated, solving the decision bias caused by inconsistencies in detection, segmentation, and prediction results. Based on the robustness evaluation model, stability indicators under noise, occlusion, and illumination disturbances are calculated, vulnerable schemes are identified and converted into negative reward signals, and by updating the loss function and parameters of the multimodal fusion model, adaptive optimization of the model for complex environments is achieved.

[0016] In a preferred embodiment, the method further includes: adaptive evolution of the multimodal sensing feature library. The false detection rate, false negative rate, and system response latency of the captured runtime are mapped into perceived quality reward signals; The latent intent weights in the target dynamic behavior feature vector are updated using the perceived quality reward signal, and the confidence weights of the associated environmental semantic feature vectors are boosted in reverse. Monitor external event streams, calculate the semantic similarity between event keywords and existing environmental semantic features, and if the similarity exceeds the dynamic injection threshold, generate a timeliness-enhanced perception feature vector. The timeliness-enhanced sensing feature vector is written into the real-time cache of the multimodal sensing feature library, and removed or downgraded after its lifespan expires.

[0017] By adopting the above technical solutions, the false detection rate, false negative rate, and response latency of the operation end are captured and mapped into a perception quality reward signal. The potential intent weight in the target dynamic behavior feature vector is dynamically adjusted, and the confidence weight of the associated environmental semantic features is enhanced in reverse, realizing the enhanced memory of high-frequency correct perception patterns in the feature library. External event streams (such as traffic control and sudden weather changes) are monitored and the similarity with existing environmental semantic features is calculated. Timely enhanced perception feature vectors are dynamically injected, solving the problem of knowledge lag in the feature library under sudden scenarios. Timely features are written into the real-time cache and a life cycle is set. After expiration, they are automatically removed or downgraded, avoiding the interference of outdated features on perception results and realizing the adaptive evolution of the multimodal perception feature library.

[0018] In a preferred embodiment of this application, the method of using graph neural networks to aggregate entity relationships and combining intelligent scheduling algorithms for resource allocation and perception strategy optimization includes: The entity objects and their attributes are extracted from the target visual perception scheme as graph nodes, and directed edges between nodes are constructed based on spatial distance and semantic co-occurrence relationship to form the dynamic scene graph. The dynamic scene graph is processed by a graph neural network for message passing and aggregation, and the node embedding is updated to reflect the latest scene state. The state space is defined based on the updated dynamic scene graph, and the state-action value function is learned by combining the intelligent scheduling algorithm to output the perceived resource scheduling instructions and strategy optimization instructions.

[0019] By adopting the above technical solutions, entity objects and attributes are extracted from the target visual perception scheme to construct dynamic scene graph nodes. Directed edges are constructed based on spatial distance, motion trajectory similarity, and semantic co-occurrence relationships, realizing the structured expression and dynamic updating of scene entity relationships. Graph neural networks are used to perform message passing and aggregation on the scene graph, updating node embeddings to reflect the latest scene state, thus enhancing the system's ability to understand the scene context. Based on the updated dynamic scene graph, a state space is defined, and a state-action value function is learned by combining an intelligent scheduling algorithm to output perception resource scheduling instructions and strategy optimization instructions. This realizes on-demand allocation of computing resources and dynamic optimization of perception strategies, significantly improving resource utilization and system response efficiency in multi-task concurrent scenarios, and meeting the high-concurrency requirements of large-scale infrastructure inspections such as railways and power plants.

[0020] Secondly, the objective of this invention is achieved through the following technical solution: A visual perception system based on multi-source data and AI decision-making, the system comprising: The data acquisition module is used to collect multi-source heterogeneous sensing data and perform spatiotemporal alignment on the multi-source heterogeneous sensing data to construct a multimodal fusion data stream; The feature library construction module is used to extract environmental semantic features and target dynamic behavior features based on historical high-precision labeled samples, analyze the style constraint features of each sensor, and construct a multimodal perception feature library. The cross-modal feature enhancement module is used to enhance the features of the multimodal fused data stream based on the cross-modal attention module, and obtain the multimodal joint representation vector; The initial perception scheme generation module is used to obtain the target perception feature vector from the multimodal perception feature library based on the task instructions, and to process the multimodal joint representation vector and the target perception feature vector using a pre-trained multimodal fusion model to obtain the initial visual perception scheme. The perception scheme adaptation module is used to perform feature space transformation on the initial visual perception scheme according to the preset platform deployment constraints and task scenario tone parameters to obtain candidate perception schemes. The verification and iterative optimization module is used to perform closed-loop verification of candidate perception schemes based on security, logical consistency and perception robustness indicators. If the verification fails, a feedback signal is sent back to trigger secondary reasoning until the target visual perception scheme is determined. The graph construction and resource scheduling module is used to construct dynamic scene graphs based on target visual perception schemes, aggregate entity relationships using graph neural networks, and optimize resource allocation and perception strategies using intelligent scheduling algorithms.

[0021] Thirdly, the objective of this invention is achieved through the following technical solution: A computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described visual perception method based on multi-source data and AI decision-making.

[0022] Fourthly, the objective of this invention is achieved through the following technical solution: A computer program product includes a computer program / instructions that, when executed by a processor, implement the steps of the visual perception method based on multi-source data and AI decision-making as described above.

[0023] In summary, this application includes at least one of the following beneficial technical effects: 1. Collect multi-source heterogeneous sensing data such as visible light, infrared, lidar and millimeter wave radar and perform high-precision spatiotemporal alignment to construct a unified multimodal fusion data stream, effectively overcoming the perception blind spots of a single sensor in complex environments; combine a multimodal sensing feature library built based on bbbbbbb historical annotations with a cross-modal attention enhancement mechanism to achieve deep fusion of semantic and structural information; 2. This application further introduces task-driven target perception features to guide the generation of initial schemes, and forms a "perception-decision-optimization" closed loop through working condition adaptation, closed-loop verification and dynamic map scheduling, thereby improving the perception integrity, semantic consistency and operational robustness of the system in all weather and all scenarios, while supporting efficient deployment under edge resource constraints. Attached Figure Description

[0024] Figure 1 This is a flowchart of a visual perception method based on multi-source data and AI decision-making in one embodiment of this application; Figure 2 This is a flowchart of step S2 in a visual perception method based on multi-source data and AI decision-making in one embodiment of this application. Detailed Implementation

[0025] The present application will be further described in detail below with reference to the accompanying drawings.

[0026] In one embodiment, such as Figure 1 As shown, this application discloses a visual perception method based on multi-source data and AI decision-making. Specifically, in the context of railway inspection, this application includes the following steps: S1: Collect multi-source heterogeneous sensing data and perform spatiotemporal alignment on the multi-source heterogeneous sensing data to construct a multimodal fusion data stream.

[0027] In this embodiment, the application uses a railway track intelligent inspection scenario as an example. When applied to eight types of railway safety risk intelligent inspection scenarios, the multi-source heterogeneous sensing data includes visible light high-definition cameras, infrared thermal imagers, lidar, and trackside fixed monitoring equipment carried by the track inspection vehicle. In this embodiment, spatiotemporal alignment adopts a dual reference of "BeiDou time synchronization + track mileage markers". The hardware layer synchronizes the clocks of each sensor through the BeiDou time synchronization module, and the software layer maps the data of each sensor to a unified mileage coordinate system based on the absolute coordinates of the track mileage markers, in meters, and aligns them to a mileage window of 0.5m.

[0028] Specifically, taking the daily inspection of high-speed rail as an example, the inspection system connects to: one track inspection vehicle (equipped with a visible light camera, infrared thermal imager, and lidar) and more than 3,000 fixed monitoring stations along the line. The raw data collected is transmitted to the edge computing node through the railway 5G private network. First, the BeiDou timestamp of each data is extracted, and then the absolute mileage is converted through the mileage marker coordinates (such as "K456+789.1"). The multi-source data within ±0.25m of the same mileage are grouped into the same mileage frame. After completing the spatiotemporal alignment, the data is stitched together to form a multimodal fusion data stream that includes the dynamic video stream of the inspection vehicle, the static monitoring images of the trackside, the infrared thermal radiation sequence, and the point cloud coordinates.

[0029] Furthermore, the construction of multimodal fusion data streams needs to retain the structured characteristics of railway infrastructure, such as sleeper spacing, catenary elevation, and signal display status, to avoid distortion of engineering parameters caused by general alignment methods. In this embodiment, a two-level index structure of "mileage-modality" is adopted. The first dimension is the aligned track mileage with an accuracy of 0.01m. The second dimension is the engineering feature parameters of each modality at that mileage, such as the ROI coordinates of the track area in the visible light image, the track surface elevation value of the point cloud, and the signal light position status of the trackside monitoring. Specifically, in the above-mentioned high-speed rail inspection scenario, the aligned multimodal fusion data stream contains the following for every 0.5m mileage frame: visible light image of the inspection vehicle (3840×2160), trackside fixed monitoring image (3840×2160), infrared thermal imager data (640×512), and a three-dimensional point cloud of approximately 80,000 points, including x / y / z coordinates and reflection intensity; at the same time, parameters such as standard track gauge (1435mm) and catenary height (5300mm±30mm) at each mileage are recorded through pre-stored track engineering drawings.

[0030] S2: Based on historical high-precision labeled samples, extract environmental semantic features and target dynamic behavior features, analyze the style constraint features of each sensor, and construct a multimodal perception feature library.

[0031] In this embodiment, the historical high-precision annotation samples are from the China State Railway Group's "Railway Infrastructure Defect Database," "Railway Safety Risk Case Database," and the inspection annotation data of this line over the past 5 years. The annotation content includes environmental semantic features and target dynamic behavior features. Environmental semantic features include track geometric deformation, abnormal catenary suspension, signal status, turnout position, foreign object encroachment area, integrity of protective fence, signs of roadbed settlement, and illegal intrusion by personnel. Target dynamic behavior features include abnormal train running posture, construction personnel not wearing reflective vests, unplanned track work, and large machinery operating beyond its designated area.

[0032] Environmental semantic feature extraction adopts the Rail-ResNet model (Rail-ResNet is a dedicated model fine-tuned for 8 types of railway risk tasks based on the standard ResNet-101, and does not propose a new network topology). Based on ImageNet pre-training, a new semantic branch for 8 types of railway risk points is added. The input is the visible light image and corresponding point cloud of historical samples, and the output is the category, location and confidence of 8 types of environmental semantic entities. The co-occurrence frequency of each semantic entity in the historical samples is calculated. The co-occurrence frequency is normalized and used as the semantic confidence weight to generate a weighted environmental semantic feature vector (4096 dimensions, the first 2048 dimensions are the one-hot encoding of 8 semantic categories, and the last 2048 dimensions are the corresponding confidence weights).

[0033] Specifically, the extraction of target dynamic behavior features needs to be combined with railway operation procedures and train operation specifications to avoid missing violations due to relying solely on static features. In this embodiment, the target historical trajectory sequence is extracted using a sliding window of "operation period + 30 minutes before and after", with a window length of 60 minutes and a step size of 5 minutes. When constructing the spatiotemporal interaction graph, nodes represent the dynamic parameters of the target, and edges represent the operational logic relationships between targets. Dynamic parameters include train acceleration, construction personnel location, and machinery operation radius. Operational logic relationships include the conflict relationship between the construction area and the train operation section, and the coordination relationship between personnel and machinery. The state information of neighboring targets is aggregated through Rail-GCN (Railway-Specific Graph Convolutional Network) to extract the explicit behavioral features of the target, such as train speeding and construction personnel not wearing safety helmets. At the same time, using the environmental semantic feature vector as the query key, cosine similarity retrieval is performed in the behavioral state space, and the state with the highest similarity is selected as the latent intention feature. The explicit behavioral features and the latent intention features are concatenated to generate the target dynamic behavior feature vector, which has a dimension of 2048. The behavioral state space is predefined into three categories: "compliant operation", "general violation", and "serious violation".

[0034] Specifically, taking the high-speed rail construction monitoring scenario as an example, the visible light video stream, personnel positioning data, and large machinery GPS trajectory of a certain construction area over the past 60 minutes are extracted. After constructing a spatiotemporal interaction map, Rail-GCN aggregates the explicit behavioral features as "personnel not wearing reflective vests + large machinery crossing the boundary" (probability 0.91). Then, using the environmental semantic feature vector as the query, the potential intent feature is retrieved as "unplanned illegal operation" (similarity 0.93). The two are then concatenated to generate a 2048-dimensional target dynamic behavioral feature vector.

[0035] S3: Feature enhancement is performed on the multimodal fusion data stream based on the cross-modal attention module to obtain the multimodal joint representation vector.

[0036] In this embodiment, the cross-modal attention module adopts a "railway scene adaptive enhancement multi-head cross-attention" mechanism. The query is the track geometry parameters, such as track gauge, elevation, and alignment; the key and value are the texture features of the visible light image, the temperature gradient features of the infrared image, and the normal vector features of the point cloud. The contribution of each modality to the eight types of risk features is dynamically adjusted through attention weights. In this embodiment, under strong light, rain, fog, or nighttime conditions, the system automatically activates the multi-modal adaptive enhancement module: Retinex dehazing and HDR synthesis are performed on the visible light image, non-uniformity correction is performed on the infrared image, and dynamic voxel filtering is performed on the point cloud data. The input of the cross-modal attention module is the enhanced multi-modal feature tensor (visible light feature map is 2048×32×32, infrared feature map is 1024×64×48, and point cloud feature map is 1024×N). First, the attention score is calculated, and then the railway scene constraint prior is introduced for weighted Softmax normalization. Prior constraints in railway scenarios, such as "foreign object intrusion," primarily rely on the fusion features of visible light and lidar.

[0037] Specifically, the multimodal joint representation vector needs to integrate multi-physics coupling information of the railway scene. A "railway semantic gating unit" is added after the cross-modal attention module. The gating coefficients (between 0 and 1) of each modality feature are generated by the Sigmoid function. The attention output is multiplied element by element to filter out low-confidence modality features. The training of the gating unit adopts a "risk-normal" contrastive learning strategy. Positive samples are multimodal data of 8 risk areas labeled, and negative samples are normal state data of the same area. The gating coefficients are optimized by the Triplet Loss function.

[0038] S4: Based on the task instructions, obtain the target perception feature vector from the multimodal perception feature library, and use the pre-trained multimodal fusion model to process the multimodal joint representation vector and the target perception feature vector to obtain the initial visual perception scheme.

[0039] In this embodiment, the task instruction is issued by the railway operation and maintenance scheduling system, and includes the inspection section, inspection items, and output requirements. The target perception feature vector is retrieved from the multimodal perception feature library. The retrieval method is as follows: the task instruction is converted into a railway domain semantic embedding vector (the railway domain semantic embedding vector is encoded using the Rail-BERT pre-trained model, with a dimension of 1024, specifically optimized for railway safety regulations), and cosine similarity matching is performed with the semantic embeddings of each feature vector in the feature library. The top k with the highest similarity (k=8 in this embodiment, corresponding to 8 types of risk) are selected as the target perception feature vector.

[0040] Specifically, the pre-trained multimodal fusion model adopts the Rail-Fusion Transformer architecture, based on improvements to BEiT-3, and adds eight new pre-training tasks for railway risks. It is pre-trained on a multimodal dataset of railway safety risks containing 8 million frames of visible light images, 3 million frames of infrared video, 2 million sets of point clouds, and 1 million operational behavior data points, supporting joint inference of image, point cloud, and behavioral features.

[0041] In this embodiment, the initial visual perception scheme generation process is as follows: First, the multimodal joint representation vector (2048 dimensions) and the target perception feature vector (8×1024 dimensions) are concatenated into a 10240-dimensional input vector, which is then input into the encoder part of the Rail-FusionTransformer model. After outputting intermediate features, eight parallel decoding heads are used to detect eight types of safety risks, such as the track geometry deformation detection head outputting track gauge deviation, the catenary suspension anomaly detection head outputting conductor height deviation, and the personnel illegal intrusion detection head outputting intrusion location and number of people. Specifically, in the above-mentioned high-speed rail inspection scenario, the initial visual perception scheme output by the pre-trained model includes: track geometry deformation, catenary suspension anomaly, foreign object encroachment into restricted areas, damaged protective fences, personnel illegal intrusion, construction workers not wearing reflective vests, unplanned track operations, and large machinery operating beyond its designated area.

[0042] S5: Based on the preset platform deployment constraints and task scenario tone parameters, perform feature space transformation on the initial visual perception scheme to obtain candidate perception schemes.

[0043] In this embodiment, the platform deployment constraints include the computing power of edge computing nodes, data storage capacity (≤5TB / 100km), and real-time requirements (inspection time per kilometer ≤5 minutes); the task scenario tuning parameters are determined by the line level, such as high-speed railway mainline focusing on detection accuracy, and conventional railway focusing on risk coverage.

[0044] In this embodiment, the perceptual adaptation loss function is:

[0045] In this context, α, β, and γ are weighting coefficients, set to 0.6, 0.2, and 0.2 respectively for the high-speed rail scenario. Risk location deviation is the difference between the initial plan and the manually measured value, single-frame time is the actual inference time, and risk detection rate is the accuracy of the historical validation set. Specifically, taking the inspection of permafrost sections of plateau railways as an example, the platform is deployed at containerized edge nodes along the line. The task scenario parameters are "permafrost subgrade settlement and protective fence integrity monitoring," requiring settlement detection accuracy ≤1mm and time per kilometer ≤4 minutes. In this case, α is set to 0.7 (emphasizing settlement accuracy), β to 0.2, and γ to 0.1.

[0046] Specifically, the lightweight discriminator module adopts the MobileNetV3-Rail architecture. The input is the visualization results of the initial visual perception scheme, such as a risk box overlay and parameter deviation curve. The output is the working condition adaptability score, ranging from 0 to 1, with higher values ​​indicating better adaptation. The environment adaptation gradient is obtained through backpropagation of the discriminator, reflecting the shortcomings of the initial scheme in permafrost sections. In this embodiment, when adjusting the attention weights of the multimodal fusion model, a gradient inversion layer (GRL) is used to backpropagate the environment adaptation gradient to the cross-modal attention module. Simultaneously, the threshold of railway semantic gating is adjusted to minimize the perception adaptation loss function without changing the core structure of the model. For example, in the above-mentioned high-altitude permafrost region scenario, the initial solution's settlement detection accuracy was 1.5mm (not meeting the ≤1mm requirement), and the time taken per kilometer was 4.5 minutes (exceeding the 4-minute threshold). The lightweight discriminator gave a working condition adaptability score of 0.65 (low). By adjusting the weights of the cross-modal attention module through backpropagation and increasing the point cloud feature weights (from 0.35 to 0.50) to enhance settlement detection capabilities, and simultaneously reducing the hidden layer dimension of the Rail-Fusion Transformer encoder head from 2048 to 1536, after two iterations, the settlement detection accuracy improved, the time taken per kilometer decreased, and the environment adaptation gradient converged, resulting in a candidate perception solution.

[0047] S6: Based on security, logical consistency and perception robustness indicators, perform closed-loop verification of candidate perception schemes. If the verification fails, a feedback signal is sent back to trigger secondary reasoning until the target visual perception scheme is determined.

[0048] In this embodiment, the safety verification uses a railway-specific anomaly detection model, outputting a safety risk level of 0-4, with level 4 representing a risk of speed limit / operational shutdown. The railway-specific anomaly detection model is trained based on a 1D-CNN+LSTM model, with the input being a time series of track geometric parameters. Logical consistency verification calculates the matching degree between the risk detection box and the BIM design parameters; if the matching degree is <0.8, it is considered a mismatch, otherwise a successful match. Perceptual robustness verification adds railway-specific perturbations to the candidate scheme input, such as salt-and-pepper noise σ=0.15 in the visible light image simulating sandstorms and a random offset of 10% in the point cloud simulating permafrost thawing. The stability index of the output results is calculated; for example, a risk IoU ≥ 0.75 indicates stability.

[0049] For example, taking the inspection of protective fences along a coastal railway as an example, the candidate sensing scheme detected a 0.8m damage to the protective fence at K200+300. The deviation between the center point of the detection box and the position of the protective fence in the BIM model was 2 pixels (matching degree 0.96>0.8, logically consistent). After adding a 10% point cloud offset perturbation, the risk IoU was 0.86 (≥0.75, robust). However, the safety risk level output by the 1D-CNN+LSTM model was level 3. Since the damage may lead to personnel intrusion, it was determined that a review was required, triggering a feedback signal.

[0050] The secondary inference is achieved by adjusting the "risk saliency weight" of the multimodal fusion model. For example, the weight of local contrast features is increased to address minor risks, or the weight of temporal behavioral features is enhanced to verify the development trend of risks. In this embodiment, the feedback signal includes mismatch types, such as security risks, logical mismatches, and poor robustness. If the mismatch type is a security risk, the weight of the target dynamic behavior feature vector in the multimodal joint representation vector is increased from 0.3 to 0.6 because the dynamic behavior features contain the evolution trend of risks, and the data is re-input into the pre-trained model for inference. If the verification still fails after two iterations, a bidirectional iterative mechanism of "perception-reasoning-decision-feedback" is triggered, and the result is uploaded to the cloud center deployment platform for secondary verification. The cloud verification result is sent back to optimize the edge perception parameters and decision model. When any of the eight types of risk events is detected, the system generates a structured alarm package, which is sent back to the dispatch center through the railway 5G private network, and at the same time, a local audio-visual alarm is triggered at the edge.

[0051] S7: Construct a dynamic scene map based on the target visual perception scheme, aggregate entity relationships using graph neural networks, and combine intelligent scheduling algorithms to optimize resource allocation and perception strategies.

[0052] In this embodiment, the nodes of the dynamic scene graph include static and dynamic attributes of railway entities. Static attributes include type, mileage, and design parameters, while dynamic attributes include risk status, geometric deviation, and environmental parameters. The types of directed edges include spatial adjacency, functional association, and causal association.

[0053] Specifically, the graph neural network uses GATv2-Rail with a hidden layer dimension of 1024, and the aggregation method is multi-head attention with 16 heads. When updating node embedding, the weight of the edges is considered. For example, the weight of spatially adjacent edges is 1 / distance, and the weight of functionally related edges is the load transfer coefficient. Taking the inspection of the throat area of ​​a high-speed railway station as an example, the extracted entity nodes include: mainline track (type "arrival / departure track", mileage K0+000), turnout (model 18#, straight track), signal (red light), catenary (conductor height 5300mm), and protective fence (damaged state). The constructed directed edges include: mainline track → turnout (spatially adjacent, distance 0m, weight 1.0), turnout → signal (functionally related, interlocking relationship, weight 0.9), catenary → mainline track (causal relationship, "conductor height deviation" affects "pantograph current collection", weight 0.85), and protective fence → mainline track (spatially adjacent, distance 3m, weight 0.33).

[0054] The intelligent scheduling algorithm adopts PPO-Rail (Railway Dedicated Near-End Strategy Optimization Algorithm). The state space is the node embedding vector of the updated dynamic scene graph (dimension 1024 × number of nodes). The action space includes perception resource scheduling instructions and strategy optimization instructions (such as "increase the confidence threshold of foreign object intrusion detection from 0.85 to 0.90"). In this embodiment, the loss function of PPO-Rail is the surrogate objective function, and the reward function is: R = risk detection rate increase × 0.7 + resource consumption reduction × 0.3. During training, historical inspection data is stored through the railway dedicated network, and the network parameters are updated every 50 frames. Specifically, in the aforementioned station throat area scenario, the node embedding after GATv2-Rail aggregation shows that turnout #18 has the dynamic characteristic of "poor contact". PPO-Rail outputs the scheduling instructions: "The inspection vehicle's visible light camera focuses on turnout #18, zooms to 2x to collect details of the switch rail area" and "The ground lidar scanning frequency is increased from 10Hz to 40Hz, focusing on collecting point clouds in the turnout area". At the same time, the strategy optimization instruction is "The turnout contact detection confidence threshold is increased from 0.85 to 0.90". After execution, the detection rate of poor turnout contact is improved.

[0055] The central deployment platform adopts a cloud-edge collaborative architecture to uniformly manage the algorithm repository, computing power scheduling, and task distribution. In this embodiment, the algorithm repository integrates ≥50 railway-specific detection algorithms (covering 8 types of safety risks, power violations, etc.) and supports dynamic loading according to line level and inspection tasks. The computing power scheduling module implements containerized resource management based on Kubernetes, supports concurrent access of 3000+ cameras, and each camera can be adapted to 1-5 detection items, such as "track geometric deformation + foreign object intrusion + protective fence integrity".

[0056] In one embodiment, such as Figure 2As shown, a multimodal sensing feature library is constructed, including: S21: Use a multimodal pre-trained model to extract high-level semantic entities of the environment from the raw observation data of each sensor, and generate an environmental semantic feature vector with semantic confidence weights based on the co-occurrence frequency of high-level semantic entities of the environment in historical high-precision labeled samples.

[0057] In this embodiment, the high-level semantic entities in the environment strictly correspond to eight types of railway safety risk scenarios: track geometric deformation, abnormal overhead contact line suspension, signal status, turnout position, foreign object encroachment on restricted areas, integrity of protective fences, signs of roadbed settlement, and illegal intrusion by personnel. This embodiment uses Rail-ResNet-101 as a multimodal pre-trained model. The input is a high-definition visible light image (3840×2160) from a track inspection vehicle and a lidar point cloud (128 lines). The output is the bounding boxes, pixel-level masks, and initial confidence scores of the above eight types of semantic entities.

[0058] Specifically, the calculation of semantic confidence weight needs to be combined with the special characteristics of the railway scenario. In this embodiment, the co-occurrence frequency is corrected by introducing the line section coefficient: straight section coefficient 1.0, curved section coefficient 1.2, and turnout area coefficient 1.5.

[0059] S22: Construct a spatiotemporal interaction graph based on the target's historical trajectory sequence, use a graph neural network to aggregate the state information of neighboring targets, extract the target's explicit behavioral features, and use the environmental semantic feature vector as the query key to retrieve semantically aligned potential intent features in the behavioral state space, and fuse them to generate the target's dynamic behavioral feature vector.

[0060] In this embodiment, the target historical trajectory sequence covers four types of dynamic railway behaviors: abnormal train operation posture, construction workers not wearing reflective vests, unplanned track work, and large machinery operating beyond its designated boundaries. In this embodiment, the nodes of the spatiotemporal interaction graph are defined as the dynamic parameters of the target, including train acceleration, construction worker location, machinery operating radius, and time window. Edges are defined as the operational logic relationships between targets, such as the conflict between the construction area and the train operating section, and the collaborative relationship between personnel and machinery.

[0061] S23: Analyze the high-quality perception output of each sensor in typical scenarios, extract point cloud density distribution, thermal imaging contrast curve and image texture complexity features, and cluster to generate sensor style prototype vectors; calculate the perception compatibility matrix between sensor style prototype vectors and environmental semantic feature vectors, and select style constraint parameters with compatibility higher than a preset threshold as sensor style constraint features.

[0062] In this embodiment, eight typical railway scenarios were selected: straight sections, curved sections, turnout areas, tunnels, bridges, rainy / foggy days, nighttime, and strong light. High-quality sensor outputs (manually marked as "high-quality") were collected for each scenario: point cloud density distribution (number of points per unit area) was extracted from LiDAR; thermal imaging contrast curve (temperature difference ΔT between target and background) was extracted from infrared thermal imagers; and image texture complexity (LBP feature entropy) was extracted from visible light cameras. Sensor style prototype vectors were generated using K-means++ clustering (K=8, corresponding to the eight scenario categories), with a dimension of 1536, 192 dimensions for each scenario category.

[0063] Specifically, the calculation of the perception compatibility matrix requires quantifying the degree of matching between sensor style and environmental semantics. The calculation formula is as follows: ,in Let i be the prototype vector of the style of the i-th type of sensor. Let j be the semantic feature vector of the environment of class j. This represents the scene weight. The preset compatibility threshold is 0.85, and style constraint parameters that are higher than the preset compatibility threshold are filtered out.

[0064] S24: Associate and map the target dynamic behavior feature vector with the sensor style constraint features to the same feature space index to construct a multimodal perception feature library.

[0065] In this embodiment, the feature space index adopts a three-level index structure of "mileage-risk type-sensor". The target dynamic behavior feature vector (1024 dimensions) and the sensor style constraint features (1536 dimensions) are mapped to the same 2048-dimensional feature space through a linear projection layer, and then associated with the three-level index: the first-level index is track mileage, the second-level index is risk type (8 types of environmental semantics), and the third-level index is sensor type. The multimodal perception feature library is stored in a MySQL cluster, and the feature vectors are updated incrementally every morning at midnight, and a full feature clustering is performed once a month.

[0066] In one embodiment, in step S4, the multimodal joint representation vector and the target perception feature vector are processed using a pre-trained multimodal fusion model to obtain an initial visual perception scheme, specifically including: S41: Based on the Transformer architecture, scene semantic token sequences are generated using target-aware feature vectors.

[0067] In this embodiment, the Transformer architecture employs a Rail-BERT pre-trained model (12-layer encoder, 768-dimensional hidden layers). The input consists of target-aware feature vectors and semantic embeddings of task instructions, while the output is a sequence of scene semantic tokens. These scene semantic tokens must contain railway-specific engineering semantics, such as clearance, track surface elevation, and overhead contact line height. In this embodiment, 200 railway-specific tokens are added to the Rail-BERT vocabulary, and the token embedding is optimized through domain-adaptive pre-training (fine-tuned on 1 million railway inspection texts).

[0068] S42: Based on the improved multi-scale U-Net structure, the semantic embedding of the scene semantic token sequence and the multimodal joint representation vector are fused across modalities to generate a semantic segmentation mask.

[0069] In this embodiment, the improved multi-scale U-Net adds a railway structure-aware module on the basis of the classic U-Net, which includes track parallelism constraints (ensuring that the segmented area extends along the track direction) and catenary symmetry constraints.

[0070] Specifically, cross-modal fusion employs attention-gated skip connections, introducing attention weights of semantic tokens into the encoder-decoder skip path of U-Net. The attention weights are calculated from the cosine similarity between the scene semantic tokens and the multimodal features, with higher similarity resulting in greater weights.

[0071] S43: Based on spatiotemporal graph convolutional networks, it uses scene semantic token sequences and target historical trajectory features to autoregressively generate behavior prediction trajectories.

[0072] In this embodiment, the input of the Spatiotemporal Graph Convolutional Network (ST-GCN) includes two types of features: static semantic scene semantic token sequences and dynamic temporal target historical trajectory features. The graph structure of ST-GCN is defined as a "human skeleton graph" (for human behavior) or a "mechanical joint graph" (for large machinery), and outputs the predicted trajectory of behavior in the next 5 seconds. The predicted trajectory of behavior includes position, speed, and posture. Taking the scene of construction workers not wearing reflective vests as an example, the scene semantic token sequence is: [CLS] Construction workers not wearing reflective vests [SEP] K306+050 [SEP] 2 people [SEP]. The target historical trajectory is a feature (past 30 seconds, sampling rate 10Hz): Person A's position (K306+048, shoulder), Person B's position (K306+052, center of the road). After ST-GCN constructs the "human skeleton graph" (17 key points), it generates the trajectory for the next 5 seconds through autoregression: Person A moves towards the safe zone (speed 1.2m / s), Person B is stationary (possibly performing work).

[0073] Among these, behavior prediction needs to be combined with railway operation safety regulations (such as "personnel in the track center must evacuate 10 minutes before the train arrives"). The ST-GCN output layer adds a rule constraint branch to compare the predicted trajectory with the regulations and output a compliance score. In this embodiment, the compliance score formula is: , where d is the distance between the person and the center of the track, v is the train speed, and t is the remaining time.

[0074] S44: If the confidence level of the generated result is lower than the preset confidence threshold, the inference space is resampled based on the deterministic results of other modalities to determine the initial visual perception scheme.

[0075] In this embodiment, in high-risk railway scenarios, a confidence threshold of 0.9 can be preset. The generated results include the IoU of the semantic segmentation mask and the ADE (average displacement error) of the behavior prediction trajectory. When the confidence level is lower than the threshold, a multimodal resampling mechanism is activated: the results of the high-confidence modality are preferentially selected as priors, and the inference space of other modalities is resampled under constraints. For example, taking foreign object intrusion detection in rainy and foggy weather as an example, the initial generated results are: visible light segmentation IoU=0.82 (<0.9), infrared segmentation IoU=0.94 (>0.9), and point cloud segmentation IoU=0.91 (>0.9). The system uses the infrared segmentation mask (plastic bag area) and the point cloud location (K305+200, 2m outside the protective fence) as priors to resample the inference space of the visible light image, such as searching for foreign object features only in the prior area.

[0076] Specifically, the resampling process uses Monte Carlo dropout uncertainty estimation, and calculates the variance of the generated results through multiple forward propagations (with dropout enabled). The smaller the variance, the higher the certainty. In this embodiment, the number of resampling times is set to 10, and the average of the 5 results with the smallest variance is taken as the final output.

[0077] In one embodiment, step S5 involves performing a feature space transformation on the initial visual perception scheme, including: S51: Construct a perceptual adaptation loss function that includes an output granularity deviation term, a delay over-limit penalty term, and a confidence stability term.

[0078] In this embodiment, the output granularity deviation term matches the detection accuracy requirements of the perception adaptation loss function in railway inspection, the delay over-limit penalty term matches the real-time requirements, and the confidence stability term matches the reliability requirements.

[0079] Specifically, the expression for the perceptual adaptation loss function is: ,in The granularity of the risk area output by the model, such as foreign object size and crack length; This refers to the manually labeled truth granularity, such as 0.1mm; For single-frame inference latency, For the scene latency threshold, the high-speed rail mainline =200ms, conventional railway =500ms. Var(C) is the variance of the confidence level of the output for 5 consecutive frames; α, β, and γ are weighting coefficients, set to 0.5, 0.3, and 0.2 for high-speed rail scenarios and 0.4, 0.4, and 0.2 for conventional speed scenarios. In this embodiment, differentiated granular weights are set for different risk types: track geometric deformation weight is 1.2, and foreign object intrusion weight is 0.8.

[0080] S52: The lightweight discriminator module is used to perform condition-adaptive discrimination on the target detection box density, semantic segmentation region smoothness and behavior prediction trajectory continuity of the initial visual perception scheme, and the environmental adaptation gradient is obtained.

[0081] In this embodiment, the lightweight discriminator module uses MobileNetV3-Rail, optimized for railway scenarios, with only 3.5M parameters. The input is the visualization results of the initial scheme, including the detection box overlay map, the segmentation mask heatmap, and the trajectory prediction curve. The output is the working condition adaptability score S∈[0,1], where 1 is a perfect fit. The environment adaptability gradient is obtained through backpropagation of the discriminator, reflecting the shortcomings of the initial scheme under specific working conditions. Taking the inspection of a certain railway section as an example, the initial scheme detects large machinery crossing the boundary at K500+050, with a detection box density of 0.8 boxes / m² (standard ≥ 1.2 boxes / m²), a semantic segmentation region smoothness (edge ​​gradient variance) of 0.15 (standard ≤ 0.08), and a behavior prediction trajectory continuity (adjacent frame displacement standard deviation) of 0.3m (standard ≤ 0.1m). The MobileNetV3-Rail output adaptability score S=0.62 (lower), and the environment adaptability gradient is obtained through backpropagation. .

[0082] Specifically, the condition adaptability assessment needs to be combined with the priority of the eight types of railway risks. A risk priority weight layer is added to the discriminator to impose a greater penalty on the adaptability score of high-priority risks. In the case above, "large machinery crossing the boundary" is a high risk (priority 1), with a weight of 1.5, and the environmental adaptability gradient is amplified to ∇S′=[0.6, 1.05, 0.75].

[0083] S53: Based on the environmental adaptation gradient and task scenario tone parameters, the attention weights and feature selection gating of the multimodal fusion model are adjusted through gradient backpropagation to minimize the perceptual adaptation loss function while maintaining the integrity of the original semantics.

[0084] In this embodiment, the task scenario tone parameters are determined by the line level and inspection type, and the adjustment strategy is as follows: if the environment adaptation gradient is in... A value >0.3 indicates insufficient detection box density, in which case the attention weights for visible light and LiDAR should be increased; if A value >0.3 indicates uneven segmentation, which is the gating coefficient for enhancing infrared texture features; if A value greater than 0.3 indicates that the trajectory is discontinuous, so the weight of the dynamic behavior feature vector should be increased.

[0085] Specifically, the adjustment process must maintain the integrity of the original semantics, so a semantic consistency constraint is introduced: the cosine similarity between the adjusted feature vector and the original feature vector must be ≥0.9.

[0086] S54: When deploying at the edge, introduce a computing resource constraint to limit inference energy consumption and memory usage; iteratively execute the above transformation until the environment adaptation gradient converges and meets the real-time requirements.

[0087] In this embodiment, the computing resource constraint is defined as follows: Where E is inference power consumption in J / frame and M is memory usage in GB. The constraint coefficients are set to 0.6 and 0.4 for the edge end. In this embodiment, the edge end deployment platform is Huawei Atlas 800 (256 TOPS INT8 computing power, 256GB memory, 2kW power consumption), requiring single-frame inference energy consumption ≤0.5J and memory usage ≤4GB.

[0088] In one embodiment, step S6 specifically includes: S61: Use a pre-trained anomaly detection model to scan candidate sensing schemes, identify and label abnormal segments, and eliminate schemes with potential security risks.

[0089] In this embodiment, the pre-trained anomaly detection model adopts the Rail-Anomaly-Transformer architecture. The input consists of time series data of track geometry parameters (gauge, elevation, alignment, and level) and dynamic data on catenary tension / conductor height from candidate sensing schemes. The output is a safety risk level (0-4, with level 4 indicating immediate speed limit / operational shutdown risk). In this embodiment, the anomaly detection model is pre-trained on the "Track Anomaly Database" published by China State Railway Group (containing 100,000 anomaly samples such as bulging tracks, broken rails, and broken catenary wires), focusing on identifying structural hazards among eight types of safety risks. In this embodiment, anomaly segments are labeled using a "mileage-risk type-confidence" triplet format.

[0090] S62: Calculate the distance between the center point of the target detection box and the centroid of the semantic segmentation mask, and quantify the consistency score between the behavior prediction trajectory and the historical motion pattern. If it exceeds the preset tolerance range, it is judged as a multimodal logic mismatch.

[0091] Specifically, multimodal logic consistency verification aims to ensure that different modalities have a unified understanding of the same physical entity. For risks related to foreign object intrusion or the integrity of protective fences, it calculates the center point of the target detection box in the visible light image. Centroid of the point cloud segmentation mask with lidar Euclidean distance between them: For the behavior of construction workers, the consistency score between the future trajectory predicted by ST-GCN and the historical movement patterns (such as walking at a constant speed and standing still) is calculated using the DTW dynamic time warping algorithm.

[0092] For example, taking the inspection of damaged protective fences along a coastal railway as an example, the candidate solution detects a crack in the protective fence mesh at K200+300. The center point of the visible light detection frame is (512, 256), and the centroid of the corresponding point cloud segmentation mask is (510, 258), with an Euclidean distance D = 2.83 pixels. The preset tolerance range is 5 pixels. Since D < 5, the judgment logic is consistent. If D > 5, it is judged as a logical mismatch.

[0093] S63: Calculate the stability index of candidate sensing schemes under noise and occlusion disturbances. If it is lower than the preset baseline, it is judged as a vulnerable scheme.

[0094] In this embodiment, the perception robustness verification targets the complex natural environment of railways (rain, fog, sandstorms, nighttime) and equipment interference. In this embodiment, stability testing is conducted under railway-specific disturbances: 1) Simulating wind and sand / rain and snow: Salt-and-pepper noise with σ=0.15 is added to the visible light image, and 10% random outliers are added to the point cloud data to simulate rain and snow noise; 2) Simulating occlusion: A rectangular occlusion covering 20% ​​of the image area is randomly generated to simulate tree branches or train occlusion; 3) Simulating strong light / backlight: The image Gamma value and contrast are adjusted. The stability index is defined as the IoU (Intersection over Union) of the detection boxes before and after the disturbance being ≥0.75.

[0095] For nighttime or infrared scenes, the consistency of thermal imaging contrast also needs to be verified. In this embodiment, non-uniform noise (simulating detector temperature drift) is applied to the infrared image. If the target temperature gradient change is less than 10%, it is still considered stable.

[0096] S64: Convert the mismatch type or performance gap value into a negative reward signal and update the loss function and model parameters of the multimodal fusion model.

[0097] In this embodiment, the feedback signal is customized according to the mismatch type. If the mismatch type is a security risk, the weight of the target dynamic behavior feature vector in the multimodal joint representation vector is increased by 0.3. If it is a logical mismatch, the weight of the alignment loss in the cross-modal attention module is increased. If it is a vulnerable scheme, the perturbation intensity of the data augmentation layer is increased.

[0098] Furthermore, if the edge fails the verification after two iterations, a second review is triggered in the cloud. In this embodiment, the candidate scheme and mismatch log are uploaded to the central deployment platform. The cloud uses a more accurate model (such as Rail-Fusion-Large with a larger number of parameters) for review and sends the optimized parameters back to the edge.

[0099] In one embodiment, the visual perception method based on multi-source data and AI decision-making further includes: S201: Capture the false detection rate, false negative rate, and system response delay at the runtime and map them into a perceived quality reward signal.

[0100] In this embodiment, the operational indicator capture needs to target eight types of railway safety risks. The false detection rate is defined as the percentage of samples where "the edge detection identifies it as a risk, but the cloud verification shows it as normal." The false negative rate is defined as the percentage of samples where "the cloud verification identifies it as a risk, but the edge detection fails to detect it." The system response latency is defined as the total time consumed in the entire process from "data acquisition → edge inference → cloud alarm." Perceived quality reward signal. Using a negative mapping function: Where FDR is the false positive rate and FNR is the false negative rate. To respond to the delay, Baseline delay; High-speed rail scenario =5s; weight =0.4, =0.5, =0.1, the penalty for missed detection has the highest weight, which is in line with the principle of prioritizing railway safety.

[0101] Specifically, the indicator capture needs to dynamically adjust the weights according to risk type; in this embodiment, the weights are set for high-risk types. =0.7, low-risk type (such as "signal device status abnormal") setting =0.3.

[0102] S202: Update the latent intent weights in the target dynamic behavior feature vector using the perceived quality reward signal, and inversely enhance the confidence weights of the associated environmental semantic feature vectors.

[0103] In this embodiment, the target dynamic behavior feature vector (1024 dimensions) includes explicit behavior features (512 dimensions) and latent intention features (512 dimensions). The latent intention weight is updated through backpropagation of the reward signal, and the update formula is: ,in, To achieve a perceptual adaptation loss function, =0.01 is the learning rate. Meanwhile, the confidence weights of the associated environmental semantic feature vectors (4096 dimensions) are inversely enhanced through the "intent-semantic" association matrix A (predefined association degrees between 8 types of intents and 8 types of semantics, such as the association degree of 0.95 between "unplanned operation" and "track operation area"). .

[0104] Weight updates need to set upper and lower limits, such as ∈[0.1, 1.0]. In this embodiment, when When the value is less than -0.05 (severely negative), an emergency update mode is triggered, and the learning rate is increased to η=0.05.

[0105] S203: Monitor external event streams, calculate the semantic similarity between event keywords and existing environmental semantic features, and if the similarity exceeds the dynamic injection threshold, generate a timeliness-enhanced perception feature vector.

[0106] In this embodiment, the external event stream includes weather warnings, dispatching commands, and equipment fault notifications, which are accessed in real time through the railway dispatching system via API interfaces.

[0107] Specifically, the Rail-BERT model is used to calculate the semantic similarity between event keywords (such as "red rainstorm warning") and eight existing environmental semantic features (such as "roadbed settlement signs" and "protective fence integrity"). The dynamic injection threshold is set to 0.85, and enhancement is triggered when the similarity is >0.85.

[0108] The enhanced timeliness perception feature vector is generated by combining "basic semantic features" with "event modulation parameters." The modulation parameters include a risk sensitivity enhancement coefficient and a detection threshold reduction. In this embodiment, the enhanced vector has a dimension of 4096, consistent with the original environmental semantic feature vector. The first 2048 dimensions are basic semantic features, and the last 2048 dimensions are event modulation parameters.

[0109] S204: Write the timeliness-enhanced perception feature vector into the real-time cache of the multimodal perception feature library, and perform removal or weight reduction processing after the life cycle expires.

[0110] In this embodiment, the real-time cache is deployed using a Redis cluster, supporting millisecond-level read and write operations. The time-to-live (TTL) is dynamically set according to the event type: weather warning TTL = 24 hours, scheduling command TTL = until the command ends, and equipment failure TTL = 48 hours (such as the overhead contact line repair period). In this embodiment, the cache index structure is "event ID-risk type-mileage range", such as "rainstorm warning-K300+000 to K310+000-foreign object intrusion area".

[0111] Specifically, after the feature lifecycle expires, a "soft deweighting" process is performed (instead of direct deletion) to avoid feature library oscillation; in this embodiment, the weight decays exponentially after expiration: The attenuation coefficient λ = 0.1, and t is the time after expiration; if the enhancement is not triggered again within 72 hours, it will be completely removed.

[0112] In one embodiment, a graph neural network is used to aggregate entity relationships, and an intelligent scheduling algorithm is combined to optimize resource allocation and perception strategies, including: S71: Extract entity objects and their attributes from the target visual perception scheme as graph nodes, construct directed edges between nodes based on spatial distance and semantic co-occurrence relationships to form a dynamic scene graph; entity objects strictly correspond to the core elements of the 8 types of railway safety risk scenarios, including: static entities and dynamic entities, where static entities include track geometry, catenary suspension, signal lights, turnouts, protective fences, roadbed sections, etc., and dynamic entities include trains, construction workers, large machinery, inspection vehicles, etc.; entity attributes include static attributes and dynamic attributes, where static attributes include mileage K value, equipment model, design parameters, and dynamic attributes include real-time geometric deviation, temperature, behavioral trajectory, risk confidence level.

[0113] Specifically, taking the inspection of the throat area of ​​a high-speed railway station as an example, the extracted entity nodes are as follows: Static entities: No. 18 turnout (mileage K0+150, model No. 18, straight track, design tightness ≤0.5mm), signal (mileage K0+148, red light, interlocking status normal), and protective fence (mileage K0+145, good integrity). Dynamic entities: G1 train (mileage K0+120, speed 300km / h, normal operating posture), construction worker A (mileage K0+152, not wearing a reflective vest, located on the road shoulder), and inspection vehicle (mileage K0+100, speed 25km / h, equipped with visible light / infrared / lidar).

[0114] The construction of directed edges is based on spatial adjacency (requiring distance <10m), semantic co-occurrence (requiring co-occurrence frequency >80% in historical samples), and functional association (such as the "turnout-signal" interlocking relationship and the "train-track" load-bearing relationship). The edge weight is jointly determined by the reciprocal of the distance (spatial edge), the co-occurrence frequency (semantic edge), and the load transfer coefficient (functional edge). Specifically, the directed edges constructed in the above case include: Spatial adjacent edge: Construction worker A → No. 18 turnout (distance 2m, weight 0.5); Semantic co-occurrence edge: 18# turnout → signal (historical co-occurrence frequency 92%, weight 0.92); Functional association edge: G1 train → 18# turnout (load-bearing relationship, load transfer coefficient 0.95, weight 0.95).

[0115] S72: Utilize graph neural networks to perform message passing and aggregation on dynamic scene graphs, and update node embeddings to reflect the latest scene state.

[0116] In this embodiment, the neural network adopts a railway-specific graph attention network (GATv2-Rail), which aggregates neighbor node information through a multi-head attention mechanism and automatically learns the importance of different types of edges. During message transmission, each node collects information sent by its neighbor nodes and performs weighted fusion based on edge weights and attention scores to update its own feature representation.

[0117] Taking a high-speed train inspection in rainy and foggy weather as an example, the dynamic scene map includes nodes such as "foreign object intrusion (plastic bag, K305+200)", "protective fence (damaged, K305+198)", and "inspection vehicle (K305+150)". When GATv2-Rail aggregates, it focuses on the spatial adjacency relationship between the foreign object and the protective fence (distance 2 meters, attention weight 0.8), the semantic co-occurrence relationship between the foreign object and the inspection vehicle (historical co-occurrence frequency 88%, attention weight 0.7), and the functional association relationship between the protective fence and the foreign object (protection failure leads to foreign object entry, attention weight 0.9). After two rounds of message passing, the feature representation of the foreign object intrusion node is significantly enhanced, including composite features such as "lightweight material (visible light texture)", "no heat source (infrared)" and "suspended outside the fence (point cloud)".

[0118] The node embedding update also incorporates multimodal perception features. The multimodal joint representation vector output from step S3 (containing visible light, infrared, and point cloud features) is mapped to the same feature space as the graph neural network through linear projection, and then concatenated with the aggregated node features to form the final node embedding. Specifically, in the above case, the multimodal joint representation vector of the foreign object intrusion node is projected and concatenated with the node embedding output by GATv2-Rail. The final embedding vector simultaneously includes traditional image processing features, thermal imaging features, and 3D spatial features, making the description of the foreign object more comprehensive.

[0119] S73: Define the state space based on the updated dynamic scene graph, learn the state-action value function by combining the intelligent scheduling algorithm, and output the perceived resource scheduling instruction and strategy optimization instruction.

[0120] In this embodiment, the state space is a concatenated vector embedded from all nodes in the dynamic scene graph (dimension = number of nodes × 2048); the action space includes perceptual resource scheduling instructions (such as camera zoom, LiDAR scanning frequency adjustment, and UAV hovering position) and strategy optimization instructions (such as risk detection threshold adjustment and model weight update); the intelligent scheduling algorithm adopts PPO-Rail (railway-specific near-end strategy optimization algorithm), and the reward function combines risk handling priority and resource consumption. ,in For risk management benefits, high-risk event handling increases by 1.0, while missed detection decreases by 2.0; This represents the cost of resource consumption (such as drone battery life and edge computing power usage, which are negative values). .

[0121] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0122] In one embodiment, a visual perception system based on multi-source data and AI decision-making is provided, which corresponds to the visual perception method based on multi-source data and AI decision-making in the above embodiments.

[0123] The visual perception system based on multi-source data and AI decision-making includes a data acquisition module, a feature library construction module, a cross-modal feature enhancement module, an initial perception scheme generation module, a perception scheme adaptation module, a verification and iterative optimization module, and a map construction and resource scheduling module. Detailed descriptions of each functional module are as follows: The data acquisition module is used to collect multi-source heterogeneous sensing data and perform spatiotemporal alignment on the multi-source heterogeneous sensing data to construct a multimodal fusion data stream; The feature library construction module is used to extract environmental semantic features and target dynamic behavior features based on historical high-precision labeled samples, analyze the style constraint features of each sensor, and construct a multimodal perception feature library. The cross-modal feature enhancement module is used to enhance the features of the multimodal fused data stream based on the cross-modal attention module, and obtain the multimodal joint representation vector; The initial perception scheme generation module is used to obtain the target perception feature vector from the multimodal perception feature library based on the task instructions, and to process the multimodal joint representation vector and the target perception feature vector using a pre-trained multimodal fusion model to obtain the initial visual perception scheme. The perception scheme adaptation module is used to perform feature space transformation on the initial visual perception scheme according to the preset platform deployment constraints and task scenario tone parameters to obtain candidate perception schemes. The verification and iterative optimization module is used to perform closed-loop verification of candidate perception schemes based on security, logical consistency and perception robustness indicators. If the verification fails, a feedback signal is sent back to trigger secondary reasoning until the target visual perception scheme is determined. The graph construction and resource scheduling module is used to construct dynamic scene graphs based on target visual perception schemes, aggregate entity relationships using graph neural networks, and optimize resource allocation and perception strategies using intelligent scheduling algorithms.

[0124] For specific limitations regarding the visual perception system based on multi-source data and AI decision-making, please refer to the limitations of the visual perception method based on multi-source data and AI decision-making mentioned above, which will not be repeated here. Each module in the aforementioned visual perception system based on multi-source data and AI decision-making can be implemented entirely or partially through software, hardware, or a combination thereof. Each module can be embedded in or independent of the processor in the computer device in hardware form, or it can be stored in the memory of the computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0125] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the visual perception method based on multi-source data and AI decision-making as described above.

[0126] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0127] In one embodiment, particularly according to embodiments of the present invention, the processes described above with reference to the flowchart can be implemented as a computer software program. For example, embodiments of the present invention include a computer program product comprising a computer program / instructions that, when executed by a processor, implement the steps of the visual perception method based on multi-source data and AI decision-making as described above. In such embodiments, the computer program can be downloaded and installed from a network via a communication module, and / or installed from a removable medium. When the computer program is executed by a central processing unit (CPU), it performs the various functions defined in the present invention.

[0128] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0129] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A visual perception method based on multi-source data and AI decision-making, characterized in that, include: Collect multi-source heterogeneous sensing data and perform spatiotemporal alignment on the multi-source heterogeneous sensing data to construct a multimodal fusion data stream; Based on historical high-precision labeled samples, environmental semantic features and target dynamic behavior features are extracted, style constraint features of each sensor are analyzed, and a multimodal perception feature library is constructed. The multimodal fused data stream is enhanced with features based on a cross-modal attention module to obtain a multimodal joint representation vector; Based on the task instructions, the target perception feature vector is obtained from the multimodal perception feature library. The multimodal joint representation vector and the target perception feature vector are processed using a pre-trained multimodal fusion model to obtain an initial visual perception scheme. Based on the preset platform deployment constraints and task scenario tone parameters, the initial visual perception scheme is transformed in feature space to obtain candidate perception schemes. The candidate perception schemes are verified in a closed loop based on security, logical consistency and perception robustness indicators. If the verification fails, a feedback signal is sent back to trigger secondary reasoning until the target visual perception scheme is determined. Based on the target visual perception scheme, a dynamic scene map is constructed, and a graph neural network is used to aggregate entity relationships. Combined with an intelligent scheduling algorithm, resource allocation and perception strategy optimization are performed.

2. The visual perception method based on multi-source data and AI decision-making according to claim 1, characterized in that, The construction of the multimodal perception feature library includes: A multimodal pre-trained model is used to extract high-level semantic entities of the environment from the raw observation data of each sensor. Based on the co-occurrence frequency of the high-level semantic entities of the environment in historical high-precision labeled samples, an environmental semantic feature vector with semantic confidence weight is generated. A spatiotemporal interaction graph is constructed based on the target's historical trajectory sequence. A graph neural network is used to aggregate the state information of neighboring targets and extract the target's explicit behavioral features. The environmental semantic feature vector is used as a query key to retrieve semantically aligned potential intent features in the behavioral state space and fuse them to generate a target dynamic behavioral feature vector. The high-quality perception output of each sensor in typical scenarios is analyzed, and the point cloud density distribution, thermal imaging contrast curve and image texture complexity features are extracted. The sensor style prototype vector is generated by clustering. The perception compatibility matrix between the sensor style prototype vector and the environmental semantic feature vector is calculated, and style constraint parameters with compatibility higher than a preset threshold are selected as sensor style constraint features. The target dynamic behavior feature vector and the sensor style constraint feature are associated and mapped to the same feature space index to construct the multimodal perception feature library.

3. The visual perception method based on multi-source data and AI decision-making according to claim 1, characterized in that, The process of using a pre-trained multimodal fusion model to process the joint multimodal representation vector and the target perception feature vector to obtain an initial visual perception scheme specifically includes: Based on the Transformer architecture, scene semantic token sequences are generated using the target-aware feature vectors. Based on the improved multi-scale U-Net structure, the semantic embedding of the scene semantic token sequence and the multimodal joint representation vector are fused across modally to generate a semantic segmentation mask; Based on a spatiotemporal graph convolutional network, the behavior prediction trajectory is generated by autoregression using the scene semantic token sequence and target historical trajectory features. If the confidence level of the generated result is lower than the preset confidence threshold, the inference space is resampled based on the deterministic results of other modalities to determine the initial visual perception scheme.

4. The visual perception method based on multi-source data and AI decision-making according to claim 1, characterized in that, The feature space transformation of the initial visual perception scheme includes: Construct a perceptual adaptation loss function that includes an output granularity deviation term, a time delay exceedance penalty term, and a confidence stability term; The lightweight discriminator module is used to perform condition-adaptive discrimination on the target detection box density, semantic segmentation region smoothness, and behavior prediction trajectory continuity of the initial visual perception scheme, and the environmental adaptation gradient is obtained. Based on the environment adaptation gradient and the task scene tone parameters, the attention weights and feature selection gating of the multimodal fusion model are adjusted through gradient backpropagation to minimize the perceptual adaptation loss function while maintaining the integrity of the original semantics. When deployed at the edge, a computing resource constraint is introduced to limit inference energy consumption and memory usage; the above transformation is iteratively executed until the environment adaptation gradient converges and meets the real-time requirements.

5. The visual perception method based on multi-source data and AI decision-making according to claim 1, characterized in that, Closed-loop verification of candidate sensing schemes includes: The candidate perception schemes are scanned using a pre-trained anomaly detection model to identify and mark abnormal segments, and schemes with potential security risks are eliminated. Calculate the distance between the center point of the target detection box and the centroid of the semantic segmentation mask, and quantify the consistency score between the behavior prediction trajectory and the historical motion pattern. If it exceeds the preset tolerance range, it is judged as a multimodal logic mismatch. The stability index of the candidate sensing scheme under noise and occlusion disturbances is calculated. If it is lower than the preset baseline, it is determined to be a fragile scheme. The mismatch type or performance gap value is converted into a negative reward signal, and the loss function and model parameters of the multimodal fusion model are updated.

6. The visual perception method based on multi-source data and AI decision-making according to claim 1, characterized in that, The method further includes: adaptive evolution of the multimodal sensing feature library: The false detection rate, false negative rate, and system response latency of the captured runtime are mapped into perceived quality reward signals; The latent intent weights in the target dynamic behavior feature vector are updated using the perceived quality reward signal, and the confidence weights of the associated environmental semantic feature vectors are boosted in reverse. Monitor external event streams, calculate the semantic similarity between event keywords and existing environmental semantic features, and if the similarity exceeds the dynamic injection threshold, generate a timeliness-enhanced perception feature vector. The timeliness-enhanced sensing feature vector is written into the real-time cache of the multimodal sensing feature library, and removed or downgraded after its lifespan expires.

7. The visual perception method based on multi-source data and AI decision-making according to claim 1, characterized in that, The method of using graph neural networks to aggregate entity relationships and combining intelligent scheduling algorithms to optimize resource allocation and perception strategies includes: The entity objects and their attributes are extracted from the target visual perception scheme as graph nodes, and directed edges between nodes are constructed based on spatial distance and semantic co-occurrence relationship to form the dynamic scene graph. The dynamic scene graph is processed by a graph neural network for message passing and aggregation, and the node embedding is updated to reflect the latest scene state. The state space is defined based on the updated dynamic scene graph, and the state-action value function is learned by combining the intelligent scheduling algorithm to output the perceived resource scheduling instructions and strategy optimization instructions.

8. A visual perception system based on multi-source data and AI decision-making, characterized in that, The system includes: The data acquisition module is used to collect multi-source heterogeneous sensing data and perform spatiotemporal alignment on the multi-source heterogeneous sensing data to construct a multimodal fusion data stream; The feature library construction module is used to extract environmental semantic features and target dynamic behavior features based on historical high-precision labeled samples, analyze the style constraint features of each sensor, and construct a multimodal perception feature library. The cross-modal feature enhancement module is used to enhance the features of the multimodal fused data stream based on the cross-modal attention module, and obtain the multimodal joint representation vector; The initial perception scheme generation module is used to obtain the target perception feature vector from the multimodal perception feature library based on the task instructions, and to process the multimodal joint representation vector and the target perception feature vector using a pre-trained multimodal fusion model to obtain the initial visual perception scheme. The perception scheme adaptation module is used to perform feature space transformation on the initial visual perception scheme according to the preset platform deployment constraints and task scenario tone parameters to obtain candidate perception schemes. The verification and iterative optimization module is used to perform closed-loop verification of candidate perception schemes based on security, logical consistency and perception robustness indicators. If the verification fails, a feedback signal is sent back to trigger secondary reasoning until the target visual perception scheme is determined. The graph construction and resource scheduling module is used to construct dynamic scene graphs based on target visual perception schemes, aggregate entity relationships using graph neural networks, and optimize resource allocation and perception strategies using intelligent scheduling algorithms.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the visual perception method based on multi-source data and AI decision-making as described in any one of claims 1 to 7.

10. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the visual perception method based on multi-source data and AI decision-making as described in any one of claims 1 to 7.

Images