An automatic inspection robot traffic inspection method based on an urban traffic scene

Abstract

The application provides a kind of automatic inspection robot traffic inspection method based on urban traffic scene, it is related to intelligent traffic technology field, the method comprises: obtaining original multi-modal sensor data from inspection robot hardware platform and pre-processing, standard multi-modal sensor data is constructed BEV space representation learning module, simultaneously constructs VLA model module, constructs traffic event detection and anomaly identification module, and constructs end-to-end motion planning and navigation module and man-machine interaction and remote management module, define model training and deployment strategy to carry out model training and deployment control automatic inspection robot traffic inspection to multiple modules.Solve the technical problems that the prior art has insufficient perception, semantic understanding and navigation planning capability, which leads to the difficulty of efficient and reliable autonomous inspection of the inspection robot in complex traffic environment.The technical effects of autonomous perception, intelligent decision-making and efficient inspection of the inspection robot in complex traffic environment are realized.

Description

Technical Field

[0001] This invention relates to the field of intelligent transportation technology, specifically to a traffic inspection method using an automated inspection robot based on urban traffic scenarios. Background Technology

[0002] With the accelerating pace of urbanization and the rapid development of intelligent transportation systems, urban traffic management is becoming increasingly complex and challenging. Traditional traffic management primarily relies on a combination of fixed surveillance cameras and manual patrols. This involves installing monitoring equipment at intersections and key road sections to achieve real-time traffic monitoring. While this approach improves efficiency to some extent, it still has many limitations. For example, fixed cameras can only cover specific areas, resulting in numerous blind spots. Furthermore, the fixed installation locations of fixed cameras prevent flexible adjustments to the monitoring scope and focus based on changes in traffic conditions.

[0003] In response to the limitations of traditional traffic management methods, automated inspection robots have gained increasing attention in recent years as a novel means of traffic inspection. An automated inspection robot is a mobile platform integrating multiple sensors and possessing autonomous navigation and intelligent decision-making capabilities. It can autonomously cruise in urban road environments, achieving comprehensive perception and inspection of traffic conditions.

[0004] However, existing robots generally rely on single sensors for perception, resulting in limited perception capabilities under varying lighting conditions, inclement weather, or complex traffic conditions. Furthermore, the immaturity of multimodal information fusion technology restricts a comprehensive understanding of traffic scenarios. Robots also have limited capabilities in semantic understanding and decision-making, struggling to accurately identify complex traffic events and anomalies, and lack the ability to understand and flexibly execute natural language commands. Their navigation and planning largely depend on rule-based algorithms, exhibiting poor adaptability to dynamically changing traffic environments. In addition, the various functional modules are mostly designed independently, lacking end-to-end joint optimization, making it difficult to improve overall system performance. Moreover, their insufficient generalization ability across different traffic scenarios limits their application scope and intelligent inspection efficiency.

[0005] Therefore, existing technologies suffer from insufficient perception, semantic understanding, and navigation planning capabilities, which makes it difficult for inspection robots to achieve efficient and reliable autonomous inspection in complex traffic environments. Summary of the Invention

[0006] The purpose of this application is to provide an automatic inspection robot traffic inspection method based on urban traffic scenarios, which is used to solve the technical problem that the existing technology has insufficient perception, semantic understanding and navigation planning capabilities, which makes it difficult for inspection robots to achieve efficient and reliable autonomous inspection in complex traffic environments.

[0007] In view of the above problems, this application provides a traffic inspection method for an automated inspection robot based on urban traffic scenarios. The method includes: constructing a hardware platform for the inspection robot in an urban traffic scenario, the hardware platform including a mobile chassis, a multimodal sensor system, a computing unit, and a communication module; acquiring raw multimodal sensor data from the hardware platform; preprocessing the raw multimodal sensor data to obtain standard multimodal sensor data; and performing feature transformation and BEV feature fusion on the standard multimodal sensor data to construct a BEV space. The system includes a representation learning module and a VLA model module. Based on the BEV spatial representation learning module and the VLA model module, a traffic event detection and anomaly recognition module is constructed, along with an end-to-end motion planning and navigation module and a human-computer interaction and remote management module. A model training and deployment strategy is defined, and the BEV spatial representation learning module, the VLA model module, the traffic event detection and anomaly recognition module, the end-to-end motion planning and navigation module, and the human-computer interaction and remote management module are trained and deployed using this strategy. The deployed modules are then used to control the automatic inspection robot for traffic inspection.

[0008] Optionally, the original multimodal sensor data is time-synchronized and spatially aligned to obtain usable multimodal sensor data; a multimodal data preprocessing step is constructed, which includes image preprocessing, point cloud preprocessing, and radar data preprocessing; the usable multimodal sensor data is preprocessed based on the multimodal data preprocessing step to obtain standard multimodal sensor data; the standard multimodal sensor data is represented as: D_t={I_t,L_t,R_t,T_t}, where I_t represents the image data at time t, L_t represents the point cloud data, R_t represents the radar data, and T_t represents the sensor timestamp.

[0009] Optionally, a convolutional neural network is used to extract visual features from the image data I_t in the standard multimodal sensor data to obtain a visual feature representation: F_vis = CNN_vis(I_t), where F_vis represents a visual feature map with a shape of (C_vis, H_vis, W_vis), and C_vis, H_vis, and W_vis represent the number of channels, height, and width, respectively. The visual feature representation is then transformed using the LSS method to obtain a BEV visual feature representation: F_vis_bev = LSS(F_vis). The lidar point cloud L_t in the standard multimodal sensor data is projected onto the XY plane using a cylindrical projection, and the Z-axis information is encoded as feature channels to obtain a point cloud BEV feature representation: F_lidar_bev = Pillar(L_t). The BEV visual feature representation and the point cloud BEV feature representation are then fused to construct a BEV spatial representation learning module.

[0010] Optionally, the BEV visual feature representation and the point cloud BEV feature representation are fused, represented as: F_bev=Fuse(F_vis_bev,F_lidar_bev), where Fuse represents the fusion operation, which can be achieved by concatenation, addition, or attention mechanisms. The fused BEV spatial representation learning module is obtained as: F_bev∈R^{C_bev×H_bev×W_bev}, where C_bev represents the number of BEV feature channels, H_bev and W_bev represent the height and width of the BEV grid, and each BEV grid corresponds to a fixed region in physical space.

[0011] Optionally, a VLA model module is constructed, the core components of which include a visual encoder, a language encoder, and an action decoder; wherein, the visual encoder adopts a Transformer-based visual encoder, and for the input image data I_t and BEV features F_bev, the visual encoder outputs visual feature representations: Z_vis=VisionEncoder(I_t) and Z_vis=VisionEncoder(F_bev), where Z_vis∈R^{N×D_vis}, N represents the number of tokens, and D_vis represents the feature dimension; the language encoder adopts a pre-trained language model, and for the input natural language instruction Q, the language encoder outputs a language feature representation: Z_lang=Langu `ageEncoder(Q)`, where `Z_lang` ∈ R^{M×D_lang}, M represents the number of tokens, and D_lang represents the feature dimension; the language feature representation and the language feature representation are fused through a cross-modal attention mechanism, represented as: `Z_fused = CrossAttention(Z_vis, Z_lang)`, where `CrossAttention` represents cross-modal attention computation, which can be bidirectional or unidirectional attention; the action decoder generates action instructions based on the fused feature `Z_fused`, and the action decoder generates action sequences using an autoregressive approach: `A = {a_1, a_2, ..., a_T}`, where `a_t` represents the action at time `t`, and `T` represents the length of the action sequence.

[0012] Optionally, the traffic incident detection module adopts a multi-task learning architecture, sharing the BEV feature extractor of the BEV spatial representation learning module, with each task having an independent detection head. For the detection task, an object detection framework is used, and the detection head outputs the category, bounding box, and confidence score for each object, represented as: D={(b_i,c_i,s_i)|i=1,...,N}, where b_i represents the bounding box coordinates, c_i represents the category, s_i represents the confidence score, and N represents the number of detected objects. For the semantic segmentation task, a segmentation network is used, outputting the category label for each BEV grid and segmenting... The result is expressed as: S∈{0,1,...,C}^{H_bev×W_bev}, where C represents the number of categories; the anomaly identification module, based on the VLA model module, uses anomaly detection or contrastive learning to identify anomalies. The output of the anomaly identification includes anomaly type, anomaly location, and anomaly degree, expressed as: E={(type_j,loc_j,score_j)|j=1,...,M}, where type_j represents the anomaly type, loc_j represents the anomaly location, score_j represents the anomaly confidence, and M represents the number of detected anomalies.

[0013] Optionally, an end-to-end motion planning and navigation module is constructed. This module adopts a layered architecture, including a global planning layer, a local planning layer, and a control layer. The global planning layer generates a global inspection path based on the inspection task and map information. The global inspection path is represented as: P_global={p_1,p_2,...,p_L}, where p_l represents the l-th path point, containing its position coordinates and desired velocity, and L represents the number of path points. The local planning layer generates local obstacle avoidance and path tracking instructions based on local perception information and the global inspection path. The local planning output is represented as: P_local={τ_1,τ_2,...,τ_H}, where τ_h represents the local action at the h-th step, and H represents the prediction time domain. The control layer converts the local planning output into control instructions to control the robot's speed and direction. The control instructions are represented as: u=(v,ω), where v represents linear velocity and ω represents angular velocity.

[0014] Optionally, a world model is introduced, denoted as: s_{t+1}=WorldModel(s_t,a_t), where s_t represents the current state, a_t represents the current action, and s_{t+1} represents the predicted next state. The world model is based on a sequence prediction model, with inputs including: current BEV features, robot state, and planned actions, and outputs including: future BEV feature predictions, other vehicle behavior predictions, and traffic event predictions. Based on the prediction output of the world model, the optimal action sequence is selected. A planning objective function is constructed, denoted as: J=∑_{t=1}^{T}[r(s_t,a_t)+γ·V(s_{t+1})], where r(s_t,a_t) represents the immediate reward, V(s_{t+1}) represents the value function, and γ represents the discount factor.

[0015] Optionally, the model training adopts a pre-training-fine-tuning paradigm, including a multimodal pre-training stage, a scene fine-tuning stage, and an online learning stage; the model deployment strategies include: model compression, mixed precision inference, operator optimization, and pipeline parallelism, and the model deployment architecture includes: robot-side, edge-side, and cloud-side.

[0016] Optionally, the training of the VLA model module adopts a pre-training-fine-tuning paradigm. During the pre-training phase, it is pre-trained on a large-scale vision-language-action dataset. The pre-training tasks include image-text alignment, action prediction, and instruction understanding. During the fine-tuning phase, it is fine-tuned on traffic inspection scene data. The fine-tuning data includes real traffic scene images, inspection instructions, and corresponding action sequences. The loss function of the VLA model module is expressed as: L_VLA=λ_align·L_align+λ_action·L_action+λ_lang·L_lang, where L_align represents the image-text alignment loss, L_action represents the action prediction loss, L_lang represents the language understanding loss, and λ is the weight coefficient.

[0017] One or more technical solutions provided in this application have at least the following technical effects or advantages:

[0018] The method provided in this application embodiment constructs a hardware platform for an inspection robot in an urban traffic scenario. The inspection robot hardware platform includes a mobile chassis, a multimodal sensor system, a computing unit, and a communication module. Raw multimodal sensor data is acquired from the inspection robot hardware platform. This raw multimodal sensor data is preprocessed to obtain standard multimodal sensor data. Feature transformation and BEV feature fusion are performed on the standard multimodal sensor data to construct a BEV spatial representation learning module and a VLA model module. Based on the BEV spatial representation learning module and the VLA model module, a traffic event detection and anomaly recognition module is constructed, along with an end-to-end motion planning and navigation module and a human-computer interaction and remote management module. A model training and deployment strategy is defined, and this strategy is used to train and deploy the BEV spatial representation learning module, the VLA model module, the traffic event detection and anomaly recognition module, the end-to-end motion planning and navigation module, and the human-computer interaction and remote management module. The deployed modules then control the automatic inspection robot for traffic inspection. It achieves the technical effect of using multimodal sensors to acquire comprehensive environmental information, using VLA model to realize deep semantic understanding and decision-making, using BEV representation to unify data from different sensors, and using end-to-end motion planning to realize adaptive navigation, thereby enabling the inspection robot to autonomously perceive, make intelligent decisions and carry out efficient inspections in complex traffic environments.

[0019] The above description is merely an overview of the technical solution of this application. To better understand the technical means of this application and to facilitate its implementation according to the description, and to make the above and other objects, features, and advantages of this application more apparent, specific embodiments of this application are described below. It should be understood that the content described in this section is not intended to identify key or important features of the embodiments of this application, nor is it intended to limit the scope of this application. Other features of this application will become readily apparent through the following description. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely exemplary. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0021] Figure 1 This application provides a flowchart illustrating an automatic inspection robot traffic inspection method based on urban traffic scenarios.

[0022] Figure 2 This is a schematic diagram illustrating the process of obtaining standard multimodal sensor data in an automatic inspection robot traffic inspection method based on urban traffic scenarios provided in this application. Detailed Implementation

[0023] This application provides a traffic inspection method for an automated inspection robot based on urban traffic scenarios. It addresses the technical problem that existing technologies suffer from insufficient perception, semantic understanding, and navigation planning capabilities, making it difficult for inspection robots to achieve efficient and reliable autonomous inspection in complex traffic environments. The method achieves the technical effect of utilizing multimodal sensors to acquire comprehensive environmental information, realizing deep semantic understanding and decision-making through a VLA model, using BEV representation to unify data from different sensors, and achieving adaptive navigation through end-to-end motion planning. This enables the inspection robot to achieve autonomous perception, intelligent decision-making, and efficient inspection in complex traffic environments.

[0024] The technical solutions of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. It should be understood that the present invention is not limited to the exemplary embodiments described herein. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention. It should also be noted that, for ease of description, only the parts related to the present invention are shown in the accompanying drawings, not all of them.

[0025] like Figure 1 , Figure 2 As shown, this application provides a traffic inspection method using an automated inspection robot based on urban traffic scenarios. The method includes:

[0026] A hardware platform for an inspection robot in urban traffic scenarios is constructed. The inspection robot hardware platform includes a mobile chassis, a multimodal sensor system, a computing unit, and a communication module.

[0027] Specifically, a hardware platform for an inspection robot in urban traffic scenarios is constructed based on a mobile chassis, a multimodal sensor system, a computing unit, and a communication module. The mobile chassis adopts a wheeled or tracked structure and has autonomous navigation and obstacle avoidance capabilities. The chassis is equipped with an odometer and an inertial measurement unit (IMU) for pose estimation. The chassis's motion control system receives control commands and executes speed and steering control.

[0028] The multimodal sensor system includes at least a visible light camera, a lidar, a millimeter-wave radar, an infrared thermal imaging camera, and an ultrasonic sensor. The visible light camera acquires color image information and includes multiple cameras with different perspectives, such as a front-facing camera and a surround-view camera. The visible light camera employs a high-resolution CMOS sensor, supports wide dynamic range (WDR), and can acquire clear images under complex lighting conditions. The lidar acquires accurate 3D point cloud information, including single-line or multi-line lidar, for distance measurement, obstacle detection, and environmental mapping. The millimeter-wave radar is used for target detection and speed measurement in adverse weather conditions, such as rain, fog, and snow, compensating for the limitations of optical sensors in such conditions. The infrared thermal imaging camera is used for target detection at night or in low-light conditions, capturing the thermal radiation information of objects. The ultrasonic sensor is used for near-range obstacle detection and avoidance, such as in low-speed inspection and parking scenarios. Furthermore, the installation positions and orientations of multiple sensors are precisely calibrated, and spatiotemporal synchronization between sensors is achieved through hardware triggering and timestamp alignment.

[0029] The computing unit employs a high-performance embedded computing platform, such as the NVIDIA Jetson series and GPU accelerator cards, for running deep learning algorithms and real-time data processing. The computing unit also possesses sufficient computing power to support tasks such as multimodal data processing, deep learning inference, and path planning. The communication module supports multiple communication methods, including 5G / 4G cellular networks, Wi-Fi, V2X communication, and satellite positioning. 5G / 4G cellular networks are used for remote data transmission and cloud computing; Wi-Fi is used for high-speed local data transmission; V2X communication is used for communication with vehicles and infrastructure; and satellite positioning, such as GPS or BeiDou, is used for global positioning.

[0030] By constructing a hardware platform for inspection robots in urban traffic scenarios, a physical foundation and operational support are provided for automated robot inspection: the mobile chassis is responsible for mobile inspection in complex road environments, ensuring that the robot can reach various inspection points; the multimodal sensor system is used to perceive the surrounding environment and achieve accurate detection of roads, traffic facilities, and potential anomalies; the computing unit undertakes core computing tasks such as sensor data processing, feature extraction, model reasoning, and decision planning; and the communication module ensures real-time interaction between the robot and remote management platforms or other collaborative systems, enabling remote monitoring, command issuance, and data uploading, thereby achieving autonomous, intelligent, and manageable urban traffic inspection.

[0031] Raw multimodal sensor data is obtained from the inspection robot hardware platform, and the raw multimodal sensor data is preprocessed to obtain standard multimodal sensor data.

[0032] Furthermore, the raw multimodal sensor data is preprocessed to obtain standard multimodal sensor data, including: performing time synchronization and spatial alignment on the raw multimodal sensor data to obtain usable multimodal sensor data; constructing a multimodal data preprocessing step, which includes image preprocessing, point cloud preprocessing, and radar data preprocessing; preprocessing the usable multimodal sensor data based on the multimodal data preprocessing step to obtain standard multimodal sensor data; and representing the standard multimodal sensor data as: D_t={I_t,L_t,R_t,T_t}, where I_t represents the image data at time t, L_t represents the point cloud data, R_t represents the radar data, and T_t represents the sensor timestamp.

[0033] Specifically, the inspection robot hardware platform utilizes a multimodal sensor system to acquire raw multimodal sensor data, including image data, point cloud data, and radar data. Since different sensors have different sampling frequencies and triggering methods, time alignment is achieved through hardware triggering or software interpolation. A unified reference clock is defined, and the timestamps of each sensor's data are aligned to this reference clock. Furthermore, for sensors with different sampling rates, interpolation or resampling is used to unify them to the same time base, thus synchronizing the raw multimodal sensor data.

[0034] Simultaneously, extrinsic parameters of each sensor, including rotation and translation matrices, are obtained through sensor calibration, transforming data from different sensors to the robot coordinate system or the world coordinate system. Specifically, the robot coordinate system is defined as a right-handed coordinate system with the robot center as the origin, the forward direction as the X-axis, the left direction as the Y-axis, and the upward direction as the Z-axis. For visible light camera images, its intrinsic parameter matrix K_c and extrinsic parameter matrix [R_c|t_c] are defined, and the mapping relationship between image coordinates and robot coordinates is established through a perspective projection model. For LiDAR point clouds, its extrinsic parameter matrix [R_l|t_l] is defined, transforming the point cloud coordinates to the robot coordinate system. For millimeter-wave radar detection results, its extrinsic parameter matrix [R_r|t_r] is defined, transforming the detection results to the robot coordinate system, achieving spatial alignment of the original multimodal sensor data. Through time synchronization and spatial alignment, usable multimodal sensor data is obtained.

[0035] Then, a multimodal data preprocessing step is constructed, including image preprocessing, point cloud preprocessing, and radar data preprocessing. Among them, image preprocessing includes distortion correction, noise reduction, white balance, and exposure correction, point cloud preprocessing includes noise reduction, filtering, ground point segmentation, and clustering, and radar data preprocessing includes clutter suppression, target clustering, and track association.

[0036] The available multimodal sensor data is preprocessed according to the multimodal data preprocessing steps. First, distortion correction is performed on the original images from the available multimodal sensor data. Barrel or pincushion distortion caused by fisheye or wide-angle lenses is corrected using the camera intrinsic parameter matrix and distortion coefficients, ensuring that straight lines in the image remain straight in physical space. Next, noise reduction is performed, using median filtering to remove random noise while preserving edge and texture information. After noise reduction, white balance correction is performed on the image, using a grayscale world algorithm to adjust the color cast to a natural state. Finally, exposure correction is performed on the image, using histogram equalization to achieve uniform image brightness and preserve details in both dark and bright areas, thus outputting a standard image I_t for visual feature extraction.

[0037] For the raw point cloud acquired by LiDAR from available multimodal sensor data, denoising is first performed using statistical filtering or radius filtering methods to remove isolated points and reflection anomalies. Then, filtering and downsampling are performed, and the point cloud density is reduced using a voxel grid method to decrease computational load while preserving spatial structure. Next, ground point segmentation is performed on the point cloud data using ground model fitting, such as RANSAC plane fitting or height thresholding, to separate ground points from non-ground points and extract targets such as vehicles and pedestrians. Finally, the processed point cloud data is clustered using Euclidean clustering to group spatially dense points into the same target, obtaining point cloud data L_t, which provides independent point cloud clusters for subsequent target detection and recognition.

[0038] Clutter suppression is performed on the radar data acquired by the radar. A constant false alarm rate (CFAR) algorithm is used to remove static background and low-intensity noise, retaining only possible moving target points. Furthermore, the DBSCAN clustering method is employed to cluster targets in the clutter-suppressed radar data, classifying adjacent radar echo points with similar motion states into individual targets. Finally, the multiple hypothesis tracking (MHT) method is used to correlate the observation points of each target at different time steps, forming a complete radar data set R_t.

[0039] The preprocessed multimodal data is represented as: D_t={I_t,L_t,R_t,T_t}, where I_t represents the image data at time t, L_t represents the point cloud data, R_t represents the radar data, and T_t represents the sensor timestamp.

[0040] By synchronizing and aligning the raw multimodal sensor data in time and space, and by denoising, correcting and structuring the image, point cloud and radar data respectively, a unified and standardized multimodal data representation is formed, ensuring the spatiotemporal consistency and reliability of the collected data, and providing high-quality input for feature extraction, environmental perception and decision planning.

[0041] The standard multimodal sensor data is subjected to feature transformation and BEV feature fusion to construct a BEV spatial representation learning module, and a VLA model module is constructed simultaneously.

[0042] Furthermore, feature transformation and BEV feature fusion are performed on the standard multimodal sensor data to construct a BEV spatial representation learning module. This includes: using a convolutional neural network to extract visual features from the image data I_t in the standard multimodal sensor data to obtain a visual feature representation: F_vis = CNN_vis(I_t), where F_vis represents a visual feature map with a shape of (C_vis, H_vis, W_vis), where C_vis, H_vis, and W_vis represent the number of channels, height, and width, respectively; performing a viewpoint transformation on the visual feature representation based on the LSS method to obtain a BEV visual feature representation: F_vis_bev = LSS(F_vis); projecting the lidar point cloud L_t in the standard multimodal sensor data onto the XY plane using a cylindrical projection, encoding the Z-axis information as feature channels to obtain a point cloud BEV feature representation: F_lidar_bev = Pillar(L_t); and fusing the BEV visual feature representation and the point cloud BEV feature representation to construct a BEV spatial representation learning module.

[0043] Specifically, the BEV spatial representation learning module unifies multimodal data into a bird's-eye view representation space, providing globally consistent spatiotemporal scene understanding. First, visual feature extraction is performed using a Convolutional Neural Network (CNN) to extract visual features from the image data I_t in the standard multimodal sensor data. The CNN consists of three convolutional layers, pooling layers, and normalization layers stacked alternately. Each convolutional layer uses a kernel of different sizes to scan the local receptive field of the input feature map, extracting edges, textures, and higher-level semantic information. The kernel size is 3×3 or 5×5, with a stride of 1. After the convolution operation, a non-linear activation function, such as ReLU, is used to increase the model's non-linear expressive power, enabling the CNN to learn complex visual patterns. The lower-level convolutional layers mainly extract local structural features such as edges and corners; the middle-level convolutional layers extract textures, shapes, and local combination patterns; and the higher-level convolutional layers extract semantic information, such as high-level semantic features of traffic signs, vehicles, or road areas. The pooling layers reduce spatial resolution through max pooling while preserving key features, improving the model's translation invariance. Normalization layers, such as BatchNorm, are used to accelerate training convergence and mitigate internal covariance bias.

[0044] CNNs are trained using supervised learning, optimizing network parameters through backpropagation on labeled image datasets. Cross-entropy loss is chosen as the loss function to measure the difference between the predicted class distribution and the true labels. In segmentation or class imbalance tasks, Dice Loss can be combined to improve the model's sensitivity to minority classes. The Adam optimizer is used during training, updating parameters through momentum terms and adaptive learning rates to accelerate convergence and enhance training stability. The initial learning rate can be set to 1×10⁻⁶. -3 Furthermore, a learning rate scheduling strategy, such as cosine annealing, is employed to gradually decrease the learning rate with each iteration, thereby improving convergence accuracy while avoiding gradient oscillations. Training convergence is determined by the performance on the validation set. When the cross-entropy loss on the validation set changes by less than a preset threshold of 0.01 for several consecutive rounds (e.g., 5 rounds), the CNN network is considered to have converged. An early stopping strategy can also be used to prevent overfitting.

[0045] Image data I_t from standard multimodal sensor data is input into a trained CNN for visual feature extraction, resulting in a visual feature representation: F_vis = CNN_vis(I_t), where F_vis represents the visual feature map with shape (C_vis, H_vis, W_vis), where C_vis, H_vis, and W_vis represent the number of channels, height, and width, respectively.

[0046] Then, the LSS method is used to perform a viewpoint transformation on the visual feature representation, upscaling the features of each pixel to 3D space according to its depth distribution to form 3D features. These 3D features are then flattened onto the BEV mesh. Finally, by summing or averaging the features of the same BEV mesh, the BEV visual feature representation is obtained as: F_vis_bev = LSS(F_vis), realizing the transformation from visual features to BEV. For the LiDAR point cloud L_t in standard multimodal sensor data, a BEV feature map is generated through voxelization or cylindrical projection. The point cloud is projected onto the XY plane, and the Z-axis information is encoded into feature channels, obtaining the point cloud BEV feature representation as: F_lidar_bev = Pillar(L_t), realizing the transformation from point cloud features to BEV. Furthermore, the BEV visual feature representation and the point cloud BEV feature representation are fused to construct a BEV spatial representation learning module.

[0047] By constructing a BEV spatial representation learning module, the data representation of different sensors can be unified, allowing data from different modalities such as vision and LiDAR to be represented as BEV feature maps. Simultaneously, by eliminating scale variations caused by perspective projection, the scale of objects represented by BEV is independent of distance, facilitating target detection and tracking. Furthermore, BEV representation directly corresponds to the real physical space, facilitating spatial reasoning and path planning, and improving the effectiveness and accuracy of traffic inspection by automated inspection robots.

[0048] Furthermore, a BEV spatial representation learning module is constructed by fusing the BEV visual feature representation and the point cloud BEV feature representation. This includes fusing the BEV visual feature representation and the point cloud BEV feature representation as: F_bev=Fuse(F_vis_bev,F_lidar_bev), where Fuse represents the fusion operation, which can be achieved through concatenation, addition, or attention mechanisms. The fused BEV spatial representation learning module is then obtained as: F_bev∈R^{C_bev×H_bev×W_bev}, where C_bev represents the number of BEV feature channels, H_bev and W_bev represent the height and width of the BEV grid, and each BEV grid corresponds to a fixed region in physical space.

[0049] Specifically, visual BEV features and point cloud BEV features are fused along the channel dimension, represented as: F_bev = Fuse(F_vis_bev, F_lidar_bev), where Fuse represents the fusion operation, which can be achieved through methods such as concatenation, addition, or attention mechanisms. Through fusion, the fused BEV spatial representation learning module is obtained, represented as: F_bev ∈ R^{C_bev × H_bev × W_bev}, where C_bev represents the number of BEV feature channels, H_bev and W_bev represent the height and width of the BEV grid, and each BEV grid corresponds to a fixed region in physical space.

[0050] Taking the concatenation method as an example, the two types of features are first concatenated along the channel dimension to form a fused feature: F_bev=Concat(F_vis_bev,F_lidar_bev). If F_vis_bev∈R^{C_vis×H_bev×W_bev} and F_lidar_bev∈R^{C_lidar×H_bev×W_bev}, then the concatenated F_bev∈R^{(C_vis+C_lidar)×H_bev×W_bev} retains the complete features of both visual and point cloud information. Here, C_vis represents the number of channels of the visual features, such as the visual feature dimension extracted by CNN, C_lidar represents the number of channels of the point cloud BEV features, that is, the number of channels after the point cloud features are encoded, and H_bev and W_bev represent the height and width of the BEV grid. Then, the spliced ​​channels are reduced in dimension or fused through 1×1 convolution or linear transformation to obtain a unified number of channels C_bev, forming a fused BEV spatial representation learning module, represented as: F_bev∈R^{C_bev×H_bev×W_bev}, which comprehensively reflects the visual and geometric information of the region, and each BEV grid corresponds to a fixed region in physical space, such as 0.5m×0.5m.

[0051] By integrating visual image information and point cloud information, a spatially consistent, dense, and structured BEV feature representation is generated, providing high-quality environmental perception input for traffic event detection, anomaly recognition, and end-to-end motion planning, ensuring that the inspection robot can accurately understand urban traffic scenarios and make intelligent decisions.

[0052] Furthermore, a VLA model module is constructed, including: The core components of the VLA model module include: a visual encoder, a language encoder, and an action decoder; wherein, the visual encoder adopts a Transformer-based visual encoder, and for the input image data I_t and BEV features F_bev, the visual encoder outputs visual feature representations: Z_vis=VisionEncoder(I_t) and Z_vis=VisionEncoder(F_bev), where Z_vis∈R^{N×D_vis}, N represents the number of tokens, and D_vis represents the feature dimension; the language encoder adopts a pre-trained language model, and for the input natural language instruction Q, the language encoder outputs a language feature representation: Z_lan g = LanguageEncoder(Q), where Z_lang ∈ R^{M×D_lang}, M represents the number of tokens, and D_lang represents the feature dimension; the language feature representation and the language feature representation are fused through a cross-modal attention mechanism, represented as: Z_fused = CrossAttention(Z_vis, Z_lang), where CrossAttention represents cross-modal attention calculation, which can be bidirectional or unidirectional attention; the action decoder generates action instructions based on the fused feature Z_fused, and the action decoder generates action sequences in an autoregressive manner: A = {a_1, a_2, ..., a_T}, where a_t represents the action at time t, and T represents the length of the action sequence.

[0053] Specifically, the VLA (Vision-Language-Action) model module includes three core components: a visual encoder for encoding visual information into high-dimensional feature representations, a language encoder for encoding natural language instructions into feature representations, and an action decoder for generating specific action instructions based on visual and linguistic features.

[0054] The visual encoder employs a Transformer-based visual encoder, such as ViT or SwingTransformer. For the input image I_t or BEV feature F_bev, it is first divided into fixed-size patches, for example, each patch is 16×16 pixels, or a 16×16 grid for EV features. The partitioning method involves dividing the entire image or BEV feature into non-overlapping small blocks in both the height and width directions. Each patch is flattened into a one-dimensional vector and mapped to a fixed dimension through a linear projection, i.e., through a learnable weight matrix, forming a token sequence. Each token corresponds to the feature representation of a patch, ensuring that images or BEV features of different sizes can be unified into a sequence format that the Transformer can process. After obtaining the token sequence, it is input into a multi-layer Transformer encoder. The input token features are mapped to a query matrix Q, a key matrix K, and a value matrix V, achieved through learnable weights. Then, the attention weights of each token with all tokens are calculated. A multi-head implementation divides the attention mechanism into multiple heads, such as five heads. Each head learns the correlations of different subspaces, and the results are concatenated and then output through a linear transformation, enhancing the model's ability to capture different feature patterns. Each token undergoes two independent linear transformations, and an activation function, such as GELU, is used to perform a non-linear mapping on each token's features, improving expressive power. Residual connections are used to add the module inputs and outputs, mitigating the vanishing or exploding gradient problems in deep networks, ensuring information can be transferred across layers, and stabilizing training. After the residual connection, the token features are normalized to output visual feature representations: Z_vis=VisionEncoder(I_t) and Z_vis=VisionEncoder(F_bev), where Z_vis∈R^{N×D_vis}, N represents the number of tokens, and D_vis represents the feature dimension, that is, each token corresponds to a local region of the image or BEV, while encoding global spatial dependencies, providing rich visual context information for cross-modal fusion and action decision-making.

[0055] The language encoder uses a pre-trained language model, such as BERT or GPT. For the input natural language instruction Q, such as checking the traffic flow at the intersection ahead, it is first segmented and mapped into a sequence of tokens. Each token is converted into a fixed-dimensional embedding vector and positional encoding is added to preserve the order information. The language encoder outputs the language feature representation: Z_lang=LanguageEncoder(Q), where Z_lang∈R^{M×D_lang}, M represents the number of tokens, and D_lang represents the feature dimension.

[0056] The linguistic feature representations are fused using a cross-modal attention mechanism, with linguistic features as query Q and visual features as key K and value V. Alternatively, a bidirectional attention mechanism can be used, allowing visual features and linguistic features to act as query and key-value pairs for each other, thereby establishing a correspondence between semantics and vision. The fused feature representation is: Z_fused = CrossAttention(Z_vis, Z_lang), where CrossAttention represents cross-modal attention computation, which can be bidirectional or unidirectional attention.

[0057] The action decoder generates action instructions based on the fused feature Z_fused. The action decoder employs an autoregressive approach, meaning that the generation of each action depends on the output of the previous step, generating an action sequence: A = {a_1, a_2, ..., a_T}, where a_t represents the action at time t, and T represents the length of the action sequence. The action decoder can be a Transformer decoder. Each decoder layer includes a self-attention module, a fused feature attention module, and a feedforward fully connected network. The input to the self-attention module is the currently generated action sequence or the starting token, such as "start inspection." First, the self-attention mechanism models the dependencies between steps in the sequence. The self-attention module uses a multi-head design, dividing the query, key, and value matrix into multiple heads. Each head learns different action patterns, and the outputs are concatenated and a linear transformation is used to obtain the final feature representation. The fusion feature attention module is used to interact with the fused visual-linguistic features Z_fused and the internal state of the decoder, ensuring that the generated action at each step references historical actions and combines current environmental information and instruction semantics. Here, the query Q comes from the output of the previous layer of the decoder, and the key K and value V come from Z_fused. Through attention calculation, hidden states aligned with environmental semantics are generated, realizing multimodal information to guide action generation and ensuring that robot actions are consistent with the actual scene and instructions.

[0058] The feedforward fully connected network performs two independent linear transformations on each token and incorporates a non-linear activation function, such as GELU, to enhance the model's expressive power. Simultaneously, residual connections and layer normalization ensure gradient stability and avoid the training difficulties of deep networks. During generation, the decoder predicts the next action based on the current fused features and uses this prediction as the input for the next step, until a complete sequence is generated. Actions can be discrete commands, such as forward, left turn, deceleration, or continuous control commands, linear velocity, and angular velocity. The final numerical commands are implemented through a softmax or regression output layer, ensuring that the inspection robot can perform end-to-end motion planning and task operations in urban traffic environments.

[0059] By constructing a VLA model module, the integration of visual perception, language understanding, and action decision-making is achieved, enabling the inspection robot to understand natural language commands and generate corresponding actions based on visual perception information.

[0060] Furthermore, the method also includes: the training of the VLA model module adopts a pre-training-fine-tuning paradigm. During the pre-training phase, pre-training is performed on a large-scale vision-language-action dataset. The pre-training tasks include image-text alignment, action prediction, and instruction understanding. During the fine-tuning phase, fine-tuning is performed on traffic inspection scene data. The fine-tuning data includes real traffic scene images, inspection instructions, and corresponding action sequences. The loss function of the VLA model module is expressed as: L_VLA=λ_align·L_align+λ_action·L_action+λ_lang·L_lang, where L_align represents the image-text alignment loss, L_action represents the action prediction loss, L_lang represents the language understanding loss, and λ is the weight coefficient.

[0061] Specifically, the VLA model module is trained using a pre-training-fine-tuning paradigm. In the pre-training phase, the VLA model is trained end-to-end on a large-scale vision-language-action dataset, which includes thousands of images, text commands, and corresponding action sequences. Image inputs are processed by a visual encoder to extract visual features, while text command inputs are processed by a pre-trained language model to extract linguistic features. These features are then fused and input into the action decoder to generate predicted action sequences.

[0062] During training, the image-text alignment task uses cosine similarity or InfoNCE loss to optimize the correspondence between vision and language. The action prediction task calculates cross-entropy or mean squared error loss between the generated action sequence and the real action sequence. The instruction understanding task supervises the model's encoding of text semantics, enabling the model to learn general visual perception, language understanding and action generation capabilities on large-scale data.

[0063] During the fine-tuning phase, the VLA model is specifically optimized using traffic inspection scenario data to adapt to the specific needs of real-world urban traffic environments. The fine-tuning data includes real traffic scene images, such as images of urban roads, intersections, vehicles, and pedestrians, as well as inspection instructions and corresponding action sequences. The visual encoder and language encoder are fine-tuned by reducing the learning rate, while the cross-modal attention and action decoder are fully trained. This optimizes image-text alignment, action prediction, and instruction understanding losses, enabling the model to accurately understand inspection task instructions, capture complex traffic environment features, and generate safe and reasonable inspection action sequences. This ensures the reliability and efficiency of the inspection robot in end-to-end task execution in real-world scenarios.

[0064] The VLA model employs a multi-level marketing strategy. First, it measures the consistency between image and text descriptions by calculating the cosine similarity of visual and linguistic features. InfoNCE loss is used for optimization to obtain the image-text alignment loss L_align. Then, the action sequences generated by the decoder are compared with real inspection actions. Cross-entropy loss is used for discrete actions, and mean squared error is used for continuous control commands to obtain the action prediction loss L_action. Finally, the semantic representation of input commands and the model's internal understanding are supervised using cross-entropy or sequence prediction loss to ensure the model correctly understands and parses the task objectives in the commands, resulting in the action prediction loss L_lang. Multiple loss functions are weighted and summed, resulting in the loss function of the VLA model module: L_VLA = λ_align·L_align + λ_action·L_action + λ_lang·L_lang, where L_align represents the image-text alignment loss, L_action represents the action prediction loss, L_lang represents the language understanding loss, and λ is a weighting coefficient used to balance the importance of each task. During training, gradient descent is used to optimize this loss function, and the visual encoder, language encoder, cross-modal attention module, and action decoder are jointly updated to achieve end-to-end multimodal learning.

[0065] By employing a pre-training-fine-tuning paradigm, the VLA model learns general image understanding, language understanding, and action prediction capabilities on large-scale visual-language-action data. Fine-tuning allows the model to adapt to specific traffic inspection scenarios, enabling accurate understanding of instructions, multimodal perception of the environment, and accurate generation of inspection actions. This ensures that the inspection robot can intelligently execute end-to-end inspection tasks in complex urban traffic environments.

[0066] Based on the BEV spatial representation learning module and the VLA model module, a traffic event detection and anomaly recognition module is constructed, along with an end-to-end motion planning and navigation module and a human-computer interaction and remote management module.

[0067] Furthermore, based on the BEV spatial representation learning module and the VLA model module, a traffic event detection and anomaly recognition module is constructed, including: the traffic event detection module adopts a multi-task learning architecture, sharing the BEV feature extractor of the BEV spatial representation learning module, with each task having an independent detection head; for the detection task, an object detection framework is adopted, and the detection head outputs the category, bounding box, and confidence score of each object, represented as: D={(b_i,c_i,s_i)|i=1,...,N}, where b_i represents the bounding box coordinates, c_i represents the category, s_i represents the confidence score, and N represents the number of detected objects; for the semantic segmentation task, a... The segmentation network outputs the category label for each BEV grid. The segmentation result is represented as: S∈{0,1,...,C}^{H_bev×W_bev}, where C represents the number of categories. The anomaly detection module, based on the VLA model module, uses anomaly detection or contrastive learning to identify anomalies. The output of the anomaly detection includes the anomaly type, anomaly location, and anomaly degree, represented as: E={(type_j,loc_j,score_j)|j=1,...,M}, where type_j represents the anomaly type, loc_j represents the anomaly location, score_j represents the anomaly confidence, and M represents the number of detected anomalies.

[0068] Specifically, traffic incident detection includes the following tasks: Vehicle detection: detecting various types of vehicles, including cars, trucks, buses, motorcycles, etc.; Pedestrian detection: detecting pedestrians on the road, including pedestrians crossing the street and pedestrians on the roadside; Traffic sign recognition: recognizing various traffic signs, such as speed limit signs and no-entry signs; Traffic light recognition: recognizing the status of traffic lights, such as red, yellow, and green; Violation detection: detecting various traffic violations, such as running red lights, driving against traffic, and illegal parking; Accident detection: detecting traffic accidents, such as vehicle collisions and rear-end collisions; Congestion detection: detecting traffic congestion and assessing the degree of congestion; Abnormal behavior detection: detecting abnormal behavior, such as pedestrians suddenly crossing the road and vehicles changing lanes abnormally.

[0069] Based on the BEV spatial representation learning module and the VLA model module, a traffic incident detection module and an anomaly recognition module are constructed. The traffic incident detection module adopts a multi-task learning architecture, simultaneously performing object detection and semantic segmentation tasks by sharing the BEV feature extractor from the BEV spatial representation learning module. For the object detection task, object detection frameworks such as Faster R-CNN and YOLO are used. Loss is calculated and the network is optimized by regressing bounding boxes and using classification branches. Each task has an independent detection head, and each detection head outputs the category, bounding box, and confidence score for each object, represented as: D={(b_i,c_i,s_i)|i=1,...,N}, where b_i represents the bounding box coordinates, c_i represents the category, s_i represents the confidence score, and N represents the number of detected objects.

[0070] For semantic segmentation tasks, such as road region segmentation and lane line segmentation, segmentation networks, such as DeepLab and SegFormer, are used to output the category label for each BEV grid. The segmentation result is represented as: S∈{0,1,...,C}^{H_bev×W_bev}, where C represents the number of categories. The segmentation network predicts the category label for each BEV grid and is trained using cross-entropy loss, enabling the model to distinguish environmental elements such as roads, lane lines, and obstacles.

[0071] The anomaly detection module leverages the semantic understanding capabilities of the VLA model module, employing anomaly detection or contrastive learning methods to identify anomalies by comparing features of normal and abnormal scenes. For example, a reconstruction-based method learns feature representations of normal scenes, calculates the differences between the current scene and normal scenes, and identifies anomalies. A contrastive learning-based method uses normal and abnormal samples as positive and negative pairs, respectively, to learn features that distinguish between normal and abnormal scenes, thus identifying anomalies. A language-guided method describes anomaly patterns using natural language, and leverages the language understanding capabilities of the VLA model for matching and anomaly identification. The output of the anomaly detection includes anomaly type, anomaly location, and anomaly severity, represented as: E={(type_j,loc_j,score_j)|j=1,...,M}, where type_j represents the anomaly type, loc_j represents the anomaly location, score_j represents the anomaly confidence level, and M represents the number of detected anomalies.

[0072] By combining BEV features with the multimodal understanding capabilities of the VLA model, the system enables accurate detection of various types of targets in urban traffic scenarios, complete understanding of environmental semantics, and timely identification of abnormal events. This provides inspection robots with comprehensive and real-time environmental perception and risk assessment capabilities, ensuring the safe and efficient execution of inspection tasks in complex traffic environments. It also supports remote management and human-computer interaction.

[0073] Furthermore, an end-to-end motion planning and navigation module is constructed, including: constructing an end-to-end motion planning and navigation module, which adopts a layered architecture, including a global planning layer, a local planning layer, and a control layer; wherein, the global planning layer generates a global inspection path based on the inspection task and map information, the global inspection path is represented as: P_global={p_1,p_2,...,p_L}, where p_l represents the l-th path point, including position coordinates and desired velocity, and L represents the number of path points; the local planning layer generates local obstacle avoidance and path tracking instructions based on local perception information and the global inspection path, the local planning output is represented as: P_local={τ_1,τ_2,...,τ_H}, where τ_h represents the local action at the h-th step, and H represents the prediction time domain; the control layer converts the local planning output into control instructions to control the robot's speed and direction, the control instructions are represented as: u=(v,ω), where v represents linear velocity and ω represents angular velocity.

[0074] Specifically, an end-to-end motion planning and navigation module is constructed to generate the robot's motion trajectory and control commands based on current perception information, inspection tasks, and VLA model instructions. This module adopts a layered architecture, including a global planning layer, a local planning layer, and a control layer. The global planning layer generates a global inspection path based on the inspection task requirements and city map information. This global path is represented as: P_global={p_1,p_2,...,p_L}, where p_l represents the l-th path point, containing its position coordinates and desired velocity, and L represents the number of path points. Path planning employs graph search algorithms, such as A* and Dijkstra's algorithm, to search for the shortest path among discrete map nodes, or sampling planning algorithms, such as RRT and PRM, to generate feasible trajectories in continuous space. Each path point p_L contains its position coordinates and desired velocity information. The global planning layer is used to decompose the task and generate the global trajectory, ensuring complete overall task coverage for the inspection robot and providing reference paths for local planning.

[0075] The local planning layer combines the global inspection path and local perception information, such as BEV feature maps, traffic incident detection results, and anomaly identification information, and uses the model predictive control (MPC) method to generate local action sequences within the short-term prediction range. That is, the local planning output is represented as: P_local={τ_1,τ_2,...,τ_H}, where τ_h represents the local action at step h, H represents the prediction time domain, and each local action τ_h corresponds to the robot's position, velocity, and obstacle avoidance adjustments within the prediction time domain.

[0076] The control layer converts the local programming generated by the local programming layer into control commands that the robot can actually execute. These control commands are represented as u = (v, ω), where v represents linear velocity and ω represents angular velocity. The robot's speed and steering are controlled by these commands. A PID controller can be used for closed-loop regulation of speed and steering to achieve precise trajectory tracking and stable motion, while ensuring smooth robot movement in complex urban traffic environments. Furthermore, the control layer can also restrict the robot's actions based on actual dynamic constraints to ensure safety and feasibility.

[0077] Meanwhile, the human-machine interaction and remote management module displays inspection status, path, traffic events, and anomaly information through a remote interface, allowing operators to send instructions, adjust inspection strategies, or intervene in robot actions to achieve human-machine collaboration. The human-machine interaction module includes the following functions: receiving natural language instructions from management personnel, such as reporting current traffic conditions or detecting illegal parking; generating inspection status reports, including current location, inspection progress, detected traffic events, and anomalies; generating status reports based on the current status and detection results, such as moderate traffic flow and no traffic accidents detected; and a visualization interface for providing visual displays of the inspection process, including: display of the robot's position and trajectory on a map, real-time video stream and BEV feature map display, detection result annotations, anomaly highlighting, and inspection task progress display.

[0078] Furthermore, the remote management module supports remote task scheduling, remote monitoring, remote intervention, and multi-robot collaboration. Remote task scheduling allows managers to issue inspection tasks through a remote interface, specifying inspection areas, inspection content, and inspection priorities. Remote monitoring enables real-time viewing of the robot's inspection status and sensor information. Remote intervention allows managers to remotely take over control of the robot in emergency situations. Multi-robot collaboration supports the collaborative work of multiple inspection robots, enabling task allocation and collaborative planning.

[0079] The human-computer interaction and remote management module synchronizes the robot's status in real time through a communication link, ensuring information visualization, task controllability, and remote monitoring capabilities.

[0080] By constructing an end-to-end motion planning and navigation module, an end-to-end closed loop from task planning to robot control is achieved. This enables the inspection robot to navigate safely and efficiently autonomously based on global task planning, while simultaneously performing real-time obstacle avoidance using local perception, ensuring inspection path coverage and task execution accuracy. The human-machine interaction and remote management module provides real-time monitoring, command interaction, and task intervention capabilities, improving the safety, controllability, and reliability of inspection operations, enabling the robot to complete autonomous inspection tasks in complex urban traffic environments.

[0081] Furthermore, the method also includes: introducing a world model, denoted as: s_{t+1}=WorldModel(s_t,a_t), where s_t represents the current state, a_t represents the current action, and s_{t+1} represents the predicted next state; the world model is based on a sequence prediction model, with inputs including: current BEV features, robot state, and planned action, and outputs including: future BEV feature prediction, other vehicle behavior prediction, and traffic event prediction, and selecting the optimal action sequence based on the prediction output of the world model; constructing a planning objective function, denoted as: J=∑_{t=1}^{T}[r(s_t,a_t)+γ·V(s_{t+1})], where r(s_t,a_t) represents the immediate reward, V(s_{t+1}) represents the value function, and γ represents the discount factor.

[0082] Specifically, in order to achieve adaptive motion planning, a world model is introduced to predict future states. The world model is an environmental prediction model that can predict future environmental states based on the current state and actions. The world model is represented as: s_{t+1}=WorldModel(s_t,a_t), where s_t represents the current state, a_t represents the current action, and s_{t+1} represents the predicted next state.

[0083] The world model is constructed using a temporal prediction model, such as LSTM, GRU, or Transformer architecture. Its inputs include current BEV features, robot state, and planned actions. Outputs include future BEV feature predictions, other vehicle behavior predictions, and traffic event predictions. For LSTM or GRU, the input at each time step is the current state vector and action vector. Internally, a gating mechanism captures long-short-term dependencies, outputting a predicted vector for the next state. For Transformer, the input is a sequence of state and action tokens. A multi-head self-attention mechanism calculates the state-action relationship, and a multi-layer encoder stack is used to obtain the future state sequence. Multiple stacks, such as 4-6 layers, ensure the model can capture complex environmental dynamics and multi-step dependencies. The output of the sequence prediction module generates state predictions for the next time step or multiple future steps through a decoder, including future BEV feature maps, the positions of other traffic participants, and traffic event states. The decoder can use fully connected layers to map high-dimensional feature vectors back to the BEV grid and output a predicted probability distribution or continuous values ​​through softmax.

[0084] Based on the predictions of the world model and combined with the planning objective function, the motion planning and navigation module can evaluate the consequences of different actions and select the optimal action sequence. The planning objective function is expressed as: J = ∑_{t=1}^{T}[r(s_t,a_t) + γ·V(s_{t+1})], where r(s_t,a_t) represents the immediate reward, which includes multiple components: positive reward for completing the inspection task, positive reward for moving along the global path, positive reward for avoiding obstacles and danger zones, positive reward for completing the task quickly, positive reward for smooth movement, and negative rewards for collisions, traffic violations, etc. V(s_{t+1}) represents the value function of the predicted state, obtained through Monte Carlo estimation, used to evaluate the contribution of future states to the overall inspection task, and γ represents the discount factor used to balance immediate and future rewards. In other words, after the predicted state is output, the world model interface combines with the planning objective function to calculate the immediate reward and value function for each future state, used to evaluate the merits of the action sequence.

[0085] By optimizing this objective function, the robot can select the optimal sequence of actions, taking into account future environmental evolution and dynamic risks, to achieve safe and efficient end-to-end motion planning and task execution.

[0086] By leveraging the environmental prediction capabilities of the world model, robot decision-making is closely integrated with future traffic conditions, enabling prediction-driven planning strategies. This allows inspection robots to proactively avoid potential risks in dynamic and complex urban traffic environments, while maximizing task completion efficiency and enhancing the safety and reliability of autonomous decision-making.

[0087] Define a model training and deployment strategy, and use the model training and deployment strategy to train and deploy the BEV spatial representation learning module, the VLA model module, the traffic event detection and anomaly recognition module, the end-to-end motion planning and navigation module, and the human-computer interaction and remote management module. Then, control the automatic inspection robot to conduct traffic inspection through the deployed modules.

[0088] Furthermore, the model training and deployment strategies are defined, including: model training adopts a pre-training-fine-tuning paradigm, including a multimodal pre-training stage, a scenario fine-tuning stage, and an online learning stage; the model deployment strategies include: model compression, mixed precision inference, operator optimization, and pipeline parallelism, and the model deployment architecture includes: robot-side, edge-side, and cloud-side.

[0089] Specifically, the model training adopts a pre-training-fine-tuning paradigm, including a multimodal pre-training stage, a scene fine-tuning stage, and an online learning stage. The multimodal pre-training stage involves pre-training on large-scale multimodal datasets to learn a general spatiotemporal representation. Pre-training data includes: large-scale image-text data such as LAION and Conceptual Captions; driving scene datasets such as NuScenes and Waymo; and general robot datasets. Pre-training tasks include: mask modeling: randomly masking the input image or point cloud and learning to reconstruct the masked content; contrastive learning: using different modalities of the same scene as positive sample pairs and data from different scenes as negative sample pairs; cross-modal alignment: aligning visual and linguistic features; and action prediction: predicting the correct action sequence given a state and a target.

[0090] Scenario fine-tuning stage: Fine-tuning is performed on specific traffic inspection scenario data. The fine-tuning data includes: real urban traffic scenario data, simulated environment data, and labeled inspection task data. Fine-tuning uses a small learning rate and fewer training rounds to avoid catastrophic forgetting.

[0091] Online learning phase: During the actual operation of the robot, it continuously learns and optimizes. Through online learning, the robot can constantly adapt to new scenarios and environmental changes. Online learning adopts incremental learning or meta-learning methods to ensure that new knowledge is learned without destroying existing knowledge.

[0092] To meet the requirements of limited computing resources and real-time performance, the model deployment strategy includes: model compression, mixed-precision inference, operator optimization, and pipelined parallelism. Specifically, the model is compressed through methods such as knowledge distillation, quantization, and pruning to enable it to run efficiently on edge devices. FP16 or INT8 quantization is used for mixed-precision inference to improve inference speed. Hardware acceleration libraries, such as TensorRT, are used to optimize core operations such as convolution and matrix multiplication to optimize computation operators. Furthermore, different modules are deployed to different computing units to achieve pipelined parallel computing.

[0093] The model deployment architecture includes: robot end, edge end, and cloud end. The robot end runs real-time tasks such as perception, planning, and control, the edge end runs some computationally intensive tasks such as deep learning inference, and the cloud end runs large-scale model training, data storage, and remote management tasks.

[0094] This training and deployment strategy was adopted to train and deploy the BEV spatial representation learning module, VLA model module, traffic event detection and anomaly recognition module, end-to-end motion planning and navigation module, and human-computer interaction and remote management module, enabling each module to work efficiently and collaboratively in actual inspection tasks. Through the deployed modules, the inspection robot can autonomously complete inspection tasks in urban traffic environments based on multimodal perception information, language commands, and planning strategies. This allows for real-time perception, prediction, and response to road conditions, traffic events, and abnormal behaviors, improving the robot's autonomous perception, intelligent decision-making, and efficient inspection capabilities in complex traffic environments.

[0095] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0096] Obviously, those skilled in the art can make several improvements and modifications to this application without departing from the principles of this application, and these improvements and modifications also fall within the protection scope of this application.

Claims

1. A traffic inspection method using an automated inspection robot based on urban traffic scenarios, characterized in that, The method includes: A hardware platform for an inspection robot in urban traffic scenarios is constructed. The inspection robot hardware platform includes a mobile chassis, a multimodal sensor system, a computing unit, and a communication module. The raw multimodal sensor data is obtained from the inspection robot hardware platform, and the raw multimodal sensor data is preprocessed to obtain standard multimodal sensor data. The standard multimodal sensor data is subjected to feature transformation and BEV feature fusion to construct a BEV spatial representation learning module, and a VLA model module is constructed simultaneously. Based on the BEV spatial representation learning module and the VLA model module, a traffic event detection and anomaly recognition module is constructed, as well as an end-to-end motion planning and navigation module and a human-computer interaction and remote management module. Define a model training and deployment strategy, and use the model training and deployment strategy to train and deploy the BEV spatial representation learning module, the VLA model module, the traffic event detection and anomaly recognition module, the end-to-end motion planning and navigation module, and the human-computer interaction and remote management module. Then, control the automatic inspection robot to conduct traffic inspection through the deployed modules.

2. The traffic inspection method using an automated inspection robot based on urban traffic scenarios as described in claim 1, characterized in that, The raw multimodal sensor data is preprocessed to obtain standard multimodal sensor data, including: The original multimodal sensor data is time-synchronized and spatially aligned to obtain usable multimodal sensor data; A multimodal data preprocessing step is constructed, which includes image preprocessing, point cloud preprocessing, and radar data preprocessing; Based on the multimodal data preprocessing steps, the available multimodal sensor data is preprocessed to obtain standard multimodal sensor data; The standard multimodal sensor data is represented as: D_t={I_t,L_t,R_t,T_t}, where I_t represents the image data at time t, L_t represents the point cloud data, R_t represents the radar data, and T_t represents the sensor timestamp.

3. The traffic inspection method using an automated inspection robot based on urban traffic scenarios as described in claim 2, characterized in that, The standard multimodal sensor data undergoes feature transformation and BEV feature fusion to construct a BEV spatial representation learning module, including: A convolutional neural network is used to extract visual features from the image data I_t in the standard multimodal sensor data, resulting in a visual feature representation: F_vis = CNN_vis(I_t), where F_vis represents the visual feature map with the shape (C_vis, H_vis, W_vis), and C_vis, H_vis, and W_vis represent the number of channels, height, and width, respectively. The visual feature representation is transformed by LSS method to obtain BEV visual feature representation: F_vis_bev=LSS(F_vis); The lidar point cloud L_t in the standard multimodal sensor data is projected onto the XY plane using a cylindrical projection, and the Z-axis information is encoded into a feature channel to obtain the point cloud BEV feature representation: F_lidar_bev=Pillar(L_t); A BEV spatial representation learning module is constructed by fusing the BEV visual feature representation and the point cloud BEV feature representation.

4. The traffic inspection method using an automated inspection robot based on urban traffic scenarios as described in claim 3, characterized in that, A BEV spatial representation learning module is constructed by fusing the BEV visual feature representation and the point cloud BEV feature representation, including: The BEV visual feature representation and the point cloud BEV feature representation are fused together, represented as: F_bev=Fuse(F_vis_bev,F_lidar_bev), where Fuse represents the fusion operation, which can be achieved through concatenation, addition, or attention mechanisms. The fused BEV spatial representation learning module is obtained as: F_bev∈R^{C_bev×H_bev×W_bev}, where C_bev represents the number of BEV feature channels, H_bev and W_bev represent the height and width of the BEV grid, and each BEV grid corresponds to a fixed region in physical space.

5. The traffic inspection method using an automated inspection robot based on urban traffic scenarios as described in claim 4, characterized in that, The VLA model building module includes: A VLA model module is constructed, the core components of which include: a visual encoder, a language encoder, and an action decoder; The visual encoder is a Transformer-based visual encoder. For the input image data I_t and BEV features F_bev, the visual encoder outputs visual feature representations: Z_vis=VisionEncoder(I_t) and Z_vis=VisionEncoder(F_bev), where Z_vis∈R^{N×D_vis}, N represents the number of tokens, and D_vis represents the feature dimension. The language encoder uses a pre-trained language model. For the input natural language instruction Q, the language encoder outputs a language feature representation: Z_lang=LanguageEncoder(Q), where Z_lang∈R^{M×D_lang}, M represents the number of tokens, and D_lang represents the feature dimension. The language feature representations are fused together using a cross-modal attention mechanism, represented as: Z_fused = CrossAttention(Z_vis, Z_lang), where CrossAttention represents cross-modal attention computation, which can be bidirectional or unidirectional attention; The action decoder generates action instructions based on the fused feature Z_fused, and the action decoder generates action sequences in an autoregressive manner: A={a_1,a_2,...,a_T}, where a_t represents the action at time t, and T represents the length of the action sequence.

6. The traffic inspection method using an automated inspection robot based on urban traffic scenarios as described in claim 5, characterized in that, Based on the BEV spatial representation learning module and the VLA model module, a traffic incident detection and anomaly recognition module is constructed, including: The traffic incident detection module adopts a multi-task learning architecture, sharing the BEV feature extractor of the BEV spatial representation learning module, with each task having an independent detection head; For the detection task, an object detection framework is adopted. The detection head outputs the category, bounding box and confidence score of each object, which is represented as: D={(b_i,c_i,s_i)|i=1,...,N}, where b_i represents the bounding box coordinates, c_i represents the category, s_i represents the confidence score and N represents the number of detected objects; For semantic segmentation tasks, a segmentation network is used to output the category label for each BEV grid. The segmentation result is represented as: S∈{0,1,...,C}^{H_bev×W_bev}, where C represents the number of categories; The anomaly identification module, based on the VLA model module, uses anomaly detection or contrastive learning methods to identify anomalies. The output of the anomaly identification includes the anomaly type, anomaly location, and anomaly degree, represented as: E={(type_j,loc_j,score_j)|j=1,...,M}, where type_j represents the anomaly type, loc_j represents the anomaly location, score_j represents the anomaly confidence level, and M represents the number of detected anomalies.

7. The traffic inspection method using an automated inspection robot based on urban traffic scenarios as described in claim 1, characterized in that, Construct an end-to-end motion planning and navigation module, including: An end-to-end motion planning and navigation module is constructed, which adopts a layered architecture, including a global planning layer, a local planning layer, and a control layer. The global planning layer generates a global inspection path based on the inspection task and map information. The global inspection path is represented as: P_global={p_1,p_2,...,p_L}, where p_l represents the l-th path point, which includes the location coordinates and the expected speed, and L represents the number of path points. The local planning layer generates local obstacle avoidance and path tracking instructions based on local perception information and the global inspection path. The local planning output is represented as: P_local={τ_1,τ_2,...,τ_H}, where τ_h represents the local action at step h and H represents the prediction time domain. The control layer converts the local planning output into control commands to control the robot's speed and direction. The control commands are represented as: u=(v,ω), where v represents linear velocity and ω represents angular velocity.

8. The traffic inspection method using an automated inspection robot based on urban traffic scenarios as described in claim 7, characterized in that, The method further includes: Introducing the world model, represented as: s_{t+1}=WorldModel(s_t,a_t), where s_t represents the current state, a_t represents the current action, and s_{t+1} represents the predicted next state; The world model is based on a sequence prediction model. The inputs include: current BEV characteristics, robot state, and planned actions. The outputs include: future BEV characteristic predictions, other vehicle behavior predictions, and traffic event predictions. Based on the prediction outputs of the world model, the optimal action sequence is selected. Construct the planning objective function, expressed as: J=∑_{t=1}^{T}[r(s_t,a_t)+γ·V(s_{t+1})], where r(s_t,a_t) represents the immediate reward, V(s_{t+1}) represents the value function, and γ represents the discount factor.

9. The traffic inspection method using an automated inspection robot based on urban traffic scenarios as described in claim 1, characterized in that, Define model training and deployment strategies, including: The model training adopts a pre-training-fine-tuning paradigm, which includes a multimodal pre-training stage, a scene fine-tuning stage, and an online learning stage. The model deployment strategies include model compression, mixed-precision inference, operator optimization, and pipelined parallelism, and the model deployment architecture includes robot-side, edge-side, and cloud-side.

10. A traffic inspection method using an automated inspection robot based on urban traffic scenarios as described in claim 9, characterized in that, The method further includes: The VLA model module is trained using a pre-training-fine-tuning paradigm. During the pre-training phase, it is pre-trained on a large-scale vision-language-action dataset. The pre-training tasks include image-text alignment, action prediction, and instruction understanding. During the fine-tuning phase, fine-tuning is performed on traffic inspection scenario data, which includes real traffic scene images, inspection instructions, and corresponding action sequences. The loss function of the VLA model module is expressed as: L_VLA=λ_align·L_align+λ_action·L_action+λ_lang·L_lang, where L_align represents the image-text alignment loss, L_action represents the action prediction loss, L_lang represents the language understanding loss, and λ is the weight coefficient.

Images