Deep learning based live working multi-modal perception system

Abstract

The present application relates to the technical field of deep learning and live working environment perception. Disclosed is a live working multi-modal perception system based on deep learning, comprising a multi-modal perception component, an edge intelligent reasoning component and a safety decision output component. The multi-modal perception component collects image streams, point cloud streams and time sequence data, and generates a multi-modal fusion tensor through multi-source space-time synchronous preprocessing subcomponents. The edge intelligent reasoning component is embedded with a deep learning model, which extracts spatial features and state features through a spatial feature extraction network and a time sequence state prediction network respectively, and generates a unified high-dimensional feature representation vector through a multi-modal deep fusion layer. The system synchronously completes the fusion calculation of spatial and state features in the same node, eliminates the time difference, makes the data basis for safety level determination consistent in the time dimension, and solves the problem of deviation of motion risk assessment results from the actual physical state of the work site.

Description

Technical Field

[0001] This invention relates to the field of deep learning and live-line working environment perception technology, and discloses a deep learning-based multimodal perception system for live-line working. Background Technology

[0002] Existing live-line working platforms typically employ visual sensors and lidar for environmental monitoring. Conventional solutions use visual sensors to acquire two-dimensional planar images and lidar to acquire three-dimensional point clouds. These two types of data, along with platform attitude data from attitude sensors, are directly transmitted to a cloud server. On the cloud server, the conventional approach uses independent visual and point cloud algorithms to process the two-dimensional planar images and three-dimensional point clouds respectively, extracting obstacle outlines and position coordinates. Subsequently, the cloud server inputs the obstacle position coordinates and platform attitude data into a pre-defined rule-based judgment model. By comparing the distance values ​​between the two, a safety level is determined, and the safety level is then transmitted back to the working platform.

[0003] In the aforementioned conventional technical solutions, the visual algorithm, point cloud algorithm, and attitude data processing operate independently in the cloud. Due to network transmission time and the multi-threaded serial computing mechanism in the cloud, there is a time lag between the obstacle position coordinates processed by the cloud server and the received platform attitude data. When the work platform is in motion, this time lag causes a misalignment between the obstacle spatial position on which the safety level output by the cloud is based and the platform's current actual physical attitude, thus leading to the problem that the motion risk assessment results deviate from the actual physical state of the work site. Summary of the Invention

[0004] The purpose of this invention is to provide a deep learning-based multimodal sensing system for live-line work, which can solve the problems mentioned in the background art.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0006] A deep learning-based multimodal perception system for live-line work includes a multimodal perception component, an AI edge intelligent inference component, and a safety decision output component.

[0007] The multimodal perception components include a visual sensor, a lidar, an attitude sensor, a distance sensor, and an environmental sensor, used to collect low-level data on the work scene and platform status in real time.

[0008] The AI ​​edge intelligent inference component embeds a deep learning model, receives the underlying data, performs real-time semantic segmentation and obstacle recognition of the work scene locally, and predicts the platform's attitude, load and working conditions.

[0009] The safety decision output component is connected to the AI ​​edge intelligent inference component. Based on the deep fusion and inference analysis results of multimodal data, it performs rapid motion risk assessment and safety level determination, and outputs low-latency decision results that do not require cloud access.

[0010] Preferably, the multimodal perception component further includes a multi-source spatiotemporal synchronous preprocessing sub-component. The visual sensor acquires a two-dimensional image stream, the lidar acquires a three-dimensional point cloud stream, and the attitude sensor, distance sensor, and environment sensor acquire temporal one-dimensional sequence data respectively. The multi-source spatiotemporal synchronous preprocessing sub-component performs spatiotemporal registration on the two-dimensional image stream, the three-dimensional point cloud stream, and the temporal one-dimensional sequence data based on a unified timestamp and spatial coordinate system, generates a multimodal fusion tensor carrying a spatiotemporal alignment label, and transmits the multimodal fusion tensor to the AI ​​edge intelligent inference component.

[0011] Preferably, the deep learning model includes a spatial feature extraction network. The spatial feature extraction network receives a two-dimensional image stream and a three-dimensional point cloud stream from the multimodal fusion tensor. It extracts planar features from the image through a two-dimensional convolutional layer and extracts spatial geometric features from the point cloud through a three-dimensional sparse convolutional layer. The spatial feature extraction network uses a cross-attention mechanism to concatenate the planar features from the image and the spatial geometric features from the point cloud at the pixel-by-pixel and voxel-by-voxel levels to generate a multi-scale spatial feature map containing depth and texture information. Based on the multi-scale spatial feature map, it outputs a real-time semantic segmentation mask and a three-dimensional obstacle bounding box for the task scene.

[0012] Preferably, the deep learning model further includes a temporal state prediction network. The temporal state prediction network receives temporal one-dimensional sequence data from the multimodal fusion tensor, extracts the hidden state vector of the temporal one-dimensional sequence data in the time dimension using a long short-term memory network, inputs the hidden state vector into a graph convolutional network, and constructs a state topology graph with platform joint nodes as vertices and physical connection relationships as edges. Based on the graph convolution calculation results of the state topology graph, the temporal state prediction network outputs a platform attitude change trajectory, load distribution matrix, and working condition classification label containing a future set time step.

[0013] Preferably, the AI ​​edge intelligent inference component further includes a multimodal deep fusion layer. The multimodal deep fusion layer receives the graph convolution calculation results of the multi-scale spatial feature map and the state topology map, constructs a cross-modal interaction matrix of spatial flow and state flow using a two-stream Transformer architecture, captures the cross-modal correlation between the real-time semantic segmentation mask, the 3D obstacle bounding box and the platform posture change trajectory through multi-head self-attention operation, and generates a unified high-dimensional feature representation vector containing spatial constraints and state constraints.

[0014] Preferably, the safety decision output component includes a risk dynamic assessment network and a safety level mapper. The risk dynamic assessment network receives the unified high-dimensional feature representation vector and calculates the interference probability value of the spatial interference between the three-dimensional obstacle bounding box and the platform attitude change trajectory in the current working scenario, as well as the overload probability value of the load distribution matrix exceeding the safety threshold, through a fully connected layer and a softmax activation function. The safety level mapper pre-constructs a two-dimensional risk plane with the interference probability value and the overload probability value as coordinate axes, and outputs discrete safety level judgment results based on the divided regions in the two-dimensional risk plane.

[0015] Preferably, the multi-source spatiotemporal synchronous preprocessing subcomponent embeds a joint calibration network. The joint calibration network uses the initial extrinsic parameter matrices of the two-dimensional image stream and the three-dimensional point cloud stream as learnable parameters to construct a similarity loss function based on maximizing mutual information. During the operation of the multi-source spatiotemporal synchronous preprocessing subcomponent, the initial extrinsic parameter matrix is ​​iteratively updated through backpropagation, so that the mutual information value between the edge points of the three-dimensional point cloud stream projected onto the plane where the two-dimensional image stream is located and the corresponding gradient edge pixels in the two-dimensional image stream reaches the maximum, thereby obtaining the optimized target extrinsic parameter matrix, and the spatiotemporal registration is performed using the target extrinsic parameter matrix.

[0016] Preferably, the cross-attention mechanism includes a spatial position encoder and a feature interaction layer. The spatial position encoder adds multi-head position encoding based on two-dimensional relative position to the image planar features and adds position encoding based on three-dimensional absolute coordinates to the point cloud spatial geometric features. The feature interaction layer uses the point cloud spatial geometric features with added position encoding as a query vector and the image planar features with added position encoding as a key vector and a value vector. It generates a feature weight map through dot product attention calculation and uses the feature weight map to enhance the point cloud spatial geometric features, outputting the multi-scale spatial feature map.

[0017] Preferably, the graph convolutional network includes an uncertainty quantization branch, which superimposes Gaussian noise onto each vertex feature of the state topology graph. The vertex features after superimposing Gaussian noise are input into a multi-layer graph convolutional layer for forward propagation calculation. The forward propagation calculation is repeated a set number of times to obtain multiple prediction samples. The multiple prediction samples are statistically distributed, and the mean tensor and covariance matrix of the platform attitude change trajectory are output. The temporal state prediction network uses the mean tensor as the platform attitude change trajectory and the covariance matrix as the confidence index of the working condition classification label.

[0018] Preferably, the safety level mapper includes a dynamic boundary adjustment module. The dynamic boundary adjustment module records the sequence change trajectory of the interference probability value and the overload probability value within a historical period, uses a sliding window to extract the probability sequence with a set window length before the current moment, calculates the local variance and rate of change of the probability sequence, and when the local variance exceeds the dynamic variance threshold or the rate of change exceeds the dynamic rate of change threshold, the dynamic boundary adjustment module performs translation and scaling processing on the boundary coordinates of the divided region along the coordinate axis direction of the two-dimensional risk plane, and re-outputs the safety level determination result based on the updated boundary after translation and scaling processing.

[0019] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0020] 1. This invention executes deep learning models locally via an edge-based intelligent inference component, eliminating the need for data uploading to the cloud. A multi-source spatiotemporal synchronous preprocessing sub-component, based on a unified timestamp and spatial coordinate system, registers 2D image streams, 3D point cloud streams, and time-series 1D sequence data into a multimodal fusion tensor. The multimodal deep fusion layer utilizes a dual-stream transformer architecture to construct a cross-modal interaction matrix, correlating the features of the spatial stream and the state stream to generate a unified high-dimensional feature representation vector. This structure synchronously completes the fusion calculation of obstacle spatial features and platform attitude features within the same computing node, eliminating the time difference caused by data transmission and independent computation. This ensures that the data used to determine the safety level remains consistent across time, resolving the issue of motion risk assessment results deviating from the actual physical state of the work site.

[0021] 2. The joint calibration network iteratively updates the extrinsic parameter matrix with the goal of maximizing mutual information, aligning the edge points of the 3D point cloud with the gradient edge pixels of the 2D image, thus improving the coordinate overlap between the semantic segmentation mask and the 3D obstacle bounding box in the multi-scale spatial feature map. The temporal state prediction network extracts the temporal hidden state vector using a long short-term memory network and constructs a state topology graph through a graph convolutional network to calculate physical connectivity. Combined with the covariance matrix output by the uncertainty quantization branch, it provides a numerical confidence index for the working condition classification label. The risk dynamic assessment network calculates the interference probability value and overload probability value based on the fully connected layer. The dynamic boundary adjustment module in the safety level mapper translates and scales the boundary of the divided region of the 2D risk plane according to the local variance and rate of change of the probability sequence, so that the safety level judgment result changes accordingly with the fluctuation trend of the probability sequence. Attached Figure Description

[0022] Figure 1 System overall workflow diagram

[0023] Figure 2 Flowchart of multi-source spatiotemporal synchronization preprocessing

[0024] Figure 3 Flowchart for spatial feature extraction and semantic segmentation obstacle detection

[0025] Figure 4 Flowchart for Time Series State Prediction and Uncertainty Quantification

[0026] Figure 5 Flowchart for generating multimodal deep fusion and unified feature representation

[0027] Figure 6 This is a flowchart for dynamic risk assessment and dynamic safety level determination. Detailed Implementation

[0028] The technical solution described in this specific embodiment addresses the multimodal perception and safety decision-making needs in live-line working scenarios by constructing a system architecture for edge-to-edge local processing. The technical solution is fully and completely disclosed below in conjunction with specific implementation scenarios. The described implementation methods are only used to explain the technical solution and do not constitute an undue limitation on the scope of protection.

[0029] Please refer to Figure 1 In one embodiment, the deep learning-based live-line working multimodal perception system includes a multimodal perception component, an AI edge intelligent inference component, and a safety decision output component. The multimodal perception component includes a vision sensor, a lidar, an attitude sensor, a distance sensor, and an environmental sensor. These components are deployed at corresponding installation locations on the live-line working platform to collect low-level data on the working scene and platform status in real time. Specifically, the vision sensor is positioned facing the working area of ​​the platform and its surrounding environment, collecting two-dimensional image data within the working scene. This two-dimensional image data includes planar texture information of live conductors, grounding conductors, working tools, workers, fixed facilities, and moving objects within the scene. The lidar's scanning range covers the acquisition area of ​​the vision sensor, collecting three-dimensional point cloud data within the working scene. This three-dimensional point cloud data includes the spatial coordinate information and reflection intensity information of each object within the scene. The attitude sensor is deployed at the joints and actuators of the working platform, collecting motion state data of the platform body, including rotation parameters of each joint and motion parameters of the platform body. The distance sensor is deployed at the execution end and edge of the working platform, collecting relative distance data between key locations on the working platform and objects within the scene. Environmental sensors are deployed on the main body of the work platform to collect environmental parameter data of the work scene. The multimodal perception component transmits all the collected low-level data to the AI ​​edge intelligent inference component via a local data bus.

[0030] The AI ​​edge intelligent inference component is deployed on the local edge computing unit of the live-line working platform. Embedded with a deep learning model, it receives low-level data from the multimodal perception component and performs real-time semantic segmentation and obstacle recognition of the work scene within the local edge computing unit, while also predicting the platform's attitude, load, and operating conditions. Specifically, after receiving the low-level data, the AI ​​edge intelligent inference component first performs format normalization, converting different types of low-level data into tensor formats that meet the input requirements of the deep learning model. This normalization process is completed within the local computing unit, and the processed data is not transmitted to the cloud server via a public network. The deep learning model receives the normalized tensor data and extracts spatial features from both 2D image data and 3D point cloud data, identifying object categories within the work scene and generating corresponding semantic segmentation results. Simultaneously, it extracts the spatial location information of obstacles within the scene, generating spatial boundary information for the obstacles. The deep learning model simultaneously extracts temporal features from time-series data collected by attitude sensors, distance sensors, and environmental sensors, uncovering patterns of change in the time-series data. Based on the extracted temporal features, it predicts the subsequent attitude change trend, load distribution, and operating condition type of the work platform, generating corresponding prediction results. The AI ​​edge intelligent inference component transmits the results of semantic segmentation and obstacle recognition, along with the predicted results of platform attitude, load, and operating condition, to the safety decision output component within the local computing unit.

[0031] The safety decision output component is deployed on the same local edge computing unit and connects to the AI ​​edge intelligent inference component via a local data interface. The safety decision output component receives the multimodal data processing and inference analysis results output by the AI ​​edge intelligent inference component. Based on the deep fusion and inference analysis results of the multimodal data, it performs rapid motion risk assessment and safety level determination, outputting low-latency decision results that do not require cloud migration. Specifically, after receiving the inference analysis results, the safety decision output component first matches the spatial boundary information of scene obstacles with the platform's attitude change trend in a spatial dimension. Combined with the predicted results of load distribution status and work condition type, it constructs a risk assessment model that includes scene spatial constraints and platform state constraints. The risk assessment model completes all calculations within the local computing unit, without calling cloud computing resources or transmitting any work scene or platform state data to the cloud. Based on the input inference analysis results, the risk assessment model calculates the probability of the work platform interfering with obstacles in the scene during subsequent movement, as well as the probability of the work platform's load state exceeding the safe range. Based on the calculated probability values, it completes a quantitative assessment of motion risk. The safety decision output component has preset safety level classification rules. Based on the results of risk quantification assessment, it matches the corresponding safety level classification rules and outputs discrete safety level judgment results. The safety level judgment results are directly transmitted to the motion control unit of the live working platform through the local bus, providing a decision basis for the motion control of the working platform. The entire process of data acquisition, reasoning analysis, and decision output is completed locally on the working platform, and there is no data transmission link to the cloud.

[0032] In this embodiment, the type definition, physical meaning, data format and time granularity correspondence of each data source of the multimodal sensing component are shown in Table 1.

[0033] Table 1. Type definition, physical meaning, data format, and time granularity of each data source for the multimodal sensing component.

[0034] Visual sensors The planar texture and contour information of objects within the task scene are used for object category recognition and contour extraction. Frame data in two-dimensional matrix format, where each matrix element corresponds to the value of a single pixel in the image plane. The temporal granularity of consecutive frames matches the motion state of objects within the scene, with adjacent frame data carrying corresponding sampling time identifiers. LiDAR The three-dimensional spatial position and reflection characteristics of objects within the work scene are used for object spatial positioning and geometric contour construction. Frame data in a three-dimensional sparse matrix format, where each non-zero matrix element corresponds to the coordinates and reflection intensity value of a single spatial point. The continuous sampling time granularity corresponds to the visual sensor frame data, with each frame of point cloud data carrying a corresponding sampling time identifier. Attitude sensor The motion state information of each joint and the main body of the work platform is used for platform posture description and motion trend analysis. The sampled data is in one-dimensional time-series vector format, where each vector element corresponds to the value of a single state parameter. The sampling time granularity is matched to the joint motion frequency of the platform, and each set of sampled data carries a corresponding sampling time identifier. Distance sensor The relative distance information between key locations on the work platform and objects within the scene is used to determine spatial relative positional relationships. The sampled data is in one-dimensional time-series vector format, where each vector element corresponds to the distance value of a single measurement location. The sampling time granularity is matched to the platform's motion state, and each set of sampled data carries a corresponding sampling time identifier. Environmental sensors Environmental parameter information of the work scenario is used to assist in determining the work condition. Sampling data in one-dimensional time-series vector format, where each vector element corresponds to the value of a single environmental parameter. A continuous sampling time granularity matching the rate of change of environmental parameters, with each set of sampled data carrying a corresponding sampling time identifier.

[0035] The correspondence recorded in Table 1 clarifies the core attributes of each data source of the multimodal perception component, providing a unified data foundation for subsequent data processing and inference analysis, and ensuring that different types of underlying data can achieve attribute matching in subsequent processing.

[0036] In this embodiment, the multimodal perception component completes the real-time acquisition of the underlying data of the work scenario and platform status, the AI ​​edge intelligent inference component completes all deep learning inference operations locally, and the safety decision output component completes risk assessment and safety level determination locally. The entire process does not require data to be uploaded to the cloud, eliminating the time difference caused by cloud data transmission and multi-threaded serial computing, so that the scenario data and platform status data on which the safety level determination results are based remain synchronized in the time dimension.

[0037] refer to Figure 2 In another embodiment, the multimodal perception component further includes a multi-source spatiotemporal synchronization preprocessing sub-component, which is deployed at the output end of the multimodal perception component and connected to the input end of the AI ​​edge intelligent inference component. The underlying data acquired by the visual sensor is a two-dimensional image stream, composed of multiple consecutive frames of two-dimensional image data arranged in the order of sampling time. The underlying data acquired by the lidar is a three-dimensional point cloud stream, composed of multiple consecutive frames of three-dimensional point cloud data arranged in the order of sampling time. The underlying data acquired by the attitude sensor, distance sensor, and environment sensor are time-series one-dimensional sequence data, composed of multiple sets of consecutive sampled values ​​arranged in the order of sampling time. The multi-source spatiotemporal synchronization preprocessing sub-component receives the two-dimensional image stream, the three-dimensional point cloud stream, and all time-series one-dimensional sequence data. Based on a unified timestamp and spatial coordinate system, it performs spatiotemporal registration on all received data streams, generates a multimodal fusion tensor carrying spatiotemporal alignment labels, and transmits the multimodal fusion tensor to the AI ​​edge intelligent inference component.

[0038] Specifically, the multi-source spatiotemporal synchronization preprocessing subcomponent first performs timestamp synchronization processing to generate a unified synchronization time reference for all data streams. The subcomponent extracts the sampling time identifiers corresponding to each frame of 2D image stream, each frame of 3D point cloud stream, and each group of 1D temporal sequence data. Using the common time axis of all sampling time identifiers as a reference, it generates a unified synchronization timestamp. The calculation process of the unified synchronization timestamp follows the following expression:

[0039]

[0040] in, To unify and synchronize timestamps, This is the sampling time identifier for the sampled data corresponding to the i-th data source. This represents the total number of data sources participating in spatiotemporal registration. The multi-source spatiotemporal synchronization preprocessing subcomponent performs interpolation and resampling processing on all data streams based on the calculated unified synchronization timestamp. This ensures that the sampled data from different data sources are aligned in the time dimension under the unified synchronization timestamp. During the interpolation and resampling process, the physical meaning of the original sampled data is preserved, and the numerical distribution characteristics of the original data are not changed.

[0041] Furthermore, the multi-source spatiotemporal synchronous preprocessing subcomponent performs unified spatial coordinate system processing to generate a unified spatial coordinate reference for all data streams. The multi-source spatiotemporal synchronous preprocessing subcomponent constructs the platform's coordinate system as the unified target spatial coordinate system, using the center of the live-line working platform as the origin. It establishes transformation relationships between the original coordinate systems of the visual sensors, LiDAR, attitude sensors, distance sensors, and environmental sensors and the target spatial coordinate system. Specifically, the original coordinate system of the 3D point cloud stream is the LiDAR coordinate system, the original coordinate system of the 2D image stream is the image pixel coordinate system, and the attitude and distance data in the time-series 1D sequence data are generated based on the platform's coordinate system. Environmental data does not involve spatial coordinate transformation. The process of transforming the 3D point cloud stream from the LiDAR coordinate system to the image pixel coordinate system follows the following expression:

[0042]

[0043] in, These are the pixel coordinates of the 3D point cloud data projected onto the image plane. This is the intrinsic parameter matrix of the vision sensor. Let be the rotation extrinsic parameter matrix of the LiDAR coordinate system relative to the vision sensor coordinate system. Let be the translation extrinsic parameter vector of the LiDAR coordinate system relative to the vision sensor coordinate system. This is the extrinsic parameter matrix between the LiDAR and the vision sensor. This refers to the 3D point cloud coordinates in the lidar coordinate system. The multi-source spatiotemporal synchronous preprocessing subcomponent transforms all data streams involving spatial location information into a unified target spatial coordinate system based on the coordinate system transformation relationship corresponding to each data source, thereby achieving spatial dimension alignment.

[0044] After aligning the time and spatial dimensions, the multi-source spatiotemporal synchronization preprocessing subcomponent adds corresponding spatiotemporal alignment tags to all aligned data streams. These tags include a unified synchronization timestamp identifier and a unified spatial coordinate system identifier. The subcomponent then concatenates all data streams with the added alignment tags into a tensor, generating a multimodal fusion tensor. Each element of this tensor carries a corresponding spatiotemporal alignment tag, ensuring that data from different modalities maintains spatiotemporal consistency during subsequent inference. Finally, the subcomponent transmits the generated multimodal fusion tensor to the AI ​​edge intelligent inference component via a local data interface. The entire spatiotemporal registration process is completed locally, without involving cloud data transmission.

[0045] In this embodiment, a joint calibration network is embedded within the multi-source spatiotemporal synchronous preprocessing subcomponent. This network optimizes the extrinsic parameter matrix between the LiDAR and the visual sensor, improving the accuracy of spatial coordinate system transformation. The joint calibration network uses the initial extrinsic parameter matrices of the 2D image stream and the 3D point cloud stream as learnable parameters to construct a similarity loss function based on maximizing mutual information. During the operation of the multi-source spatiotemporal synchronous preprocessing subcomponent, the initial extrinsic parameter matrix is ​​iteratively updated through backpropagation, maximizing the mutual information between the edge points of the 3D point cloud stream projected onto the plane of the 2D image stream and the corresponding gradient edge pixels in the 2D image stream. This yields the optimized target extrinsic parameter matrix, which is then used to perform spatial coordinate system transformation during the spatiotemporal registration process.

[0046] Specifically, the joint calibration network first extracts edges from single-frame image data of the 2D image stream, extracting edge pixels in the image through gradient operations to generate an image edge feature map. Simultaneously, the joint calibration network extracts edges from single-frame point cloud data of the 3D point cloud stream, extracting edge points in the point cloud through curvature operations to generate a point cloud edge feature set. Based on the current extrinsic parameter matrix, the joint calibration network projects the 3D edge points in the point cloud edge feature set onto the image plane, generating a projected edge point set. The joint calibration network calculates the mutual information between the projected edge point set and the corresponding gradient edge pixels in the image edge feature map, constructing a loss function based on maximizing mutual information. The calculation process of the loss function follows the following expression:

[0047]

[0048] in, The loss function for the joint calibration network, For the projection edge point set Image edge feature map Mutual information between them The edge entropy is the projection edge point set. The edge entropy of the image edge feature map. This represents the joint entropy of the projected edge point set and the image edge feature map. The joint calibration network optimizes by minimizing the loss function. It iteratively updates the learnable parameters in the extrinsic parameter matrix using the backpropagation algorithm. After each iteration, the mutual information between the projected edge point set and the image edge feature map is recalculated until the loss function converges to a preset range. The corresponding extrinsic parameter matrix at this point is the optimized target extrinsic parameter matrix. The joint calibration network then transmits the optimized target extrinsic parameter matrix to the multi-source spatiotemporal synchronization preprocessing sub-component for subsequent spatial coordinate system transformation and spatiotemporal registration processing.

[0049] In this embodiment, the spatiotemporal registration rules and coordinate system mapping relationships of each data source during the multi-source spatiotemporal synchronization preprocessing are shown in Table 2.

[0050] Table 2. Spatiotemporal registration rules and coordinate system mapping relationships of each data source during multi-source spatiotemporal synchronization preprocessing.

[0051] Visual sensors Frame interpolation alignment of the 2D image stream is performed based on a unified synchronization timestamp, and a corresponding synchronization timestamp tag is added to each frame of image data. Image pixel coordinate system Platform body coordinate system First, the pixel coordinate system is transformed to the camera coordinate system using an intrinsic parameter matrix, and then transformed to the platform body coordinate system using an optimized extrinsic parameter matrix. LiDAR Frame interpolation alignment of the 3D point cloud stream is performed based on a unified synchronization timestamp, and a corresponding synchronization timestamp tag is added to each frame of point cloud data. LiDAR coordinate system Platform body coordinate system The lidar coordinate system is directly transformed to the platform body coordinate system using the optimized extrinsic parameter matrix. Attitude sensor Interpolation resampling and alignment of one-dimensional time-series data is performed based on a unified synchronization timestamp, and a corresponding synchronization timestamp label is added to each group of sampled data. Platform body coordinate system Platform body coordinate system No coordinate transformation required; directly matches the unified spatial coordinate system datum. Distance sensor Interpolation resampling and alignment of one-dimensional time-series data is performed based on a unified synchronization timestamp, and a corresponding synchronization timestamp label is added to each group of sampled data. Platform body coordinate system Platform body coordinate system No coordinate transformation required; directly matches the unified spatial coordinate system datum. Environmental sensors Interpolation resampling and alignment of one-dimensional time-series data is performed based on a unified synchronization timestamp, and a corresponding synchronization timestamp label is added to each group of sampled data. No spatial coordinate system No spatial coordinate system No spatial coordinate transformation is involved; only time dimension alignment is performed.

[0052] The correspondences recorded in Table 2 clarify the processing rules of each data source in the spatiotemporal registration process, ensuring that data streams of different modalities can achieve dual alignment in both time and space dimensions, providing a unified spatiotemporal benchmark for subsequent multimodal feature fusion.

[0053] In this embodiment, the multi-source spatiotemporal synchronization preprocessing sub-component achieves spatiotemporal registration of multimodal data through a unified timestamp and spatial coordinate system. The joint calibration network optimizes the extrinsic parameter matrix with the goal of maximizing mutual information, thereby improving the accuracy of spatial coordinate transformation. This ensures that different modal data in the multimodal fusion tensor maintain strict alignment in the spatiotemporal dimension and eliminates spatiotemporal misalignment between different data sources.

[0054] refer to Figure 3 In another preferred embodiment, the deep learning model embedded in the AI ​​edge intelligent inference component includes a spatial feature extraction network. The spatial feature extraction network receives a two-dimensional image stream and a three-dimensional point cloud stream from a multimodal fusion tensor. It extracts planar features from the image through a two-dimensional convolutional layer and extracts spatial geometric features from the point cloud through a three-dimensional sparse convolutional layer. The spatial feature extraction network uses a cross-attention mechanism to concatenate the planar features of the image and the spatial geometric features of the point cloud at the pixel-by-pixel and voxel-by-voxel levels to generate a multi-scale spatial feature map containing depth and texture information. Based on the multi-scale spatial feature map, it outputs a real-time semantic segmentation mask and a three-dimensional obstacle bounding box for the task scene.

[0055] Specifically, the spatial feature extraction network includes a two-dimensional feature extraction branch, which is composed of multiple cascaded two-dimensional convolutional layers. This branch receives two-dimensional image stream data from a multimodal fusion tensor and extracts planar texture features from the image layer by layer. The computation process of the two-dimensional convolutional layer follows the expression below:

[0056]

[0057] in, The image planar features output by the l-th two-dimensional convolutional layer. It is a non-linear activation function. Let be the kernel weight matrix of the l-th two-dimensional convolutional layer. This is a two-dimensional convolution operator. This refers to the bias term of the l-th two-dimensional convolutional layer. This is the input feature map output from layer (l-1). The two-dimensional feature extraction branch generates image planar features of different scales through downsampling operations of multiple two-dimensional convolutional layers. Image planar features of different scales correspond to the texture information of different receptive fields in the image, and all image planar features of all scales carry corresponding spatiotemporal alignment labels.

[0058] Furthermore, the spatial feature extraction network includes a 3D feature extraction branch, which is composed of multiple cascaded 3D sparse convolutional layers. This branch receives 3D point cloud flow data from the multimodal fusion tensor and extracts the spatial geometric features of the point cloud layer by layer. The 3D sparse convolutional layers only perform convolution operations on non-empty voxels containing point cloud data, ignoring empty voxels without data. The operation process of the 3D sparse convolutional layers follows the following expression:

[0059]

[0060] in, The point cloud spatial geometric features output by the l-th 3D sparse convolutional layer. The set of spatial neighborhoods corresponding to the 3D convolution kernel. Let K be the convolution kernel weight matrix corresponding to the spatial neighborhood k. The spatial coordinates of the current non-empty voxel. The output of layer l-1, with coordinates as voxel characteristics, This refers to the bias term of the l-th 3D sparse convolutional layer. This is a non-linear activation function. The 3D feature extraction branch generates point cloud spatial geometric features at different scales through downsampling operations of multiple 3D sparse convolutional layers. Point cloud spatial geometric features at different scales correspond to geometric information of different spatial ranges in the point cloud, and point cloud spatial geometric features at all scales carry corresponding spatiotemporal alignment labels.

[0061] The cross-attention mechanism in the spatial feature extraction network includes a spatial position encoder and a feature interaction layer. The spatial position encoder adds multi-head positional encoding based on two-dimensional relative positions to the image planar features. For each pixel position of the image planar features, its relative positional encoding relative to other pixel positions within the feature map is calculated, and this relative positional encoding is superimposed on the feature vector of the corresponding pixel. Simultaneously, the spatial position encoder adds positional encoding based on three-dimensional absolute coordinates to the point cloud spatial geometric features. Based on the three-dimensional absolute coordinates of point cloud voxels in a unified spatial coordinate system, a corresponding positional encoding vector is generated, and this positional encoding vector is superimposed on the feature vector of the corresponding voxel. The feature interaction layer receives the image planar features with added positional encoding and the point cloud spatial geometric features. It uses the point cloud spatial geometric features with added positional encoding as the query vector and the image planar features with added positional encoding as the key and value vectors. A feature weight map is generated through dot-product attention calculation, and this feature weight map is used to enhance the point cloud spatial geometric features, outputting a multi-scale spatial feature map. The calculation process of dot-product attention follows the following expression:

[0062]

[0063] in, For the query vector, the corresponding point cloud spatial geometric features are added with position encoding. For the key vector, the corresponding image planar features after adding positional encoding. For value vectors, corresponding to the image planar features after adding positional encoding. Let be the dimension of the key vector. The scaling factor is used to avoid the gradient vanishing problem caused by excessively large dot product results. The feature interaction layer performs the above dot product attention operation on the image planar features and point cloud spatial geometric features at different scales, respectively, to achieve pixel-level and voxel-level feature concatenation, generate fused features at the corresponding scales, and stitch together the fused features at all scales to generate a multi-scale spatial feature map containing depth and texture information.

[0064] The output of the spatial feature extraction network is connected to a semantic segmentation head and an obstacle detection head. The semantic segmentation head receives a multi-scale spatial feature map and uses upsampling operations to restore the multi-scale spatial feature map to the same size as the original 2D image stream, generating a real-time semantic segmentation mask for the task scene. Each pixel value in the semantic segmentation mask corresponds to the object category identifier at that pixel location. The obstacle detection head receives the multi-scale spatial feature map and uses regression operations to generate 3D obstacle bounding boxes for obstacles in the scene. The 3D obstacle bounding boxes contain the center coordinates, size parameters, and orientation parameters of the obstacles in a unified spatial coordinate system. The spatial feature extraction network combines the generated semantic segmentation mask and 3D obstacle bounding boxes with the multi-scale spatial feature map. Figure 1 It is then transmitted to the subsequent processing unit.

[0065] refer to Figure 4 Furthermore, the deep learning model also includes a temporal state prediction network. The temporal state prediction network receives one-dimensional temporal sequence data from the multimodal fusion tensor, extracts the hidden state vector in the time dimension of the one-dimensional temporal sequence data using a long short-term memory network, inputs the hidden state vector into a graph convolutional network, and constructs a state topology graph with platform joint nodes as vertices and physical connection relationships as edges. Based on the graph convolution calculation results of the state topology graph, the temporal state prediction network outputs the platform attitude change trajectory, load distribution matrix, and working condition classification labels for a future set time step.

[0066] Specifically, the temporal state prediction network incorporates a Long Short-Term Memory (LSTM) network at its input. The LSM receives one-dimensional temporal sequence data from the multimodal fusion tensor, including aligned temporal data collected by attitude, distance, and environment sensors. The LSM processes the input one-dimensional temporal sequence data step-by-step, extracting temporal dependency features and outputting the hidden state vector for each time step. The hidden state update process of the LSM follows the expression below:

[0067]

[0068] in, The input time series one-dimensional sequence data is at time t. , , Let be the output values ​​of the input gate, forget gate, and output gate at time t, respectively. Let t represent the candidate cell state at time t. Let t represent the cell state at time t. Let be the hidden state vector output at time t. , , , , , , , This is the corresponding weight matrix. , , , For the corresponding bias term, It is the sigmoid activation function. The hyperbolic tangent activation function is used. This is the Hadamard product operator. Long Short-Term Memory (LSTM) networks concatenate the hidden state vectors from consecutive time steps to generate a temporal hidden state feature matrix, which is then passed to the subsequent graph convolutional network.

[0069] The graph convolutional network in the temporal state prediction network receives the temporal hidden state feature matrix and first constructs a state topology graph of the work platform. The vertices of the state topology graph represent the joint nodes of the work platform, with each joint node corresponding to a vertex. The feature vector of each vertex is the corresponding feature vector in the temporal hidden state feature matrix for that joint node. The edges of the state topology graph represent the physical connections between the joint nodes of the work platform. When there is a physical connection between two joint nodes, there is an edge between their corresponding vertices. Based on these physical connections, an adjacency matrix of the state topology graph is constructed. The graph convolutional network then performs multi-layer graph convolution operations based on the constructed state topology graph to extract the association features between the joint nodes. The graph convolution operation follows the following expression:

[0070]

[0071] in, This is the vertex feature matrix output by the (l+1)th graph convolutional layer. To add a self-loop adjacency matrix, The adjacency matrix of the state topology graph. It is the identity matrix. for The corresponding degree matrix, , The vertex feature matrix is ​​the input to the l-th graph convolutional layer. Let be the weight matrix of the l-th graph convolutional layer. This is a non-linear activation function. Graph convolutional networks generate a vertex feature matrix containing the associated features of each joint node through multi-layer graph convolution operations, thus completing the graph convolution calculation of the state topology graph.

[0072] The output of the temporal state prediction network includes a trajectory regression head, a load regression head, and a working condition classification head. The trajectory regression head receives the vertex feature matrix calculated by graph convolution, performs regression operations through fully connected layers, and outputs the platform attitude change trajectory for a predetermined future time step. This trajectory includes the attitude parameters of each joint node at each future time step. The load regression head receives the vertex feature matrix calculated by graph convolution, performs regression operations through fully connected layers, and outputs the load distribution matrix of the work platform. This matrix includes the load distribution parameters of each joint node and the actuator. The working condition classification head receives the vertex feature matrix calculated by graph convolution, performs regression operations through fully connected layers and a classification activation function, and outputs the working condition classification label of the work platform. This label corresponds to the current working condition type of the work platform.

[0073] In this embodiment, the graph convolutional network includes an uncertainty quantization branch. The uncertainty quantization branch superimposes Gaussian noise onto each vertex feature of the state topology graph. The vertex features after superimposing Gaussian noise are input into a multi-layer graph convolutional layer for forward propagation calculation. The forward propagation calculation is repeated a set number of times to obtain multiple prediction samples. The statistical distribution of the multiple prediction samples is calculated, and the mean tensor and covariance matrix of the platform attitude change trajectory are output. The temporal state prediction network uses the mean tensor as the platform attitude change trajectory and the covariance matrix as the confidence index of the working condition classification label.

[0074] Specifically, the uncertainty quantization branch superimposes Gaussian-distributed random noise onto the input vertex features of each graph convolutional layer in the graph convolutional network. The noise superposition process follows the expression below:

[0075]

[0076] in, The input vertex features are used to construct the l-th graph convolutional layer after adding noise. Input vertex features into the original l-th convolutional layer. It is Gaussian distributed noise. , The preset noise standard deviation, The identity matrix is ​​used. The uncertainty quantization branch inputs the vertex features after adding noise into the graph convolutional network, performs a complete forward propagation calculation, and obtains a set of corresponding prediction samples. The prediction samples include the prediction results of the platform attitude change trajectory, load distribution matrix, and working condition classification labels. The uncertainty quantization branch repeats the above forward propagation calculation a set number of times, adding independently sampled Gaussian distributed noise in each calculation to obtain multiple sets of independent prediction samples.

[0077] The uncertainty quantification branch performs statistical distribution calculations on the obtained multiple prediction samples, and calculates the mean tensor and covariance matrix of the platform attitude change trajectory respectively. The calculation process follows the following expression:

[0078]

[0079] in, Let be the mean tensor of the platform's attitude change trajectory. Let be the covariance matrix of the platform's attitude change trajectory. This represents the total number of forward propagation calculations. The m-th forward propagation calculation yields the predicted platform attitude change trajectory sample. The temporal state prediction network uses the calculated mean tensor as the final output platform attitude change trajectory and the trace of the covariance matrix as the confidence index of the working condition classification label. The smaller the trace of the covariance matrix, the higher the confidence of the corresponding working condition classification label.

[0080] In this embodiment, the physical meaning, dimension definition, and data source correspondence of each core feature vector during the feature extraction process are shown in Table 3.

[0081] Table 3. Physical meaning, dimensional definition, and data source correspondence of each core feature vector during feature extraction.

[0082] Image planar features Planar texture and contour features extracted from a 2D image stream are used for object category recognition. The dimension of the two-dimensional feature matrix is ​​determined by the number of convolutional kernels and the downsampling rate of the two-dimensional convolutional layer. Output of the two-dimensional feature extraction branch of the spatial feature extraction network Point cloud spatial geometric features Spatial location and geometric features extracted from 3D point cloud flow are used for object spatial localization. The dimension of the three-dimensional sparse feature matrix is ​​determined by the number of convolution kernels and the downsampling rate of the three-dimensional sparse convolutional layer. Output of the 3D feature extraction branch of the spatial feature extraction network Multi-scale spatial feature map The fused features, resulting from the cross-attention cascade of image planar features and point cloud spatial geometric features, contain both scene texture and depth information. The dimension of the multi-channel feature matrix is ​​determined by the number of features fused at different scales. The feature interaction layer output of the spatial feature extraction network Temporal hidden state vector Temporal dependency features extracted from one-dimensional time series data are used for platform state change trend analysis. A one-dimensional feature vector, the dimension of which is determined by the number of hidden layer neurons in the Long Short-Term Memory network. Long Short-Term Memory Network Output of Temporal State Prediction Network Vertex features of state topology graph The state characteristics of each joint node on the platform and the correlation characteristics between nodes are used to predict the platform's attitude and load. A two-dimensional feature matrix, where rows correspond to the number of platform joint nodes and columns correspond to the number of output channels of the graph convolutional layer. Output of graph convolutional network for temporal state prediction network Platform attitude change trajectory tensor The predicted attitude parameters of each joint node of the platform with a predetermined time step will be used to describe the platform's motion trend. A three-dimensional tensor, where the dimensions correspond to the number of future time steps, the number of platform joint nodes, and the number of single-node pose parameters, respectively. Trajectory regression head output of a time-series state prediction network

[0083] The correspondences recorded in Table 3 clarify the attributes and sources of each core feature vector in the feature extraction process, providing a unified feature foundation for subsequent cross-modal feature fusion.

[0084] In this embodiment, the spatial feature extraction network extracts image and point cloud features through 2D convolution and 3D sparse convolution, respectively, and uses a cross-attention mechanism to achieve feature concatenation, generating a multi-scale spatial feature map containing depth and texture information, and outputting a semantic segmentation mask and 3D obstacle bounding boxes. The temporal state prediction network extracts temporal features through a long short-term memory network, constructs a platform state topology map using a graph convolutional network, and combines an uncertainty quantization branch to output attitude trajectories and operational condition classification results with confidence indices, thereby realizing the prediction of platform state.

[0085] refer to Figure 5 In another preferred embodiment, the AI ​​edge intelligent inference component further includes a multimodal deep fusion layer. The multimodal deep fusion layer receives the graph convolution calculation results of the multi-scale spatial feature map output by the spatial feature extraction network and the state topology map output by the temporal state prediction network. It constructs a cross-modal interaction matrix of spatial flow and state flow using a two-stream Transformer architecture, and captures the cross-modal correlation between real-time semantic segmentation mask, 3D obstacle bounding box and platform posture change trajectory through multi-head self-attention operation, generating a unified high-dimensional feature representation vector containing spatial constraints and state constraints.

[0086] Specifically, the dual-stream Transformer architecture of the multimodal deep fusion layer includes a spatial flow branch and a state flow branch. The spatial flow branch receives multi-scale spatial feature maps and transforms them into a token sequence that meets the Transformer input requirements through a linear projection layer. Each token corresponds to a local region feature in the multi-scale spatial feature map, and the token sequence carries a corresponding spatiotemporal alignment label. The state flow branch receives the graph convolution calculation result of the state topology graph, which is the vertex feature matrix containing the association features of each joint node. The vertex feature matrix is ​​transformed into a corresponding token sequence through a linear projection layer. Each token corresponds to the state feature of a joint node on the platform, and the token sequence carries a corresponding spatiotemporal alignment label. The dual-stream Transformer architecture inputs the token sequences from the spatial flow branch and the state flow branch into the cross-modal attention module to construct the cross-modal interaction matrix of the spatial and state flows. Multi-head self-attention operations are used to capture the association between the token sequences of the two branches. The multi-head self-attention operation follows the following expression:

[0087]

[0088] in, This is the output of the multi-head self-attention operation. For the total number of attention heads, This is the output of the i-th attention head. Let be the linear projection weight matrix of the multi-head self-attention output. The output of the i-th attention head follows the expression:

[0089]

[0090] in, , , Let be the linear projection weight matrices of the query vector, key vector, and value vector corresponding to the i-th attention head, respectively. This involves dot product attention operations. The cross-modal attention module uses the token sequence of the spatial flow branch as the query vector, key vector, and value vector simultaneously, performing self-attention operations to extract feature associations within the spatial flow; it also uses the token sequence of the state flow branch as the query vector, key vector, and value vector simultaneously, performing self-attention operations to extract feature associations within the state flow; simultaneously, it uses the token sequence of the spatial flow branch as the query vector and the token sequence of the state flow branch as the key vector and value vector, performing cross-attention operations to extract cross-modal associations between the spatial flow and state flow; and finally, it uses the token sequence of the state flow branch as the query vector and the token sequence of the spatial flow branch as the key vector and value vector, performing cross-attention operations to further extract cross-modal associations between the two branches.

[0091] The multimodal deep fusion layer concatenates the spatial flow features and state flow features output by the cross-modal attention module, and performs feature dimensionality reduction through a fully connected layer to generate a unified high-dimensional feature representation vector that includes both spatial and state constraints. This unified high-dimensional feature representation vector simultaneously contains information on spatial obstacles in the work scenario, platform attitude changes, load distribution, and work condition type. All information is based on the same spatiotemporal alignment benchmark, eliminating spatiotemporal misalignment between different modal information. The multimodal deep fusion layer then transmits the generated unified high-dimensional feature representation vector to the safety decision output component via a local data interface.

[0092] refer to Figure 6 The safety decision output component includes a risk dynamic assessment network and a safety level mapper. The risk dynamic assessment network receives a unified high-dimensional feature representation vector and calculates the interference probability value of the spatial interference between the three-dimensional obstacle bounding box and the platform attitude change trajectory in the current working scenario, as well as the overload probability value of the load distribution matrix exceeding the safety threshold, through a fully connected layer and a softmax activation function. The safety level mapper pre-constructs a two-dimensional risk plane with the interference probability value and the overload probability value as coordinate axes, and outputs discrete safety level judgment results based on the divided regions in the two-dimensional risk plane.

[0093] Specifically, the risk dynamic assessment network consists of multiple cascaded fully connected layers. Each fully connected layer receives a unified high-dimensional feature representation vector as its input, performs a nonlinear transformation on the feature vector layer by layer, and outputs the logit values ​​for the corresponding interference and overload events. The output of the risk dynamic assessment network is equipped with a softmax activation function, which activates the logit values ​​of the interference and overload events respectively, outputting the corresponding interference and overload probability values. The softmax activation function operates according to the following expression:

[0094]

[0095] in, Let be the probability value of the j-th type of event. The logit value corresponding to the j-th type of event output by the fully connected layer. This represents the total number of event categories. For the interference probability calculation, the event categories include two types: spatial interference occurring and no spatial interference occurring. The output interference probability value is the probability value for the spatial interference category. For the overload probability calculation, the event categories include two types: load exceeding the safety threshold and load not exceeding the safety threshold. The output overload probability value is the probability value for the load exceeding the safety threshold category. The risk dynamic assessment network transmits the calculated interference probability values ​​and overload probability values ​​to the safety level mapper.

[0096] The safety level mapper pre-constructs a two-dimensional risk plane. The two coordinate axes of this plane are the interference probability value axis and the overload probability value axis, both ranging from 0 to 1. The safety level mapper pre-divides the two-dimensional risk plane into multiple non-overlapping regions. Each region corresponds to a safety level, and each region corresponds to a range of interference and overload probability values. The safety level mapper receives the interference and overload probability values ​​output from the dynamic risk assessment network, uses these two probability values ​​as coordinate points within the two-dimensional risk plane, matches these coordinate points to the corresponding region, and outputs the safety level determination result for that region.

[0097] In this embodiment, the safety level mapper includes a dynamic boundary adjustment module. The dynamic boundary adjustment module records the sequence change trajectory of the interference probability value and the overload probability value within the historical period. It uses a sliding window to extract the probability sequence with a set window length before the current moment, calculates the local variance and rate of change of the probability sequence, and when the local variance exceeds the dynamic variance threshold or the rate of change exceeds the dynamic rate of change threshold, the dynamic boundary adjustment module performs translation and scaling processing on the boundary coordinates of the divided area along the coordinate axis direction of the two-dimensional risk plane, and re-outputs the safety level judgment result based on the updated boundary after translation and scaling processing.

[0098] Specifically, the dynamic boundary adjustment module includes a sequence storage unit that continuously records the output interference probability value and overload probability value at each time step, generating a corresponding probability time series. The module also employs a sliding window with a preset fixed length. The sliding window, ending at the current time step, extracts probability sequences up to the set window length, specifically the interference probability sequence and the overload probability sequence. For the extracted interference and overload probability sequences, the module calculates the local variance and rate of change. The local variance describes the fluctuation of the probability sequence within the sliding window, while the rate of change describes the trend of the probability sequence within the sliding window.

[0099] When the local variance of the interference probability sequence exceeds a preset dynamic variance threshold, or the rate of change of the interference probability sequence exceeds a preset dynamic rate of change threshold, the dynamic boundary adjustment module performs translation and scaling processing on the boundary coordinates of each divided region along the interference probability value coordinate axis of the two-dimensional risk plane. When the local variance of the overload probability sequence exceeds a preset dynamic variance threshold, or the rate of change of the overload probability sequence exceeds a preset dynamic rate of change threshold, the dynamic boundary adjustment module performs translation and scaling processing on the boundary coordinates of each divided region along the overload probability value coordinate axis of the two-dimensional risk plane. The translation and scaling process of the boundary coordinates follows the following expression:

[0100]

[0101] in, For the updated boundary coordinates, These are the original boundary coordinates. This is the scaling factor. For translation vectors, scaling factor With translation vector The value of scaling factor is calculated from the local variance and rate of change of the probability sequence. As the local variance or rate of change of the probability sequence increases, the scaling factor... When the value is less than 1, the boundary shrinks towards the origin, reducing the area corresponding to the lower security level; when the local variance or rate of change of the probability sequence decreases, the scaling factor... If the value is greater than 1, the boundary expands away from the origin, increasing the area corresponding to the lower security level. Translation vector. The value of is in the same direction as the rate of change of the probability sequence. When the probability sequence shows an upward trend, the translation vector... Shifting towards the origin, when the probability sequence shows a decreasing trend, the translation vector... It shifts away from the origin of the coordinate system.

[0102] After completing the translation and scaling of the boundary coordinates, the dynamic boundary adjustment module transmits the updated boundary coordinates to the safety level mapper. Based on the updated boundary coordinates, the safety level mapper re-matches the division regions corresponding to the current interference probability value and overload probability value, and outputs the updated safety level determination result. The dynamic boundary adjustment module continuously monitors the probability sequence, performing the aforementioned local variance and rate of change calculation at each sampling time. Based on the calculation results, it adjusts the boundary of the division region of the two-dimensional risk plane in real time, so that the safety level determination result can change accordingly with the fluctuation trend of the probability sequence.

[0103] In this embodiment, the correspondence between the safety level division area, boundary range and judgment rules of the two-dimensional risk plane is shown in Table 4.

[0104] Table 4. Correspondence between safety level classification areas, boundary ranges, and judgment rules for two-dimensional risk planes.

[0105] Security level 0 to the first interference probability boundary value 0 to the first overload probability boundary value The output safety level assessment result indicates that the work platform can execute the work actions normally according to the preset plan. Attention level From the first interference probability boundary value to the second interference probability boundary value, and not exceeding the range of the higher-level boundary. From the first overload probability boundary value to the second overload probability boundary value, and not exceeding the range of the higher-level boundary. The system outputs the attention level determination result, maintains the current operation, and continuously monitors the trend of probability value changes. Warning level From the second interference probability boundary value to the third interference probability boundary value, and not exceeding the range of the higher-level boundary. The overload probability boundary value is between the second and third overload probability boundary values, and does not exceed the range of the higher-level boundary. The system outputs a warning-level judgment result, the work platform slows down to execute the work action, and the pre-set risk avoidance plan preparation process is initiated. Danger level Exceeding the third interference probability boundary value Exceeding the third overload probability boundary value Upon outputting the hazard level assessment result, the work platform immediately ceases its current operation and executes the preset emergency avoidance actions.

[0106] The correspondence recorded in Table 4 clarifies the classification rules and judgment output logic of each safety level in the two-dimensional risk plane, providing a clear execution basis for safety level judgment. The dynamic boundary adjustment module can adjust the boundary values ​​recorded in the table by translation and scaling based on the fluctuation of the probability sequence, so as to realize the dynamic update of the safety level classification boundary.

[0107] In this embodiment, the multimodal deep fusion layer achieves cross-modal feature fusion of spatial and state flows through a dual-stream Transformer architecture, generating a unified high-dimensional feature representation vector. The risk dynamic assessment network, based on this unified high-dimensional feature representation vector, calculates the interference probability and overload probability values, achieving a quantitative assessment of operational risks. The safety level mapper implements discretized safety level determination through a two-dimensional risk plane, and the dynamic boundary adjustment module dynamically adjusts the boundary division based on the fluctuation trend of the probability sequence, enabling the safety level determination results to adapt to the dynamic changes in the operational scenario.

Claims

1. A deep learning-based multimodal sensing system for live-line work, characterized in that, This includes multimodal perception components, AI edge intelligent inference components, and security decision output components; The multimodal perception components include a visual sensor, a lidar, an attitude sensor, a distance sensor, and an environmental sensor, used to collect low-level data on the work scene and platform status in real time. The AI ​​edge intelligent inference component embeds a deep learning model, receives the underlying data, performs real-time semantic segmentation and obstacle recognition of the work scene locally, and predicts the platform's attitude, load and working conditions. The safety decision output component is connected to the AI ​​edge intelligent inference component. Based on the deep fusion and inference analysis results of multimodal data, it performs rapid motion risk assessment and safety level determination, and outputs low-latency decision results that do not require cloud access.

2. The deep learning-based multimodal sensing system for live-line work according to claim 1, characterized in that, The multimodal perception component further includes a multi-source spatiotemporal synchronous preprocessing sub-component. The visual sensor acquires a two-dimensional image stream, the lidar acquires a three-dimensional point cloud stream, and the attitude sensor, distance sensor, and environment sensor acquire temporal one-dimensional sequence data respectively. The multi-source spatiotemporal synchronous preprocessing sub-component performs spatiotemporal registration on the two-dimensional image stream, the three-dimensional point cloud stream, and the temporal one-dimensional sequence data based on a unified timestamp and spatial coordinate system, generates a multimodal fusion tensor carrying a spatiotemporal alignment label, and transmits the multimodal fusion tensor to the AI ​​edge intelligent inference component.

3. The deep learning-based multimodal perception system for live-line work according to claim 2, wherein the deep learning model includes a spatial feature extraction network, the spatial feature extraction network receives a two-dimensional image stream and a three-dimensional point cloud stream in the multimodal fusion tensor, extracts planar features of the image through a two-dimensional convolutional layer, extracts spatial geometric features of the point cloud through a three-dimensional sparse convolutional layer, the spatial feature extraction network uses a cross-attention mechanism to concatenate the planar features of the image and the spatial geometric features of the point cloud at the pixel-by-pixel and voxel-by-voxel levels to generate a multi-scale spatial feature map containing depth and texture information, and outputs a real-time semantic segmentation mask and a three-dimensional obstacle bounding box of the work scene based on the multi-scale spatial feature map.

4. The deep learning-based multimodal sensing system for live-line work according to claim 3, characterized in that, The deep learning model also includes a temporal state prediction network. The temporal state prediction network receives temporal one-dimensional sequence data from the multimodal fusion tensor, extracts the hidden state vector of the temporal one-dimensional sequence data in the time dimension using a long short-term memory network, inputs the hidden state vector into a graph convolutional network, and constructs a state topology graph with platform joint nodes as vertices and physical connection relationships as edges. Based on the graph convolution calculation results of the state topology graph, the temporal state prediction network outputs the platform attitude change trajectory, load distribution matrix, and working condition classification labels for a future set time step.

5. The deep learning-based multimodal sensing system for live-line work according to claim 4, characterized in that, The AI ​​edge intelligent inference component also includes a multimodal deep fusion layer. The multimodal deep fusion layer receives the graph convolution calculation results of the multi-scale spatial feature map and the state topology map, constructs a cross-modal interaction matrix of spatial flow and state flow using a two-stream Transformer architecture, captures the cross-modal correlation between the real-time semantic segmentation mask, the 3D obstacle bounding box and the platform posture change trajectory through multi-head self-attention operation, and generates a unified high-dimensional feature representation vector containing spatial constraints and state constraints.

6. The deep learning-based multimodal sensing system for live-line work according to claim 5, characterized in that, The safety decision output component includes a risk dynamic assessment network and a safety level mapper. The risk dynamic assessment network receives the unified high-dimensional feature representation vector and calculates the interference probability value of the spatial interference between the three-dimensional obstacle bounding box and the platform attitude change trajectory in the current working scenario, as well as the overload probability value of the load distribution matrix exceeding the safety threshold, through a fully connected layer and a softmax activation function. The safety level mapper pre-constructs a two-dimensional risk plane with the interference probability value and the overload probability value as coordinate axes, and outputs discrete safety level judgment results based on the divided regions in the two-dimensional risk plane.

7. The deep learning-based multimodal sensing system for live-line work according to claim 2, characterized in that, The multi-source spatiotemporal synchronous preprocessing subcomponent embeds a joint calibration network. The joint calibration network uses the initial extrinsic matrix of the two-dimensional image stream and the three-dimensional point cloud stream as learnable parameters to construct a similarity loss function based on maximizing mutual information. During the operation of the multi-source spatiotemporal synchronous preprocessing subcomponent, the initial extrinsic matrix is ​​iteratively updated through backpropagation so that the mutual information value between the edge points of the three-dimensional point cloud stream projected onto the plane where the two-dimensional image stream is located and the corresponding gradient edge pixels in the two-dimensional image stream reaches the maximum, thereby obtaining the optimized target extrinsic matrix. The spatiotemporal registration is then performed using the target extrinsic matrix.

8. The deep learning-based multimodal sensing system for live-line work according to claim 3, characterized in that, The cross-attention mechanism includes a spatial position encoder and a feature interaction layer. The spatial position encoder adds multi-head position encoding based on two-dimensional relative position to the image planar features and adds position encoding based on three-dimensional absolute coordinates to the point cloud spatial geometric features. The feature interaction layer uses the point cloud spatial geometric features with added position encoding as a query vector and the image planar features with added position encoding as a key vector and a value vector. It generates a feature weight map through dot product attention calculation and uses the feature weight map to enhance the point cloud spatial geometric features, outputting the multi-scale spatial feature map.

9. The deep learning-based multimodal perception system for live-line operation according to claim 4, wherein the graph convolutional network includes an uncertainty quantization branch, the uncertainty quantization branch superimposes Gaussian noise onto each vertex feature of the state topology graph, inputs the vertex features after superimposing Gaussian noise into a multi-layer graph convolutional layer for forward propagation calculation, repeats the forward propagation calculation a set number of times to obtain multiple prediction samples, performs statistical distribution calculation on the multiple prediction samples, and outputs the mean tensor and covariance matrix of the platform attitude change trajectory, wherein the temporal state prediction network uses the mean tensor as the platform attitude change trajectory and the covariance matrix as the confidence index of the working condition classification label.

10. The deep learning-based multimodal sensing system for live-line work according to claim 6, characterized in that, The safety level mapper includes a dynamic boundary adjustment module. The dynamic boundary adjustment module records the sequence change trajectory of the interference probability value and the overload probability value within a historical period. It uses a sliding window to extract the probability sequence with a set window length before the current moment, calculates the local variance and rate of change of the probability sequence, and when the local variance exceeds the dynamic variance threshold or the rate of change exceeds the dynamic rate of change threshold, the dynamic boundary adjustment module performs translation and scaling processing on the boundary coordinates of the divided region along the coordinate axis direction of the two-dimensional risk plane, and re-outputs the safety level determination result based on the updated boundary after translation and scaling processing.

Images