Deep fusion method for multi-modal data based on spatio-temporal semantic graph convolution network

Abstract

The application discloses a multi-modal data deep fusion method based on a space-time semantic graph convolution network, and belongs to the technical field of data fusion methods, and comprises the following steps: obtaining a multi-modal data set after space-time alignment, constructing a space-time semantic graph, performing preliminary judgment on transition smoothing risk, setting initial working parameters of a space-time graph convolution network, inputting the space-time semantic graph into the space-time graph convolution network for reasoning, performing adaptive optimization on space-time graph convolution network reasoning, outputting a feature vector of a target object node after the reasoning is completed, decoding a fusion state of the target object, and outputting a globally consistent fusion result. The application can avoid the problem of excessive smoothing of node features in the space-time convolution reasoning process, and maintain the sensitivity of burst state and dynamic details. Meanwhile, in the case that multi-modal information exists conflict or is missing, a globally consistent and reliable fusion result is outputted, and the recognition accuracy, response speed and generalization ability of the model in a complex dynamic scene are improved.

Description

Technical Field

[0001] This invention relates to the field of data fusion methods, and in particular to a multimodal data deep fusion method based on spatiotemporal semantic graph convolutional networks. Background Technology

[0002] With the rapid growth in demand for multimodal data acquisition and application, researchers widely adopt graph-based modeling methods to achieve the fusion and analysis of complex scene information. Among these methods, the construction of spatiotemporal semantic graphs has become an important technique. This involves aligning and unifying the representation of multi-source heterogeneous data across temporal and spatial dimensions, transforming multimodal data such as sensor data, images, speech, text, and geographic location information into graph-structured data with spatiotemporal attributes and semantic relationships. To further enhance the processing capabilities of graph-structured data, Graph Convolutional Networks (GCNs) are widely used in multimodal fusion tasks. These networks effectively extract local structural features and global semantic information by performing convolution operations within the neighborhood of nodes. Multimodal data processing methods based on spatiotemporal semantic graph convolutional networks can capture temporal evolution relationships while preserving spatial topological features, thus supporting applications such as event prediction, pattern recognition, and intelligent decision-making.

[0003] For example, Chinese invention patent CN109583569B discloses a multimodal feature fusion method and apparatus based on convolutional neural networks, comprising: extracting features of multiple modalities from different heterogeneous images to obtain a first feature set for each modality; in the multimodal convolutional neural network, selecting features that meet preset conditions from the first feature set of each modality based on the correlation between different modalities to obtain a second feature set for each modality; in the fully connected layer of the multimodal convolutional neural network, determining the weights of the second feature sets of each modality, and fusing the second feature sets of multiple modalities according to the weights, so that the fused second feature sets can be used to train the multimodal convolutional neural network for biometric recognition.

[0004] For example, Chinese invention patent CN117592556B discloses a semi-federated learning system and its operation method based on GNN. The system includes a cloud service center, multiple edge servers, and terminal devices. There is a bidirectional connection between the edge servers and the cloud service center, and a bidirectional link between the edge servers and the terminal devices. The edge servers analyze the terminal devices that need to upload local data, issue transmission decisions, determine the terminal devices for local training, and receive model parameters uploaded by these devices for aggregation. Simultaneously, it receives local data from terminal devices that cannot perform local training, performs deduplication, and then trains the model. The resulting model parameters are then aggregated and uploaded to the cloud service center. This invention improves the system's inclusiveness and the model's performance.

[0005] The above-mentioned technology has at least the following technical problems:

[0006] In the inference process of spatiotemporal semantic graph convolutional networks, spatial graph convolution and temporal series modeling are usually performed simultaneously, forming a strongly coupled spatiotemporal feature propagation mechanism. Spatial graph convolution inherently has a low-pass filtering effect, which weights and averages the features of adjacent nodes, thereby enhancing local consistency. Similarly, temporal modeling methods also have temporal low-pass characteristics, aggregating node states across consecutive time steps to extract smooth temporal evolution features. When these two low-pass effects are superimposed, it can easily lead to the problem of over-smoothing of features, especially in dynamic scenes. For example, when temporal edges are too dense or the receptive field of the temporal convolution kernel is too large, states at different time steps are over-fused, and instantaneous changes may be masked, making the response to sudden events appear as a slow, continuous change. When the weight of the temporal residual path is high, the dominance of historical states on the current state is enhanced, further weakening the expression of abrupt signals. When the number of graph convolutional layers is too deep, the information between nodes gradually becomes homogenized during long-path propagation, leading to a weakening of individual differences. The combined effect of these mechanisms may cause the model to exhibit response lag, identification confusion, or prediction bias when dealing with tasks such as drastic state changes, sudden behavior detection, or multi-target recognition, thus affecting its applicability and generalization performance in highly dynamic and complex environments. Summary of the Invention

[0007] To address the aforementioned technical problems in existing technologies, this invention provides a method for deep fusion of multimodal data based on spatiotemporal semantic graph convolutional networks. The technical solution is as follows:

[0008] S1: Obtain the raw data of the target object through a heterogeneous sensor set, and perform spatiotemporal reference alignment on the raw data of the target object to obtain a spatiotemporally aligned multimodal dataset.

[0009] S2. Based on the spatiotemporally aligned multimodal dataset, a spatiotemporal semantic graph is constructed, and a preliminary judgment on the transition smoothing risk is made. Based on the preliminary judgment result of the transition smoothing risk, the initial working parameters of the spatiotemporal graph convolutional network are set, and a start inference signal is generated after optimization.

[0010] S3, after receiving the inference start signal, inputs the spatiotemporal semantic graph into the spatiotemporal graph convolutional network for inference, evaluates the smoothness of the spatiotemporal semantic graph at a preset period, performs adaptive optimization of the spatiotemporal graph convolutional network inference, and outputs the feature vector of the target object node after inference is completed, decodes the fusion state of the target object, performs intelligent arbitration based on the confidence score, and outputs a globally consistent fusion result.

[0011] The beneficial effects of the technical solutions provided in the embodiments of the present invention include at least the following:

[0012] 1. The multimodal data deep fusion method based on spatiotemporal semantic graph convolutional network provided by this invention can achieve high-precision spatiotemporal alignment on multimodal data collected by heterogeneous sensors. By introducing an oversmoothing risk index, comprehensive smoothness evaluation and adaptive optimization mechanism, it avoids the problem of excessive smoothing of node features during spatiotemporal convolutional inference and maintains sensitivity to sudden states and dynamic details. At the same time, combined with the decoding and confidence intelligent arbitration mechanism, it can output globally consistent and reliable fusion results when there are conflicts or missing multimodal information, thereby improving the recognition accuracy, response speed and generalization ability of the model in complex dynamic scenarios.

[0013] 2. This invention sets the initial working parameters of the spatiotemporal graph convolutional network based on the preliminary judgment results of the transition smoothing risk. It can adaptively adjust key parameters such as residual injection intensity and edge sparsity ratio according to different risk levels of high, medium and low. It suppresses the occurrence of feature over-smoothing before inference starts, thereby ensuring that the model can maintain good feature resolution and dynamic response capability in the initial stage, and improve the stability and accuracy of the subsequent inference process.

[0014] 3. This invention evaluates the smoothness of the spatiotemporal semantic graph at a preset period and dynamically adjusts the edge confidence threshold and residual injection intensity based on the comprehensive smoothness deviation value when excessive smoothness is detected. This enables continuous suppression of feature homogenization during inference and maintains sensitivity to sudden changes and detailed features, thereby achieving adaptive optimization of spatiotemporal graph convolutional network inference and ensuring the accuracy and robustness of the fusion results.

[0015] 4. This invention utilizes the powerful information propagation and aggregation capabilities of graph convolutional networks. When errors, conflicts, or missing data occur in local sensor data, the system can automatically correct, arbitrate, and infer to complete the data based on global, high-confidence contextual information, thereby improving the robustness and anti-interference capabilities of perception.

[0016] 5. The end-to-end processing method based on deep learning models avoids a large number of manually designed fusion rules, enabling the system to adaptively learn the optimal fusion strategy in different scenarios through data-driven methods, and has stronger generalization ability. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a schematic diagram of the multimodal data deep fusion method based on spatiotemporal semantic graph convolutional network provided in the embodiments of the present invention.

[0019] Figure 2 This is a flowchart of the preliminary judgment and execution process for oversmoothing in an embodiment of the present invention.

[0020] Figure 3 This is a flowchart of the smoothness evaluation process involved in the embodiments of the present invention.

[0021] Figure 4 This is a diagram of the multimodal data deep fusion inference architecture involved in the embodiments of the present invention. Detailed Implementation

[0022] The technical solution of the present invention will now be described with reference to the accompanying drawings.

[0023] In embodiments of the present invention, words such as "exemplarily," "for example," etc., are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" in the present invention should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the word "exemplary" is intended to present the concept in a concrete manner. Furthermore, in embodiments of the present invention, the meaning expressed by "and / or" can be both, or either one.

[0024] In this embodiment of the invention, "image" and "picture" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction, their intended meanings are consistent. The terms “corresponding” and “relevant” can sometimes be used interchangeably. It should be noted that when the distinction is not emphasized, they convey the same meaning.

[0025] In embodiments of the present invention, sometimes the subscript is as follows: It may be written in a non-subscript form, such as When the distinction is not emphasized, the meaning they express is the same.

[0026] To make the technical problems, technical solutions and advantages of the present invention clearer, the following will describe in detail with reference to the accompanying drawings, taking the holographic perception of material pallets in a smart warehousing scenario as a specific embodiment.

[0027] like Figure 4 The diagram shown is a multimodal data deep fusion inference architecture diagram involved in an embodiment of the present invention. The overall architecture includes a data input layer, a spatiotemporal modeling layer, a graph structure inference layer, a spatiotemporal graph convolutional layer, a fusion inference layer, and a state output and arbitration layer, constituting an end-to-end deep multimodal state fusion process.

[0028] First, in the raw data input stage, a heterogeneous set of sensors deployed in different scenarios collects raw data of the target object at each time step t∈[1,T]. Abnormally missing data (marked with red nodes) is identified and recorded, and a missing data completion mechanism (marked with purple nodes) is introduced to form a unified input data sequence. To achieve spatial consistency of multi-source data, the system performs a unified spatial information transformation on the raw data of all target objects, mapping it to a preset world coordinate system, thereby completing spatiotemporal benchmark alignment.

[0029] In the spatiotemporal modeling stage, the aligned input data sequence X1, X2, ..., XT, along with its original semantic graph structure A, is fed into the multimodal spatiotemporal modeling unit. This module integrates a set of spatiotemporal feature extraction structures based on Long Short-Term Memory (LSTM) networks, which can simultaneously capture the evolution trend of the target state in the temporal dimension and spatial adjacency relationship, and output the basic spatiotemporal feature embedding representation.

[0030] To more realistically reflect the spatiotemporal relationships of the target object in a dynamic environment, this system adopts an adaptive graph modeling mechanism. Based on the spatiotemporal embedding features output in the previous stage, the graph structure at each time step is dynamically constructed. This structure not only maintains basic consistency with the original graph structure A, but also has robust reconstruction capabilities for instantaneous anomalies, occlusion, and sensor loss.

[0031] In the spatiotemporal graph convolution stage, the constructed dynamic graph structure is used to perform graph convolution operations on the features at each time step, realizing the aggregation and propagation of information between adjacent nodes in the spatial dimension, and linking with the features in the time dimension to form a strongly coupled spatiotemporal feature representation.

[0032] The above graph convolution results are fed into the fusion state inference module, which integrates the convolution outputs from multiple time steps, extracts stable target state features, and then proceeds to the state output and arbitration stage. The system integrates a confidence-based intelligent arbitration mechanism to evaluate and rank the confidence levels of multiple modal or path outputs, automatically adjudicates contradictory states according to set criteria, and outputs a globally consistent fusion situation map.

[0033] Furthermore, the right side of the system architecture includes a set of spatiotemporal graph convolutional stacked units (S refers to the number of spatiotemporal graph convolutional stacked units). This structure supports multi-layer graph convolution operations and node reconstruction mechanisms, and introduces spectral clustering to achieve node clustering initialization, further improving graph modeling efficiency and feature representation capabilities. It includes a multimodal memory feature matrix and an arbitration scoring mechanism, strengthening the model's ability to judge long-term feature dependencies and modality credibility.

[0034] like Figure 1 The flowchart shown is a multimodal data deep fusion method based on spatiotemporal semantic graph convolutional networks, which includes the following steps:

[0035] S1: Obtain the raw data of the target object through a heterogeneous sensor set, and perform spatiotemporal reference alignment on the raw data of the target object to obtain a spatiotemporally aligned multimodal dataset.

[0036] Furthermore, the spatiotemporally aligned multimodal dataset is obtained, and the specific process is as follows:

[0037] A1. In the target scenario, deploy a heterogeneous set of sensors to acquire raw data of the target object.

[0038] In a warehouse setting, a heterogeneous sensor suite including top-view cameras, LiDAR, and RFID readers is deployed to collect images, point clouds, and tag information of material pallets, respectively.

[0039] In this embodiment, the camera is installed 6 meters above the warehouse ceiling, with a 120-degree field of view, covering an area of ​​6m × 8m, and outputting 1920 × 1080 image frames. The AGV is equipped with a 2D LiDAR with a sampling frequency of 10Hz, a scanning radius of 3 meters, and outputs a point cloud format of [x,y,d] triplets. The RFID system uses passive UHF tags with a reading range of approximately 2.5 meters, and the antenna is installed in a fixed channel position.

[0040] It should be noted that the above-mentioned data limitations are only for the specific implementation of this embodiment. In specific implementation scenarios, specific settings can be made based on the relevant characteristics of the implementation scenario. This embodiment does not impose any limitations.

[0041] Visual targets are detected by the YOLOv5 network to identify the rectangular bounding box of the pallet. The pallet structure outline is extracted from the point cloud by DBSCAN clustering combined with height features. The RFID reading value directly corresponds to each pallet ID and is bound to a position in space.

[0042] A2 assigns high-precision timestamps to the raw data of the target object. Each sensor is synchronized through a network time protocol to ensure that the accuracy of the collected timestamps reaches the microsecond level, so as to achieve time consistency across devices.

[0043] In this embodiment, all devices are connected to an internal industrial Ethernet network and synchronized via the IEEE 1588 precision time protocol, with the error controlled within 50 microseconds.

[0044] A3 converts the spatial information of the target object's original data to a preset world coordinate system. Through sensor calibration, it converts the image pixel coordinates, LiDAR local coordinates, and RFID sensing area into a unified warehouse world coordinate system.

[0045] In this embodiment, the camera uses a pinhole model and acquires intrinsic and extrinsic parameters through the Tsai dual calibration method to achieve pixel coordinates to world coordinates mapping. The radar uses AGV odometer and IMU position for positioning, and the point cloud is fitted to the ground using RANSAC to convert it into world coordinates. The RFID antenna records the correspondence between its working area and world coordinates through a spatial configuration database.

[0046] A4, based on the allocated high-precision timestamps and the converted spatial information, performs spatiotemporal reference alignment on the raw data of the target object from the heterogeneous sensor set to obtain the spatiotemporally aligned multimodal dataset.

[0047] Spatiotemporally aligned multimodal datasets refer to fused data sets that are collected by multiple heterogeneous sensors under the same time reference and unified spatial coordinate system, and have associated target object identifiers and spatial states. They are used to support subsequent graph construction and state decoding tasks.

[0048] In this embodiment, the spatiotemporally aligned multimodal dataset includes image data structure items, point cloud data structure items, RFID data structure items, time tag items, and spatial reference items.

[0049] The image data structure item contains visual recognition information for each tray target in the image frame acquired by the top-view camera. The visual recognition information includes, but is not limited to, image coordinates, the position of the center point in the world coordinate system, the size of the bounding rectangle, color or texture feature vectors, etc.

[0050] The point cloud data structure item contains point cloud clusters corresponding to the material pallet in the scene point cloud data collected by LiDAR. After spatial filtering and clustering, the pallet boundary contour, height value, contour features and spatial distribution information in the world coordinate system are extracted from the point cloud clusters.

[0051] The RFID data structure item contains the pallet tag ID information collected by the RFID reader at the same timestamp. The pallet tag ID information is bound to the positioning result of the tag spatial activation area in the world coordinate system to obtain the correspondence between ID and spatial location.

[0052] The time stamp is a high-precision timestamp uniformly assigned to the aforementioned image data, point cloud data, and RFID data. It is used to identify the time point of collection and ensure the alignment of multimodal data within a microsecond time scale.

[0053] The spatial reference term represents the coordinate information of all modal data that has been transformed to a unified world coordinate system, and records the mapping relationship or transformation matrix from the original local coordinates of each mode to the world coordinate system.

[0054] S2. Based on the spatiotemporally aligned multimodal dataset, a spatiotemporal semantic graph is constructed, and a preliminary judgment on the transition smoothing risk is made. Based on the preliminary judgment result of the transition smoothing risk, the initial working parameters of the spatiotemporal graph convolutional network are set, and a start inference signal is generated after optimization.

[0055] In this embodiment, the spatiotemporal semantic graph not only reflects the static features of the tray under multimodal perception, but also reflects its dynamic behavior in dimensions such as temporal evolution, spatial relationships, and environmental interactions through various semantic edge definitions. The graph structure can be modeled in the form of heterogeneous graphs. By defining node type labels and edge type labels, different types of entities and their relationships are encoded and input into the subsequent graph convolutional network model for feature propagation and state reasoning.

[0056] In this embodiment, the spatiotemporal semantic graph has the following structural features: based on the different entities existing in the scene, the nodes in the graph are defined as material nodes, sensor nodes, and environment nodes.

[0057] Here, each Pallet Node represents a pallet object detected by the sensing system, with node names such as Pallet_A, Pallet_B, etc. The initial feature vector of this node is formed by fusing multimodal data, including but not limited to: appearance features (such as color and edges) extracted from image data through a convolutional neural network, geometric attributes such as size features and contour shape obtained from point cloud clustering, and identity ID codes read from RFID tags.

[0058] Sensor nodes represent sensing devices used to acquire data, such as Camera_1 and LiDAR_AGV1. Their feature vectors may include the sensor's spatial location (in world coordinates), device number, acquisition frequency, and current operating status.

[0059] Environment nodes represent non-target objects in the scene that have a semantic relationship with the pallet, such as warehouse shelves (Shelf_C), transport equipment (AGV_1), buffer zones, and outbound ports. Their node features may include location information, entity category encoding, and interaction state.

[0060] Furthermore, a preliminary assessment of the transition smoothing risk is conducted, and the specific assessment process is as follows:

[0061] Based on spatiotemporal semantic graphs, graph smoothing risk representation data of target objects are obtained, and oversmoothing risk index is analyzed.

[0062] Based on the spatiotemporal semantic graph, graph smoothing risk representation data of the target object is obtained. Specifically, this involves: counting the number of connecting edges of the target object nodes and calculating the average number of connections of the graph nodes; classifying the adjacent nodes of the target node into different modalities, counting the proportion of nodes in each modality, and calculating the maximum difference in the proportion of nodes in each modality; traversing the set of adjacent nodes of the target node, obtaining the adjacent node with the highest connectivity, and recording its degree as the maximum degree of the graph node.

[0063] like Figure 2 The diagram shows the preliminary judgment and execution flowchart of oversmoothing according to an embodiment of the present invention. First, the oversmoothing risk index of the target object is calculated and compared with a preset minimum threshold and a maximum threshold for oversmoothing risk. When the oversmoothing risk index is greater than or equal to the maximum threshold, it is determined to be high oversmoothing risk, and the oversmoothing risk deviation index is calculated. The working parameter optimization set is then queried, and the initial working parameter settings for the spatiotemporal graph convolutional network are finally completed. When the oversmoothing risk index is greater than the minimum threshold but less than the maximum threshold, it is determined to be medium oversmoothing risk. Simultaneously, a smoothing risk warning message is generated, and the initial parameter settings are directly completed based on the initial working parameter set of the spatiotemporal graph convolutional network. When the oversmoothing risk index is less than or equal to the minimum threshold, it is determined to be low oversmoothing risk, and the initial parameter settings are directly completed based on the initial working parameter set of the spatiotemporal graph convolutional network.

[0064] Furthermore, the oversmoothing risk index is implemented as follows:

[0065] The graph smoothing risk characterization data for the target object includes the average number of graph nodes connected, the maximum proportion difference in the number of graph modal nodes, and the maximum degree of the graph nodes.

[0066] The average number of connections per graph node refers to the average number of connections between the corresponding node of a target object and other nodes in the constructed spatiotemporal semantic graph. Specifically, for all graph nodes related to the target object, the number of their adjacent edges is counted, and the average value is taken to reflect the degree of information interaction of the target in the graph structure. The larger the average number of connections per graph node, the denser the information coupling between nodes in the graph, and the more prone it is to feature oversmoothing during convolutional propagation.

[0067] The maximum modal node ratio difference refers to the maximum difference in the proportion of nodes of different modal types (e.g., visual modality, laser modality, RFID modality, etc.) among all adjacent nodes of the node corresponding to the target object. The maximum modal node ratio difference measures the degree of structural imbalance in the modal distribution among adjacent nodes. When the number of nodes in one modality is significantly greater than that of other modalities, modal bias may occur during information aggregation in the graph neural network, leading to an imbalance in feature representation and exacerbating the risk of over-smoothing of the local graph.

[0068] The maximum degree of a node in a graph refers to the number of connections made to the node with the highest degree among all first-order adjacent nodes of the target node. The maximum degree reflects whether the target node is directly associated with highly connected central nodes in the graph. If the target node is connected to nodes with extremely high degrees, these highly connected nodes have a strong dominant role in information transmission within the graph neural network, potentially leading to over-assimilation of the target node and resulting in local information convergence or representational collapse.

[0069] The average number of graph nodes of the target object, the maximum ratio difference of the number of graph modal nodes, and the maximum degree of the graph node are compared with the corresponding reference values. Then, a smoothing risk allocation factor is introduced to perform weighted coupling processing on the comparison results, thereby obtaining the oversmoothing risk index.

[0070] The oversmoothing risk index is a quantitative representation of the impact of the average number of graph nodes, the maximum ratio difference of the number of graph modal nodes, and the maximum degree of the graph on the degree of oversmoothing risk of the target object.

[0071] Extract the average number of nodes in the reference graph, the maximum ratio difference in the number of modal nodes in the reference graph, and the maximum degree of the node in the reference graph stored in the database.

[0072] Extract the pre-defined risk allocation factors from the database, including the average number of connections of graph nodes, the maximum ratio difference of the number of graph modal nodes, and the maximum degree of the graph nodes.

[0073] In this embodiment, the system database pre-stores configuration data for multiple risk allocation factors used for quantitative assessment of graph structure smoothness risk. These factors characterize the weighting of different graph structure parameters (in this embodiment, average node connectivity, maximum difference in modal node numbers, and maximum node degree) in the overall smoothness risk score calculation. The configuration data is managed in the form of a structured factor parameter set. This parameter set is determined comprehensively based on experimental results of smoothing algorithms running on a large amount of graph data of different scales and topological features, analysis of the impact of key node structural differences on algorithm stability, and verification conclusions regarding the correlation between smoothness risk score and graph structure convergence efficiency. Based on this structured factor parameter set, the system can directly read the smoothness risk allocation factors for average node connectivity, maximum difference in modal node numbers, and maximum node degree from the database before constructing the smoothness risk scoring function and calculating the target graph structure smoothness risk score. Each allocation factor is a real value between 0 and 1, and the sum of the three is 1. This ensures that when the comparison results of each graph structure parameter and the corresponding reference value are weighted and coupled, the influence ratio of different graph structure parameters on the smoothing risk score is stable and the quantification logic is rigorous, so as to ensure that the final output smoothing risk score can accurately reflect the potential risk level of the target graph structure in the smoothing process.

[0074] In a specific embodiment, the oversmoothing risk index is represented as follows:

[0075] ,

[0076] Where C is the oversmoothing risk index, a is the average number of nodes in the target graph, b is the maximum proportion difference of the number of nodes in the target graph modalities, c is the maximum degree of the node in the target graph, a0 is the average number of nodes in the reference graph, b0 is the maximum proportion difference of the number of nodes in the reference graph modalities, c0 is the maximum degree of the node in the reference graph, s1 is the smoothing risk allocation factor for the average number of nodes in the graph, s2 is the smoothing risk allocation factor for the maximum proportion difference of the number of nodes in the graph modalities, and s3 is the smoothing risk allocation factor for the maximum degree of the node in the graph.

[0077] Extract the preset minimum threshold and maximum threshold of the oversmoothing risk index.

[0078] If the oversmoothing risk index is greater than or equal to the maximum threshold of the oversmoothing risk index, the preliminary judgment result of the oversmoothing risk is recorded as high oversmoothing risk.

[0079] If the oversmoothing risk index is greater than or equal to the maximum threshold, it indicates a severe oversmoothing risk in the spatiotemporal semantic graph of the target object. Quantitatively, this means that the combined impact of the three key indicators—average number of connections per graph node, maximum ratio difference in the number of nodes in graph modalities, and maximum degree of the graph—on the oversmoothing risk exceeds the highest warning line after weighted coupling. In this case, the differences in node features of the spatiotemporal semantic graph will be significantly weakened, potentially leading to feature confusion during subsequent spatiotemporal graph convolutional network inference, making it impossible to accurately capture the effective information of the target object. Therefore, targeted parameter optimization measures should be prioritized to reduce this risk.

[0080] If the oversmoothing risk index is greater than the minimum threshold of the oversmoothing risk index but less than the maximum threshold of the oversmoothing risk index, then the preliminary judgment result of the oversmoothing risk is recorded as medium oversmoothing risk.

[0081] If the oversmoothing risk index is greater than the minimum threshold but less than the maximum threshold, it indicates that the spatiotemporal semantic graph of the target object has a moderate degree of oversmoothing risk. Specifically, the combined effect of the average number of connections per graph node, the maximum ratio difference in the number of nodes in different graph modalities, and the maximum degree of the graph node keeps the oversmoothing risk within an acceptable but concerning range. At this point, although the node features of the spatiotemporal semantic graph have not shown serious confusion, they have begun to show a convergence trend. If this is not paid attention to, the risk may escalate further during subsequent inference. Therefore, when setting the initial parameters of the spatiotemporal graph convolutional network, it is necessary to generate smoothing risk warning information to monitor risk changes in real time.

[0082] If the oversmoothing risk index is less than or equal to the minimum threshold of the oversmoothing risk index, the preliminary judgment result of the oversmoothing risk is recorded as low oversmoothing risk.

[0083] If the oversmoothing risk index is less than or equal to the minimum threshold of the oversmoothing risk index, it indicates that the spatiotemporal semantic graph of the target object has almost no oversmoothing risk. The coupling results of the average number of connections of graph nodes, the maximum ratio difference of the number of nodes in graph modalities, and the maximum degree of the graph node have a minimal impact on the oversmoothing risk. The node features of the spatiotemporal semantic graph retain clear differences and uniqueness, which can provide high-quality input data for subsequent spatiotemporal graph convolutional network inference. No additional complex risk control measures are required, and the inference process can be started directly based on the preset initial working parameters.

[0084] Furthermore, the initial operating parameters of the spatiotemporal graph convolutional network are set, specifically as follows:

[0085] Extract the preset set of initial working parameters for the spatiotemporal graph convolutional network.

[0086] In this embodiment, the initial working parameter set of the spatiotemporal graph convolutional network refers to a set of key initial parameters set before the constructed spatiotemporal semantic graph is input into the graph convolutional neural network model for feature propagation and state decoding. These parameters are used to control the model structure, propagation strategy, and convergence behavior. They include, but are not limited to, the number of graph convolutional layers, the size of the time window, the initial feature dimension, the edge sparsity ratio, the modality fusion method, the activation function type, the residual injection strength, the dropout rate, the node normalization method, and the propagation depth limit.

[0087] If the initial assessment of oversmoothing risk is high oversmoothing risk, the oversmoothing risk deviation index is obtained by subtracting the maximum threshold of the oversmoothing risk index from the oversmoothing risk index.

[0088] Using the oversmoothing risk deviation index as the query key, the set of optimized working parameters is queried, and then combined with the initial working parameter set of the spatiotemporal graph convolutional network, the initial working parameters of the spatiotemporal graph convolutional network are set.

[0089] The optimized set of working parameters consists of residual injection strength supplement values ​​and edge sparsity ratio supplement values.

[0090] A pre-built working parameter optimization mapping database is invoked. This database pre-stores the mapping relationships between different oversmoothing risk deviation index intervals and their corresponding working parameter optimization values. The mapping relationships are constructed based on a large amount of experimental data, following the rule that the degree of risk deviation is positively correlated with the intensity of parameter adjustment: when the oversmoothing risk deviation index is small, the database stores low-level residual injection intensity supplementary values ​​and low-level edge sparsity ratio supplementary values; when the oversmoothing risk deviation index is large, the database stores high-level residual injection intensity supplementary values ​​and high-level edge sparsity ratio supplementary values. Using the calculated oversmoothing risk deviation index as the query key, interval matching is performed in the aforementioned working parameter optimization mapping database. After locating the deviation interval to which the index belongs, the corresponding residual injection intensity supplementary values ​​and edge sparsity ratio supplementary values ​​can be extracted, completing the query of the working parameter optimization set.

[0091] If the initial assessment of oversmoothing risk is medium or low, the initial working parameters of the spatiotemporal graph convolutional network are set directly based on the initial working parameter set of the spatiotemporal graph convolutional network, and a smoothing risk warning message is generated when there is medium oversmoothing risk.

[0092] S3, after receiving the inference start signal, inputs the spatiotemporal semantic graph into the spatiotemporal graph convolutional network for inference, evaluates the smoothness of the spatiotemporal semantic graph at a preset period, performs adaptive optimization of the spatiotemporal graph convolutional network inference, and outputs the feature vector of the target object node after inference is completed, decodes the fusion state of the target object, performs intelligent arbitration based on the confidence score, and outputs a globally consistent fusion result.

[0093] Once the system receives the inference start signal, it inputs the spatiotemporal semantic graph G_t, which has been constructed at the current time t, into the preset spatiotemporal graph convolutional network model for inference processing. This spatiotemporal semantic graph uses multimodal heterogeneous nodes (material pallets, sensors, environment) as graph nodes, and connects physical proximity, observation relationships, temporal continuity, and logical interaction relationships as graph edges, fully reflecting the spatiotemporal association structure of target objects in the warehousing scenario.

[0094] In this embodiment, the input to the preset spatiotemporal graph convolutional network model refers to using the spatiotemporal semantic graph constructed from multimodal data after spatiotemporal benchmark alignment as structured input, which is then fed into a pre-trained spatiotemporal graph convolutional neural network model with fixed parameters to perform feature extraction and fusion judgment of the target object state. This preset spatiotemporal graph convolutional network model is prepared before system deployment through offline training, parameter tuning, and structural optimization, as follows: First, developers select a suitable spatiotemporal graph neural network architecture (such as ST-GCN, ASTGCN, MDGCN, etc.) based on specific task requirements such as target recognition, trajectory prediction, and behavior judgment, and set structural parameters such as the number of graph convolutional layers, temporal convolutional kernel size, activation function, and residual pathways. Using the collected large-scale heterogeneous data (such as vision, RFID, accelerometer, etc.), a sample graph structure is constructed in a unified spatiotemporal coordinate system, and feature annotation and label pairing are performed. Then, based on the above samples, repeated training and validation are performed to determine the optimal learning rate, edge sparsity ratio, residual injection strength and other key hyperparameters to improve the model's generalization ability and dynamic adaptability. Finally, after the performance meets the deployment requirements, it is used as the standard model for the inference stage.

[0095] Once the system receives the processed spatiotemporal semantic graph, it uses it as input to the pre-defined spatiotemporal graph convolutional network model. The model extracts multi-scale dynamic features of target object nodes through spatial adjacency graph convolution and temporal convolution paths, and fuses multimodal information to achieve high-confidence identification and prediction of node states, providing a structured representation foundation for subsequent state decoding and intelligent arbitration.

[0096] During inference, a graph convolution operation is first performed in the spatial dimension. Taking a pallet node, Pallet_A, as an example, the model aggregates feature information from its neighboring nodes, including visual features from camera nodes (such as Camera_1), contour geometric features from LiDAR nodes (such as LiDAR_AGV1), and contextual information from neighboring material nodes (such as Pallet_B). This spatial graph convolution process fuses information from different modalities through edge weighting, enabling node features to achieve cross-modal complementarity and enhancement within a unified graph structure.

[0097] Subsequently, in the time dimension, the model processes the time series {G_{tn},...,G_t} composed of multiple historical frames. Through temporal convolution or recursion mechanisms, it learns the evolution of target node features over time, and then infers their dynamic behavior, such as velocity estimation and future location prediction. This temporal feature modeling helps improve the ability to capture changes in material state during continuous operations.

[0098] To improve reasoning quality and structural adaptability, the system performs a graph smoothness evaluation once at a preset cycle.

[0099] like Figure 3 The diagram shows a smoothness evaluation flowchart according to an embodiment of the present invention. First, the overall smoothness is calculated. Then, it is determined whether there is smoothing risk warning information in the system. If so, the first smoothness threshold correction coefficient is retrieved using the oversmoothing risk index as the query key, the oversmoothing risk deviation factor is calculated, and then the second smoothness threshold correction coefficient is retrieved using the oversmoothing risk deviation factor as the query key. Finally, the overall smoothness correction threshold is calculated based on the first smoothness threshold correction coefficient, the second smoothness threshold correction coefficient, and the overall smoothness threshold. If there is no smoothing risk warning information, only the first smoothness threshold correction coefficient is retrieved using the oversmoothing risk index as the query key, and the overall smoothness correction threshold is obtained based on this correction coefficient and the overall smoothness threshold. Finally, the overall smoothness is compared with the overall smoothness correction threshold. If the overall smoothness is lower than the overall smoothness correction threshold, it is determined to be normal smoothing; otherwise, it is determined to be oversmoothing.

[0100] Furthermore, the smoothness of the spatiotemporal semantic graph is evaluated at a preset period. The specific analysis process is as follows:

[0101] The smoothing state data of the spatiotemporal semantic graph during the inference process is obtained at a preset period, and the overall smoothness is analyzed.

[0102] Furthermore, the overall smoothness is analyzed, and the specific analysis process is as follows:

[0103] Smoothed state data of the spatiotemporal semantic graph, including Dirichlet energy, Fisher discriminant ratio, and effective rank of the node embedding matrix.

[0104] The reference values ​​of Dirichlet energy, Fisher discriminant ratio, and effective rank of node embedding matrix are compared with the reference values ​​of Dirichlet energy, Fisher discriminant ratio, and effective rank of node embedding matrix, respectively. Then, a smoothness influence factor is introduced to perform weighted coupling processing on the results of each comparison, thereby obtaining the comprehensive smoothness.

[0105] The overall smoothness is a quantitative representation of the influence of Dirichlet energy, Fisher discriminant ratio, and effective rank of the node embedding matrix on the smoothness of the inference state.

[0106] Dirichlet energy is used to measure the degree of variation in the features of nodes in a graph, representing the smoothness of the features within the graph structure. Specifically, for a graph node embedding matrix H and its corresponding graph Laplacian matrix L, the Dirichlet energy is defined as: The lower the Dirichlet energy, the more similar the features of adjacent nodes are, indicating that the graph may be overly smoothed at the current propagation level.

[0107] The Fisher discriminant ratio measures the separability of features between nodes of different classes. It is defined as the ratio of between-class variance to within-class variance, i.e., F = Tr(S_B) / Tr(S_W), where F represents the Fisher discriminant ratio, S_B represents the between-class scatter matrix, and S_W represents the within-class scatter matrix. A higher Fisher discriminant ratio indicates that the embedded features of nodes of different classes are more easily distinguished, and the model has stronger expressive power. If the Fisher discriminant ratio is too low, it indicates that the node features have a tendency to aggregate and degenerate, and there may be oversmoothing.

[0108] The effective rank of a node embedding matrix measures the dimensionality utilization of the node feature space. It is calculated as follows: Perform singular value decomposition on the node embedding matrix, take the probability distribution p formed by the normalized singular values, calculate its entropy, and convert it into an equivalent dimension: r_eff = exp(H(p)). Here, r_eff represents the effective rank of the node embedding matrix, and H(p) represents the Shannon entropy of the probability distribution p. The effective rank of the node embedding matrix reflects the number of active dimensions in the embedding space. The lower the effective rank, the more consistent the features of most nodes tend to be, resulting in a decrease in information representation ability.

[0109] Extract the reference Dirichlet energy, reference Fisher discriminant ratio, and effective rank of the reference node embedding matrix stored in the database.

[0110] Extract the pre-defined Dirichlet energy smoothness influence factor, Fisher discriminant ratio smoothness influence factor, and node embedding matrix effective rank smoothness influence factor from the database.

[0111] In this embodiment, the system database pre-stores configuration data for multiple smoothness influence factors used in the quantitative analysis of graph structure smoothness. These factors characterize the weight ratio of different graph structure parameters (Dirichlet energy, Fisher discriminant ratio, and effective rank of the node embedding matrix in this embodiment) in the overall smoothness score calculation. The configuration data is managed in the form of a structured factor parameter set. This parameter set is determined comprehensively based on the smoothing optimization experimental results of a large number of different graph structure data, the sensitivity analysis results of each parameter to the smoothness evaluation index, and the verification conclusions regarding the correlation between the smoothness score and the actual optimization convergence efficiency. Based on this parameter set, the system can directly read the Dirichlet energy smoothness influence factor, Fisher discriminant ratio smoothness influence factor, and effective rank of the node embedding matrix smoothness influence factor from the database before constructing the smoothness scoring function and calculating the smoothness score of the target graph structure. The influence factor is a real value between 0 and 1, and the sum of the three is 1. This ensures that when the comparison results of each parameter and the corresponding reference value are weighted and coupled, the influence ratio of different parameters on the smoothness score is stable and the quantification logic is rigorous, so that the final output smoothness score can accurately reflect the complexity characteristics of the target graph structure in the smoothing optimization and representation learning tasks.

[0112] In a specific embodiment, the overall smoothness is represented as follows:

[0113] ,

[0114] Where B is the overall smoothness, E is the Dirichlet energy, F is the Fisher discriminant ratio, r_eff is the effective rank of the node embedding matrix, E0 is the reference Dirichlet energy, F0 is the reference Fisher discriminant ratio, r_eff0 is the effective rank of the reference node embedding matrix, r1 is the smoothness influence factor of the Dirichlet energy, r2 is the smoothness influence factor of the Fisher discriminant ratio, and r3 is the smoothness influence factor of the effective rank of the node embedding matrix.

[0115] Extract the preset overall smoothness threshold from the database.

[0116] Using the oversmoothing risk index as the query key, query the first smoothness threshold correction coefficient.

[0117] The system invokes a pre-built mapping library for first smoothness threshold correction coefficients. This library is pre-trained with extensive experimental data and stores a one-to-one mapping relationship between different oversmoothing risk index intervals and their corresponding first smoothness threshold correction coefficients. This mapping relationship follows a negative correlation rule between the oversmoothing risk index and the first smoothness threshold correction coefficient: the higher the oversmoothing risk index, the smaller the corresponding first smoothness threshold correction coefficient; conversely, the lower the oversmoothing risk index, the larger the corresponding first smoothness threshold correction coefficient. This negative correlation makes the subsequently calculated comprehensive smoothness correction threshold more sensitive, thereby making the judgment conditions for oversmoothing more stringent, and thus identifying and preventing potential oversmoothing problems in advance. Using the obtained oversmoothing risk index as the query key, interval matching is performed in the aforementioned mapping library for first smoothness threshold correction coefficients. After locating the preset interval to which the oversmoothing risk index belongs, the first smoothness threshold correction coefficient corresponding to that interval is extracted, thus completing the query operation.

[0118] The system checks if there is a smoothing risk warning. If so, it uses the oversmoothing risk deviation factor obtained by subtracting the oversmoothing risk index from the minimum threshold of the oversmoothing risk index as the query key. It then queries the second smoothness threshold correction coefficient. Based on the first smoothness threshold correction coefficient, the second smoothness threshold correction coefficient, and the comprehensive smoothness threshold, it analyzes and obtains the comprehensive smoothness correction threshold, which is the product of the first smoothness threshold correction coefficient, the second smoothness threshold correction coefficient, and the comprehensive smoothness threshold.

[0119] It should be noted that the oversmoothing risk deviation factor is the numerical result of subtracting the minimum threshold of the oversmoothing risk index from the oversmoothing risk index.

[0120] A pre-built second smoothness threshold correction coefficient mapping library is invoked. This library is pre-trained based on experimental data from multiple sets of spatiotemporal semantic graph samples and stores a one-to-one mapping relationship between different oversmoothing risk deviation factor intervals and their corresponding second smoothness threshold correction coefficients. The mapping relationship follows a negative correlation rule between the oversmoothing risk deviation factor and the second smoothness threshold correction coefficient; that is, the larger the oversmoothing risk deviation factor, the smaller the corresponding second smoothness threshold correction coefficient. This is to further improve the sensitivity of the comprehensive smoothness correction threshold through coefficient adjustment, ensuring accurate capture of smoothing state changes in scenarios with oversmoothing risk. The calculated oversmoothing risk deviation factor is used as the query key, and interval matching is performed in the aforementioned second smoothness threshold correction coefficient mapping library. After locating the preset interval to which the deviation factor belongs, the second smoothness threshold correction coefficient corresponding to that interval is extracted, thus completing the query operation for the second smoothness threshold correction coefficient.

[0121] It should be noted that the values ​​of both the first and second smoothness threshold correction coefficients are in the range of 0-1.

[0122] If there is no smoothing risk warning information, the comprehensive smoothing correction threshold is obtained based on the first smoothing threshold correction coefficient and the comprehensive smoothing threshold. That is, the product of the first smoothing threshold correction coefficient and the comprehensive smoothing threshold is used as the comprehensive smoothing correction threshold.

[0123] If the overall smoothness is lower than the overall smoothness correction threshold, the smoothness evaluation result of the spatiotemporal semantic graph will be recorded as normal smoothness.

[0124] If the overall smoothness is lower than the overall smoothness correction threshold, it indicates that the overall smoothness is a quantified value obtained by comparing the Dirichlet energy, Fisher discriminant ratio, and effective rank of the node embedding matrix of the spatiotemporal semantic graph with corresponding reference values ​​and introducing a smoothness influence factor for weighted coupling. This value accurately represents the smoothness state of the spatiotemporal semantic graph during inference. The overall smoothness correction threshold is a judgment criterion calculated by combining the first smoothness threshold correction coefficient, the second smoothness threshold correction coefficient (when there is a smoothing risk warning information), and a preset overall smoothness threshold. When the overall smoothness is lower than this correction threshold, it indicates that the Dirichlet energy, Fisher discriminant ratio, and effective rank of the node embedding matrix of the current spatiotemporal semantic graph do not exhibit excessive smoothing. The uniqueness and differences of its node features are effectively preserved, providing high-quality data input for subsequent inference of the spatiotemporal graph convolutional network; therefore, it is judged as normal smoothing.

[0125] If the overall smoothness is not lower than the overall smoothness correction threshold, the smoothness evaluation result of the spatiotemporal semantic graph is recorded as oversmooth.

[0126] If the overall smoothness is not lower than the overall smoothness correction threshold, it indicates that the Dirichlet energy, Fisher discriminant ratio, and effective rank of the node embedding matrix of the spatiotemporal semantic graph are working together to cause oversmoothing in the spatiotemporal semantic graph. In this case, the differences in node features in the spatiotemporal semantic graph are significantly weakened, which will directly affect the accuracy and effectiveness of the inference results of the spatiotemporal graph convolutional network, and is therefore judged as oversmoothing.

[0127] Furthermore, adaptive optimization of the spatiotemporal graph convolutional network inference is performed, and the specific analysis process is as follows:

[0128] If the smoothness evaluation result of the spatiotemporal semantic graph is normal smoothness, then the inference process continues.

[0129] If the smoothness evaluation result of the spatiotemporal semantic graph is oversmooth, the difference between the overall smoothness and the overall smoothness correction threshold is processed to obtain the overall smoothness deviation value, that is, the overall smoothness minus the overall smoothness correction threshold is used as the overall smoothness deviation.

[0130] Use the overall smoothness deviation value as the query key to query the set of optimized inference parameters for spatiotemporal graph convolutional networks.

[0131] The set of optimized inference parameters for spatiotemporal graph convolutional networks includes edge confidence threshold supplementation values ​​and residual injection strength supplementation values.

[0132] A pre-built spatiotemporal graph convolutional network inference parameter optimization mapping library is invoked. This mapping library is generated based on experimental data under multiple sets of different oversmoothing scenarios. It internally stores a one-to-one mapping relationship between different comprehensive smoothness deviation value ranges and corresponding inference parameter optimization values. The mapping relationship follows the rule that the comprehensive smoothness deviation value is positively correlated with the parameter optimization intensity: that is, the larger the comprehensive smoothness deviation value (indicating a more severe degree of oversmoothing), the larger the corresponding edge confidence threshold supplement value (to improve the edge selection standard, remove more low-confidence redundant connections, and reduce feature homogenization transmission), and the larger the residual injection intensity supplement value (to further enhance the ability of nodes to retain original features and alleviate the feature convergence problem); conversely, the smaller the comprehensive smoothness deviation value (indicating a milder degree of oversmoothing), the smaller the corresponding edge confidence threshold supplement value and residual injection intensity supplement value are matched to avoid excessive parameter adjustment from interfering with the inference process. Using the calculated comprehensive smoothness deviation value as the query key, interval matching is performed in the aforementioned spatiotemporal graph convolutional network inference parameter optimization mapping library. After locating the preset interval to which the comprehensive smoothness deviation value belongs, the edge confidence threshold supplement value and residual injection intensity supplement value corresponding to the interval are extracted, thus completing the query of the spatiotemporal graph convolutional network inference parameter optimization set.

[0133] Adaptive optimization of spatiotemporal graph convolutional network inference based on spatiotemporal graph convolutional network inference parameter optimization set.

[0134] After completing graph feature propagation in both spatial and temporal dimensions, the system outputs an updated feature vector for each target object node. This feature vector integrates state information from the multimodal perception system and possesses high dynamic stability and semantic expressiveness. It is then used as input for subsequent fusion state decoding, task execution instruction generation, or abnormal behavior recognition.

[0135] Furthermore, the fusion state of the target object is decoded, and the specific execution process is as follows:

[0136] The feature vectors of each node of the target object are input into the decoding module, and the embedded features are restored into the state results by the built-in decoder of the decoding module.

[0137] The decoding module can adopt a multilayer perceptron structure, using a Softmax classifier for the identity dimension and a regression structure to output continuous values ​​for the pose and velocity dimensions.

[0138] The output formats for each state result include, but are not limited to: identity as a one-dimensional discrete value, position as a three-dimensional vector (x, y, z), attitude as an Euler angle triple (roll, pitch, yaw), and velocity as a two-dimensional vector (vx, vy).

[0139] The validity of each state result is verified, including confidence filtering and outlier removal, and the verified state results are encapsulated into a structured data format.

[0140] The fusion results can be packaged into a structured JSON format, with fields including ID, Position, Orientation, Velocity, and Confidence. It supports integration with downstream API interfaces of the warehouse control system or reporting to the intelligent scheduling module.

[0141] Confidence filtering uses a threshold method; results with a confidence level below 0.7 will be filtered out. Outlier removal is based on the standard deviation of the historical sliding window mean; when the deviation exceeds twice the standard deviation, it is marked as an outlier.

[0142] It should be noted that the above numerical judgment is based only on example data in this embodiment. In specific implementation scenarios, specific settings can be made according to the relevant characteristics of the corresponding implementation scenario. This embodiment does not impose any limitations.

[0143] Furthermore, intelligent arbitration is performed based on the confidence score to output a globally consistent fusion result. The specific execution process is as follows:

[0144] The results of each state of the target object are compared to determine whether there are any conflicts in key state dimensions such as identity and pose.

[0145] Check if any modality is missing at the current time step and mark the state dimensions that need to be completed for inference.

[0146] Summarize the state values ​​of each state result and their corresponding confidence scores to construct a fusion judgment matrix.

[0147] The fusion judgment matrix is ​​a two-dimensional matrix consisting of a state dimension and a modality source, where each cell records the predicted value of a certain modality in a certain state dimension and the corresponding confidence level.

[0148] The historical state sequence of the target object, the graph structure association strength between nodes, and the confidence propagation path are introduced as auxiliary judgment criteria.

[0149] Historical state sequences typically retain state vectors from the last 5 to 10 frames and are dynamically updated using a sliding window mechanism to predict trends and support missing reasoning.

[0150] The confidence propagation path is constructed based on the edge weight structure in the graph. The transfer coefficient of confidence from high-quality modes to associated nodes is calculated through the graph attention mechanism to strengthen the dominant role of high-confidence nodes.

[0151] In the event of a conflict, the modality with the highest confidence level is adopted as the dominant outcome.

[0152] During training, the ST-GCN model automatically learns the weight contributions of different modalities in different state dimensions through backpropagation. This means that the model gives a greater weight to the RFID node input in the identity dimension than to the visual node input, thus automatically biasing towards the data source with higher confidence when there is a conflict.

[0153] When a missing dimension exists, the missing dimension is inferred by combining the spatial characteristics and historical evolution trends of adjacent nodes.

[0154] When the information is consistent and the confidence score meets the threshold requirement, the fusion result is output directly.

[0155] The final output includes the fused identity, pose, velocity, and other state information, along with the corresponding confidence scores, forming a globally consistent fusion result.

[0156] The following example illustrates conflict handling in the identity recognition dimension. Suppose that at a certain moment, the visual sensor Camera_1, due to dirt on the tray surface, misidentifies the actual tray as Pallet_C, while the RFID system accurately reads the electronic tag information of the tray as Pallet_A. At this point, a conflict arises between different modalities in the identity dimension of the same target object.

[0157] During graph convolutional inference in the spatiotemporal graph convolutional network model, Camera_1 and RFID, acting as sensor nodes, establish observation edges with the target object Pallet_A, and transmit their respective identity feature information through edge weights. Since the model has mastered the confidence differences of each modality across different state dimensions through supervised learning during the training phase, it recognizes that the confidence of the RFID signal in the identity dimension is significantly higher than that of the visual modality. Therefore, during graph feature propagation, the model adaptively increases the feature aggregation weights from the RFID node, suppressing information interference from misidentified paths of Camera_1.

[0158] Ultimately, the updated features of the target node Pallet_A primarily reflect the correct identity information Pallet_A from the high-confidence channel, while mitigating the impact of erroneous visual input. This conflict arbitration process does not rely on manually defined rule logic; it is automatically learned by the graph neural network through end-to-end training, demonstrating the model's adaptive processing capability for conflict information in multimodal inputs and its robustness in data-driven integration.

[0159] The above embodiments can be implemented, in whole or in part, by software, hardware (such as circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the flow or function according to the embodiments of the present invention is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. Computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. A computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. Available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media. Semiconductor media can be solid-state drives.

[0160] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. A and B can be singular or plural. Additionally, the character " / " in this article generally indicates an "or" relationship between the preceding and following related objects, but it can also represent an "and / or" relationship. Please refer to the context for a more accurate understanding.

[0161] In this invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of a single item or a plurality of items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be a single item or multiple items.

[0162] It should be understood that, in various embodiments of the present invention, the order of the above-mentioned process numbers does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0163] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0164] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the devices, apparatuses, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0165] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0166] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0167] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for deep fusion of multimodal data based on spatiotemporal semantic graph convolutional networks, characterized in that, Includes the following steps: S1. Obtain the raw data of the target object through a heterogeneous sensor set, and perform spatiotemporal reference alignment on the raw data of the target object to obtain a spatiotemporally aligned multimodal dataset; S2. Based on the spatiotemporally aligned multimodal dataset, construct a spatiotemporal semantic graph, make a preliminary judgment on the transition smoothing risk, set the initial working parameters of the spatiotemporal graph convolutional network based on the preliminary judgment results of the transition smoothing risk, and generate a start inference signal after optimization. S3, after receiving the inference start signal, input the spatiotemporal semantic graph into the spatiotemporal graph convolutional network for inference, evaluate the smoothness of the spatiotemporal semantic graph at a preset period, perform adaptive optimization of the spatiotemporal graph convolutional network inference, and output the feature vector of the target object node after inference is completed, decode the fusion state of the target object, and perform intelligent arbitration based on the confidence score to output a globally consistent fusion result. The smoothness evaluation of the spatiotemporal semantic graph at a preset period is specifically analyzed as follows: The smoothing state data of the spatiotemporal semantic graph during the inference process is acquired at a preset period, and the overall smoothness is analyzed. Extract the preset overall smoothness threshold from the database; Using the oversmoothing risk index as the query key, query the first smoothness threshold correction coefficient; The system checks whether there is a smoothing risk warning. If there is, the system uses the oversmoothing risk deviation factor obtained by subtracting the oversmoothing risk index from the minimum threshold of the oversmoothing risk index as the query key. The system then queries the second smoothness threshold correction coefficient. Based on the first smoothness threshold correction coefficient, the second smoothness threshold correction coefficient, and the comprehensive smoothness threshold, the system analyzes and obtains the comprehensive smoothness correction threshold. That is, the product of the first smoothness threshold correction coefficient, the second smoothness threshold correction coefficient, and the comprehensive smoothness threshold is used as the comprehensive smoothness correction threshold. If there is no smoothing risk warning information, the comprehensive smoothing correction threshold is obtained by analyzing the first smoothness threshold correction coefficient and the comprehensive smoothness threshold. If the overall smoothness is lower than the overall smoothness correction threshold, the smoothness evaluation result of the spatiotemporal semantic graph will be recorded as normal smoothness. If the overall smoothness is not lower than the overall smoothness correction threshold, the smoothness evaluation result of the spatiotemporal semantic graph is recorded as oversmooth. The adaptive optimization for inference using spatiotemporal graph convolutional networks is analyzed in detail below: If the smoothness evaluation result of the spatiotemporal semantic graph is normal smoothness, then continue to execute the inference process; If the smoothness evaluation result of the spatiotemporal semantic graph is oversmooth, the difference between the overall smoothness and the overall smoothness correction threshold is processed to obtain the overall smoothness deviation value, that is, the overall smoothness minus the overall smoothness correction threshold is used as the overall smoothness deviation. Use the overall smoothness deviation value as the query key to query the set of optimized inference parameters for spatiotemporal graph convolutional networks; The set of optimized inference parameters for the spatiotemporal graph convolutional network includes edge confidence threshold supplementation values ​​and residual injection strength supplementation values. Adaptive optimization of spatiotemporal graph convolutional network inference based on spatiotemporal graph convolutional network inference parameter optimization set.

2. The multimodal data deep fusion method based on spatiotemporal semantic graph convolutional networks according to claim 1, characterized in that, The specific process for obtaining the spatiotemporally aligned multimodal dataset is as follows: A1. In the target scene, deploy a heterogeneous set of sensors to acquire raw data of the target object; A2 assigns a high-precision timestamp to the original data of the target object; A3, uniformly converts the spatial information of the target object's original data to the preset world coordinate system; A4, based on the allocated high-precision timestamps and the converted spatial information, performs spatiotemporal reference alignment on the raw data of the target object from the heterogeneous sensor set to obtain the spatiotemporally aligned multimodal dataset.

3. The multimodal data deep fusion method based on spatiotemporal semantic graph convolutional networks according to claim 1, characterized in that, The preliminary assessment of transition smoothing risk is conducted, and the specific assessment process is as follows: Based on spatiotemporal semantic graphs, graph smoothing risk representation data of target objects are obtained, and oversmoothing risk index is analyzed. Extract the preset minimum threshold and maximum threshold of the oversmoothing risk index; If the oversmoothing risk index is greater than or equal to the maximum threshold of the oversmoothing risk index, the preliminary judgment result of the oversmoothing risk is recorded as high oversmoothing risk. If the oversmoothing risk index is greater than the minimum threshold of the oversmoothing risk index but less than the maximum threshold of the oversmoothing risk index, then the preliminary judgment result of the oversmoothing risk is recorded as medium oversmoothing risk. If the oversmoothing risk index is less than or equal to the minimum threshold of the oversmoothing risk index, the preliminary judgment result of the oversmoothing risk is recorded as low oversmoothing risk.

4. The multimodal data deep fusion method based on spatiotemporal semantic graph convolutional networks according to claim 3, characterized in that, The specific implementation process of the oversmoothing risk index is as follows: The graph smoothing risk characterization data of the target object includes the average number of graph nodes connected to the target object, the maximum ratio difference of the number of graph modal nodes, and the maximum degree of the graph nodes. The average number of graph nodes of the target object, the maximum ratio difference of the number of graph modal nodes, and the maximum degree of the graph node are compared with the corresponding reference values. Then, a smoothing risk allocation factor is introduced to perform weighted coupling processing on the comparison results, thereby obtaining the oversmoothing risk index. The oversmoothing risk index is a quantitative representation of the impact of the target object's average number of graph nodes, the maximum ratio difference of the number of graph modal nodes, and the maximum degree of the graph node on the degree of oversmoothing risk of the target object.

5. The multimodal data deep fusion method based on spatiotemporal semantic graph convolutional networks according to claim 1, characterized in that, The specific process for setting the initial operating parameters of the spatiotemporal graph convolutional network is as follows: Extract the pre-defined set of initial working parameters for the spatiotemporal graph convolutional network; If the initial assessment result of the oversmoothing risk is high oversmoothing risk, the oversmoothing risk deviation index is obtained by taking the difference between the oversmoothing risk index and the maximum threshold of the oversmoothing risk index. Using the oversmoothing risk deviation index as the query key, the set of optimized working parameters is queried, and then combined with the initial working parameter set of the spatiotemporal graph convolutional network, the initial working parameters of the spatiotemporal graph convolutional network are set. The optimized set of working parameters consists of residual injection strength supplementary values ​​and edge sparsity ratio supplementary values. If the initial assessment of oversmoothing risk is medium or low, the initial working parameters of the spatiotemporal graph convolutional network are set directly based on the initial working parameter set of the spatiotemporal graph convolutional network, and a smoothing risk warning message is generated when there is medium oversmoothing risk.

6. The multimodal data deep fusion method based on spatiotemporal semantic graph convolutional networks according to claim 1, characterized in that, The analysis integrates smoothness, and the specific analysis process is as follows: The smoothed state data of the spatiotemporal semantic graph includes Dirichlet energy, Fisher discriminant ratio, and effective rank of the node embedding matrix; The reference values ​​of Dirichlet energy, Fisher discriminant ratio, and effective rank of node embedding matrix are compared with the reference values ​​of Dirichlet energy, Fisher discriminant ratio, and effective rank of node embedding matrix, respectively. Then, a smoothness influence factor is introduced to perform weighted coupling processing on the results of each comparison, so as to obtain the comprehensive smoothness. The overall smoothness is a quantitative representation of the influence of Dirichlet energy, Fisher discriminant ratio, and effective rank of the node embedding matrix on the smoothness of the inference state.

7. The multimodal data deep fusion method based on spatiotemporal semantic graph convolutional networks according to claim 1, characterized in that, The decoding process yields the fusion state of the target object, and the specific execution process is as follows: The feature vectors of each node of the target object are input into the decoding module, and the embedded features are restored into the state results by the built-in decoder of the decoding module. The validity of each state result is verified, including confidence filtering and outlier removal, and the verified state results are encapsulated into a structured data format.

8. The multimodal data deep fusion method based on spatiotemporal semantic graph convolutional networks according to claim 1, characterized in that, The intelligent arbitration based on the confidence score outputs a globally consistent fusion result. The specific execution process is as follows: Compare the various state results of the target object to determine whether there are any conflicts in the key state dimensions, including identity and pose. Check if any modality is missing at the current time step and mark the state dimensions that need to be completed for inference. Summarize the state values ​​of each state result and their corresponding confidence scores to construct a fusion judgment matrix; The historical state sequence of the target object, the graph structure association strength between nodes, and the confidence propagation path are introduced as auxiliary judgment criteria. In the event of a conflict, the modality with the highest confidence level is adopted as the dominant result. When a missing dimension exists, the missing dimension is inferred by combining the spatial characteristics and historical evolution trends of adjacent nodes. When the information is consistent and the confidence score meets the threshold requirement, the fusion result is output directly; The final output includes state information such as identity, pose, and velocity, along with corresponding confidence scores, forming a globally consistent fusion result.

Images