A method, system, computer, and storage medium for predicting cross-modal spatiotemporal events.

Abstract

This invention provides a cross-modal spatiotemporal event prediction method, system, computer, and storage medium. The method includes the following steps: receiving a multimodal high-level structured semantic feature sequence of monitoring data output from multiple edge nodes; mapping each semantic feature in the feature sequence to a unified embedding space; summing the weighted results of each modality according to a gating fusion mechanism to obtain joint representation parameters; performing average pooling operation on the joint representation parameters corresponding to the time step within the current sliding time window, and inputting the result into a lightweight prediction head to output decision parameters for the probability of event occurrence. By implementing spatiotemporal modeling through a sliding time window and a memory mechanism, the robustness and adaptability of the system are enhanced, detection sensitivity is improved, and false alarms are reduced. This solution, with its low power consumption, low latency, and high accuracy, solves the core bottlenecks of high computational cost, large latency, and poor scalability in existing technologies, providing a sustainable solution for smart city monitoring.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a cross-modal spatiotemporal event prediction method, system, computer, and storage medium. Background Technology

[0002] Urban environments are home to a wide variety of heterogeneous sensing devices, including high-definition cameras, LiDAR, air quality sensors, and noise monitors, which continuously generate multimodal raw data such as video, point clouds, and time-series signals. In existing intelligent analysis systems, the mainstream approach typically adopts an "edge acquisition + cloud-based heavy-duty model" model: edge devices are only responsible for data compression and uploading, while event recognition tasks rely entirely on large-scale deep learning models (such as 3D-CNN, spatiotemporal Transformer, etc.) deployed in the cloud.

[0003] However, this model faces significant bottlenecks in large-scale deployment scenarios:

[0004] High computational cost: Heavy models often have more than 50 million parameters, requiring a large amount of GPU resources and making it difficult to support hundreds of concurrent inferences;

[0005] The latency is insufficient to meet real-time requirements: single-path inference often takes tens to hundreds of milliseconds, which cannot meet the second-level response requirements for sudden events;

[0006] Poor scalability: Each new monitoring channel requires a corresponding increase in cloud resource quota, leading to a linear increase in operation and maintenance costs.

[0007] To reduce cloud load, some solutions attempt to deploy lightweight models (such as MobileNet and Tiny-YOLO) at the edge for end-to-end event prediction. However, limited by edge computing power and model capacity, such methods struggle to effectively model cross-modal correlations and long-term dynamic evolution, leading to a significant decrease in prediction accuracy, especially in low-frequency, high-risk event scenarios.

[0008] Another approach to improvement involves extracting intermediate features from the neural network at the edge (such as the hundreds-dimensional vectors output by CNNs) and uploading them to the cloud to reduce the amount of original data transmitted. However, these features are high-dimensional, redundant, and have unclear physical meanings. The cloud still needs to introduce complex fusion mechanisms (such as attention networks) and heavy-duty temporal modules (such as LSTM and Transformer) to maintain reasonable performance, failing to fundamentally solve the computational overhead problem. Summary of the Invention

[0009] To address the shortcomings of existing technologies, the present invention aims to provide a cross-modal spatiotemporal event prediction method, system, computer, and storage medium, thereby solving the technical problems of high model cost, low prediction efficiency, and high latency in existing technologies.

[0010] To achieve the above objectives, in a first aspect, the present invention provides: a cross-modal spatiotemporal event prediction method, comprising the following steps:

[0011] The system receives a multimodal high-level structured semantic feature sequence of monitoring data output from multiple edge nodes, wherein the multimodal high-level structured semantic feature sequence is low-dimensional data generated by the monitoring devices of the edge nodes through a perception algorithm.

[0012] For each time step, each semantic feature in the feature sequence is mapped to a unified embedding space. Each semantic feature is linearly transformed by a weight vector and a bias. The output is mapped to a confidence interval based on the sigmoid function to generate the confidence weights of the modality.

[0013] Based on the gated fusion mechanism, the confidence weights are multiplied by the semantic features using element-wise multiplication, and the weighted results of each modality are summed to obtain the joint representation parameters:

[0014] The average pooling operation is performed based on the joint representation parameters corresponding to the time step within the current sliding time window, compressing them into a fixed-length average feature vector, which is then input into a lightweight prediction head to output decision parameters for the probability of event occurrence.

[0015] According to one aspect of the above technical solution, the expression for calculating the probability of an event occurring is:

[0016] ;

[0017] In the formula, Let y represent the probability of the event occurring, and y represent the event label. This is the predicted weight vector, where T is the transpose sign and c is the scalar bias. The time window length, For average pooling operation, It is the sigmoid activation function. For joint characterization parameters, These are the joint characterization parameters of the previous time step.

[0018] According to one aspect of the above technical solution, after obtaining the multimodal high-level structured semantic feature sequence, the method further includes:

[0019] A feature refinement model is established to filter semantic features from multimodal high-level structured semantic features. The computational expression for the feature refinement model is as follows:

[0020] ;

[0021] In the formula, To filter and refine the semantic features, X is the optimization variable. For multimodal high-level structured semantic features, Describing the L1 norm, Represents the L2 norm. These are sparse constraint weights.

[0022] According to one aspect of the above technical solution, the expression for calculating the confidence weight is:

[0023] ;

[0024] ;

[0025] In the formula, Here, k represents the confidence weight of the k-th mode at time step t, where k is the mode index and t is the time step. It is the sigmoid activation function. Let T be the learnable weight vector, and let T be the transpose. This is a vector concatenation operation. This is the current memory vector for the gating fusion mechanism. For bias, This is the memory vector from the previous time step. To learn the decay factor, These are the joint characterization parameters of the previous time step.

[0026] According to one aspect of the above technical solution, the calculation expression for the joint characterization parameter is as follows:

[0027] ;

[0028] In the formula, For joint characterization parameters, This represents element-wise multiplication, where m is the total number of modes.

[0029] According to one aspect of the above technical solution, the calculation expression for the multimodal high-level structured semantic features is as follows:

[0030] ;

[0031] In the formula, , respectively, are the feature vectors of image semantic mode, point cloud statistical mode, and sensor reading mode, where d is the unified embedding dimension and m is the total number of modes.

[0032] According to one aspect of the above technical solution, the lightweight prediction head consists of a fully connected layer and a sigmoid activation function.

[0033] Secondly, the present invention provides a cross-modal spatiotemporal event prediction system, comprising:

[0034] The edge acquisition module is used to receive multimodal high-level structured semantic feature sequences of monitoring data output from multiple edge nodes, wherein the multimodal high-level structured semantic feature sequences are low-dimensional data generated by the monitoring devices of the edge nodes through a perception algorithm;

[0035] The confidence module is used to map each semantic feature in the feature sequence to a unified embedding space at each time step. It performs a linear transformation on each semantic feature through a weight vector and a bias, and maps the output to a confidence interval based on the sigmoid function to generate the confidence weights of the modality.

[0036] The joint module is used to multiply the confidence weights by the semantic features through element-wise multiplication according to the gating fusion mechanism, and sum the weighted results of each modality to obtain the joint representation parameters:

[0037] The decision module performs average pooling based on the joint representation parameters corresponding to the time step within the current sliding time window, compresses them into a fixed-length average feature vector, and inputs it into the lightweight prediction head to output decision parameters for the probability of event occurrence.

[0038] According to one aspect of the above technical solution, the system further includes:

[0039] The refinement module is used to build a feature refinement model to filter semantic features from multimodal high-level structured semantic features. The calculation expression for the feature refinement model is as follows:

[0040] ;

[0041] In the formula, To filter and refine the semantic features, X is the optimization variable. For multimodal high-level structured semantic features, Describing the L1 norm, Represents the L2 norm. These are sparse constraint weights.

[0042] Thirdly, the present invention provides a computer, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the cross-modal spatiotemporal event prediction method as described in the above technical solution.

[0043] Fourthly, the present invention provides a storage medium on which a computer program is stored, which, when executed by a processor, implements the cross-modal spatiotemporal event prediction method as described in the above technical solution.

[0044] Compared with existing technologies, the beneficial effects of this invention are as follows: By using high-level structured semantic features as input and combining a lightweight gating fusion mechanism with dynamic adaptive optimization, the following multiple advantages are achieved: First, in terms of performance, the total number of model parameters does not exceed 1.5 million, and it can be run on standard cloud servers (such as NVIDIA T4). The single-path inference latency on GPUs is less than 15 milliseconds, supporting concurrent processing of over 600 edge node requests, ensuring efficient real-time response under large-scale deployment and meeting the second-level early warning requirements for emergencies. Secondly, in terms of cost-effectiveness, compared with traditional heavy model solutions, the monthly cloud computing resource cost is reduced by approximately 45%. At the same time, through end-to-end training and sparse constraint optimization, the model's comprehensive prediction capability on a self-built urban event dataset reaches 91.8%, comparable to mainstream heavy models (93.1%), but avoiding high computational overhead. In addition, the lightweight design of the solution does not contain recurrent neural networks or self-attention structures. Spatiotemporal modeling is achieved through sliding time windows and memory mechanisms, enhancing the robustness and adaptability of the system. It can be widely applied in traffic, environmental, and public safety scenarios, such as automatically optimizing thresholds and resource utilization, improving detection sensitivity, and reducing false alarms. With its low power consumption, low latency, and high accuracy, this solution solves the core bottlenecks of high computational cost, large latency, and poor scalability in existing technologies, providing a sustainable solution for smart city monitoring. Attached Figure Description

[0045] Figure 1 This is a flowchart illustrating the cross-modal spatiotemporal event prediction method in the first embodiment of the present invention;

[0046] Figure 2 This is a structural block diagram of the cross-modal spatiotemporal event prediction system in the second embodiment of the present invention;

[0047] Figure 3 This is a schematic diagram of the hardware structure of the computer in the third embodiment of the present invention;

[0048] The following detailed description, in conjunction with the accompanying drawings, will further illustrate the present invention. Detailed Implementation

[0049] To facilitate understanding of the present invention, a more complete description will be given below with reference to the accompanying drawings. Several embodiments of the invention are illustrated in the drawings. However, the invention can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete.

[0050] It should be noted that when a component is said to be "fixed to" another component, it can be directly on the other component or there may be an intervening component. When a component is said to be "connected to" another component, it can be directly connected to the other component or there may be an intervening component. The terms "vertical," "horizontal," "left," "right," and similar expressions used in this document are for illustrative purposes only.

[0051] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0052] Example 1

[0053] Please see Figure 1 The figure shows a flowchart of the cross-modal spatiotemporal event prediction method in the first embodiment of the present invention. As shown in the figure, the method includes the following steps:

[0054] Step S100: Receive a multimodal high-level structured semantic feature sequence of monitoring data output from multiple edge nodes. The multimodal high-level structured semantic feature sequence is low-dimensional data generated by the monitoring devices of the edge nodes using a perception algorithm. Specifically, the multimodal high-level structured semantic features are low-dimensional, interpretable business-level numerical values ​​(such as target quantity, motion state, and anomaly score) generated by the edge nodes using a perception algorithm, rather than raw sensor data such as original video, point clouds, or sensor time-series signals.

[0055] To make it easy to understand, this solution consists of three parts: the first part is the edge side, which consists of cameras, LiDAR, and temperature / humidity / PM2.5 sensors deployed at intersections, factories, and parks; the second part is the edge side, which consists of local servers or MEC nodes responsible for data caching and optional preprocessing (optional compression or feature extraction of raw data, such as extracting the number of targets, the average speed of ByteTrack tracking trajectories, and population density estimates as semantic features of images); and the third part is the cloud side, which consists of virtual computing instances (such as virtual machines or containerized service nodes based on x86 architecture) deployed on public or private cloud platforms. The edge layer, acting as the perception layer, acquires raw multimodal data (video streams, point cloud frames, sensor time series) and uploads it to the edge via wired / wireless networks. The edge layer, acting as the preprocessing layer, is used for caching multimodal data, optional feature extraction or data compression, and uploading raw data, intermediate features or compressed streams to the cloud. The cloud platform, acting as a lightweight prediction layer, is used for cross-modal fusion, gating weight calculation, spatiotemporal sequence modeling, event probability prediction and alarm triggering engine.

[0056] Step S200: For each time step, map each semantic feature in the feature sequence to a unified embedding space, perform a linear transformation on each semantic feature through a weight vector and a bias, and map the output to a confidence interval based on the sigmoid function to generate the confidence weights of the modality.

[0057] Preferably, in this embodiment, after obtaining the multimodal high-level structured semantic feature sequence, the method further includes:

[0058] A feature refinement model is established to filter semantic features from multimodal high-level structured semantic features. The computational expression for the feature refinement model is as follows:

[0059] ;

[0060] In the formula, To filter and refine the semantic features, X is the optimization variable. For multimodal high-level structured semantic features, Describing the L1 norm, Represents the L2 norm. For sparse constraint weights, after obtaining the multimodal high-level structured semantic feature sequence, a preprocessing step is added, namely, filtering feature dimensions based on sparse constraints to automatically remove redundant or noisy dimensions, retain key information, and enhance the robustness of subsequent fusion.

[0061] The expression for calculating the confidence weight is:

[0062] ;

[0063] ;

[0064] In the formula, Here, k represents the confidence weight of the k-th mode at time step t, where k is the mode index and t is the time step. It is the sigmoid activation function. Let T be the learnable weight vector, and let T be the transpose. This is a vector concatenation operation. This is the current memory vector for the gating fusion mechanism. For bias, This is the memory vector from the previous time step. To learn the decay factor, These are the joint representation parameters from the previous time step. Memory vector. By capturing temporal dependencies through weighted updates, weight calculations can not only respond to current features but also consider historical trends. For example, in public safety scenarios, if historical data consistently shows anomalies, the memory vector will increase the sensitivity of the current weights. This mechanism enables lightweight temporal modeling without the need for recurrent neural networks, meeting the low-latency requirements of documents.

[0065] Specifically, for each semantic feature, a memory vector is used. Learnable weight vector and bias Perform a linear transformation. Then, use the sigmoid function ( σ ( The output is mapped to the (0,1) interval to generate confidence weights. These weights represent the reliability score of the modality; the closer the value is to 1, the greater the contribution of that modality at the current time step. This mechanism replaces complex attention networks, achieving adaptive fusion through lightweight computation. This ensures that the model prioritizes high-reliability modalities in real-time prediction. For example, in traffic monitoring, if an image modality shows an anomaly, its weight will automatically increase, thereby enhancing the accuracy of event detection.

[0066] In step S300, the confidence weights are multiplied by the semantic features using element-wise multiplication according to the gating fusion mechanism, and the weighted results of each modality are summed to obtain the joint representation parameters. The "gating" mechanism refers to dynamically calculating the confidence weights of each modality using learnable parameters to adaptively adjust their contribution to the fusion.

[0067] The calculation expression for the joint characterization parameter is as follows:

[0068] ;

[0069] In the formula, For joint characterization parameters, This represents element-wise multiplication, where m is the total number of modes.

[0070] Step S400: Perform average pooling operation based on the joint representation parameters corresponding to the time steps within the current sliding time window, compress it into a fixed-length average feature vector, and input it into the lightweight prediction head to output decision parameters of the event occurrence probability.

[0071] Specifically, the expression for calculating the probability of the above event is:

[0072] ;

[0073] In the formula, Let y represent the probability of the event occurring, and y represent the event label. This is the predicted weight vector, where T is the transpose sign and c is the scalar bias. The time window length, For average pooling operation, It is the sigmoid activation function. For joint characterization parameters, These are the joint characterization parameters from the previous time step. The length of the sliding time window... This is a configurable parameter, with a value range of 5 to 30 time steps. The "event prediction" in the solution refers to using structured semantic features (such as vehicle speed, spacing, group density, or abnormal behavior indicators) provided by edge nodes, and employing a lightweight cloud-based prediction model to estimate the probability of a target event occurring within a short future window before it actually happens, and proactively triggering an early warning when the risk exceeds a threshold.

[0074] The lightweight prediction head consists of fully connected layers and a sigmoid activation function, and does not contain recurrent neural networks, 3D convolutions, or self-attention mechanisms. When predicting probabilities... Exceeding the preset threshold θ When the value is 0.7, an event alarm is triggered and pushed to the management terminal.

[0075] The above weight vector Bias Prediction weights These parameters are obtained through end-to-end training on labeled event datasets, enabling the model to automatically learn the reliability weights of different modalities in different scenarios, rather than relying on manually set rules or fixed coefficients.

[0076] Furthermore, the aforementioned multimodal high-level structured semantic features are represented as follows:

[0077] ;

[0078] in, These are the feature vectors for the image semantic modality, point cloud statistical modality, and sensor reading modality, respectively. d is the unified embedding dimension, and m is the total number of modalities. Preferably, in some application scenarios of this embodiment, the unified embedding dimension d = 128: that is, the original semantic features of each modality are compressed or expanded into vectors of length 128 through linear mapping, ensuring that multimodal data can be fused and computed in the same space; the number of modalities m = 3 (image semantic, point cloud statistical, sensor reading): indicating that the system simultaneously processes three types of structured information from different sources; time window. (Corresponding to 15 seconds, sampling rate 1Hz), the model refers to the historical feature sequence of the most recent 15 seconds for each prediction; the total number of parameters of the model is about 1.2×106 (that is, about 1.2 million learnable values), which is much smaller than that of a typical Transformer model (usually more than 50 million parameters), significantly reducing the computational burden.

[0079] On a self-built urban event dataset, this invention achieves a comprehensive prediction capability of 91.8% for anomalous events (balancing hit rate and false alarm rate), comparable to mainstream heavyweight models (93.1%). On a standard cloud server (such as a virtual machine equipped with an NVIDIA T4 GPU), a single cloud deployment unit can simultaneously handle real-time prediction requests from more than 600 edge nodes, with an end-to-end inference latency of less than 15 milliseconds. Compared to traditional heavyweight model solutions, this invention reduces the monthly cloud computing resource costs by approximately 45% while maintaining the same prediction performance.

[0080] In summary, the cross-modal spatiotemporal event prediction method in the above embodiments of the present invention achieves the following multiple advantages by using high-level structured semantic features as input and combining a lightweight gating fusion mechanism with dynamic adaptive optimization: First, in terms of performance, the total number of model parameters does not exceed 1.5 million, and it can be implemented on standard cloud servers (such as NVIDIA T4). The single-path inference latency on GPUs is less than 15 milliseconds, supporting concurrent processing of over 600 edge node requests, ensuring efficient real-time response under large-scale deployment and meeting the second-level early warning requirements for emergencies. Secondly, in terms of cost-effectiveness, compared with traditional heavy model solutions, the monthly cloud computing resource cost is reduced by approximately 45%. At the same time, through end-to-end training and sparse constraint optimization, the model's comprehensive prediction capability on a self-built urban event dataset reaches 91.8%, comparable to mainstream heavy models (93.1%), but avoiding high computational overhead. In addition, the lightweight design of the solution does not contain recurrent neural networks or self-attention structures. Spatiotemporal modeling is achieved through sliding time windows and memory mechanisms, enhancing the robustness and adaptability of the system. It can be widely applied in traffic, environmental, and public safety scenarios, such as automatically optimizing thresholds and resource utilization, improving detection sensitivity, and reducing false alarms. With its low power consumption, low latency, and high accuracy, this solution solves the core bottlenecks of high computational cost, large latency, and poor scalability in existing technologies, providing a sustainable solution for smart city monitoring.

[0081] Example 2

[0082] A second embodiment of this application also provides a cross-modal spatiotemporal event prediction system for implementing the embodiments and preferred embodiments, which have already been described and will not be repeated hereafter. As used below, the terms "module," "unit," "subunit," etc., can refer to a combination of software and / or hardware that performs a predetermined function. Although the system described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0083] like Figure 2 As shown, the system includes: an edge acquisition module 100, a confidence module 200, a joint module 300, and a decision module 400.

[0084] The edge acquisition module 100 is used to receive multimodal high-level structured semantic feature sequences of monitoring data output from multiple edge nodes, wherein the multimodal high-level structured semantic feature sequences are low-dimensional data generated by the monitoring devices of the edge nodes through a perception algorithm.

[0085] The confidence module 200 is used to map each semantic feature in the feature sequence to a unified embedding space at each time step, perform a linear transformation on each semantic feature through a weight vector and a bias, and map the output to a confidence interval based on the sigmoid function to generate the confidence weights of the modality.

[0086] The joint module 300 is used to multiply the confidence weights by the semantic features through element-wise multiplication according to the gating fusion mechanism, and sum the weighted results of each modality to obtain the joint representation parameters:

[0087] The decision module 400 is used to perform average pooling operation based on the joint representation parameters corresponding to the time step within the current sliding time window, compress it into a fixed-length average feature vector, and input it into the lightweight prediction head to output decision parameters of the probability of event occurrence.

[0088] Preferably, in this embodiment, the system further includes:

[0089] The refinement module is used to build a feature refinement model to filter semantic features from multimodal high-level structured semantic features. The calculation expression for the feature refinement model is as follows:

[0090] ;

[0091] In the formula, To filter and refine the semantic features, X is the optimization variable. For multimodal high-level structured semantic features, Describing the L1 norm, Represents the L2 norm. These are sparse constraint weights.

[0092] It should be noted that the modules can be functional modules or program modules, and can be implemented in software or hardware. For modules implemented in hardware, the modules can reside in the same processor; or the modules can be located in different processors in any combination.

[0093] Example 3

[0094] A third embodiment of this application provides a computer that may include a processor 81 and a memory 82 storing computer program commands.

[0095] Specifically, the processor 81 may include a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0096] The memory 82 may include a large-capacity storage device for data or commands. For example, and not limitingly, the memory 82 may include a hard disk drive (HDD), a floppy disk drive, a solid-state drive (SSD), flash memory, an optical disk drive, a magneto-optical disk drive, magnetic tape, or a Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, the memory 82 may include removable or non-removable (or fixed) media. Where appropriate, the memory 82 may be internal or external to a data processing device. In a particular embodiment, the memory 82 is non-volatile memory. In a particular embodiment, the memory 82 includes read-only memory (ROM) and random access memory (RAM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), an electrically alterable read-only memory (EAROM), or flash memory, or a combination of two or more of these. Where appropriate, the RAM can be Static Random-Access Memory (SRAM) or Dynamic Random-Access Memory (DRAM). DRAM can be Fast Page Mode Dynamic Random-Access Memory (FPMDRAM), Extended Data Out Dynamic Random-Access Memory (EDODRAM), Synchronous Dynamic Random-Access Memory (SDRAM), etc.

[0097] The memory 82 can be used to store or cache various data files that need to be processed and / or communicated, as well as possible computer program commands executed by the processor 81.

[0098] The processor 81 reads and executes computer program commands stored in the memory 82 to implement any of the cross-modal spatiotemporal event prediction methods in the above embodiments.

[0099] In some embodiments, the computer may further include a communication interface 83 and a bus 80. For example, Figure 3 As shown, the processor 81, memory 82, and communication interface 83 are connected through bus 80 and complete communication with each other.

[0100] The communication interface 83 is used to enable communication between the various modules, devices, units, and / or equipment in the embodiments of this application. The communication interface 83 can also enable data communication with other components such as external devices, image / data acquisition devices, databases, external storage, and image / data processing workstations.

[0101] Bus 80 includes hardware, software, or both, that couples computer components together. Bus 80 includes, but is not limited to, at least one of the following: data bus, address bus, control bus, expansion bus, and local bus. For example, and not as a limitation, bus 80 may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an InfiniBand interconnect, a Low Pin Count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus, or other suitable buses, or a combination of two or more of these. Where appropriate, bus 80 may include one or more buses. Although specific buses are described and illustrated in the embodiments of this application, this application considers any suitable bus or interconnection.

[0102] Example 4

[0103] The fourth embodiment of this application provides a readable storage medium. This readable storage medium stores computer program commands; when executed by a processor, these computer program commands implement any of the cross-modal spatiotemporal event prediction methods described in the above embodiments.

[0104] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0105] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A method for predicting cross-modal spatiotemporal events, characterized in that, Includes the following steps: The system receives a multimodal high-level structured semantic feature sequence of monitoring data output from multiple edge nodes, wherein the multimodal high-level structured semantic feature sequence is low-dimensional data generated by the monitoring devices of the edge nodes through a perception algorithm. For each time step, each semantic feature in the feature sequence is mapped to a unified embedding space. Each semantic feature is linearly transformed by a weight vector and a bias. The output is mapped to a confidence interval based on the sigmoid function to generate the confidence weights of the modality. Based on the gated fusion mechanism, the confidence weights are multiplied by the semantic features using element-wise multiplication, and the weighted results of each modality are summed to obtain the joint representation parameters: The average pooling operation is performed based on the joint representation parameters corresponding to the time step within the current sliding time window, which is compressed into a fixed-length average feature vector and input into a lightweight prediction head to output decision parameters for the probability of event occurrence. After obtaining the multimodal high-level structured semantic feature sequence, the method further includes: A feature refinement model is established to filter semantic features from multimodal high-level structured semantic features. The computational expression for the feature refinement model is as follows: ； In the formula, To filter and refine the semantic features, X is the optimization variable. The multimodal high-level structured semantic features obtained from the k-th modality at time step t. Describing the L1 norm, Represents the L2 norm. For sparse constraint weights; The expression for calculating the confidence weight is: ； ； In the formula, Here, k represents the confidence weight of the k-th mode at time step t, where k is the mode index and t is the time step. It is the sigmoid activation function. Let T be the learnable weight vector corresponding to the k-th mode, and let T be the transpose. This is a vector concatenation operation. This is the current memory vector for the gating fusion mechanism. This is the bias corresponding to the k-th mode. This is the memory vector from the previous time step. To learn the decay factor, These are the joint characterization parameters of the previous time step; The calculation expression for the joint characterization parameter is as follows: ； In the formula, For joint characterization parameters, This represents element-wise multiplication, where m is the total number of modes; The lightweight prediction head consists of a fully connected layer and a sigmoid activation function. The calculation expression for the multimodal high-level structured semantic features is as follows: ； In the formula, , respectively, are the feature vectors of image semantic mode, point cloud statistical mode, and sensor reading mode, where d is the unified embedding dimension and m is the total number of modes.

2. The cross-modal spatiotemporal event prediction method according to claim 1, characterized in that, The expression for calculating the probability of an event occurring is: ； In the formula, Let y represent the probability of the event occurring, and y represent the event label. This is the predicted weight vector, where T is the transpose sign and c is the scalar bias. The time window length, For average pooling operation, It is the sigmoid activation function. Let be the joint characterization parameters at time step t. For time step Joint characterization parameters at time.

3. A cross-modal spatiotemporal event prediction system for implementing the cross-modal spatiotemporal event prediction method according to any one of claims 1-2, characterized in that, include: The edge acquisition module is used to receive multimodal high-level structured semantic feature sequences of monitoring data output from multiple edge nodes, wherein the multimodal high-level structured semantic feature sequences are low-dimensional data generated by the monitoring devices of the edge nodes through a perception algorithm; The confidence module is used to map each semantic feature in the feature sequence to a unified embedding space at each time step. It performs a linear transformation on each semantic feature through a weight vector and a bias, and maps the output to a confidence interval based on the sigmoid function to generate the confidence weights of the modality. The joint module is used to multiply the confidence weights by the semantic features through element-wise multiplication according to the gating fusion mechanism, and sum the weighted results of each modality to obtain the joint representation parameters: The decision module performs average pooling based on the joint representation parameters corresponding to the time step within the current sliding time window, compresses them into a fixed-length average feature vector, and inputs it into the lightweight prediction head to output decision parameters for the probability of event occurrence.

4. A computer, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the cross-modal spatiotemporal event prediction method as described in any one of claims 1-2.

5. A storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the cross-modal spatiotemporal event prediction method as described in any one of claims 1-2 above.

Images