Data processing method, system, and storage medium

Abstract

The application provides a data processing method, system and storage medium, and relates to the technical field of data processing. The method comprises the following steps: acquiring a plurality of data streams, and determining a corresponding feature extraction strategy based on the data modalities of the plurality of data streams; the data modalities comprise a video data modality, an audio data modality and a sensor data modality; for each data stream, acquiring a feature vector of the data stream according to the feature extraction strategy of the data stream; determining a feature matching degree between the plurality of data streams based on the corresponding feature vectors of the plurality of data streams; in the case that the feature matching degree of any two data streams is greater than a matching threshold, triggering a target event; the target event comprises any one of an alarm event, a state change event and a control instruction event.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a data processing method, system and storage medium. Background Technology

[0002] In complex application scenarios such as smart parks and industrial IoT, the ability to perform real-time correlation analysis and processing of multimodal and heterogeneous data, including video, audio, sensor data, and text, has become a key technical requirement. Such data typically exhibits characteristics such as diverse protocols, heterogeneous formats, and strong scenario dependence.

[0003] In related technologies, general data stream processing frameworks are difficult to adapt to efficient real-time processing of such diverse and non-standardized data. In particular, when data traffic experiences peak fluctuations (such as a surge in video streams due to large events) or when business scenarios need to be dynamically switched (such as switching from routine operation and maintenance mode to emergency response mode), their processing efficiency, system flexibility, and resource utilization are all insufficient. Summary of the Invention

[0004] This application provides a data processing method, system, and storage medium to improve processing efficiency, system flexibility, and resource utilization.

[0005] To achieve the above objectives, this application adopts the following technical solution: In a first aspect, this application provides a data processing method, the method comprising: acquiring multiple data streams and determining corresponding feature extraction strategies based on the data modes of the multiple data streams; the data modes include video data modes, audio data modes, and sensor data modes; for each data stream, acquiring a feature vector of the data stream according to the feature extraction strategy of the data stream; determining the feature matching degree between the multiple data streams based on the feature vectors corresponding to the multiple data streams; and triggering a target event when the feature matching degree between any two data streams is greater than a matching threshold; the target event includes any one of an alarm event, a state change event, and a control command event.

[0006] In one possible implementation, the feature matching degree between multiple data streams is determined based on the feature vectors corresponding to multiple data streams, including: obtaining feature vectors within a sliding time window from the feature vectors corresponding to multiple data streams based on a sliding time window; mapping the feature vectors within the sliding time window to a common feature metric space to obtain the feature representation of the feature vectors within the sliding time window in the common feature metric space; determining the similarity between the feature representations from different data streams in the common feature metric space, and using the similarity as the feature matching degree of the feature representations.

[0007] In one possible implementation, for each data stream, a feature vector of the data stream is obtained according to the feature extraction strategy of the data stream, including: obtaining the real-time resource utilization of multiple computing nodes; determining the computation type of the feature extraction operator corresponding to the data mode of the data stream; determining a target computing node from multiple computing nodes based on the real-time resource utilization, wherein the target computing node has hardware computing resources that match the computation type; and running the corresponding feature extraction operator on the target computing node to obtain the feature vector.

[0008] In one possible implementation, a target computing node is determined from multiple computing nodes based on real-time resource utilization, including: in response to the resource utilization of a first node running a first type of feature extraction operator exceeding a load threshold, a second node that matches the computing type of the first type of feature extraction operator is determined as the target computing node.

[0009] In one possible implementation, a corresponding feature extraction strategy is determined based on the data modalities of multiple data streams, including: when the data stream is a video data stream, determining the feature extraction strategy as a video feature extraction operator; when the data stream is an audio data stream, determining the feature extraction strategy as an audio feature extraction operator; and when the data stream is a sensor data stream, determining the feature extraction strategy as a sensor feature extraction operator.

[0010] In one possible implementation, the data modality of a data stream is determined by: performing protocol parsing and data format analysis on each data stream to obtain analysis results; the analysis results include the communication protocol of the data stream and the encoding format of the data stream's payload content; and based on the analysis results, the data modality of the data stream is determined.

[0011] In one possible implementation, after determining the data modality to which the data stream belongs, the method further includes: adding meta tags to the data units in the data stream according to the data modality; the meta tags include at least one of the following: data source device identifier, data acquisition timestamp, and spatial location information of the data source.

[0012] In one possible implementation, the method further includes: receiving rule logic input through a configuration interface; the rule logic being used to indicate the conditions for triggering a target event; applying the rule logic to the running data processing flow and updating the original rule logic; wherein updating the rule logic only restarts the data processing steps related to the rule judgment.

[0013] In one possible implementation, the rule logic has a corresponding version identifier; the method also includes: in response to a rollback instruction, rolling back the currently running rule logic to the historical rule logic based on the version identifier.

[0014] In one possible implementation, the method further includes: obtaining multiple processing rule templates corresponding to different business scenarios; in response to a scenario switching instruction, determining a target template from the multiple processing rule templates; switching the currently running rule logic to the rule logic defined in the target template, and adjusting the parameters in the feature extraction strategy.

[0015] In one possible implementation, different business scenarios include daily office scenarios, exhibition scenarios, and emergency security scenarios; when switching from a daily office scenario to an emergency security scenario, the parameters to be adjusted include the analysis frame rate parameter of the video data stream and / or the matching threshold parameter of the temperature data in the sensor data stream.

[0016] Secondly, a data processing system is provided, comprising: a processing unit and a communication unit; the communication unit is configured to: acquire multiple data streams and determine corresponding feature extraction strategies based on the data modalities of the multiple data streams; the data modalities include video data modalities, audio data modalities, and sensor data modalities; the processing unit is configured to: for each data stream, acquire a feature vector of the data stream according to the feature extraction strategy of the data stream; the processing unit is further configured to: determine the feature matching degree between the multiple data streams based on the feature vectors corresponding to the multiple data streams; the processing unit is further configured to: trigger a target event when the feature matching degree between any two data streams is greater than a matching threshold; the target event includes any one of an alarm event, a state change event, and a control command event.

[0017] In one possible implementation, the processing unit is specifically configured to: obtain feature vectors within the sliding time window from feature vectors corresponding to multiple data streams based on the sliding time window; map the feature vectors within the sliding time window to a common feature metric space to obtain the feature representation of the feature vectors within the sliding time window in the common feature metric space; determine the similarity between feature representations from different data streams in the common feature metric space, and use the similarity as the feature matching degree of the feature representation.

[0018] In one possible implementation, the processing unit is specifically configured to: acquire the real-time resource utilization of multiple computing nodes; determine the computation type of the feature extraction operator corresponding to the data modality of the data stream; based on the real-time resource utilization, determine a target computing node from the multiple computing nodes, wherein the target computing node has hardware computing resources that match the computation type; and run the corresponding feature extraction operator on the target computing node to acquire feature vectors.

[0019] In one possible implementation, the processing unit is specifically configured to: in response to the resource utilization rate of the first node running the first type of feature extraction operator exceeding the load threshold, determine the second node that matches the computation type of the first type of feature extraction operator as the target computation node.

[0020] In one possible implementation, the processing unit is specifically configured to: determine a video feature extraction strategy as a video feature extraction operator when the data stream is a video data stream; determine an audio feature extraction strategy as an audio feature extraction operator when the data stream is an audio data stream; and determine a sensor feature extraction strategy as a sensor feature extraction operator when the data stream is a sensor data stream.

[0021] In one possible implementation, the processing unit is specifically configured to: perform protocol parsing and data format analysis on each data stream to obtain analysis results; the analysis results include the communication protocol of the data stream and the encoding format of the data stream's payload content; and determine the data mode of the data stream based on the analysis results.

[0022] In one possible implementation, the processing unit is further configured to add meta tags to data units in the data stream according to the data modality; the meta tags include at least one of the following: data source device identifier, data acquisition timestamp, and spatial location information of the data source.

[0023] In one possible implementation, the processing unit is further configured to: receive rule logic input through a configuration interface; the rule logic is used to indicate the conditions for triggering the target event; apply the rule logic to the running data processing flow and update the original rule logic; wherein, updating the rule logic only restarts the data processing step related to the rule judgment.

[0024] In one possible implementation, the rule logic has a corresponding version identifier; the processing unit is further configured to: in response to a rollback instruction, roll back the currently running rule logic to the historical rule logic based on the version identifier.

[0025] In one possible implementation, the processing unit is further configured to: acquire multiple processing rule templates corresponding to different business scenarios; in response to a scenario switching instruction, determine a target template from the multiple processing rule templates; switch the currently running rule logic to the rule logic defined in the target template, and adjust the parameters in the feature extraction strategy.

[0026] In one possible implementation, different business scenarios include daily office scenarios, exhibition scenarios, and emergency security scenarios; when switching from a daily office scenario to an emergency security scenario, the parameters to be adjusted include the analysis frame rate parameter of the video data stream and / or the matching threshold parameter of the temperature data in the sensor data stream.

[0027] The technical effects of any implementation method in the second aspect can be found in the technical effects of any implementation method in the first aspect mentioned above, and will not be repeated here.

[0028] Thirdly, a computer device is provided, comprising: a processor and a memory, wherein the memory stores at least one computer program, and the at least one computer program is loaded and executed by the processor to implement the above-mentioned report generation method for civil aviation operations.

[0029] Fourthly, a computer-readable storage medium is provided, wherein at least one computer program is stored in the computer-readable storage medium, and the at least one computer program is loaded and executed by a processor to implement the above-mentioned report generation method for civil aviation operations.

[0030] Fifthly, a computer program product is provided, comprising a computer program or instructions, which, when executed by a processor, implement the above-mentioned report generation method for civil aviation operations.

[0031] The solutions provided in aspects two through five above are used to implement the method provided in aspect one above, and their specific implementations will not be described in detail here. The technical effects corresponding to any implementation method of the solutions provided in aspects two through five above can be found in the technical effects corresponding to any implementation method in aspect one above, and will not be described in detail here.

[0032] It should be noted that any of the possible implementations of any of the above aspects can be combined, provided that the solutions do not contradict each other.

[0033] The technical solution provided in this application offers at least the following benefits: First, by determining the corresponding feature extraction strategy based on data modality, the system can uniformly access and differentiate the processing of video, audio, and other data streams with varying protocols and formats, solving the initial challenge of adapting multi-source heterogeneous data. Then, feature vectors of each data stream are obtained according to the strategy, transforming unstructured raw stream data into structured, quantifiable feature vectors, providing a unified basis for subsequent correlation analysis. Next, the matching degree between different data streams is determined based on the feature vectors, thereby achieving deep correlation and fusion analysis of cross-modal data at the feature level, enabling insights into the intrinsic connections between data such as video images, audio signals, and sensor readings. Finally, when the matching degree exceeds a threshold, a target event is triggered, allowing the system to automatically and accurately identify meaningful events (such as security police linkage) from complex data correlations, completing a closed loop from raw data processing to real-time decision response, significantly improving the automation level and decision-making efficiency of multimodal data processing. Attached Figure Description

[0034] Figure 1 This is a schematic diagram of the structure of a data processing system provided in an embodiment of this application; Figure 2 A flowchart illustrating a data processing method provided in an embodiment of this application; Figure 3 A flowchart illustrating another data processing method provided in an embodiment of this application; Figure 4 A flowchart illustrating another data processing method provided in an embodiment of this application; Figure 5 This is a schematic diagram of another data processing system provided in an embodiment of this application; Figure 6 This is a schematic diagram of another data processing system provided in an embodiment of this application. Detailed Implementation

[0035] In the embodiments of this application, in order to clearly describe the technical solutions of the embodiments of this application, the terms "first" and "second" are used to distinguish identical or similar items with essentially the same function and effect. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, and the terms "first" and "second" are not necessarily different. The technical features described by "first" and "second" have no sequential or size order.

[0036] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner to facilitate understanding.

[0037] In the embodiments of this application, at least one can also be described as one or more, and multiple can be two, three, four or more, and this application does not impose any restrictions.

[0038] Furthermore, the network architecture and scenarios described in the embodiments of this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided in the embodiments of this application. As those skilled in the art will know, with the evolution of network architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0039] In complex scenarios such as smart parks and industrial IoT, where real-time fusion processing of multimodal heterogeneous data including video, audio, sensor data, and text is required, existing stream processing solutions suffer from significant shortcomings in terms of efficiency, flexibility, and adaptability. Specifically, existing stream processing frameworks, such as StreamPipes, have at least the following deficiencies in practical applications: First, its operator adaptability is limited. The default data processing operators provided by the framework are mainly designed for structured numerical or log data, lacking dedicated feature extraction and processing capabilities for unstructured data such as video and audio. This forces developers to manually write dedicated processing operators for various types of unstructured data when dealing with multimodal data, resulting in long development cycles and problems such as poor cross-version compatibility, complex system integration and maintenance.

[0040] Secondly, its business rule adjustment mechanism is rigid. The association rules between multimodal data (e.g., combining temperature sensor readings with video smoke features to determine fires) need to be hard-coded into the data processing topology. When business logic needs to be modified or optimized, the entire data processing pipeline must be stopped and restarted, causing significant business service interruptions and failing to meet the urgent need for dynamic and real-time adjustment of business rules in scenarios such as smart parks.

[0041] Furthermore, its resource scheduling and load balancing strategies are not intelligent enough. The framework's default resource allocation strategies (such as round-robin scheduling) fail to fully consider the inherent differences in computational characteristics of different data processing operators (for example, video analytics is typically computationally intensive and relies on GPU acceleration). When faced with sudden fluctuations in data traffic, it can easily cause overload of computing nodes that rely on specific hardware (such as GPUs), while other nodes are underutilized, resulting in unbalanced overall system resource utilization and bottlenecks in processing performance.

[0042] Furthermore, existing solutions lack the ability to self-awareness and adapt to business scenarios. Data processing strategies and rule priorities typically differ across different areas (such as office areas, laboratories, and exhibition halls) or modes (such as daily operations and emergency response) within a smart park. Existing technologies lack a dynamic adaptation mechanism that links with these scenarios, relying on manual configuration file switching, which is cumbersome, error-prone, and makes it difficult to achieve rapid and accurate switching of processing strategies.

[0043] Based on this, this application provides a data processing method that determines corresponding feature extraction strategies based on data modalities. This enables the system to uniformly access and differentiate the processing of video, audio, and other data streams with varying protocols and formats, solving the initial challenge of adapting multi-source heterogeneous data. Then, feature vectors of each data stream are obtained according to the strategy, transforming unstructured raw stream data into structured, quantifiable feature vectors, providing a unified basis for subsequent correlation analysis. Next, the matching degree between different data streams is determined based on the feature vectors, thereby achieving deep correlation and fusion analysis of cross-modal data at the feature level, revealing the intrinsic connections between data such as video images, audio signals, and sensor readings. Finally, when the matching degree exceeds a threshold, a target event is triggered, enabling the system to automatically and accurately identify meaningful events (such as security police linkage) from complex data correlations, completing a closed loop from raw data processing to real-time decision response, significantly improving the automation level and decision-making efficiency of multimodal data processing.

[0044] The solution provided in this application can be applied to Figure 1 In the data processing system shown, such as Figure 1 This is a schematic diagram of the structure of a data processing system provided in an embodiment of this application.

[0045] For example, Figure 1 The data processing system shown includes a data input unit 101, multiple computing nodes 102, and a rule logic unit 103. The data input unit 101, the multiple computing nodes 102, and the rule logic unit 103 are connected via a network.

[0046] In some embodiments, the data input unit 101 is used to receive and preprocess raw data streams from multiple heterogeneous data sources within the smart park.

[0047] Specifically, the data input unit 101 has built-in access adaptation operators that support multiple communication protocols. For example, for network cameras deployed throughout the park, the data input unit 101 accesses the continuous video streams they send via Real Time Streaming Protocol (RTSP); for temperature and humidity sensors installed in rooms, it subscribes to their periodically reported JSON-formatted sensor data via Message Queuing Telemetry Transport (MQTT); and for access logs generated by the access control system, it receives the text data pushed by the system via Hypertext Transfer Protocol (HTTP).

[0048] Upon receiving the raw data, the access adaptation operator performs protocol parsing and data format analysis to automatically identify the data modality. For example, by parsing the Content-Type field in the RTSP protocol header as video / H264 and combining it with the binary characteristics of the data payload, the data stream can be determined to be in video data modality. After determining the data modality, the operator adds metadata tags to each data unit (such as a video frame or a sensor reading) containing a data source device identifier (such as CAM-001), a high-precision timestamp (such as 2023-10-27T14:30:00.123Z), and the spatial location of the data source (such as the lobby on the first floor of Building 3), providing contextual information for subsequent processing.

[0049] For example, when a large-scale event is held at the convention center, the data input unit 101 will simultaneously process RTSP video streams from hundreds of cameras, MQTT data streams from hundreds of environmental sensors throughout the venue, and HTTP log streams from the ticketing system, and accurately label the source, time, and location of each data item to form a standardized data stream with rich metadata, which will then be supplied to downstream computing nodes for processing.

[0050] In some embodiments, multiple computing nodes 102 are equipped with corresponding feature extraction operators to extract high-level feature vectors from a data stream with metadata, and the system dynamically schedules the running instances of these operators according to the operator type and node load.

[0051] Specifically, feature extraction operators are pre-developed and registered in the system, mainly falling into three categories: The video feature operator, internally based on the lightweight convolutional neural network (CNN) model MobileNetV3, is used to extract 128-dimensional target feature vectors from video frames, such as the outline of a flame and its color distribution.

[0052] Audio feature operators extract 64-dimensional sound feature vectors, such as energy peaks within a specific frequency range, by calculating the Mel-spectrogram of an audio signal and analyzing its spectral characteristics.

[0053] Sensor feature operators perform sliding window (e.g., 5-second window) statistics on time-series numerical data such as temperature, humidity, and smoke concentration, calculating the maximum, average, and volatility within the window, and outputting a 32-dimensional feature vector of numerical changes. These operators differ in their computational type: video feature operators are GPU-intensive, while audio and sensor feature operators are CPU-intensive.

[0054] The system collects the CPU, GPU, and memory usage of each computing node 102 in real time through a monitoring interface. When a new video data stream needs to be processed, the system first determines that a video feature operator (GPU-intensive) needs to be called. Then, based on the real-time resource usage, it selects a computing node (e.g., Node-GPU-02) with idle GPU resources and low load from the resource pool, and schedules and starts a video feature operator instance on that node to process the data stream, ultimately outputting a feature vector.

[0055] For example, during peak exhibition periods, when the system detects that the GPU utilization of the Node-GPU-01 node running the video feature operator continuously exceeds the 80% load threshold, the scheduling module will automatically launch a new video feature operator instance on a backup node Node-GPU-02 with idle GPU resources. Subsequently, some newly connected video streams will be dynamically allocated to the new instance on Node-GPU-02 for processing, thereby effectively distributing the load, avoiding processing delays, and ensuring that the load deviation of each node is maintained within 10%, achieving intelligent resource balancing and elastic scaling.

[0056] In some embodiments, the rule logic unit 103 is used to perform multimodal feature association analysis and to determine and trigger events based on dynamically updatable business rules.

[0057] The core of the rule logic unit 103 is the association inference operator, which receives feature vectors output from various feature extraction operators on different computing nodes 102. The association inference operator has a configurable sliding time window (default 5 seconds) that collects all feature vectors arriving within the same time window. Since the feature vectors of different modalities have different dimensions (e.g., 128-dimensional, 64-dimensional, 32-dimensional), the operator first projects them to a unified 128-dimensional common feature metric space through a mapping function, obtaining standardized feature representations. Subsequently, it calculates the cosine similarity between the feature representations from any two different data streams (e.g., a video feature vector and a temperature sensor feature vector) within this common space, using this as the feature matching degree. The rule logic unit 103 also provides a visual configuration interface for operations and maintenance personnel to define business rules.

[0058] For example, a rule logic expression can be configured as: "(video feature matching 'flame' similarity > 0.7) and (temperature feature > 75℃)", specifying that it triggers the "fire alarm" target event. Once configured, this expression is compiled into an independent, hot-loadable rule topology fragment (or processing module). Through a dynamic update interface, this new rule fragment is injected (inserted) into the running association inference operator, replacing the original fire alarm judgment logic. This replacement process only causes the association inference operator itself to restart within milliseconds to load the new logic, while the preceding data access, feature extraction, and other operators continue to run, ensuring uninterrupted business operations and achieving hot rule updates.

[0059] For example, when maintenance personnel switch from daily office mode to emergency security mode, the rule logic unit 103 can quickly load a pre-set "emergency" template via a scene switching command. This template not only lowers the temperature threshold in the fire alarm rules from 80℃ to 65℃, improving sensitivity, but may also automatically adjust the analysis frame rate parameters of the video feature extraction operator for more intensive video analysis.

[0060] Meanwhile, if problems are found in the newly deployed rules (V2 version), the rules can be quickly rolled back to the historical stable version (V1 version) through the version management function. The entire rollback process is completed within hundreds of milliseconds, ensuring the stability and flexibility of the system.

[0061] Figure 2 This is a flowchart illustrating a data processing method provided in an embodiment of this application. The method is applied to... Figure 1 The data processing system in the middle.

[0062] like Figure 2 As shown, the data processing method provided in this application embodiment may include: S201: Acquire multiple data streams and determine the corresponding feature extraction strategy based on the data modalities of the multiple data streams.

[0063] In the embodiments of this application, the data modalities include video data modalities, audio data modalities, and sensor data modalities.

[0064] Among them, video data mode usually refers to a continuous sequence of image data streams generated by devices such as cameras, whose data payload is compressed by video encoding (such as H.264, H.265).

[0065] Audio data modality refers to a continuous audio signal data stream collected by devices such as microphones, whose data payload is encoded with audio (such as AAC, G.711).

[0066] Sensor data modalities refer to numerical or state data streams periodically or event-wise reported by various IoT sensors, and their payloads are often in structured text formats (such as JSON, XML).

[0067] For example, the data modality of the above data stream can be determined by performing protocol parsing and data format analysis on each data stream, obtaining analysis results, and then determining the data modality of the data stream based on the analysis results.

[0068] The analysis results include the communication protocol of the data stream and the encoding format of the data stream's payload content.

[0069] For example, for a data stream transmitted via Real-Time Streaming Protocol (RTSP), by parsing the response header of the DESCRIBE or SETUP command in the RTSP protocol, if the Content-Type field is set to application / sdp and the media description includes a video field, the data stream is initially determined to be in video mode. Further, by parsing the payload type of subsequent RTP packets in the data stream, or by detecting the start code of the data payload, if the start code matches a predetermined start code (0x00000001 or 0x000001) in the H.264 encoding format, the data stream is ultimately determined to be in video mode, and its video encoding format is determined to be H.264.

[0070] For a data stream arriving via the MQTT protocol, the subject (e.g., sensor / floor1 / temperature) and payload in the MQTT protocol message are parsed. If the payload is text data in JSON format and contains key-value pairs representing numerical values ​​such as "value": 25.6, then the data stream is determined to be a sensor data mode.

[0071] For a data stream uploaded via an HTTP POST request, check the Content-Type field in the request header. If the field value is audio / wav or audio / mpeg, further parse the header information of the payload. If the RIFF identifier unique to WAV format files or the frame synchronization word of MP3 format is identified, then the data stream is determined to be an audio data modality.

[0072] Furthermore, after determining the data modality, meta tags can be added to the data units in the data stream based on the data modality.

[0073] The meta tag includes at least one of the following: data source device identifier, data collection timestamp, and spatial location information of the data source.

[0074] For example, once a data stream is identified as a video data stream from the lobby on the first floor of Building A in the park, the system adds a structured meta tag to each video frame data unit in it.

[0075] For example, for a video frame captured at 10:30:00:123 UTC on October 27, 2023, the added meta tag might include: device identifier CAM-101, used to uniquely identify the source camera; data acquisition timestamp 2023-10-27T10:30:00.123Z, precisely recording the acquisition time; and spatial location information of the data source: Building A-1F-Lobby-East, used to identify the physical source location of the data.

[0076] For a JSON data set {“sensor_id”: “TEMP-205”, “value”: 22.5, “unit”: “℃”} received from the temperature sensor at the same time, the system will also add meta tags to it.

[0077] For example, the device identifier TEMP-205, the timestamp 2023-10-27T10:30:00.123Z, and the location information Building A-1F-Lobby-Central. By uniformly attaching such meta-tags to all modal data units, subsequent processing steps can perform correlation, traceability, and fusion analysis of heterogeneous data based on multiple dimensions such as device, time, and space.

[0078] In some embodiments, when the data stream is a video data stream, the feature extraction strategy is determined to be a video feature extraction operator; when the data stream is an audio data stream, the feature extraction strategy is determined to be an audio feature extraction operator; and when the data stream is a sensor data stream, the feature extraction strategy is determined to be a sensor feature extraction operator.

[0079] For example, the system automatically selects a dedicated operator based on a predefined mapping relationship between data modalities and feature extraction strategies.

[0080] Specifically, when the data input unit determines that a data stream is a video data modality (e.g., an RTSP stream encoded in H.264), it determines to apply a video feature extraction strategy to it. This strategy corresponds to a pre-registered video feature extraction operator that internally encapsulates a MobileNetV3 convolutional neural network model. The function of this operator is to output a 128-dimensional image feature vector.

[0081] When a data stream is determined to be in audio data mode (e.g., an HTTP stream encoded in WAV format), an audio feature extraction strategy is adopted, which corresponds to an audio feature extraction operator. This operator will calculate the Mel spectrum and extract frequency domain features to output a 64-dimensional sound feature vector.

[0082] When a data stream is determined to be sensor data mode (e.g., MQTT messages in JSON format), a sensor feature extraction strategy is adopted, which corresponds to a sensor feature operator. This operator performs sliding window statistics on the numerical sequence and outputs a 32-dimensional feature vector of numerical changes.

[0083] This application embodiment, through this deterministic mapping of one-to-one correspondence between modality and strategy, ensures that each type of heterogeneous data can be processed by its most suitable feature extraction algorithm, thereby providing high-quality, standardized feature input for subsequent multimodal association analysis.

[0084] S202: For each data stream, obtain the feature vector of the data stream according to the feature extraction strategy of the data stream.

[0085] In some embodiments, the real-time resource utilization of multiple computing nodes is obtained, the computation type of the feature extraction operator corresponding to the data mode of the data stream is determined, and the target computing node is determined from the multiple computing nodes based on the real-time resource utilization. Then, the corresponding feature extraction operator is run on the target computing node to obtain the feature vector.

[0086] The target computing node possesses hardware computing resources that match the type of computing.

[0087] For example, the system includes a resource pool of multiple heterogeneous computing nodes (e.g., some nodes are equipped with high-performance graphics processing units (GPUs), and other nodes are equipped with multiple central processing units (CPU) cores), and continuously collects real-time resource utilization indicators such as CPU utilization, memory usage, and GPU utilization of each node through the monitoring interface provided by the framework.

[0088] When it is necessary to extract features from a newly arrived data stream that has been identified as a video data mode, the system first determines the video feature extraction operator to be invoked based on the video feature extraction strategy, and determines that the computation type of the operator is GPU intensive.

[0089] Subsequently, the scheduling module selects nodes with low current GPU utilization (e.g., below 50%) and available GPU memory from the resource pool as candidate target nodes based on real-time resource utilization. Assuming node Node-GPU-02 meets the above conditions, it is determined as the target computing node.

[0090] The system then loads and runs an instance of the video feature extraction operator on the Node-GPU-02 node, scheduling the video data stream to this instance for processing. This operator instance decodes the input video frames and uses its built-in convolutional neural network model to extract visual features frame by frame, ultimately outputting a 128-dimensional feature vector, thus completing the acquisition of the feature vector.

[0091] Furthermore, in response to the resource utilization rate of the first node running the first type of feature extraction operator exceeding the load threshold, the second node that matches the computation type of the first type of feature extraction operator is identified as the target computation node.

[0092] For example, during peak exhibition periods in a smart park, assume that the video feature extraction operator instances that initially process all video streams are running on node Node-GPU-01. System monitoring reveals that the GPU utilization of Node-GPU-01 consistently exceeds the preset load threshold of 80%, indicating that the node is overloaded and may become a processing bottleneck. At this point, the dynamic scheduling mechanism is triggered.

[0093] The scheduling module first identifies the overload operator instance as a GPU-intensive video feature extraction operator. Then, it searches the resource pool for another node with matching hardware computing resources, specifically a node also equipped with a GPU but currently with low GPU utilization, such as Node-GPU-02.

[0094] The system designates Node-GPU-02 as the new target computing node and automatically launches a new video feature extraction operator instance on it. Subsequently, newly received video streams or portions of existing video streams are dynamically allocated to the new instance on Node-GPU-02 for processing, thus reducing the load on Node-GPU-01. This process achieves horizontal elastic scaling for specific operator types, ensuring balanced overall system load and linear scaling of processing capacity during data peaks, avoiding processing delays caused by overload of a single node.

[0095] S203: Determine the feature matching degree between multiple data streams based on the feature vectors corresponding to multiple data streams.

[0096] In some embodiments, based on a sliding time window, feature vectors within the sliding time window are obtained from feature vectors corresponding to multiple data streams, and the feature vectors within the sliding time window are mapped to a common feature metric space to obtain the feature representation of the feature vectors within the sliding time window in the common feature metric space. Then, the similarity between feature representations from different data streams in the common feature metric space is determined, and the similarity is used as the feature matching degree of the feature representation.

[0097] For example, the system sets a sliding time window with a configurable duration (e.g., a default duration of 5 seconds), which slides continuously on the timeline (e.g., a new window is created every 1 second).

[0098] The correlation inference operator collects feature vectors from all different modalities that arrive within the same sliding time window. For example, within a 5-second window, the system may collect: three video feature vectors from the CAM-101 camera (corresponding to video frames at times t1, t2, and t3, respectively), two sensor feature vectors from the TEMP-205 temperature sensor (corresponding to readings at times t1 and t4), and one audio feature vector from the MIC-301 microphone (corresponding to an audio segment at time t2).

[0099] Because these feature vectors have different dimensions (128-dimensional for video, 64-dimensional for audio, and 32-dimensional for sensors), direct similarity comparison is not possible. Therefore, the system uses a pre-trained multilayer perceptron mapping function to project all feature vectors of different dimensions into a unified 128-dimensional common feature metric space, thus obtaining their standardized feature representations.

[0100] In the public space, calculate the cosine similarity between feature representations from any two different data streams.

[0101] Cosine similarity is calculated using the formula similarity = Calculate, where A and B represent two feature representation vectors, A B is their dot product. A and B These are their modulus lengths.

[0102] The physical meaning of this formula is to calculate the cosine of the angle between two vectors in their directions. Its value ranges from [-1, 1]. The closer the value is to 1, the more consistent the directions of the two vectors are and the higher their similarity.

[0103] For example, the cosine similarity between a feature representation of a flame from a video and a feature representation of a high-temperature anomaly from a temperature sensor in a public space is calculated. If the result is 0.85, this value of 0.85 is determined as the feature matching degree between the two feature representations within a sliding time window. This process achieves semantic alignment and association measurement of cross-modal features.

[0104] S204: Trigger the target event if the feature matching degree between any two data streams is greater than the matching threshold.

[0105] The target event includes any one of the following: alarm event, status change event, and control command event.

[0106] For example, the association inference operator compares the continuously calculated feature matching degree with a preset matching threshold to make event decisions. Assume the system sets the matching threshold to 0.8 for a fire early warning scenario. Within a certain sliding time window, if the calculated feature matching degree between the video flame feature from camera CAM-101 and the high-temperature feature from temperature sensor TEMP-205 is 0.85, since 0.85 > 0.8, the triggering condition is met, and the system immediately triggers the corresponding target event. Which target event is triggered is determined by pre-configured rule logic.

[0107] For example, it can be configured to trigger an alarm event, specifically by highlighting the real-time image of CAM-101 on the alarm screen in the monitoring center, and sending an alarm message containing the time, location, and alarm type (such as "suspected fire") to the safety manager via SMS, application push, or other means.

[0108] It can also be configured to trigger state change events, for example, automatically change the global state of the park security system from normal to fire alarm monitoring, and record this state change to the log database for subsequent auditing and analysis.

[0109] Furthermore, it can be configured to trigger a control command event, such as automatically sending a control command to the fire alarm control panel to activate the audible and visual alarms installed in the lobby area on the first floor of Building A, and automatically shut down the ventilation system in that area to prevent the fire from spreading. In this way, the system achieves a seamless closed loop from multimodal data correlation analysis to automated decision-making and execution, significantly improving the real-time performance and accuracy of event response.

[0110] For example, Figure 3 This is a flowchart illustrating another data processing method provided in an embodiment of this application. The method is applied to... Figure 1 The data processing system in [the context]. Specifically, it includes the following steps: S301: Receives rule logic input through the configuration interface.

[0111] The rule logic is used to indicate the conditions that trigger the target event; the configuration interface is a visual web page.

[0112] For example, the configuration interface provides logical operators (such as "AND", "OR", and "NOT"), comparison operators (such as ">", "<", and "="), and parameter input boxes, which can be combined into business rules by operation and maintenance personnel through drag-and-drop and filling. For example, for the fire protection scenario of a smart park, operation and maintenance personnel can configure a rule logical expression in the interface as: (video.feature.flame similarity>0.7) AND (sensor.temperature.latest value>75) AND (sensor.smoke concentration.latest value>0.5).

[0113] The rule specifies the following parameters: `video.feature.flame similarity` is an output variable from the video feature extraction operator; `sensor, temperature, latest value` and `sensor, smoke concentration, latest value` are output variables from the sensor feature extraction operator; and `0.7`, `75`, and `0.5` are user-defined matching and numerical thresholds, respectively. This rule logic indicates the combined conditions for triggering a fire alarm. After configuration, the user submits this rule logic by clicking the "Save and Deploy" button on the interface.

[0114] In some embodiments, multiple processing rule templates corresponding to different business scenarios are obtained. In response to a scenario switching instruction, a target template is determined from the multiple processing rule templates, and the currently running rule logic is switched to the rule logic defined in the target template. The parameters in the feature extraction strategy are also adjusted.

[0115] For example, system administrators can predefine and save multiple processing rule templates according to different operational stages of the smart park.

[0116] For example, a daily office template, the rules of which focus on energy management (such as turning off lights when the light is too strong and no one is around) and general security (such as triggering alarms when movement is detected outside of working hours).

[0117] The exhibition template focuses on pedestrian flow statistics and abnormal behavior detection in public areas; the emergency security template focuses on sensitive detection and coordinated control of emergencies such as fires and intrusions. When the park is about to transition from daily office mode to large-scale exhibition mode, the administrator can select exhibition mode through the drop-down menu in the management backend and click the switch button to issue a scene switch command.

[0118] In response to the command, the system immediately loads the target template named "Exhibition" from the template library. Subsequently, the system replaces all currently running rule logic in the data processing flow (such as multiple energy management rules) with multiple rule logic defined in the "Exhibition" template (such as pedestrian flow statistics rules). Simultaneously, the system automatically adjusts the parameters of relevant feature extraction strategies based on the pre-configuration in the template. For example, it increases the analysis frame rate of the video feature extraction operator from the usual 5 frames per second to 15 frames per second to obtain more continuous pedestrian movement footage for analysis.

[0119] In this application's embodiments, scenarios include daily office work, exhibitions, and emergency security. When switching from a routine office scenario to an emergency security scenario, the parameters to be adjusted include the analysis frame rate parameter of the video data stream and / or the matching threshold parameter of the temperature data in the sensor data stream.

[0120] Specifically, when switching to an emergency security scenario, the template loaded by the system will automatically adjust two key parameters.

[0121] First, the analysis frame rate parameter of the video data stream is significantly increased from a lower frame rate (such as 5 FPS) in daily office scenarios to a higher value (such as 25 FPS). This means that the video feature extraction operator will sample and analyze the raw video stream captured by the camera more intensively, thereby enabling the detection of abnormal changes in the scene (such as the initial flames or people running) more quickly.

[0122] Second, the matching threshold parameter for temperature data used in fire detection in the sensor data stream (i.e., the matching threshold in step S204) is appropriately lowered from the higher threshold (e.g., 0.8) in daily office scenarios (e.g., adjusted to 0.65). This means that during correlation inference, the matching degree between video flame features and high-temperature sensor features only needs to reach 0.65 to trigger an alarm, lowering the alarm threshold and improving the system's sensitivity to potential fires, in order to meet the zero-tolerance and early detection requirements for safety incidents in emergency situations.

[0123] S302: Apply the rule logic to the running data processing flow and update the original rule logic.

[0124] The update rule logic only restarts the data processing steps related to rule judgment.

[0125] For example, when a new fire alarm rule logic configured in S301 is submitted, the system's rule compilation engine compiles it into an independent, executable rule topology fragment (or rule processing module). Subsequently, through the dynamic topology update interface provided by a stream processing framework (such as StreamPipes), the rule topology fragment is hot-swappably injected into the running data processing flow.

[0126] Specifically, the new fragments are deployed to the associative reasoning operators responsible for executing S203 and S204, replacing the original rule logic code within these operators used to determine whether fire alarm conditions are met. The entire update (replacement) process is localized, causing only a single, rapid restart of a specific part of the associative reasoning operator to load and enable the new logic, with a restart time of extremely short duration (e.g., within 50 milliseconds).

[0127] It should be noted that other parts of the data processing flow, including data access and modality recognition in S201 and feature extraction in S202, continue to operate normally and are not affected by this rule update, thus achieving zero or near-zero interruption hot updates at the business level.

[0128] In some embodiments, in response to a rollback instruction, the currently running rule logic is rolled back to the historical rule logic based on the version identifier.

[0129] For example, when assigning a version identifier (e.g., Fire_Rule_V2) to each newly deployed rule logic, the system archives and saves the old rule logic to be replaced (e.g., Fire_Rule_V1) and its version identifier. Suppose that the newly deployed Fire_Rule_V2 rule generates a false alarm during actual operation due to an overly sensitive threshold. Operations personnel can view the rule's historical versions through the management interface and issue a rollback command for Fire_Rule_V2. In response to this rollback command, the system first locates the historical rule logic corresponding to its direct predecessor, Fire_Rule_V1, in the version repository based on the version identifier Fire_Rule_V2 of the currently running rule. Then, the system dynamically redeploys the rule topology fragment corresponding to Fire_Rule_V1 into the associated inference operator, replacing the problematic Fire_Rule_V2 logic.

[0130] The entire rollback operation also only triggers a rapid restart of the associated inference operator and is completed in a very short time (e.g., within 100 milliseconds), allowing the system to quickly return to a stable and reliable historical rule state, which greatly enhances the robustness and maintainability of the system in the process of frequent rule iteration.

[0131] For example, such as Figure 4 As shown, the overall process of the data processing method is introduced. Specifically, it includes the following steps: S401: Acquire multiple data streams.

[0132] S402: Determine the access adaptation operator.

[0133] It should be understood that this access adaptation operator is responsible for protocol parsing (such as parsing different protocols such as RTSP, MQTT, and HTTP) and adding meta tags (such as data source ID, timestamp, and location information) to the data, thereby uniformly encapsulating the raw data into a standardized data stream with rich contextual information.

[0134] S403: Determine the corresponding feature extraction strategy based on the data modality of the data flow.

[0135] S404: Receive rule logic.

[0136] It should be understood that the business rule logic configured by the user through the visual interface (such as "an alarm will be triggered if both flame and high temperature characteristics appear simultaneously") is compiled in this module.

[0137] S405: Dynamic rule topology.

[0138] It should be understood that this step receives feature vectors from various feature extraction operators, performs cross-modal feature matching and similarity calculation within a sliding time window, and makes judgments based on the loaded rule logic.

[0139] Continuously monitor the utilization of each computing node (the resource running operator instances) and dynamically schedule tasks or elastically scale instances according to operator type (CPU / GPU intensive) to ensure efficient and stable system operation.

[0140] S406: Output result.

[0141] In other words, based on the judgment result of the dynamic rule topology, the corresponding event is triggered and output.

[0142] This application provides a data processing method that determines corresponding feature extraction strategies based on data modalities. This enables the system to uniformly access and differentiate the processing of video, audio, and other data streams with varying protocols and formats, solving the initial challenge of adapting multi-source heterogeneous data. Furthermore, feature vectors of each data stream are obtained according to the strategy, transforming unstructured raw stream data into structured, quantifiable feature vectors, providing a unified basis for subsequent correlation analysis. Then, the matching degree between different data streams is determined based on the feature vectors, thereby achieving deep correlation and fusion analysis of cross-modal data at the feature level, revealing the intrinsic connections between data such as video images, audio signals, and sensor readings. Finally, when the matching degree exceeds a threshold, a target event is triggered, enabling the system to automatically and accurately identify meaningful events (such as security alerts) from complex data correlations, completing a closed loop from raw data processing to real-time decision response, significantly improving the automation level and decision-making efficiency of multimodal data processing.

[0143] For example, this application also provides a data processing system, which is used to implement the above-described method embodiments.

[0144] like Figure 5 This is a schematic diagram of another data processing system provided in an embodiment of this application. The data processing system includes a processing module 501 and a communication module 502. The communication module 502 is used to execute... Figure 2 The operation S201 in the illustrated method is used to execute... Figure 3 The illustrated method includes the operation of S301 and the method used for execution. Figure 4 The illustrated method involves the operation of S401; the processing module 501 is used to execute... Figure 2 The operations S202, S203, and S204 in the illustrated method are used to execute Figure 3 The illustrated method includes the operation of S302 and the method used for execution. Figure 4 The illustrated method includes operations S402 to S406.

[0145] In some embodiments, the data processing system described above includes hardware structures and / or software modules corresponding to the execution of each function in order to achieve the above-described functions. Those skilled in the art will readily recognize that, based on the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0146] This application embodiment can divide the data processing system into functional modules according to the above method embodiment. For example, each function can be divided into its own functional module, or two or more functions can be integrated into one processing module. The integrated module can be implemented in hardware or as a software functional module. It should be noted that the module division in this application embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods.

[0147] like Figure 6 As shown, another data processing system provided in this application embodiment may include a processor 1101, a bus 1102, a communication interface 1103, and a memory 1104. The processor 1101, the memory 1104, and the communication interface 1103 communicate with each other via the bus 1102. It should be understood that this application does not limit the number of processors and memories in the network device.

[0148] Bus 1102 can be a PCI bus, an Extended Industry Standard Architecture (EISA) bus, or a UB bus, etc. Buses can be divided into address buses, data buses, control buses, etc. For ease of representation, Figure 6 The bus 1102 may be represented by a single line, but this does not mean that there is only one bus or a bus of one data mode. The bus 1102 may include a path for transmitting information between various components of the network device (e.g., memory 1104, processor 1101, communication interface 1103).

[0149] Processor 1101 may include any one or more processors such as CPU, graphics processing unit (GPU), microprocessor (MP), or digital signal processor (DSP).

[0150] The memory 1104 may include volatile memory, such as random access memory (RAM). The processor 1101 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0151] The communication interface 1103 uses transceiver modules such as, but not limited to, network interface cards and transceivers to enable communication between network devices and other devices or communication networks.

[0152] The memory 1104 stores executable program code, and the processor 1101 executes the executable program code to implement the functions of the aforementioned method embodiments. That is, the memory 1104 stores instructions for executing the aforementioned report generation method for civil aviation operations.

[0153] On the one hand, a computer-readable storage medium is provided, in which at least one computer program is stored, the at least one computer program being loaded and executed by a processor to implement the report generation method for civil aviation operations as provided in the above-described method embodiments.

[0154] In another aspect, a computer program product is provided, which includes a computer program or instructions that, when executed by a processor, implement the report generation method for civil aviation operations provided in the above-described method embodiments.

[0155] Through the above description of the implementation methods, those skilled in the art will clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the module can be divided into different functional modules to complete all or part of the functions described above. The specific working process of the system, modules, and units described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0156] Since the indicator generation module, computer-readable storage medium, and computer program product in the embodiments of the present invention can be applied to the above method, the technical effects obtained can also be referred to the above method embodiments. The embodiments of the present invention will not be described again here.

[0157] The method steps in this embodiment can be implemented in hardware or by a processor executing software instructions. The software instructions can consist of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, portable hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and storage medium can reside in an ASIC. Alternatively, the ASIC can reside in a network device. Of course, the processor and storage medium can also exist as discrete components in the network device.

[0158] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer programs or instructions. When a computer program or instruction is loaded and executed on a computer, the processes or functions of the embodiments of this application are performed, in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, a network device, a user equipment, or other programmable module. The computer program or instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another. For example, a computer program or instructions can be transferred from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium, such as a floppy disk, hard disk, or magnetic tape; it can also be an optical medium, such as a digital video disc (DVD); or it can be a semiconductor medium, such as a solid-state drive (SSD). The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A data processing method, characterized in that, The method includes: Multiple data streams are acquired, and a corresponding feature extraction strategy is determined based on the data modalities of the multiple data streams; the data modalities include video data modalities, audio data modalities, and sensor data modalities. For each data stream, obtain the feature vector of the data stream according to the feature extraction strategy of the data stream; Based on the feature vectors corresponding to the multiple data streams, the feature matching degree between the multiple data streams is determined; A target event is triggered when the feature matching degree between any two data streams is greater than the matching threshold; the target event includes any one of an alarm event, a status change event, and a control command event.

2. The method according to claim 1, characterized in that, Determining the feature matching degree among the multiple data streams based on their corresponding feature vectors includes: Based on the sliding time window, the feature vector within the sliding time window is obtained from the feature vectors corresponding to the multiple data streams; The feature vectors within the sliding time window are mapped to a common feature metric space to obtain the feature representations of the feature vectors within the sliding time window in the common feature metric space. Determine the similarity between feature representations from different data streams within the common feature metric space, and use the similarity as the feature matching degree of the feature representation.

3. The method according to claim 1, characterized in that, For each data stream, obtaining the feature vector of the data stream according to the feature extraction strategy of the data stream includes: Obtain real-time resource utilization rates for multiple computing nodes; Determine the computation type of the feature extraction operator corresponding to the data modality of the data stream; Based on the real-time resource utilization rate, a target computing node is determined from the plurality of computing nodes, wherein the target computing node has hardware computing resources that match the computing type; The corresponding feature extraction operator is run on the target computing node to obtain the feature vector.

4. The method according to claim 3, characterized in that, The step of determining the target computing node from the plurality of computing nodes based on the real-time resource utilization rate includes: In response to the resource utilization rate of the first node running the first type of feature extraction operator exceeding the load threshold, the second node that matches the computation type of the first type of feature extraction operator is determined as the target computation node.

5. The method according to claim 1, characterized in that, Based on the data modalities of the multiple data streams, a corresponding feature extraction strategy is determined, including: When the data stream is a video data stream, the feature extraction strategy is determined to be a video feature extraction operator; When the data stream is an audio data stream, the feature extraction strategy is determined to be an audio feature extraction operator; When the data stream is a sensor data stream, the feature extraction strategy is determined to be a sensor feature extraction operator.

6. The method according to claim 1, characterized in that, The data mode of the data stream is determined in the following way: Each data stream is parsed and its data format is analyzed to obtain the analysis results; the analysis results include the communication protocol of the data stream and the encoding format of the payload content of the data stream. Based on the analysis results, the data mode of the data stream is determined.

7. The method according to claim 6, characterized in that, After determining the data modality of the data stream, the method further includes: According to the data modality, a meta tag is added to the data unit in the data stream; the meta tag includes at least one of the following: data source device identifier, data acquisition timestamp, and spatial location information of the data source.

8. The method according to claim 1, characterized in that, The method further includes: Receive rule logic input through the configuration interface; the rule logic is used to indicate the conditions that trigger the target event; The rule logic is applied to the running data processing flow to update the original rule logic; wherein, updating the rule logic only restarts the data processing steps related to rule judgment.

9. The method according to claim 8, characterized in that, The rule logic has a corresponding version identifier; the method further includes: In response to the rollback command, the currently running rule logic is rolled back to the historical rule logic based on the version identifier.

10. The method according to claim 1 or 8, characterized in that, The method further includes: Obtain multiple processing rule templates corresponding to different business scenarios; In response to a scene switching command, a target template is determined from the plurality of processing rule templates; Switch the currently running rule logic to the rule logic defined in the target template, and adjust the parameters in the feature extraction strategy.

11. The method according to claim 10, characterized in that, The different business scenarios include daily office scenarios, exhibition scenarios, and emergency security scenarios; When switching from the daily office scenario to the emergency security scenario, the parameters to be adjusted include the analysis frame rate parameter of the video data stream and / or the matching threshold parameter of the temperature data in the sensor data stream.

12. A data processing system, characterized in that, The system includes: a processing unit and a communication unit; The communication unit is configured to: acquire multiple data streams and determine corresponding feature extraction strategies based on the data modalities of the multiple data streams; the data modalities include video data modalities, audio data modalities, and sensor data modalities; The processing unit is configured to: for each data stream, obtain the feature vector of the data stream according to the feature extraction strategy of the data stream; The processing unit is further configured to: determine the feature matching degree between the multiple data streams based on the feature vectors corresponding to the multiple data streams; The processing unit is further configured to trigger a target event when the feature matching degree between any two data streams is greater than a matching threshold; the target event includes any one of an alarm event, a status change event, and a control command event.

13. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing at least one computer program, the at least one computer program being loaded and executed by the processor to implement the method as claimed in any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one computer program, which is loaded and executed by a processor to implement the method as described in any one of claims 1 to 11.

15. A computer program product, characterized in that, The computer program product includes a computer program or instructions that, when executed by a processor, implement the method as described in any one of claims 1 to 11.

Images