Scene data processing system and method
By introducing a scenario data processing system into the Internet of Things platform, the combination of receiving module, data definition module, execution engine module and output module is solved, and the problem of data processing delay in traditional systems is achieved, achieving fast and efficient data processing and business response.
Patent Information
- Application Number
- CN202411924635.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-25
- Publication Date
- 2025-05-13
AI Technical Summary
The independent processing methods in traditional IoT platforms lead to long processing links between systems, and time delays for data interaction and processing, affecting the user experience and business efficiency of quickly responding to business scenarios.
A scenario data processing system is proposed, including a receiving module, a data definition module, an execution engine module and an output module. By defining processing rules for the received scene data and performing target processing on the basis of dynamic path selection, fast and efficient data fusion or enhancement can be achieved.
Ensure that data can be correctly parsed and utilized according to business needs, improve the accuracy and efficiency of data processing, optimize data flow management, speed up data processing speed, and reduce latency and potential errors.
Smart Images
Figure CN119996344A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data processing technology, and in particular to a scene data processing system and method. Background Art
[0002] In traditional solutions, the IoT platform is mainly responsible for data collection, the business rule platform is mainly responsible for rule setting and logic processing, the business platform is responsible for basic data maintenance and business logic display, and the data analysis platform is responsible for data aggregation and analysis.
[0003] However, this independent processing approach results in a long processing link between systems, resulting in time delays in data interaction and processing. For business scenarios that require quick response, this delay may result in untimely service, affecting user experience and business efficiency. Summary of the invention
[0004] In order to solve the above-mentioned technical problem of low efficiency in scene data processing, the present application proposes a scene data processing system and method, which can quickly and efficiently fuse or enhance various types of scene data.
[0005] In a first aspect, the present application provides a scene data processing system, the system comprising:
[0006] A receiving module, used for receiving scene data from each terminal device;
[0007] A data definition module, used to define processing rules for received scenario data, wherein the processing rules include data classification rules, data format rules and matching rules with business scenarios;
[0008] The execution engine module is used to perform target processing on the scene data after the defined processing rules based on dynamic path selection;
[0009] Output module outputs the target processed data.
[0010] In some embodiments, the scene data includes real-time data, event messages, command messages, and resource status data.
[0011] In some embodiments, the execution engine module includes a routing module and a data conversion module; the routing module is used to perform dynamic path selection on the scene data after the processing rules are defined; and the data conversion module is used to perform target processing on the scene data for which the path selection has been completed.
[0012] In some embodiments, the routing module performs dynamic path selection on the scene data after the processing rule is defined, including:
[0013] The routing module allocates paths according to key tags corresponding to the scene data based on a preset routing strategy.
[0014] In some embodiments, the target processing includes a fusion process and an enhancement process.
[0015] In some embodiments, fusing the scene data for which path selection has been completed includes analyzing the scene data for which path selection has been completed, and fusing the analyzed scene data according to preset scene rules.
[0016] In some embodiments, the enhancement processing includes at least one of the following: translation, packaging or normalization processing.
[0017] In some embodiments, the system further includes a message channel module, and the message channel module is used to transfer the scene data after the processing rules are defined by the data definition module to the execution engine module according to different types of scene data.
[0018] In some embodiments, the system further includes an execution monitoring module, which is used to monitor the execution process of the execution engine module performing target processing on the scene data after the processing rules are defined based on dynamic path selection.
[0019] A second aspect of the present application provides a scene data processing method, the method comprising:
[0020] Receive scene data from each terminal device;
[0021] Defining processing rules for the received scenario data, wherein the processing rules include data classification rules, data format rules, and matching rules with business scenarios;
[0022] Target processing is performed on the scene data after the defined processing rules based on dynamic path selection;
[0023] Output the processed data of the target.
[0024] The third aspect of the present application provides an electronic device, including a memory and a processor, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the processor executes the scene data processing method described in the embodiments of the present application.
[0025] The fourth aspect of the present application provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the scene data processing method described in the application embodiment is implemented.
[0026] The technical solution provided in the embodiments of the present application has at least the following technical effects or advantages:
[0027] The scene data processing system described in each embodiment of the present application includes a receiving module for receiving scene data from each terminal device, a data definition module for defining processing rules for the received scene data, wherein the processing rules include data classification rules, data format rules and matching rules with business scenarios, an execution engine module for performing target processing on the scene data after the processing rules are defined on the basis of dynamic path selection, and an output module for outputting the target processed data. In this way, the system ensures that the data can be correctly parsed and utilized according to business needs, thereby increasing the accuracy and efficiency of data processing; and by using the dynamic path selection method through the execution engine module, the data that meets the processing rules is accurately processed, which not only optimizes the dynamic management of data flow within the system, but also speeds up the data processing speed, and reduces waiting time and potential errors.
[0028] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS
[0029] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present application. Also, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:
[0030] Figure 1 is a structural schematic diagram of a scene data processing system in an exemplary embodiment of the present application;
[0031] Figure 2 is a schematic diagram of the structure of the execution engine module in the exemplary embodiment of the present application;
[0032] Figure 3 is a schematic diagram of the steps of a scene data processing method in an exemplary embodiment of the present application;
[0033] Figure 4 It is a schematic diagram of the structure of an electronic device provided by an exemplary embodiment of the present application.
[0034] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. DETAILED DESCRIPTION
[0035] The present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It is understood that the embodiments described herein are only used to explain the relevant invention, rather than to limit the invention. It should also be noted that, for ease of description, only the parts related to the relevant invention are shown in the accompanying drawings.
[0036] In the existing technology, the Internet of Things platform is mainly responsible for data collection, the business rule platform is mainly responsible for rule setting and logic processing, the business platform is responsible for basic data maintenance and business logic display, and the data analysis platform is responsible for data aggregation and analysis.
[0037] However, this independent processing approach results in a long processing link between systems, resulting in time delays in data interaction and processing. For business scenarios that require quick response, this delay may result in untimely service, affecting user experience and business efficiency.
[0038] To this end, in some embodiments of the present application, a scene data processing system is provided, such as Figure 1 As shown, the system includes: a receiving module 101, which is used to receive scene data from each terminal device; a data definition module 102, which is used to define processing rules for the received scene data, wherein the processing rules include data classification rules, data format rules and matching rules with business scenarios; an execution engine module 104, which is used to perform target processing on the scene data after the processing rules are defined based on dynamic path selection; and an output module 106, which outputs the data after the target processing.
[0039] The receiving module 101 of the present system not only receives data from various terminal devices at close range, but also processes data from a wider network, such as data streams from cloud platforms or other business systems. These data streams may include real-time monitoring data, event trigger messages, device command responses, and real-time status data of devices. When receiving these data, the receiving module uses advanced data recognition technologies, such as machine learning algorithms, to perform real-time classification and preprocessing of data, ensuring that the data is accurately identified according to business needs and quickly forwarded to the data definition module.
[0040] The data definition module 102 plays an important role as a bridge in this system. It not only defines how data is classified and processed according to different business needs, but also presets processing rules that are closely related to various business scenarios. In the intelligent manufacturing scenario, for example, this module will define how sensor data such as temperature and pressure are processed, including the real-time and accuracy requirements of the data and how to handle abnormal values detected in the data. The setting of these rules ensures that data can flow accurately between different business logics, thereby supporting real-time decision-making and long-term strategic planning.
[0041] In one possible implementation, Figure 2As shown, the execution engine module 104 includes a routing module 1041 and a data conversion module 1042; the routing module 1041 is used to perform dynamic path selection on the scene data after the processing rules are defined; the data conversion module 1042 is used to perform target processing on the scene data that has completed path selection. It can be understood that the execution engine module 104 of the present system is a core module, in which the routing module 1041 and the data conversion module 1042 play a key role. The routing module 1041 selects the most appropriate data processing path based on the dynamic path selection algorithm according to the data content and business requirements. The data conversion module 1042 performs necessary conversion processing on the data on the selected path to ensure that the data input into the back-end business logic is accurate and meets the specifications.
[0042] In a possible implementation, the routing module 1041 performs dynamic path selection on the scene data after the processing rules are defined, including: the routing module allocates paths according to the key tags corresponding to the scene data based on the preset routing strategy. The routing module 1041 identifies the data and performs appropriate routing by setting key tags, such as device ID, event type, emergency level, etc. For example, for high-priority emergency event messages (such as factory equipment overheating alarms), the routing module will preferentially route the data to the emergency response processor for rapid processing. As a convertible implementation, the routing module will also select a specific path for more professional processing, such as a data analysis module or an early warning module, based on the data content itself (such as the temperature reading in the resource status data is too high or too low). It can be understood that the specific fusion processing method is an advanced feature of the scene data processing system, which allows the system to achieve in-depth analysis across data sources by comprehensively analyzing and processing the data from each source. In this way, the system can use data from different devices and sensors to obtain an overall and dynamic business environment state to support higher-level business decisions and policy deployment. For example, by comprehensively analyzing data from sensors at different locations within the factory, the system can adjust the temperature and humidity of the space in real time to ensure the optimization of the production environment.
[0043] In another possible implementation, the data conversion module 1042 follows the routing module and processes the data forwarded by the routing module. These processes include data format conversion: for example, converting the collected real-time data (which may be an analog signal) into a digital format, or converting the European date format into the American format to meet the processing standards of the system, or translating English data into Chinese data.
[0044] The above-mentioned target processing includes fusion processing and enhancement processing. In a specific implementation, the fusion processing of the scene data for which the path selection has been completed includes analyzing the scene data for which the path selection has been completed, and fusing the analyzed scene data according to a preset scene rule.
[0045] In another specific implementation, the enhancement processing includes at least one of the following: translation, packaging or normalization processing. For example, the packaging method may be to add metadata such as timestamps and geographic locations to enhance the availability and traceability of the data to achieve data enhancement effects; for better data display and subsequent processing, packaging may be to encapsulate or compress the original data. Normalization processing may be to standardize the data format and range, such as unifying the temperature output by all sensors from different units to degrees Celsius to facilitate comparison and analysis. For a multilingual operating environment, warning messages and commands can be translated from one language to another. Fusion processing is to combine data from different sources (such as event messages and resource status data) to form a more comprehensive view, which is convenient for performing more complex analysis and decision support.
[0046] In one embodiment, the specific implementation of the fusion processing is to combine the multi-source data after data conversion for in-depth analysis and processing, i.e., cross-source data analysis, such as using data from multiple sensors (e.g., temperature and humidity sensor data at different locations in the same factory workshop) to evaluate the state of the entire environment. Then, according to the preset scenario rules, the relevant data is fused to generate business insights according to specific business rules. For example, if the temperature of the machine exceeds the safety threshold three times in a row, the system will automatically generate a maintenance schedule.
[0047] In one embodiment, Figure 1 As shown, the system also includes a message channel module 103, and the message channel module 103 is used to transfer the scene data after the processing rules defined by the data definition module to the execution engine module according to the different types of scene data. Optionally, in this system, the message channel module uses Kafka as the core message queue system to achieve efficient data flow management and optimize data transmission performance. The message channel module 103 is responsible for transferring the scene data after the processing rules defined by the data definition module to the execution engine module through the set Kafka topics (topics) according to the type and priority of the data. This transmission process involves the filtering and type identification of messages. The message channel module 103 can also automatically adjust the priority and routing of the data flow according to the current load of the system to ensure the robustness and response speed of the system when facing large amounts of data.
[0048] In one embodiment, the system further includes an execution monitoring module 105, which is used to monitor the execution process of the execution engine module 105 performing target processing on the scene data after the processing rules are defined based on dynamic path selection. Specifically, the execution monitoring module 105 evaluates the overall performance of the system by collecting and analyzing the operating indicators (such as processing time, processing speed and success rate) of each sub-module (such as routing module and data conversion module). Monitoring these indicators helps to timely discover performance bottlenecks or potential system overloads, ensuring that the system can continue to operate stably.
[0049] The execution monitoring module 105 has a log management function. All data processing activities, from data access to data output, are recorded in detail in log files. This includes the processing path, processing time, processing results and any errors encountered for each data packet. This comprehensive log management is not only conducive to subsequent troubleshooting and system analysis, but also an important means of complying with data governance specifications.
[0050] Moreover, the execution monitoring module 105 also has the function of abnormality detection and response. The execution monitoring module has advanced abnormality detection capabilities and can identify in real time any behavior that deviates from the normal processing flow, such as data format errors, data processing delays, or data loss. Once an abnormal situation is detected, the monitoring module will automatically trigger a preset response mechanism, such as issuing an alarm, starting a backup system, or rerouting the task to other processing units. In addition, through monitoring and data analysis of system operation, the execution monitoring module 105 can also provide suggestions on system configuration and parameter adjustment to optimize the data processing flow and improve overall efficiency.
[0051] Through these functions, the execution monitoring module ensures that the system can maintain high efficiency and high reliability when processing complex and changing scenario data, while quickly responding to various emergencies and protecting the security and integrity of data processing activities.
[0052] The output module 106 ensures that the data after target processing can be accurately delivered to the downstream system or storage solution. The processed data is optimized and integrated according to the target format and structure to adapt to different business needs and application scenarios, ensuring that the data after target processing can be quickly and accurately distributed to various demand parties.
[0053] In some other embodiments of the present application, a scene data processing method is provided, such as Figure 3 As shown, the method includes S1-S4:
[0054] S1. Receive scene data from each terminal device.
[0055] The reception of scene data is the first step in data processing. The key lies in collecting data from various terminal devices with high efficiency and accuracy. The reception here can be a specialized subscription. Through efficient communication protocols and network configurations, the system can receive data transmitted by terminal devices in real time, reducing the delay of data transmission. In addition, preferably, a security check is performed on the received data, including security measures such as data encryption and prevention of data leakage, to protect the security of data during transmission.
[0056] S2. Define processing rules for the received scenario data, wherein the processing rules include data classification rules, data format rules, and matching rules with business scenarios.
[0057] Defining processing rules for received data is a critical part of data processing, ensuring that data can be classified and processed according to preset business rules. This step mainly covers the setting of data classification rules, data format rules, and matching rules with business scenarios.
[0058] By setting precise classification rules, the system can correctly classify large amounts of disorganized data for subsequent specialized processing. This helps improve the pertinence and efficiency of data processing, especially when real-time processing of data streams in a big data environment is particularly important.
[0059] Data format rules can ensure that all received data is converted into a unified format before processing, which not only reduces format compatibility issues in subsequent processing, but also improves the level of automation in data processing and reduces manual intervention.
[0060] The matching rules with business scenarios enable data to be accurately located according to its business significance and purpose before reaching the appropriate processing flow. Such rule settings ensure the accuracy and usefulness of data processing, ensuring that the data can subsequently serve the most appropriate business scenario, thereby enhancing the relevance and effectiveness of business decisions.
[0061] S3. Target processing is performed on the scene data after the processing rules are defined based on dynamic path selection.
[0062] This step is the core link of scene data processing, that is, target processing of data that meets the processing rules based on dynamic path selection. This step includes data routing decisions and conversion processing. In specific implementation, data routing decisions can use advanced algorithms (such as machine learning decision trees, rule engines, etc.) to determine the most suitable processing path for data in real time. This dynamic selection mechanism adjusts the data flow according to the real-time characteristics of the data and the current state of the system, optimizing processing efficiency and response speed.
[0063] Furthermore, according to the preset conversion rules, the data will be processed into a format and structure that can directly support business decisions and operations. For example, raw data is processed by algorithms and transformed into business insight reports, trend forecasts, etc., directly providing support for decision-makers. In addition, during the data processing process, any abnormal situations, such as data format errors, processing delays, etc., can be monitored and handled in real time, and the processing flow can be adjusted in time or alternative paths can be started to ensure the continuity of the data flow and the robustness of the system.
[0064] Through the S2 and S3 stages, the above scenario data processing method can effectively handle complex data environments. Through intelligent classification and path selection, the data processing process is made flexible and accurate, thereby improving the operating efficiency of the entire system and the business value of the data.
[0065] S4. Output the target processed data.
[0066] Through efficient data transmission solutions, such as distributed file systems or data transmission networks, the data processed by the target can be quickly and accurately distributed to various demand parties. In addition, the output step includes recording the tracking information of the data, such as processing time, data source and destination, etc., to enhance data traceability and management efficiency.
[0067] Please refer to the following Figure 4 , which shows a schematic diagram of an electronic device provided by some embodiments of the present application. Figure 4 As shown, the electronic device 2 includes: a processor 200, a memory 201, a bus 202 and a communication interface 203, wherein the processor 200, the communication interface 203 and the memory 201 are connected via the bus 202; the memory 201 stores a computer program that can be run on the processor 200, and the processor 200 executes the scene data processing method provided in any of the aforementioned embodiments of the present application when running the computer program, wherein the method includes: receiving scene data from each terminal device; defining processing rules for the received scene data, wherein the processing rules include data classification rules, data format rules and matching rules with business scenarios; performing target processing on the scene data after the processing rules are defined based on dynamic path selection; and outputting the target processed data.
[0068] The memory 201 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk memory. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 203 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used.
[0069] The bus 202 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. The memory 201 is used to store programs, and the processor 200 executes the programs after receiving execution instructions. The control method disclosed in any implementation of the above-mentioned embodiment of the present application may be applied to the processor 200, or implemented by the processor 200.
[0070] The processor 200 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 200. The above processor 200 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware decoding processor to be executed, or the hardware and software modules in the decoding processor can be executed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 201, and the processor 200 reads the information in the memory 201 and completes the steps of the scene data processing method in combination with its hardware.
[0071] An embodiment of the present application also provides a computer-readable storage medium corresponding to the scene data processing method provided in the aforementioned embodiment, on which a computer program is stored. When the computer program is run by a processor, it will execute the scene data processing method provided in any of the aforementioned embodiments.
[0072] In addition, examples of the computer-readable storage medium may also include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical or magnetic storage media, which are not listed here one by one.
[0073] In addition, an embodiment of the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the scene data processing method provided by any of the aforementioned embodiments, the method including: receiving scene data from each terminal device; defining processing rules for the received scene data, wherein the processing rules include data classification rules, data format rules and matching rules with business scenarios; performing target processing on the scene data after the processing rules are defined based on dynamic path selection; and outputting the data after the target processing.
[0074] Those skilled in the art will appreciate that the various component embodiments of the present application may be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. Those skilled in the art will appreciate that a microprocessor or digital signal processor (DSP) may be used in practice to implement some or all functions of some or all components of the virtual machine creation device according to the embodiments of the present application.
[0075] The above is only a preferred specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.
Claims
1. A scene data processing system, characterized in that: The system comprises: A receiving module, used for receiving scene data from each terminal device; A data definition module, used to define processing rules for received scenario data, wherein the processing rules include data classification rules, data format rules and matching rules with business scenarios; The execution engine module is used to perform target processing on the scene data after the defined processing rules based on dynamic path selection; Output module outputs the target processed data.
2. The scene data processing system according to claim 1, characterized in that: The scene data includes real-time data, event messages, command messages and resource status data.
3. The scene data processing system according to claim 1, characterized in that: The execution engine module includes a routing module and a data conversion module; the routing module is used to perform dynamic path selection on the scene data after the processing rules are defined; the data conversion module is used to perform target processing on the scene data after the path selection has been completed.
4. The scene data processing system according to claim 3, characterized in that: The routing module dynamically selects a path for the scene data after the processing rule is defined, including: The routing module allocates paths according to key tags corresponding to the scene data based on a preset routing strategy.
5. The scene data processing system according to claim 3, characterized in that: The target processing includes fusion processing and enhancement processing.
6. The scene data processing system according to claim 5, characterized in that: The fusion processing of the scene data for which the path selection has been completed includes analyzing the scene data for which the path selection has been completed, and fusing the analyzed scene data according to preset scene rules.
7. The scene data processing system according to claim 5, characterized in that: The enhancement process includes at least one of the following: translation, packaging or normalization.
8. The scene data processing system according to claim 1, characterized in that: The system further comprises a message channel module, which is used to transfer the scene data after the processing rules are defined by the data definition module to the execution engine module according to different types of scene data.
9. The scene data processing system according to claim 1, characterized in that: The system further comprises an execution monitoring module, which is used for monitoring the execution process of the execution engine module performing target processing on the scene data after the processing rules are defined based on the dynamic path selection.
10. A scene data processing method, characterized in that: The method comprises: Receive scene data from each terminal device; Defining processing rules for the received scenario data, wherein the processing rules include data classification rules, data format rules, and matching rules with business scenarios; Target processing is performed on the scene data after the defined processing rules based on dynamic path selection; Output the processed data of the target.