Data processing method, system, medium and equipment

Abstract

The invention discloses a data processing method and system, a medium and equipment, and aims to solve the problems of packet loss and software lagging easily caused by multi-data-source and high-real-time data flow in a rocket test by constructing a multi-thread parallel processing architecture. The method specifically comprises the following steps: receiving data from a plurality of data sources through a plurality of independent data receiving threads, and carrying out preliminary analysis on the received data by the data receiving threads according to a frame format protocol to obtain frame data; performing secondary analysis on the frame data according to a communication protocol through a plurality of independent data analysis threads to obtain message data; message data is processed by a plurality of independent data consumption threads, the data consumption threads configured to perform data display and / or data storage operations. According to the invention, complete parallelism of data receiving, analysis and display / storage processes is realized, blocking interference among data processing links is effectively avoided, and data processing efficiency and system reliability are improved.

Description

Technical Field

[0001] This invention relates to the field of aerospace measurement and control technology, and in particular to a data processing method, system, medium, and device. Background Technology

[0002] Rockets often generate a large amount of data during a single flight test. Ground software needs to receive, analyze, display, and store this data. As rocket control systems become increasingly complex, the amount of test data also increases, often generating hundreds or even thousands of test files in a single test.

[0003] When multiple data sources receive a large amount of data simultaneously, problems such as data loss, software lag, and system unresponsiveness often occur due to the inability to process the data, which places high demands on the performance of the software.

[0004] Therefore, designing an efficient and reliable ground data processing system that can ensure the real-time, complete, and error-free reception, parsing, and storage of rocket test data from multiple data sources, with high concurrency and large data volumes, while also ensuring a smooth software interface response and avoiding data loss and system lag, has become a key technical problem that urgently needs to be solved in this field. Summary of the Invention

[0005] In view of this, embodiments of the present invention provide a data processing method system to solve problems such as lag and data loss during software operation.

[0006] To achieve the above objectives, firstly, a data processing method is provided, comprising the following steps: Data reception and preliminary parsing steps: Data is received from multiple data sources through multiple independent data receiving threads, and the received data is initially parsed according to the frame format protocol by the data receiving threads to obtain frame data; Secondary parsing step: The frame data is parsed a second time according to the communication protocol through multiple independent data parsing threads to obtain message data; Data consumption steps: Message data is processed through multiple independent data consumption threads, which are configured to perform data display and / or data storage operations.

[0007] Secondly, a data processing system is provided, comprising: The data receiving and preliminary parsing module is configured to receive data from multiple data sources through multiple independent data receiving threads, and the data receiving threads perform preliminary parsing of the received data according to the frame format protocol to obtain frame data; The secondary parsing module, connected to the data receiving and preliminary parsing module, is configured to perform secondary parsing on the frame data from the data receiving and preliminary parsing module according to the communication protocol through multiple independent data parsing threads to obtain message data; The data consumption module, connected to the secondary parsing module, is configured to process message data from the secondary parsing module through multiple independent data consumption threads. The data consumption module is further configured to perform data display and / or data storage operations.

[0008] Thirdly, an electronic device is provided, comprising: One or more processors; Storage device for storing one or more programs. When one or more programs are executed by one or more processors, the one or more processors implement the data processing method as described in the first aspect.

[0009] Fourthly, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the data processing method of the first aspect.

[0010] Fifthly, a computer program product is provided, comprising a computer-readable storage medium on which a computer program is stored, wherein when the computer program is run, the data processing method of the first aspect is executed.

[0011] The above technical solution has the following beneficial technical effects: This method constructs a multi-level independent dedicated thread pipeline for receiving, parsing, and consuming data, and introduces an asynchronous buffer data cache between threads. This achieves efficient parallel processing and decoupling of data receiving, parsing, and display / storage stages, fundamentally avoiding mutual interference between different data sources and different processing stages. It effectively solves the problems of data loss and software lag in high real-time and large data volume scenarios, and significantly improves the overall reliability and efficiency of rocket test data processing.

[0012] This invention employs multi-threaded processing for receiving, parsing, and displaying data. Multiple data sources and multiple display interfaces are processed using multi-threading to ensure that data processing occurs simultaneously without interference.

[0013] This invention accumulates frequently communicated data over a period of time before storing and displaying it, thereby reducing resource consumption caused by frequent file reading and writing and interface refresh. Attached Figure Description

[0014] The accompanying drawings are provided to better understand the invention and are not intended to unduly limit the scope of the invention. Wherein: Figure 1 This is a flowchart of a data processing method according to an embodiment of the present invention; Figure 2 This is a flowchart illustrating a data processing method as an example in an embodiment of the present invention; Figure 3 This is a flowchart of a data processing method as another example of an embodiment of the present invention; Figure 4 This is an example diagram illustrating the two parsing processes in an embodiment of the present invention; Figure 5 This is a functional block diagram of a data processing system according to an embodiment of the present invention; Figure 6 This is a functional block diagram of another data processing system according to an embodiment of the present invention; Figure 7 This is a schematic diagram of the structure of a data processing system according to an embodiment of the present invention; Figure 8 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0015] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of the present invention, including various details to aid understanding. These details should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the invention. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0016] Example 1

[0017] Figure 1 This is a flowchart of a data processing method according to an embodiment of the present invention. Figure 1 As shown, the data processing method includes the following steps: S11. Data reception and preliminary parsing steps: Data is received from multiple data sources through multiple independent data receiving threads, and the received data is initially parsed according to the frame format protocol by the data receiving threads to obtain frame data; S12, Secondary parsing step: The frame data is parsed a second time according to the communication protocol through multiple independent data parsing threads to obtain message data; S13. Data Consumption Steps: Message data is processed through multiple independent data consumption threads, which are configured to perform data display and / or data storage operations.

[0018] This invention constructs a highly parallelized processing architecture through a three-tiered independent thread pipeline for data reception and preliminary parsing, secondary parsing, and data consumption. This fundamentally solves the system processing capacity bottleneck problem under multiple data sources and high-concurrency data influx. This architecture allows data reception, deep parsing, and final consumption (display / storage), three tasks with different computational intensiveities, to be executed in parallel without mutual waiting or blocking. This significantly improves the overall system throughput and real-time performance, ensuring the system remains responsive under high load and avoiding data loss or software lag caused by the saturation of a single processing stage. It provides a reliable software performance foundation for high-speed data scenarios such as rocket experiments.

[0019] In a further embodiment, the data receiving and preliminary parsing steps include: The data receiving thread stores the received data into the first data buffer and performs preliminary parsing on the data in the first data buffer. After parsing out frame data, the frame data is stored into the second data buffer.

[0020] This scheme introduces a first data buffer and a second data buffer, decoupling the data receiving thread from the data parsing thread and acting as a data buffer. The first buffer allows the receiving thread to continuously write raw data streams at high speed without waiting for parsing operations; the second buffer stores pre-parsed frame data with complete logic. This design effectively smooths out instantaneous fluctuations in the data stream, preventing starvation or overload of the parsing thread due to uneven packet arrival, thereby improving thread efficiency and system stability.

[0021] In a further embodiment, the secondary parsing step includes: The data parsing thread reads frame data from the second data buffer, performs secondary parsing, and stores the parsed message data into the third data buffer.

[0022] This solution further introduces a third data buffer, achieving complete decoupling between data parsing and data consumption, thus enhancing the system's robustness and modularity. The data parsing thread can quickly store the parsing results (message data) into the third buffer and immediately process the next frame of data, without needing to worry about the processing speed of the consumer end (e.g., interface display or data storage). This effectively isolates the impact of fluctuations in the parsing process on the real-time performance of the consumer end, ensuring that even if the consumer end experiences occasional I / O latency or slightly longer interface rendering time, it will not block the critical data parsing process.

[0023] In a further embodiment, the data consumption step includes: The data consumption thread reads the required message data from the third data buffer and consumes it.

[0024] This solution provides a flexible, on-demand data acquisition mechanism for data consumers. Multiple consumer threads can independently and asynchronously subscribe to and retrieve the message data they need from the third buffer, achieving efficient data distribution. This design supports flexible configuration and expansion of the display interface. For example, one display interface only needs to subscribe to the speed parameter, while another interface only needs to subscribe to the attitude parameter, thereby avoiding unnecessary data processing and improving the efficiency and targeting of the consumer threads.

[0025] In a further embodiment, in the data receiving and preliminary parsing step, multiple data receiving threads are created for at least one data source to receive and process data from the data source in parallel.

[0026] This solution allows for the creation of multiple receiving threads for a single data source, enhancing the ability to receive and process data from high-bandwidth data sources. When the data rate of a data source (such as the primary telemetry data source) is abnormally high, the load can be distributed by having multiple threads receive and parse data in parallel, thus preventing a single thread from becoming a bottleneck and ensuring that the system can keep up with the maximum output rate of the data source, preventing data loss.

[0027] In a further embodiment, different data parsing threads are configured to process frame data from different data sources.

[0028] This solution configures data parsing threads specifically for each data source, optimizing parsing efficiency and reducing system complexity by avoiding context switching and resource contention. Dedicated threads handle data from fixed data sources, resulting in higher cache hit rates and purer processing logic. Furthermore, this design simplifies data routing logic, making system debugging and performance monitoring more direct and improving system maintainability.

[0029] In a further embodiment, during the data consumption step, independent display threads are created for different display interfaces, and each display thread is configured to subscribe to and read message data corresponding to its display interface only.

[0030] This solution creates an independent display thread for each display interface on the consumer side and adopts a subscription mechanism to achieve isolation between display tasks and high performance. Each display interface may have different refresh rates, data volumes, and rendering complexities; the independent thread ensures that stuttering or abnormalities in one interface will not affect the smooth display of other interfaces. The subscription mechanism guarantees that the display thread only processes the data it cares about, significantly reducing unnecessary memory copying and computational overhead, making the display interface respond faster and smoother.

[0031] In a further embodiment, during the data consumption step, message data is written to a database or file for storage via a dedicated storage thread.

[0032] This solution separates time-consuming I / O operations from real-time data display, ensuring the responsiveness of the user interface. Data storage (especially writing to databases or files) is typically a relatively slow process; if it's executed in the same thread as display operations, it can directly cause the display to freeze. By handling storage tasks in a dedicated thread, the display thread can focus on updating the interface, fundamentally avoiding UI lag caused by storage operations and improving the user experience.

[0033] In a further embodiment, after the secondary parsing step and before the data consumption step, a batch transfer step is also included: All message data stored in the third data cache within a predetermined time period is accumulated, and then all message data accumulated within the predetermined time period is batch-transferred to the fourth data cache. The data consumption steps include: reading message data from the fourth data cache and consuming it.

[0034] This solution transforms high-frequency, fragmented data write operations into low-frequency, batch block operations, significantly improving the write efficiency of downstream consumers (especially database storage) and the overall system performance. By packaging a large number of small messages into a single batch for writing within a short period, it drastically reduces I / O operations and transaction overhead, avoiding frequent lock contention in the database, thus resolving the most common performance bottleneck on the data consumer side. Simultaneously, this also provides a smoother data source for the display, avoiding frequent interface refreshes and jitter caused by real-time processing of each message.

[0035] The following example, using data processing methods in the rocket field (but not limited to this), will provide a more detailed explanation of the above technical solutions: Figure 2 This is a flowchart of a data processing method according to an embodiment of the present invention. Figure 1 As shown, an embodiment of the present invention provides a data processing method comprising the following steps: S21. Data is received from multiple data sources through multiple independent data receiving threads, with each data source corresponding to a different independent data receiving thread. S22. The data receiving thread first stores the received data into the first data buffer, and performs preliminary parsing of the data in the first data buffer according to the frame format protocol. When a complete frame of data is parsed, a complete frame of data is extracted from the first data buffer and stored into the second data buffer. S23. Create a corresponding independent data parsing thread for each data source. The data parsing thread reads a complete frame of data from the second data cache used by the data source corresponding to itself, performs secondary parsing according to the communication protocol, parses out multiple types of message data, and stores the parsed message data into the third data cache. S24. Create a corresponding independent data display thread for the display interface. The data display thread is configured to: read the parsed message data required by the display interface corresponding to the data display thread from the third data cache and display it on the display interface; and / or store the parsed message data read from the third data cache through the data storage module.

[0036] Furthermore, the multiple data sources include at least two of wired Ethernet, wireless Ethernet, and 1553B bus; the multiple types of message data include various types of servo device messages, inertial navigation system messages, and flight controller device messages from the rocket control system; the sizes of the first data buffer, the second data buffer, and the third data buffer are configurable to adapt to the data traffic.

[0037] Specifically, the rocket has a first-stage flight controller, a first-stage engine controller, a second-stage flight controller, and a second-stage engine controller. Each device transmits data via 1553B and Ethernet. To ensure reliable communication, 1553B and Ethernet are usually configured with dual redundancy backups.

[0038] Specifically, the flight controller typically needs to communicate with servo mechanisms, engine controllers, inertial navigation systems (INS), rate gyroscopes, measurement center programmers, storage devices, ground-based measurement and control systems, air cushion pressure transducers, batteries, etc. Depending on the communication protocol, there are often hundreds of types of data, such as INS measurement data, INS calibration parameters, rate gyroscope measurement data, servo mechanism motion angles, engine controller solenoid valve commands, engine controller switch input data, battery self-test information, battery telemetry data, battery control commands, flight controller switch input data, flight controller timing data, air cushion pressure transducer data, etc.

[0039] The data processing method provided in this invention establishes independent data receiving and parsing threads for each data source, and independent data display threads for each display interface, while also employing a multi-level data caching mechanism to construct a highly parallel processing architecture. This design enables key processes such as data receiving, protocol parsing, interface display, and data storage to truly operate synchronously and without interference, fundamentally avoiding data loss and software lag caused by processing capacity bottlenecks, a common problem in traditional single-threaded or insufficient-thread architectures. In particular, this method effectively mitigates competition and interference between multiple data sources through thread isolation, ensuring that the system maintains stable and efficient real-time processing capabilities even when facing high-frequency, bursty data streams generated by rocket tests, thereby improving the reliability and response efficiency of the entire ground telemetry and control system.

[0040] Figure 3 This is a flowchart of another data processing method according to an embodiment of the present invention. Figure 3As shown, the data processing method includes the following steps: S31. Data is received from multiple data sources through multiple independent data receiving threads, with each data source corresponding to a different independent data receiving thread. S32. The data receiving thread first stores the received data into the first data buffer, and performs preliminary parsing of the data in the first data buffer according to the frame format protocol. When a frame of data is parsed, a complete frame of data is extracted from the first data buffer and stored into the second data buffer. S33. Create a corresponding independent data parsing thread for the data source. The data parsing thread reads a complete frame of data from the second data cache used by the data source corresponding to itself, performs secondary parsing according to the communication protocol, parses out multiple types of message data, and stores the parsed message data into the third data cache. like Figure 4 As shown, this solution implements an efficient and modular two-level data parsing method. First, the system processes the raw data stream from the physical link (e.g., Ethernet) through a single parsing step. Based on its fundamental communication protocol (including fields such as frame header, data length, data type, data area, and frame trailer), it performs verification and frame integrity checks, extracting the valid data area and storing it as a complete data frame in the second data buffer corresponding to the data source. Then, the system enters a second parsing stage: an independent parsing thread is created for each data source. Each thread is responsible for reading data frames from its own second data buffer and performing deep parsing according to the specific application layer protocol (e.g., when the data type is identified as 0x13, it follows the protocol regarding the servo mechanism's motion angle in the onboard device). This process accurately extracts multiple parameters with clear physical meaning from a single data frame (e.g., time, motion angles of different actuators, i.e., multiple types of message data). Finally, these structured and semantically clear message data are classified and stored in a third data buffer, providing clear and directly usable data support for upper-layer applications (e.g., status monitoring, data analysis). This design improves data processing efficiency, maintainability, and system scalability by decoupling communication frame processing from business data parsing and using independent threads to achieve parallel processing.

[0041] S34. Accumulate all parsed message data stored in the third data cache within a predetermined time period, then batch transfer all the accumulated parsed message data within the predetermined time period to the fourth data cache, and clear the parsed message data that has been transferred from the third data cache after batch transfer. S35. Create a corresponding independent data display thread for the display interface. The data display thread is configured to: read the parsed message data required by the display interface corresponding to the data display thread from the fourth data cache and display it on the display interface; and / or store the message data in the fourth data cache through the data storage module.

[0042] Furthermore, in step S34, the predetermined time period is 1 second; in step S5, the displayed data is refreshed on the display interface at predetermined time intervals; the predetermined time interval is 1 second.

[0043] This embodiment effectively reduces frequent data write operations and lowers the read / write burden on the computer's file system by increasing the data cache or cache space and writing the data uniformly after accumulating a certain amount of data. Specifically, for data with short communication cycles (e.g., 10ms), storing each received data packet immediately would lead to frequent file read / write operations, causing system resource consumption and wasted time. By temporarily storing these small data packets in a larger data cache and then writing them uniformly to the storage medium after accumulating a second's worth of data, the number of writes can be significantly reduced, lowering system I / O overhead. This also improves the real-time performance of data processing and system stability, achieving more efficient storage performance.

[0044] Furthermore, the multiple data sources include at least two of wired Ethernet, wireless Ethernet, and 1553B bus; the multiple types of message data include various types of servo device messages, inertial navigation system messages, and flight controller device messages from the rocket control system; the sizes of the first data buffer, second data buffer, third data buffer, and fourth data buffer are configurable to adapt to data traffic.

[0045] The data processing method provided in this invention, based on a multi-threaded parallel architecture, introduces a data accumulation and batch transfer mechanism. By adding a fourth data cache and accumulating message data from multiple third data caches within a specific time period before batch transfer, this method reduces frequent write operations to storage devices and the number of display refreshes. This batch processing strategy effectively solves the I / O efficiency bottleneck problem in high-frequency, low-data-volume scenarios, significantly improving the overall throughput of data storage and avoiding display lag and data flickering by reducing the interface refresh frequency. This solution enables the system to further optimize computational resource consumption while ensuring real-time performance, and is particularly suitable for the long-term stable processing of massive, high-concurrency data streams generated in rocket experiments, demonstrating higher engineering practical value and system reliability.

[0046] Rockets typically generate massive amounts of data during a single flight test, receiving data from various sources such as wired Ethernet, wireless Ethernet, and 1553B bus. Ground-based data processing software needs to receive, parse, display, and store this data. As rocket control systems become increasingly complex and real-time requirements become more stringent, the amount of test data generated also increases dramatically. Dozens of communication messages are generated every 10ms or even shorter intervals, and a single flight test can often produce hundreds or even thousands of test files. This increase in the number of messages and data sources puts pressure on the software's data processing capabilities, leading to data loss, software lag, and a range of other problems.

[0047] like Figure 5 and Figure 6 As shown, the embodiments of the present invention mainly solve the above problems from two aspects: multi-threaded processing and data caching.

[0048] First, separate threads are created for receiving, parsing, and displaying data to ensure that these processes can occur simultaneously without interference. When data arrives from the data source, the receiving thread first receives the data and parses it according to the frame format protocol (the receiving thread performs initial data parsing according to the transmission protocol to obtain preliminary parsing results. The subsequent parsing thread performs secondary parsing on the preliminary parsing results from the receiving thread). When a complete data frame is obtained, it is copied to data buffer 2. (When data arrives from the data source, the receiving thread first receives the data and stores it in data buffer 1. It then parses it according to the frame format protocol. When a complete data frame is obtained, it is copied to data buffer 2. The buffer has a certain size; data exceeding the limit will be overwritten. Simultaneous parsing while buffering removes the data.) The data parsing thread parses the data in data cache 2 according to the communication protocol, extracting messages of each type (for example, a rocket control system has multiple devices such as servos, inertial navigation systems, and flight controllers; these devices need to communicate directly, and communication involves communication protocols, such as servo angle control commands and device self-test commands; after parsing, the result of each message or type of communication is obtained), and then storing or displaying them separately (for example, storing them in data cache 3 as shown in the diagram). This allows for parsing while receiving data. When a large amount of data arrives instantly, parsing may be delayed; multi-threaded processing stores the data in the data cache, preventing data loss due to the computer being busy parsing data and unable to receive it in time.

[0049] Simultaneously, multiple data sources and multiple display interfaces (or display pages) each create their own independent threads for processing. Data obtained from different IP addresses or from 1553B are referred to as different data sources, and the data from different data sources has overlapping and different parts. Since the communication protocol is the same, different data sources can share the data receiving and parsing methods, only needing to create their own threads, avoiding mutual interference between multiple data sources. The parsed data will be displayed in different locations as needed, or stored without being displayed. Multiple display interfaces create their own display data threads to display data as needed, and display interface refresh, data receiving, and data parsing do not interfere with each other.

[0050] Secondly, increasing the data cache avoids wasting time caused by frequent operations. For example, some data communication cycles are 10ms. Storing this type of data would involve frequent file read / write operations if each data packet is stored immediately upon arrival, increasing the computer's workload. Since the data volume of a single communication is often small, storing this data together and then storing it all at once after 1 second would significantly improve data storage efficiency (copying the accumulated cached data for 1 second to data cache 4, and then clearing the data in data cache 3). Similarly, frequent refreshes during data display increase computer consumption and can cause data flickering due to exceeding the human eye's reaction time. Using a 1-second refresh strategy for the data to be displayed solves these problems.

[0051] An experiment generates a lot of communication data. Some of this data only needs to be stored in the computer, while some needs to be displayed on a page. The specific data to be stored and displayed is specified in the task description. The data storage module represents the module that performs data storage operations.

[0052] In this embodiment of the invention, the receiving, parsing, and displaying of data are processed using multi-threading. Multiple data sources and multiple display interfaces are processed using multi-threading to ensure that data processing occurs simultaneously and without interference.

[0053] This invention addresses the issue of storing and displaying frequently communicated data only after it has accumulated over a period of time, thereby reducing resource consumption caused by frequent file reading and writing and interface refreshes.

[0054] Example 2

[0055] like Figure 7 As shown, an embodiment of the present invention provides a data processing system, which includes: The data receiving and preliminary parsing module is configured to receive data from multiple data sources through multiple independent data receiving threads, and the data receiving threads perform preliminary parsing of the received data according to the frame format protocol to obtain frame data; The secondary parsing module, connected to the data receiving and preliminary parsing module, is configured to perform secondary parsing on the frame data from the data receiving and preliminary parsing module according to the communication protocol through multiple independent data parsing threads to obtain message data; The data consumption module, connected to the secondary parsing module, is configured to process message data from the secondary parsing module through multiple independent data consumption threads. The data consumption module is further configured to perform data display and / or data storage operations.

[0056] Furthermore, the data receiving and preliminary parsing module includes: The first cache unit is configured to store the raw data received by the data receiving thread; The data receiving thread is further configured to perform preliminary parsing of the data in the first buffer unit, and after parsing out frame data, store the frame data into the second buffer unit.

[0057] Furthermore, the secondary parsing module is connected to the second cache unit and configured as follows: The frame data is read from the second buffer unit, parsed a second time, and the parsed message data is stored in the third buffer unit.

[0058] Furthermore, the data consumption module is connected to the third cache unit and configured to read message data from the third cache unit for consumption.

[0059] Furthermore, the data receiving and preliminary parsing module is configured to create multiple data receiving threads for at least one data source to receive and process the data from that data source in parallel.

[0060] Furthermore, the secondary parsing module is configured to enable different data parsing threads to process frame data from different data sources.

[0061] Furthermore, the data consumption module is configured as follows: Create independent display threads for different display interfaces, and ensure that each display thread can only subscribe to and read message data corresponding to its display interface.

[0062] Furthermore, the data consumption module includes a dedicated storage thread configured to write message data to a database or file for storage.

[0063] Furthermore, the system may also include a batch transfer module, which is located between the secondary parsing module and the data consumption module, and is configured as follows: All message data stored in the third cache unit within a predetermined time period is accumulated, and all message data accumulated within the predetermined time period is batch-transferred to the fourth cache unit. The data consumption module is connected to the fourth cache unit and is configured to read message data from the fourth cache unit for consumption.

[0064] For details on the technical effects and technical details of this system, please refer to the description in the foregoing method embodiments, which will not be repeated here.

[0065] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0066] This invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements any of the data processing methods described above.

[0067] If integrated modules / units are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. Computer-readable media can include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. Of course, there are other types of readable storage media, such as quantum memories, graphene memories, etc. It should be noted that the content contained in computer-readable media may be appropriately added to or subtracted from the requirements of legislation and patent practice in a jurisdiction. For example, in some jurisdictions, computer-readable media may not include electrical carrier signals and telecommunication signals, in accordance with legislation and patent practice.

[0068] The present invention also provides an electronic device. The electronic device according to an embodiment of the present invention includes: one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the data processing method provided by the present invention.

[0069] The following is for reference. Figure 8 It shows a schematic diagram of the structure of a computer system 800 suitable for implementing an electronic device according to embodiments of the present invention. Figure 8 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of the present invention.

[0070] like Figure 8 As shown, the computer system 800 includes a central processing unit (CPU) 801, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 802 or programs loaded from storage section 808 into random access memory (RAM) 803. The RAM 803 also stores various programs and data required for the operation of the computer system 800. The CPU 801, ROM 802, and RAM 803 are interconnected via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0071] The following components are connected to I / O interface 808: an input section 806 including a keyboard, mouse, etc.; an output section 807 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network interface card such as a LAN card, modem, etc. The communication section 809 performs communication processing via a network such as the Internet. Drive 810 is also connected to I / O interface 805 as needed. Removable media 811, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 810 as needed so that computer programs read from them can be installed into storage section 808 as needed.

[0072] In particular, according to the embodiments disclosed in this invention, the processes described in the above main step diagrams can be implemented as computer software programs. For example, embodiments of this invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the main step diagrams. In the above embodiments, the computer program can be downloaded and installed from a network via communication section 809, and / or installed from removable medium 811. When the computer program is executed by central processing unit 801, it performs the functions defined in the system of this invention.

[0073] It should be noted that the computer-readable medium shown in this invention can be a computer-readable signal medium, a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0074] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0075] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A data processing method, characterized in that, Includes the following steps: Data reception and preliminary parsing steps: Data is received from multiple data sources through multiple independent data receiving threads, and the received data is initially parsed according to the frame format protocol by the data receiving threads to obtain frame data; Secondary parsing step: The frame data is parsed a second time according to the communication protocol through multiple independent data parsing threads to obtain message data; Data consumption steps: The message data is processed by multiple independent data consumption threads, which are configured to perform data display and / or data storage operations.

2. The method as described in claim 1, characterized in that, The data reception and preliminary parsing steps include: The data receiving thread stores the received data into the first data buffer and performs preliminary parsing on the data in the first data buffer. After parsing out frame data, the frame data is stored into the second data buffer.

3. The method as described in claim 2, characterized in that, The secondary analysis step includes: The data parsing thread reads the frame data from the second data buffer, performs secondary parsing, and stores the parsed message data into the third data buffer.

4. The method as described in claim 3, characterized in that, The data consumption steps include: The data consumption thread reads the required message data from the third data cache and consumes it.

5. The method as described in claim 1, characterized in that, In the data receiving and preliminary parsing step, multiple data receiving threads are created for at least one data source to receive and process the data from the data source in parallel.

6. The method as described in claim 1, characterized in that, In the data consumption step, independent display threads are created for different display interfaces, and each display thread is configured to subscribe to and read message data corresponding to its display interface.

7. The method as described in claim 3 or 4, characterized in that, After the secondary parsing step and before the data consumption step, a batch transfer step is also included: All message data stored in the third data cache within a predetermined time period is accumulated, and then all message data accumulated within the predetermined time period is batch-transferred to the fourth data cache. The data consumption step includes: reading message data from the fourth data cache and consuming it.

8. A data processing system, characterized in that, include: The data receiving and preliminary parsing module is configured to receive data from multiple data sources through multiple independent data receiving threads, and the data receiving threads perform preliminary parsing of the received data according to the frame format protocol to obtain frame data. The secondary parsing module, connected to the data receiving and preliminary parsing module, is configured to perform secondary parsing on the frame data from the data receiving and preliminary parsing module according to the communication protocol through multiple independent data parsing threads to obtain message data; The data consumption module, connected to the secondary parsing module, is configured to process the message data from the secondary parsing module through multiple independent data consumption threads. The data consumption module is further configured to perform data display and / or data storage operations.

9. An electronic device, characterized in that, include: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the data processing method as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the data processing method as described in any one of claims 1-7.

Images