Stream data processing method and device, computer device and storage medium

Abstract

The application relates to the field of big data processing, and discloses a stream data processing method and device, computer equipment and a storage medium, the method comprising the following steps: acquiring stream data meeting preset processing standards; indexing, first filtering and first field processing the stream data to generate first processing data; acquiring associated data associated with the first processing data, and performing second filtering and second field processing on the associated data to generate second processing data; and performing a logical processing operation on the second processing data to complete updating of the stream data. The application reduces the development difficulty and maintenance cost of stream data processing.

Description

Technical Field

[0001] This invention relates to the field of big data processing, and in particular to a streaming data processing method, apparatus, computer equipment, and storage medium. Background Technology

[0002] With the rapid development of information technology, the global data volume is growing exponentially, propelling human society into the era of big data. However, current streaming computing methods based on big data suffer from drawbacks such as complex architecture, complex logic, and high implementation costs. Moreover, the data processing process also faces challenges such as diverse data processing sequences, complex interconnected data, diverse data versions, and multi-threaded modifications. Summary of the Invention

[0003] Therefore, it is necessary to provide a streaming data processing method, apparatus, computer equipment, and storage medium to address the aforementioned technical problems, so as to reduce the difficulty and cost of processing streaming data.

[0004] A streaming data processing method, comprising:

[0005] Acquire stream data that conforms to preset processing standards;

[0006] The streaming data is indexed, filtered in the first way, and processed in the first field to generate the first processed data;

[0007] Obtain associated data related to the first processed data, and perform second filtering and second field processing on the associated data to generate second processed data;

[0008] Perform logical processing operations on the second processed data to complete the update of the streaming data.

[0009] A streaming data processing apparatus, comprising:

[0010] The stream data acquisition module is used to acquire stream data that conforms to preset processing standards;

[0011] The first processing module is used to index, filter, and process the first field of the streaming data to generate first processed data.

[0012] The second processing module is used to obtain associated data related to the first processed data, and to perform second filtering and second field processing on the associated data to generate second processed data.

[0013] The logic processing module is used to perform logic processing operations on the second processed data to complete the update of the stream data.

[0014] A computer device includes a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor, wherein the processor implements the above-described streaming data processing method when executing the computer-readable instructions.

[0015] One or more readable storage media storing computer-readable instructions, which, when executed by one or more processors, cause the one or more processors to perform the streaming data processing method described above.

[0016] The aforementioned streaming data processing method, apparatus, computer equipment, and storage medium decompose streaming data processing into three main steps: generating first-processed data, generating second-processed data, and logical processing operations. Each step is independent of the others but supports each other, which can solve various data conflicts, data versioning, and data consistency problems caused by concurrency under multi-threading. Moreover, the configuration process of each step is relatively simple and does not require a lot of coding, thus reducing the development difficulty and maintenance cost of streaming data processing. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a schematic diagram of an application environment for a streaming data processing method according to an embodiment of the present invention;

[0019] Figure 2 This is a flowchart illustrating a streaming data processing method according to an embodiment of the present invention;

[0020] Figure 3 This is a schematic diagram of a streaming data processing device according to an embodiment of the present invention;

[0021] Figure 4 This is a schematic diagram of a computer device according to an embodiment of the present invention. Detailed Implementation

[0022] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] The streaming data processing method provided in this embodiment can be applied to, for example, Figure 1In this application environment, the client communicates with the server. Clients include, but are not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster consisting of multiple servers.

[0024] In one embodiment, such as Figure 2 As shown, a streaming data processing method is provided, which is applied to... Figure 1 Taking the server side as an example, the explanation includes the following steps S10-S40.

[0025] S10. Obtain stream data that conforms to the preset processing standards.

[0026] Understandably, preset processing criteria can be set according to actual needs. Here, streaming data that conforms to the preset processing criteria can be data that has been preprocessed from raw streaming data. Streaming data refers to data used for streaming computation. In some instances, the aforementioned streaming data can be received through certain data interfaces.

[0027] S20. Index the streaming data, perform first filtering and first field processing to generate first processed data.

[0028] Understandably, this step primarily involves indexing, initial filtering, and initial field processing of the streaming data to resolve version conflicts in high-concurrency, multi-threaded scenarios when storing streaming data. Here, only the normalization of the streaming data is involved; the complexity of the backend implementation and data versioning issues are not relevant. Each of the indexing, initial filtering, and initial field processing steps has its own configuration file. These configuration files can be modified.

[0029] S30. Obtain associated data related to the first processed data, and perform second filtering and second field processing on the associated data to generate second processed data.

[0030] Understandably, the processing of the first set of data can be implemented through configuration files (such as JSON). This step can resolve the issue of historical data rollback caused by related data (i.e., data after the state of the first set of data has changed), resolve data version association issues that may occur in high-concurrency, multi-threaded scenarios, resolve version conflicts during data saving, and ensure eventual data consistency. Specifically, there are corresponding configuration files for association, second filtering, and second field processing. These configuration files can be modified.

[0031] S40. Perform logical processing operations on the second processed data to complete the update of the stream data.

[0032] Understandably, the processing of the second set of data can be implemented through configuration files (such as JSON). This configuration file summarizes all the logical processing operations for the streaming data. Developers only need to focus on configuring the summarized rules using the JSON configuration file during development, without needing to worry about issues involved in other steps. Logical processing operations include, but are not limited to, adding, modifying, deleting, and querying. If the logical processing operation executes successfully, the streaming data update is complete.

[0033] In some examples, after updating the streaming data, it can be visualized based on actual business requirements. For specific visualization setup methods, please refer to existing technologies; they will not be elaborated upon here.

[0034] This embodiment decomposes streaming data processing into three main steps: generating first processing data, generating second processing data, and logical processing operations. Each step is independent of the others but supports each other, which can solve various data conflicts, data versioning, and data consistency problems caused by concurrency under multi-threading. Moreover, the configuration process of each step is relatively simple and does not require a lot of coding, which reduces the development difficulty and maintenance cost of streaming data processing.

[0035] Optionally, step S10, that is, before acquiring the stream data that conforms to the preset processing standard, further includes:

[0036] S11. Obtain raw stream data through a preset method;

[0037] S12. Convert the original stream data into the stream data according to the preset conversion program.

[0038] Optionally, the first processed data includes the primary key and index data of the streaming data;

[0039] Step S20, namely, indexing, filtering, and processing the first field of the streaming data to generate first processed data, includes:

[0040] S201. Obtain the index configuration file and primary key configuration file corresponding to the stream data;

[0041] S202. Configure the primary key of the stream data according to the primary key configuration file to generate the primary key of the stream data;

[0042] S203. Configure the index of the stream data according to the index configuration file to generate the index data of the stream data.

[0043] Understandably, the purpose of this embodiment is to achieve standardization, filtering, special processing at the field level, and resolution of version conflict issues in high-concurrency scenarios for streaming data.

[0044] Here, a primary key configuration file can be used to generate the primary key for the streaming data, and an index configuration file can be used to generate the index data for the streaming data. In some examples, the primary key configuration file and the index configuration file can be the same configuration file (i.e., this configuration file defines both how to generate the primary key and how to generate the index data). The primary key configuration file defines the fields in the streaming data used to generate the primary key. The index configuration file defines the fields in the streaming data used to generate the index data. The index configuration file and the primary key configuration file can be defined according to actual needs. The index data is mainly used for data association in subsequent data processing; this index configuration file must be configured, otherwise an error message will be displayed.

[0045] In one example, the streaming data could be the source table Employee, with the primary key being the employee ID and the index data being the department ID and the ID type + ID number (idType + idCode).

[0046] Optionally, step S20, namely, indexing, filtering, and processing the first field of the streaming data to generate first processed data, includes:

[0047] S204. Obtain the filter rule configuration file and the first field configuration file corresponding to the stream data;

[0048] S205. Filter the indexed stream data according to the filtering rule configuration file to generate filtered stream data;

[0049] S206. Process the filtered stream data according to the first field configuration file to generate the first processed data.

[0050] Understandably, filtering rule configuration files can be used to filter streaming data, resulting in filtered streaming data, i.e., filtered streaming data. A filtering rule configuration file defines several filtering rules, which can be used to filter streaming data, reducing the amount of streaming data processing. The filtering rule configuration file can be defined according to actual needs. If no filtering is required, the filtering rule configuration file can be empty.

[0051] The first field configuration file can be used to process the filtered stream data in the first field, obtaining the first processed data. The first field configuration file defines several field processing rules, which can be used to process the filtered stream data in the first field and generate the first processed data. The first field configuration file can be defined according to actual needs.

[0052] Optionally, the first field configuration file includes a special character configuration file, an enumeration value conversion mapping configuration file, and a date normalization rule configuration file.

[0053] Understandably, the first field configuration file includes, but is not limited to, special character configuration files, enumeration value conversion mapping configuration files, and date normalization rule configuration files.

[0054] The special character configuration file is used to define which fields of the filtered stream data require special processing. This involves developing simple field processing rules; simply implement the standard interface to implement the specific field-level processing rules. Leaving it unconfigured indicates no special processing.

[0055] In one example, the filtered stream data is the source table Employee information table, where the ID card number (idCard) is converted from 15 digits to 18 digits, and the name (name) is processed to remove spaces.

[0056] The enumeration value transformation mapping configuration file is used to define the mapping relationship between enumeration values ​​and standard enumeration values ​​in the filtered stream data. It can automatically standardize enumeration values ​​based on the configuration file; leaving it unconfigured means no transformation is needed.

[0057] For example, ID type (idType) and gender can be normalized using enumeration values. For instance, the enumeration value for "male" is 1, and the enumeration value for "female" is 2.

[0058] Date standardization rules configuration files are used to standardize date formats. For example, date formats can be converted to the standard YYYY-MM-DD HH:MI:SS.MI format.

[0059] Optionally, step S30, namely obtaining the associated data related to the first processed data and performing second filtering and second field processing on the associated data to generate second processed data, includes:

[0060] S301. Obtain the association relationship configuration file, association data filtering rule configuration file, and association data field configuration file corresponding to the first processed data;

[0061] S302. Obtain the associated data related to the first processed data according to the association configuration file;

[0062] S303. Filter the associated data according to the associated data filtering rule configuration file to generate associated filtered data;

[0063] S304. Perform second field processing on the associated filter data according to the associated data field configuration file to generate the second processed data.

[0064] Understandably, this embodiment typically requires joining, filtering, and processing fields across multiple tables. The relationship configuration file defines several relationship rules. These rules configure the relationships between streaming data (i.e., the first processed data) (the relationships can be between streaming data points or between streaming data and business system data; business system data can refer to historically stored streaming data or other pre-stored data). Related data associated with the first processed data can be found based on these relationship rules.

[0065] In the relationship configuration file, you can set the table name definition of the target table, such as what specific target table the associated data is written to; you can also set the relationship between one table and another table, such as the relationship between tables, i.e., left or inner; or the relationship conditions between tables, i.e., how to achieve the relationship between the two tables through fields.

[0066] If related data is found, the related data filtering rule configuration file is read, and the filtering conditions in the configuration file are used to filter the related data to obtain related filtered data (filtered streaming data). The related data filtering rule configuration file defines the filtering conditions for related data. In the related data filtering rule configuration file, you can set the selection conditions for field associations, such as using logical comparison words like AND or OR; you can also set the filtering conditions (filter) for the data table. Here, the filtering conditions can be customized. If the filter is executed first when requesting data from the table, and the filtering conditions are not met, related filtered data will not be generated, which can improve the speed and efficiency of real-time processing of streaming data.

[0067] In some examples, the configuration file for related data filtering rules includes a WHERE condition. The WHERE condition defines which data, after being related, meets the condition and can be included in the final target table. Complex combination configurations of related data can be implemented at this stage. Logically, this step involves performing final filtering after all data has been related.

[0068] The second field of the linked data filtering can be processed through the linked data field configuration file. This second field processing includes, but is not limited to, case conversion, string concatenation, whitespace removal, decode functions, and even some special processing related to specific business logic. The second field processing can be implemented using the linked data field configuration file (based on JSON configuration), thus offering high flexibility. In some scenarios, implementing this via SQL might require writing complex SQL logic. Here, a single function can achieve the complex SQL functionality, simplifying the complexity of JSON configuration.

[0069] Some examples also involve field mapping. This section primarily maps fields from the primary data table to the target table, and can be configured using appropriate field mapping configuration files. Alternatively, it can directly utilize the associated data field configuration file to implement special field handling and simplify JSON configuration.

[0070] The first step typically involves processing multiple tables. Iterate through all tables, repeatedly executing the steps described above, until all data has been processed.

[0071] In this embodiment, the second processing data can be generated by modeling. That is, a model is built according to the processing logic of the streaming data, and multiple relationship configuration files, related data filtering rule configuration files, and related data field configuration files in the model are configured. Then, these configuration files are modified according to actual needs, and the streaming data is processed using these configuration files to generate the second processing data.

[0072] Optionally, step S40, which involves performing logical processing operations on the second processed data to update the streaming data, includes:

[0073] S401, Perform the second data addition operation;

[0074] S402. If the result of the new operation is successful, then obtain the accumulation flag of the second processed data;

[0075] S403. If the accumulated flag is 1, then the stream data update is determined to be successful.

[0076] Understandably, a second data addition operation can be performed. If the data does not exist in the database, the addition operation will succeed; if the data exists in the database, the addition operation will fail and trigger an error message.

[0077] If the add operation fails, it checks if the version of the current second-processed data is greater than the version of the data in the database. If the version of the current second-processed data is greater than the version of the data in the database, the accumulation flag of the database data is set to 0 (indicating that it has not been accumulated yet), and the add operation is performed again. If the add operation succeeds, the accumulation flag is set to 1. At this time, it checks if the accumulation flag of the newly added second-processed data is 1. If it is 1, it indicates that the add operation was successful, that is, the streaming data was updated successfully; if it is not 1, the add operation failed, and an error message is triggered.

[0078] If the new operation is executed successfully, the accumulated flag of the newly added second processed data is obtained. If the accumulated flag is 1, the stream data update is considered successful. If the accumulated flag is 0, the new operation is executed unsuccessfully, and an error message is triggered.

[0079] Here, you can set up a logic processing configuration file to implement logical processing operations on the second set of data. The logic processing configuration file can be configured via JSON. It defines the conditions under which a specific accumulation operation will be performed, i.e., implementing the `count` and `sum` operations on a field using `groupby` based on a certain condition in SQL. The specific JSON format is consistent with other configuration files.

[0080] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0081] In one embodiment, a streaming data processing apparatus is provided, which corresponds one-to-one with the streaming data processing methods described in the above embodiments. For example... Figure 3 As shown, the streaming data processing device includes a streaming data acquisition module 10, a first processing module 20, a second processing module 30, and a logic processing module 40. Detailed descriptions of each functional module are as follows:

[0082] The streaming data acquisition module 10 is used to acquire streaming data that conforms to preset processing standards.

[0083] The first processing module 20 is used to index, filter, and process the first field of the streaming data to generate first processed data.

[0084] The second processing module 30 is used to obtain associated data related to the first processed data, and to perform second filtering and second field processing on the associated data to generate second processed data.

[0085] The logic processing module 40 is used to perform logic processing operations on the second processed data to complete the update of the stream data.

[0086] Optionally, the streaming data processing device also includes:

[0087] The raw data acquisition module is used to acquire raw stream data through a preset method.

[0088] The conversion module is used to convert the original stream data into the stream data according to a preset conversion program.

[0089] Optionally, the first processed data includes the primary key and index data of the streaming data;

[0090] The first processing module 20 includes:

[0091] The index configuration file acquisition unit is used to acquire the index configuration file and primary key configuration file corresponding to the stream data;

[0092] A primary key generation unit is used to configure the primary key of the stream data according to the primary key configuration file and generate the primary key of the stream data;

[0093] An index data generation unit is used to configure the index of the stream data according to the index configuration file and generate index data for the stream data.

[0094] Optionally, the first processing module 20 includes:

[0095] The first configuration file unit is used to obtain the filter rule configuration file and the first field configuration file corresponding to the stream data;

[0096] A filtered stream data unit is used to filter the indexed stream data according to the filtering rule configuration file to generate filtered stream data.

[0097] The first processing unit is used to process the filtered stream data according to the first field configuration file to generate the first processed data.

[0098] Optionally, the first field configuration file includes a special character configuration file, an enumeration value conversion mapping configuration file, and a date normalization rule configuration file.

[0099] Optionally, the second processing module 30 includes:

[0100] The second configuration file unit is used to obtain the association relationship configuration file, the association data filtering rule configuration file, and the association data field configuration file corresponding to the first processed data;

[0101] The associated data acquisition unit is used to acquire associated data related to the first processed data according to the associated relationship configuration file;

[0102] The association filtering unit is used to filter the association data according to the association data filtering rule configuration file and generate association filtered data;

[0103] The second processing unit is used to perform second field processing on the associated filtered data according to the associated data field configuration file to generate the second processed data.

[0104] Optionally, the logic processing module 40 includes:

[0105] A new unit is added to perform the operation of adding the second processed data.

[0106] An accumulation marker acquisition unit is used to acquire the accumulation marker of the second processed data if the result of the addition operation is successful.

[0107] The update success determination unit is used to determine that the stream data update is successful if the accumulated flag is 1.

[0108] For specific limitations regarding the streaming data processing device, please refer to the limitations on the streaming data processing method above, which will not be repeated here. Each module in the aforementioned streaming data processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in hardware or independently of the processor in the computer device, or stored in software in the memory of the computer device, so that the processor can call and execute the operations corresponding to each module.

[0109] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 4 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes a readable storage medium and internal memory. The readable storage medium stores an operating system, computer-readable instructions, and a database. The internal memory provides an environment for the operation of the operating system and computer-readable instructions in the readable storage medium. The database stores data involved in the streaming data processing method. The network interface communicates with external terminals via a network connection. When the computer-readable instructions are executed by the processor, a streaming data processing method is implemented. The readable storage medium provided in this embodiment includes both non-volatile and volatile readable storage media.

[0110] In one embodiment, a computer device is provided, including a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor, wherein the processor performs the following steps when executing the computer-readable instructions:

[0111] Acquire stream data that conforms to preset processing standards;

[0112] The streaming data is indexed, filtered in the first way, and processed in the first field to generate the first processed data;

[0113] Obtain associated data related to the first processed data, and perform second filtering and second field processing on the associated data to generate second processed data;

[0114] Perform logical processing operations on the second processed data to complete the update of the streaming data.

[0115] In one embodiment, one or more computer-readable storage media storing computer-readable instructions are provided. The readable storage media provided in this embodiment include non-volatile readable storage media and volatile readable storage media. The readable storage media stores computer-readable instructions, which, when executed by one or more processors, perform the following steps:

[0116] Acquire stream data that conforms to preset processing standards;

[0117] The streaming data is indexed, filtered in the first way, and processed in the first field to generate first processed data; associated data related to the first processed data is obtained, and the associated data is filtered in the second way and processed in the second field to generate second processed data.

[0118] Perform logical processing operations on the second processed data to complete the update of the streaming data.

[0119] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing related hardware with computer-readable instructions. These computer-readable instructions can be stored in a non-volatile readable storage medium or a volatile readable storage medium. When executed, these computer-readable instructions can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0120] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0121] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A streaming data processing method, wherein the streaming data processing method is used for streaming computing of big data, characterized in that, include: Acquire stream data that conforms to preset processing standards; The streaming data refers to the data used for streaming computation. The streaming data is indexed, filtered in the first way, and processed in the first field to generate the first processed data; The first processed data is the result of normalizing the streaming data; Obtain associated data related to the first processed data, and perform second filtering and second field processing on the associated data to generate second processed data; The second processed data is used to resolve the historical data rollback problem caused by the associated data; Perform logical processing operations on the second processed data to complete the update of the streaming data.

2. The streaming data processing method as described in claim 1, characterized in that, Before acquiring the stream data that conforms to the preset processing standard, the process also includes: Raw stream data is obtained through a preset method; The original stream data is converted into the stream data according to a preset conversion procedure.

3. The streaming data processing method as described in claim 1, characterized in that, The first processed data includes the primary key and index data of the streaming data; The step of indexing, filtering, and processing the first field of the stream data to generate first processed data includes: Obtain the index configuration file and primary key configuration file corresponding to the stream data; Configure the primary key of the stream data according to the primary key configuration file, and generate the primary key of the stream data; The streaming data is indexed according to the index configuration file to generate index data for the streaming data.

4. The streaming data processing method as described in claim 1, characterized in that, The step of indexing, filtering, and processing the first field of the stream data to generate first processed data includes: Obtain the filter rule configuration file and the first field configuration file corresponding to the stream data; The indexed stream data is filtered according to the filtering rule configuration file to generate filtered stream data; The filtered stream data is processed according to the first field configuration file to generate the first processed data.

5. The streaming data processing method as described in claim 4, characterized in that, The first field configuration file includes a special character configuration file, an enumeration value conversion mapping configuration file, and a date standardization rule configuration file.

6. The streaming data processing method as described in claim 1, characterized in that, The step of obtaining associated data related to the first processed data, and performing second filtering and second field processing on the associated data to generate second processed data includes: Obtain the relationship configuration file, the related data filtering rule configuration file, and the related data field configuration file corresponding to the first processed data; Obtain the associated data related to the first processed data according to the association configuration file; The associated data is filtered according to the associated data filtering rule configuration file to generate associated filtered data; The associated filtered data is processed in the second field according to the associated data field configuration file to generate the second processed data.

7. The streaming data processing method as described in claim 1, characterized in that, The step of performing logical processing operations on the second processed data to complete the update of the streaming data includes: Perform the data addition operation in the second processing step; If the result of the new operation is successful, then the accumulated flag of the second processed data is obtained; If the accumulated flag is 1, then the stream data update is considered successful.

8. A streaming data processing apparatus, the streaming data processing apparatus being used for streaming computation of big data, characterized in that, include: The stream data acquisition module is used to acquire stream data that conforms to preset processing standards; The streaming data refers to the data used for streaming computation. The first processing module is used to index, filter, and process the first field of the streaming data to generate first processed data. The first processed data is the result of normalizing the streaming data; The second processing module is used to obtain associated data related to the first processed data, and to perform second filtering and second field processing on the associated data to generate second processed data. The second processed data is used to resolve the historical data rollback problem caused by the associated data; The logic processing module is used to perform logic processing operations on the second processed data to complete the update of the stream data.

9. A computer device comprising a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor, characterized in that, When the processor executes the computer-readable instructions, it implements the streaming data processing method as described in any one of claims 1 to 7.

10. One or more readable storage media storing computer-readable instructions, which, when executed by one or more processors, cause the one or more processors to perform the streaming data processing method as described in any one of claims 1 to 7.

Images