Universal stream batch integrated processing method and system based on different storage media
By constructing data connection channels and generating stream-batch semantic read and write strategies, the high complexity of existing stream-batch integrated processing methods is solved, achieving efficient collaboration between different storage media and meeting the timeliness and throughput requirements of business needs.
Patent Information
- Application Number
- CN202410577313.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-05-10
- Publication Date
- 2024-11-08
AI Technical Summary
The current integrated processing method of stream processing and batch processing requires the reliance on a corresponding computing engine, which leads to high technical complexity, low R&D efficiency, single storage method, poor processing efficiency, and failure to fully utilize the advantages of different storage media in their specific scenarios.
By constructing data connection channels corresponding to storage media, generating stream and batch semantic reading strategies, determining data conversion strategies using the attribute parameters of storage media, and generating semantic writing strategies based on writing parameters, the integrated stream and batch processing of data in different storage media can be achieved.
It enables integrated streaming and batch processing of data across different storage media, meeting the technical requirements of business operations regarding latency, throughput, and performance for data storage, computation, retrieval, and utilization, simplifying the data processing architecture, and improving processing efficiency.
Smart Images

Figure CN118916408A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of data processing, and in particular to a general stream-batch integrated processing method and system based on different storage media. Background Art
[0002] With the development of information technology, various information and digital systems are rapidly popularizing, and the types of data generated and processed are becoming more and more abundant, which has led to the rapid development of data storage technology and the emergence of various storage media. How to achieve the self-adaptation and efficient collaboration of different storage media in stream processing and batch processing to meet the timeliness and throughput requirements in various business scenarios has become the main constraint factor for different storage media in the data processing process.
[0003] In the existing technology, the integrated processing method of stream processing and batch processing needs to rely on the corresponding computing engine. In actual scenarios, users need to connect and adapt one by one. The technology is highly complex and the R&D efficiency is low. In addition, there are problems such as single storage method and poor processing efficiency in the stream and batch processing process. Summary of the invention
[0004] In view of this, the purpose of the present invention is to provide a general stream-batch integrated processing method and system based on different storage media, which can fully utilize the advantages of different storage media in their unique scenarios, realize the stream-batch integrated processing of data in different storage media, and meet the technical requirements of the business for data storage, calculation, retrieval, and use, such as latency, throughput, and performance. At the same time, under the premise of simplifying the data processing related architecture, the data processing efficiency is improved, thereby solving the above-mentioned problems existing in the prior art.
[0005] In a first aspect, an embodiment of the present invention provides a general stream-batch integrated processing method based on different storage media, the method is applied to the stream processing and batch processing of data stored in different types of storage media, the method comprising:
[0006] Constructing a data connection channel corresponding to the storage medium according to the type parameters of the storage medium;
[0007] Generate a stream-batch semantic reading strategy corresponding to the storage data based on the reading parameters of the storage medium, and read the corresponding storage data in the data connection channel according to the stream-batch semantic reading strategy;
[0008] Determine a data conversion strategy for stored data using attribute parameters of the storage medium, and perform format conversion on the read stored data based on the data conversion strategy;
[0009] Generate a semantic write strategy corresponding to the storage data based on the write parameters of the storage medium, and determine a storage database corresponding to the storage medium according to the semantic write strategy;
[0010] The storage data that has completed format conversion is saved in the storage database according to the semantic writing strategy.
[0011] In one embodiment, the step of constructing a data connection channel corresponding to the storage medium according to the type parameter of the storage medium includes:
[0012] After receiving the business demand instructions corresponding to the stream processing and batch processing of the storage data, obtaining the read request instructions, the calculation request instructions and the write request instructions of the storage medium based on the business demand instructions;
[0013] Construct a computing task corresponding to the business demand instruction according to the read request instruction, the compute request instruction, and the write request instruction;
[0014] The type parameters corresponding to the storage medium are determined based on the computing task, and the data connection channel corresponding to the storage medium is constructed according to the type parameters.
[0015] In one embodiment, the step of determining a type parameter corresponding to a storage medium based on a computing task and constructing a data connection channel corresponding to the storage medium according to the type parameter includes:
[0016] Obtain the transaction storage medium, analysis storage medium, object storage medium, message storage medium, lake warehouse table storage medium, and time series storage medium corresponding to the computing task;
[0017] Determine the type parameters corresponding to the storage medium according to the transaction storage medium, analysis storage medium, object storage medium, message storage medium, lake warehouse table storage medium and time series storage medium;
[0018] According to the type parameters, data connection channels corresponding to transaction storage media, analysis storage media, object storage media, message storage media, lake warehouse table storage media and time series storage media are constructed respectively.
[0019] In one embodiment, the steps of generating a stream-batch semantic read strategy corresponding to the storage data based on the read parameters of the storage medium, and reading the corresponding storage data in the data connection channel according to the stream-batch semantic read strategy include:
[0020] Determine basic parameters corresponding to the storage medium using relationship parameters, timing parameters, transaction parameters and object parameters corresponding to the storage medium;
[0021] Determine the reading parameters of the storage medium according to the full parameters, incremental parameters and streaming parameters corresponding to the storage medium;
[0022] Determine the stream-batch semantic reading strategy corresponding to the stored data based on the basic parameters, reading parameters, and the stream processing parameters and batch processing parameters corresponding to the storage medium;
[0023] The storage medium corresponding to the stream-batch semantic reading strategy is obtained, and the storage data in the data connection channel corresponding to the storage medium is read based on the stream-batch semantic reading strategy.
[0024] In one embodiment, the step of determining a data conversion strategy for storage data using attribute parameters of a storage medium, and converting the format of the read storage data based on the data conversion strategy includes:
[0025] Determine the attribute parameters corresponding to the storage medium according to the numerical data, string data, date data, and Boolean data corresponding to the data stored in the storage medium;
[0026] Obtaining the policy setting parameters and calculation mode parameters corresponding to the stream processing and batch processing of the stored data, and determining the data conversion strategy corresponding to the attribute parameters using the policy setting parameters and the calculation mode parameters;
[0027] The data conversion strategy is used to format and convert the stored data.
[0028] In one embodiment, the steps of generating a semantic write strategy corresponding to the storage data based on the write parameters of the storage medium, and determining a storage database corresponding to the storage medium according to the semantic write strategy include:
[0029] Determine the writing parameters of the storage medium according to the batch data writing parameters and the line-by-line data writing parameters corresponding to the storage medium during the batch processing process;
[0030] Determine a semantic write strategy corresponding to the stored data using the write parameters;
[0031] The storage database corresponding to the storage medium is determined according to the semantic writing strategy.
[0032] In one embodiment, the step of saving the format-converted storage data to a storage database according to the semantic writing strategy includes:
[0033] Determine the update parameters of the storage medium according to the replacement parameters, append parameters, and insert update parameters corresponding to the storage medium during the stream processing;
[0034] Determine the equivalent transformation data corresponding to the storage data under the semantic write strategy according to the update parameters;
[0035] The equivalent transformation data is saved in a storage database corresponding to the storage medium.
[0036] In one embodiment, the step of determining equivalent transformation data corresponding to the stored data under the semantic write strategy according to the update parameters includes:
[0037] According to the update parameters, determine one or more transformation strategies of data format transformation, data append transformation, merge update transformation and deletion transformation corresponding to the stored data under the semantic write strategy;
[0038] Obtain equivalent transformed data corresponding to the stored data based on the transformation strategy.
[0039] In a second aspect, an embodiment of the present invention provides a general stream-batch integrated processing system based on different storage media, the system is applied to the stream processing and batch processing of data stored in different types of storage media, and the system includes:
[0040] A data connection channel generation module, used to construct a data connection channel corresponding to the storage medium according to the type parameters of the storage medium;
[0041] A stream-batch semantic reading strategy execution module is used to generate a stream-batch semantic reading strategy corresponding to the storage data based on the reading parameters of the storage medium, and read the corresponding storage data in the data connection channel according to the stream-batch semantic reading strategy;
[0042] A data conversion strategy execution module, used to determine a data conversion strategy for storage data using attribute parameters of the storage medium, and perform format conversion on the read storage data based on the data conversion strategy;
[0043] A semantic write policy execution module, used to generate a semantic write policy corresponding to the storage data based on the write parameters of the storage medium, and determine a storage database corresponding to the storage medium according to the semantic write policy;
[0044] The data stream batch execution module is used to save the storage data that has completed format conversion into the storage database according to the semantic writing strategy.
[0045] In a third aspect, an embodiment of the present invention further provides an electronic device, comprising a processor and a memory, wherein the memory stores computer-executable instructions that can be executed by the processor, and the processor executes the computer-executable instructions to implement the steps of the general stream-batch integrated processing method based on different storage media provided in the first aspect.
[0046] In a fourth aspect, an embodiment of the present invention further provides a storage medium storing computer executable instructions. When the computer executable instructions are called and executed by a processor, the computer executable instructions prompt the processor to implement the steps of the general stream-batch integrated processing method based on different storage media provided in the first aspect.
[0047] The embodiment of the present invention provides a method and system for integrated stream and batch processing based on different storage media. The method is applied to the stream processing and batch processing of data stored in different types of storage media. In the process of integrated stream and batch processing of storage data in different types of storage media, firstly, the data connection channel corresponding to the storage medium is constructed according to the type parameters of the storage medium; then, the stream and batch semantic reading strategy corresponding to the storage data is generated based on the read parameters of the storage medium, and the corresponding storage data in the data connection channel is read according to the stream and batch semantic reading strategy; then, the data conversion strategy of the storage data is determined by using the attribute parameters of the storage medium, and the format of the read storage data is converted based on the data conversion strategy; then, the semantic writing strategy corresponding to the storage data is generated based on the write parameters of the storage medium, and the storage database corresponding to the storage medium is determined according to the semantic writing strategy; finally, the storage data that has completed the format conversion is saved in the storage database according to the semantic writing strategy. The method converts the storage medium (matching the business scenario) according to the stream and batch computing mode to meet the characteristics of the stream and batch computing mode, realizes the self-adaptation of computing and storage, and ensures the stability and performance of the stream and batch computing under different resource specifications and the efficient access of different storage media. This method can make full use of the advantages of different storage media in their unique scenarios, realize the integrated processing of streams and batches of data in different storage media, and meet the technical requirements of the business for data storage, calculation, retrieval, and use, such as latency, throughput, and performance. At the same time, it improves data processing efficiency while simplifying the data processing related architecture, thus solving the above-mentioned problems existing in the existing technology.
[0048] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention are realized and obtained by the structures particularly pointed out in the description, claims and drawings.
[0049] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0050] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0051] Figure 1 A flowchart of a general stream-batch integrated processing method based on different storage media provided by an embodiment of the present invention;
[0052] Figure 2 A flowchart of step S101 in a general stream-batch integrated processing method based on different storage media provided in an embodiment of the present invention;
[0053] Figure 3 A flowchart of step S203 in a general stream-batch integrated processing method based on different storage media provided in an embodiment of the present invention;
[0054] Figure 4 A flowchart of step S102 in a general stream-batch integrated processing method based on different storage media provided in an embodiment of the present invention;
[0055] Figure 5 A flowchart of step S103 in a general stream-batch integrated processing method based on different storage media provided in an embodiment of the present invention;
[0056] Figure 6 A flowchart of step S104 in a general stream-batch integrated processing method based on different storage media provided in an embodiment of the present invention;
[0057] Figure 7 A flowchart of step S105 in a general stream-batch integrated processing method based on different storage media provided in an embodiment of the present invention;
[0058] Figure 8 A flowchart of step S702 in a general stream-batch integrated processing method based on different storage media provided in an embodiment of the present invention;
[0059] Fig. 9 A schematic diagram of a flow chart of a general stream-batch integrated processing method based on different storage media provided in an embodiment of the present invention;
[0060] Fig.10 A schematic diagram of the structure of a general stream-batch integrated processing system based on different storage media provided by an embodiment of the present invention;
[0061] Fig.11 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention.
[0062] icon:
[0063] 1010-data connection channel generation module; 1020-semantic reading strategy execution module; 1030-data conversion strategy execution module; 1040-semantic writing strategy execution module; 1050-data stream batch processing execution module;
[0064] 101 - processor; 102 - memory; 103 - bus; 104 - communication interface. DETAILED DESCRIPTION
[0065] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution of the present invention will be clearly and completely described in combination with the embodiments below. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0066] With the development of industrialization and informatization, relevant information storage and digital systems are penetrating faster, and the types of data generated and processed are becoming more abundant, which has promoted the rapid iteration and evolution of data storage software and hardware technologies, and storage media with their own characteristics have emerged. Taking the data storage system as an example, the current numerous relational transactions, semi-structured documents, analytical, MPP, graph, and time series database systems and middleware support the corresponding online business, retrieval business, analytical business, graph analysis, time series analysis, etc.; object storage and lake warehouse tables are the basis of the current lake warehouse integrated storage. In short, due to many factors such as cost and performance, different business scenarios and data with different characteristics must correspond to different storage media.
[0067] In addition to differences in storage methods, data formats, compression methods, indexing mechanisms, etc., the differences in the reading and writing mechanisms of different storage media are also particularly significant. For example, transactional storage is suitable for single-point random writing and random single-point reading; analytical storage is suitable for batch writing and sequential reading; time series storage is suitable for sequential batch writing and sequential batch reading; object storage is suitable for sequential reading and writing of data blocks, etc. In the current stream-batch integrated processing framework, stream processing focuses on streaming reading, processing, and writing of data one by one; batch processing focuses on sequential reading, processing, and writing of batch data. Therefore, how to achieve adaptability and efficient collaboration between different storage media and the stream-batch integrated computing framework to meet the timeliness and throughput requirements in various business scenarios has become an important issue.
[0068] The current streaming and batch integration processing methods for different storage media are different, mainly in the following categories:
[0069] (1) Relying on the plug-in mechanism of the integrated stream and batch computing engine, by extending the connectors of different storage media, the stream and batch reading and writing of data are realized for stream processing and batch processing respectively. This method is direct and has good timeliness. However, considering the characteristics of different storage media, the support levels for stream writing, stream reading, batch writing, and batch reading are different, and the maturity of support for insertion, merge update, and deletion semantics varies greatly. Users need to adapt one by one, which has high technical complexity and low R&D efficiency. At present, stream processing mostly corresponds to storage media with streaming characteristics, and batch processing corresponds to storage media with batch characteristics.
[0070] (2) Introducing message queue middleware, the data capture service reads from different storage media in a streaming or batch manner, writes to the message queue, and the streaming and batch computing engine subscribes to the message queue and performs streaming and batch processing on the data. After writing back to the message queue, the writing service writes the data to different storage media. This method is simple, but the entire link becomes longer and the latency increases. In essence, the processing of different storage media is a streaming processing method.
[0071] (3) The write end uses stream storage, combined with a stream-batch integrated computing engine, to achieve the unification of streams and batches in the computing and storage layers. However, in essence, it eliminates different storages and adopts a single storage. This method cannot fully utilize the advantages of different storage media in their specific scenarios.
[0072] In summary, the integrated processing mode of stream processing and batch processing in the prior art needs to rely on the corresponding computing engine, and users are required to dock and adapt one by one in actual scenarios. The technical complexity is high and the R&D efficiency is low. In addition, there are problems of single storage mode and poor processing efficiency in the process of stream and batch processing. Based on this, the present invention implements a general stream-batch integrated processing method and system based on different storage media. The method can make full use of the advantages of different storage media in their unique scenarios, realize the integrated stream-batch processing of data in different storage media, and meet the technical requirements of the business for data storage, calculation, retrieval, and use, such as latency, throughput, and performance. At the same time, on the premise of simplifying the data processing related architecture, the data processing efficiency is improved, thereby solving the above-mentioned problems existing in the prior art.
[0073] To facilitate understanding of this embodiment, firstly, a general stream-batch integrated processing method based on different storage media disclosed in an embodiment of the present invention is introduced in detail. The method is applied to the stream processing and batch processing of data stored in different types of storage media, such as Figure 1 As shown, including:
[0074] Step S101, constructing a data connection channel corresponding to the storage medium according to the type parameter of the storage medium.
[0075] Step S102, generating a stream-batch semantic reading strategy corresponding to the storage data based on the reading parameters of the storage medium, and reading the corresponding storage data in the data connection channel according to the stream-batch semantic reading strategy.
[0076] Step S103, determining a data conversion strategy for the stored data using the attribute parameters of the storage medium, and performing format conversion on the read stored data based on the data conversion strategy.
[0077] Step S104, generating a semantic write strategy corresponding to the storage data based on the write parameters of the storage medium, and determining a storage database corresponding to the storage medium according to the semantic write strategy.
[0078] Step S105: save the storage data that has completed the format conversion into a storage database according to the semantic writing strategy.
[0079] Specifically, the digital transformation and digital intelligent upgrading of industrial manufacturing are the general trend, and the importance of data consistency processing is prominent. Among them, the relational data supporting enterprise business management and operation is stored in the relational transaction storage system, the process data in the manufacturing process is stored in the analytical storage system, and the time series data generated by sensors and other equipment will flow to the time series storage system through the message storage system. For the above-mentioned types of storage media, it is necessary to rely on the stream-batch integrated computing engine to effectively connect, calculate, and store them in the storage system matching the business scenario. Therefore, the method is mainly divided into five steps. First, the corresponding data connection channel is established for different media; then, the stream-batch semantic reading strategy corresponding to the stream batch processing of different storage media is loaded according to the reading characteristics of different storage media; then, the data conversion strategy of the storage data is determined by the attribute parameters of the storage medium, and the format of the storage data is standardized by the strategy to adapt to the subsequent steps; then, the corresponding semantic write strategy is generated based on the write parameters of the storage medium, and the storage database corresponding to the storage medium is determined according to the write strategy, and finally, the storage data is stream-batch written according to the stream-batch write rule semantics corresponding to the semantic write strategy, and finally it is saved in the storage database.
[0080] In the actual relevant system scenario, the method may involve four layers of hardware systems according to the above steps, namely: the stream-batch integrated framework storage medium reading semantic engine, the stream-batch integrated framework data computing engine, the stream-batch integrated framework storage writing semantic engine, and the underlying storage medium. On this basis, in the process of executing the general stream-batch integrated processing method based on different storage media, firstly, a reading channel with transactional, analytical and other database systems is established, and the stream-batch reading semantic engines of different storage media implement the corresponding stream-batch reading semantics, including: data format, full volume, incremental, streaming, batch, etc. Then, according to the actual stream-batch computing mode, the reading engine applies the corresponding semantics to realize data reading and loading of different types of databases. The stream-batch integrated computing engine receives data for stream-batch computing, and the writing semantic engine performs stream-batch writing semantic equivalent transformation according to different storage databases, including data format, data append, merge update, deletion, etc., and writes the data to the underlying corresponding storage database.
[0081] It can be seen that this method can make full use of the advantages of different storage media in their unique scenarios, realize the integrated processing of stream and batch of data in different storage media, and meet the technical requirements of business for data storage, calculation, retrieval, and use in terms of latency, throughput, performance, etc. At the same time, it improves delivery efficiency under the premise of simplifying the architecture.
[0082] In one implementation, step S101 of constructing a data connection channel corresponding to the storage medium according to the type parameter of the storage medium is as follows: Figure 2 As shown, including:
[0083] Step S201, after receiving the business demand instructions corresponding to the stream processing and batch processing of the storage data, obtaining the read request instructions, the calculation request instructions and the write request instructions of the storage medium based on the business demand instructions;
[0084] Step S202, constructing a computing task corresponding to the business demand instruction according to the read request instruction, the compute request instruction and the write request instruction;
[0085] Step S203, determining a type parameter corresponding to the storage medium based on the computing task, and constructing a data connection channel corresponding to the storage medium according to the type parameter.
[0086] Specifically, when there is a business need to read, calculate, or write to a specified storage medium, the stream-batch integration framework provides an SDK to build a computing task, including registering the storage medium reading, storage medium writing, and computing logic to the stream-batch integration. The stream-batch integration framework will establish a channel with the storage medium to prepare for data reading. The read request instructions, computing request instructions, and write request instructions corresponding to the storage medium are obtained based on the business demand instructions, and then the computing tasks corresponding to the business demand instructions are built according to these instructions. Then, the type parameters corresponding to the storage medium are determined based on the computing tasks, and the corresponding data connection channels are built according to the type parameters.
[0087] In one implementation, the step S203 of determining the type parameter corresponding to the storage medium based on the computing task and constructing the data connection channel corresponding to the storage medium according to the type parameter is as follows: Figure 3 As shown, including:
[0088] Step S301, obtaining the transaction storage medium, analysis storage medium, object storage medium, message storage medium, lake warehouse table storage medium and time series storage medium corresponding to the computing task;
[0089] Step S302, determining the type parameter corresponding to the storage medium according to the transaction storage medium, the analysis storage medium, the object storage medium, the message storage medium, the lake warehouse table storage medium and the time series storage medium;
[0090] Step S303, construct data connection channels corresponding to the transaction storage medium, analysis storage medium, object storage medium, message storage medium, lake warehouse table storage medium and time series storage medium respectively according to the type parameters.
[0091] In the process of building a data connection channel, the storage objects involved, such as transaction storage, analysis storage, object storage, message storage, lake storage table storage and time series storage, are constructed by constructing corresponding data connection channels according to the type parameters corresponding to the transaction storage medium, analysis storage medium, object storage medium, message storage medium, lake storage table storage medium and time series storage medium. In one embodiment, a stream-batch semantic reading strategy corresponding to the storage data is generated based on the reading parameters of the storage medium, and the corresponding storage data in the data connection channel is read according to the stream-batch semantic reading strategy. Step S102, as shown in Figure 4 As shown, including:
[0092] Step S401, determining basic parameters corresponding to the storage medium using relationship parameters, timing parameters, transaction parameters and object parameters corresponding to the storage medium;
[0093] Step S402, determining the reading parameters of the storage medium according to the full parameters, incremental parameters and streaming parameters corresponding to the storage medium;
[0094] Step S403, determining a stream-batch semantic reading strategy corresponding to the stored data based on the basic parameters, the reading parameters, and the stream processing parameters and batch processing parameters corresponding to the storage medium;
[0095] Step S404, obtaining a storage medium corresponding to the stream-batch semantic reading strategy, and reading the storage data in the data connection channel corresponding to the storage medium based on the stream-batch semantic reading strategy.
[0096] The stream-batch semantic reading strategy mainly performs full, incremental, and streaming reading and loading processes for the relationships, delays, foods, and objects of different storage media. Specifically, it uses the basic parameters of the storage media determined by the relationship parameters, timing parameters, transaction parameters, and object parameters corresponding to the storage media, and uses the full parameters, incremental parameters, and streaming parameters to determine the storage media reading parameters. At the same time, after combining the stream processing parameters and batch processing parameters corresponding to the storage media, the corresponding stream-batch semantic reading strategy is determined, thereby reading the storage data in the data connection channel corresponding to the storage medium based on the stream-batch semantic reading strategy.
[0097] In one embodiment, a data conversion strategy for storage data is determined using attribute parameters of the storage medium, and a step S103 of format conversion is performed on the read storage data based on the data conversion strategy, such as Figure 5 As shown, including:
[0098] Step S501, determining attribute parameters corresponding to the storage medium according to numerical data, character string data, date data, and Boolean data corresponding to the data stored in the storage medium;
[0099] Step S502, obtaining the policy setting parameters and calculation mode parameters corresponding to the stream processing and batch processing of the stored data, and determining the data conversion policy corresponding to the attribute parameters by using the policy setting parameters and the calculation mode parameters;
[0100] Step S503: formatting and converting the stored data using a data conversion strategy.
[0101] For the stream-batch reading engine of different storage media, the data stream batch reading and conversion of the storage media are performed according to the policy setting parameters and calculation mode parameters corresponding to the stream-batch reading process, and the data format is standardized to adapt to the stream-batch integrated calculation engine. The data conversion strategy mainly converts the storage types such as numeric values, strings, dates, Booleans, etc. of different storage media into data types supported by the framework. Taking dates as an example, there are differences in storage formats and time zones of different storage media, which require unified conversion to form standard data.
[0102] In one embodiment, a semantic write strategy corresponding to the storage data is generated based on the write parameters of the storage medium, and a step S104 of determining a storage database corresponding to the storage medium according to the semantic write strategy is performed. Figure 6 As shown, including:
[0103] Step S601, determining the write parameters of the storage medium according to the batch data write parameters and the line-by-line data write parameters corresponding to the storage medium during the batch processing process;
[0104] Step S602, determining a semantic write strategy corresponding to the stored data using the write parameters;
[0105] Step S603: determining a storage database corresponding to the storage medium according to the semantic writing strategy.
[0106] The storage database as the target database is determined according to the semantic write strategy, which is determined according to the write parameters of the storage medium. The write parameters are mainly determined based on the batch data write parameters and the line-by-line data write parameters corresponding to the storage medium during the batch processing process. The specific processing logic can be defined according to the business scenario when the framework SDK builds the computing task, and the final streaming or batch computing can be performed in combination with the stream-batch computing scenario.
[0107] In one implementation, the step S105 of saving the format-converted storage data to the storage database according to the semantic writing strategy is as follows: Figure 7 As shown, including:
[0108] Step S701, determining update parameters of the storage medium according to the replacement parameters, additional parameters, and insertion update parameters corresponding to the storage medium during the stream processing process;
[0109] Step S702, determining equivalent transformation data corresponding to the stored data under the semantic write strategy according to the update parameters;
[0110] Step S703: Save the equivalent transformation data to a storage database corresponding to the storage medium.
[0111] During the data conversion process, different storage media stream batch writing engines perform stream batch writing on different storage media according to the semantics of stream batch writing rules. First, according to the replacement parameters, additional parameters, and insertion update parameters of the storage media during the stream processing process, the update parameters corresponding to the storage media are determined, and then the update parameters are used to perform stream batch writing semantic equivalent transformation to obtain the corresponding equivalent change data, and the equivalent transformation data is saved in the corresponding storage database to complete the entire data processing process.
[0112] In one implementation, the step S702 of determining the equivalent transformation data corresponding to the storage data under the semantic write strategy according to the update parameters is as follows: Figure 8 As shown, including:
[0113] Step S801, determining one or more transformation strategies of data format transformation, data append transformation, merge update transformation and deletion transformation corresponding to the stored data under the semantic write strategy according to the update parameters;
[0114] Step S802: acquiring equivalent transformation data corresponding to the stored data based on the transformation strategy.
[0115] Batch writing strategy refers to the writing and updating methods of the underlying storage media during batch computing. The writing methods include: batch data writing and data writing one by one, and the updating methods include: complete replacement, appending, and insert update. Finally, the stream batch writing semantic equivalent transformation is performed according to different storage databases, including data format, data appending, merge update, deletion, etc., and the data is written to the corresponding underlying storage database.
[0116] like Fig. 9 The figure shows a flow chart of a general stream-batch integrated processing method based on different storage media. This method makes full use of the current mainstream open source storage database system and the stream-batch integrated computing framework to build a stream-batch integrated big data storage and computing platform for a complete data ecosystem, covering transactional, analytical, full-text retrieval, graph analysis, time series analysis and other businesses, and providing corresponding storage and computing capabilities.
[0117] Specifically, a reading channel is established with transactional and analytical database systems. The stream-batch read semantics engines of different storage media implement the corresponding stream-batch read semantics, including: data format, full volume, incremental, streaming, batch, etc.; according to the actual stream-batch computing mode, the read engine applies the corresponding semantics to realize data reading and loading of different types of databases. The stream-batch integrated computing engine receives data for stream-batch computing, and the write semantics engine performs equivalent transformation of stream-batch write semantics according to different storage databases, including data format, data append, merge update, deletion, etc., and writes the data to the underlying corresponding storage database.
[0118] This method combines the read and write characteristics of different storage media and the characteristics of the batch-stream integrated computing mode. Through the pre-data reading (stream-batch reading semantic engine) and post-data writing conversion (stream-batch writing semantic engine) of the storage media, it completes the batch computing engine's batch reading and batch writing of streaming characteristic storage media data and the stream computing engine's streaming reading and streaming writing of batch characteristic storage media data, thereby achieving complete semantic support for different storage media in batch-stream integrated processing, breaking the restrictions of batch-stream integrated processing on storage media, and releasing the respective advantages of batch-stream integrated computing and different storage media.
[0119] It can be seen from the general stream-batch integrated processing method based on different storage media in the above embodiment that the method can make full use of the advantages of different storage media in their unique scenarios, realize the stream-batch integrated processing of data in different storage media, and meet the technical requirements of the business for data storage, calculation, retrieval, and use, such as latency, throughput, and performance. At the same time, under the premise of simplifying the data processing related architecture, the data processing efficiency is improved, thereby solving the above problems existing in the prior art.
[0120] For the general stream-batch integrated processing method based on different storage media provided in the above-mentioned embodiment, an embodiment of the present invention provides a general stream-batch integrated processing system based on different storage media, which is applied to the stream processing and batch processing of data stored in different types of storage media, such as Fig.10 As shown, the system includes:
[0121] A data connection channel generation module 1010 is used to construct a data connection channel corresponding to the storage medium according to the type parameter of the storage medium;
[0122] The semantic read strategy execution module 1020 is used to generate a stream-batch semantic read strategy corresponding to the storage data based on the read parameters of the storage medium, and read the corresponding storage data in the data connection channel according to the stream-batch semantic read strategy;
[0123] The data conversion strategy execution module 1030 is used to determine the data conversion strategy of the storage data by using the attribute parameters of the storage medium, and perform format conversion on the read storage data based on the data conversion strategy;
[0124] A semantic write policy execution module 1040, for generating a semantic write policy corresponding to the storage data based on the write parameters of the storage medium, and determining a storage database corresponding to the storage medium according to the semantic write policy;
[0125] The data stream batch processing execution module 1050 is used to save the storage data that has completed the format conversion into the storage database according to the semantic writing strategy.
[0126] From the general stream-batch integrated processing system based on different storage media mentioned in the above embodiment, it can be seen that the system can make full use of the advantages of different storage media in their unique scenarios to achieve stream-batch integrated processing of data in different storage media, and meet the technical requirements of the business for data storage, calculation, retrieval, and use, such as latency, throughput, and performance. At the same time, under the premise of simplifying the data processing related architecture, the data processing efficiency is improved, thereby solving the above-mentioned problems existing in the prior art.
[0127] The general stream-batch integrated processing system based on different storage media provided in the embodiment of the present invention has the same implementation principle and technical effects as the aforementioned general stream-batch integrated processing method embodiment based on different storage media. For the sake of brief description, for matters not mentioned in the device embodiment, reference may be made to the corresponding contents in the aforementioned general stream-batch integrated processing method embodiment based on different storage media.
[0128] This embodiment also provides an electronic device. The structural diagram of the electronic device is as follows: Fig.11 As shown, the device includes a processor 101 and a memory 102; wherein the memory 102 is used to store one or more computer instructions, and the one or more computer instructions are executed by the processor to implement the steps of the above-mentioned general stream-batch integrated processing method based on different storage media.
[0129] Fig.11 The electronic device shown further includes a bus 103 and a communication interface 104 , and the processor 101 , the communication interface 104 and the memory 102 are connected via the bus 103 .
[0130] The memory 102 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk storage. The bus 103 may be an ISA bus, a PCI bus, or an EISA bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.11 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.
[0131] The communication interface 104 is used to connect to at least one user terminal and other network units through a network interface, and send the encapsulated IPv4 message or IPv4 message to the user terminal through the network interface.
[0132] The processor 101 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 101. The above processor 101 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The disclosed methods, steps and logic block diagrams in the embodiments of the present disclosure can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in conjunction with the embodiments of the present disclosure can be directly embodied as a hardware decoding processor for execution, or a combination of hardware and software modules in the decoding processor for execution. The software module may be located in a storage medium mature in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 102, and the processor 101 reads the information in the memory 102 and completes the steps of the method of the above embodiment in combination with its hardware.
[0133] An embodiment of the present invention further provides a storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the general stream-batch integrated processing method based on different storage media in the aforementioned embodiment are executed.
[0134] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0135] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0136] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.
[0137] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present invention can essentially or in other words, the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.
[0138] Finally, it should be noted that the above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.
Claims
1. A general stream-batch integrated processing method based on different storage media, characterized in that: The method is applied to the stream processing and batch processing of data stored in different types of storage media, and the method comprises: Establishing a data connection channel corresponding to the storage medium according to the type parameter of the storage medium; Generate a stream-batch semantic reading strategy corresponding to the stored data based on the reading parameters of the storage medium, and read the corresponding stored data in the data connection channel according to the stream-batch semantic reading strategy; Determining a data conversion strategy for the stored data using the attribute parameters of the storage medium, and performing format conversion on the read stored data based on the data conversion strategy; Generating a semantic write strategy corresponding to the stored data based on the write parameters of the storage medium, and determining a storage database corresponding to the storage medium according to the semantic write strategy; The storage data that has completed the format conversion is saved in the storage database according to the semantic writing strategy.
2. The general stream-batch integrated processing method based on different storage media according to claim 1 is characterized in that: The step of constructing a data connection channel corresponding to the storage medium according to the type parameter of the storage medium includes: After receiving the business demand instructions corresponding to the stream processing and batch processing of the storage data, obtaining the read request instructions, the calculation request instructions and the write request instructions of the storage medium based on the business demand instructions; Constructing a computing task corresponding to the business demand instruction according to the read request instruction, the computing request instruction and the write request instruction; A type parameter corresponding to the storage medium is determined based on the computing task, and a data connection channel corresponding to the storage medium is constructed according to the type parameter.
3. The general stream-batch integrated processing method based on different storage media according to claim 2 is characterized in that: The step of determining the type parameter corresponding to the storage medium based on the computing task, and constructing the data connection channel corresponding to the storage medium according to the type parameter includes: Obtaining the transaction storage medium, analysis storage medium, object storage medium, message storage medium, lake warehouse table storage medium and time series storage medium corresponding to the computing task; Determine the type parameter corresponding to the storage medium according to the transaction storage medium, the analysis storage medium, the object storage medium, the message storage medium, the lake warehouse table storage medium and the time series storage medium; According to the type parameters, data connection channels corresponding to the transaction storage medium, the analysis storage medium, the object storage medium, the message storage medium, the lake warehouse table storage medium and the timing storage medium are respectively constructed.
4. The general stream-batch integrated processing method based on different storage media according to claim 1 is characterized in that: The step of generating a stream-batch semantic reading strategy corresponding to the stored data based on the reading parameters of the storage medium, and reading the corresponding stored data in the data connection channel according to the stream-batch semantic reading strategy includes: Determine basic parameters corresponding to the storage medium using relationship parameters, timing parameters, transaction parameters, and object parameters corresponding to the storage medium; Determining a reading parameter of the storage medium according to a full parameter, an incremental parameter, and a streaming parameter corresponding to the storage medium; Determine a stream-batch semantic reading strategy corresponding to the stored data based on the basic parameters, the reading parameters, and the stream processing parameters and batch processing parameters corresponding to the storage medium; The storage medium corresponding to the stream-batch semantic reading strategy is obtained, and storage data in the data connection channel corresponding to the storage medium is read based on the stream-batch semantic reading strategy.
5. The general stream-batch integrated processing method based on different storage media according to claim 1 is characterized in that: The step of determining the data conversion strategy of the stored data by using the attribute parameters of the storage medium, and converting the format of the read stored data based on the data conversion strategy includes: Determine the attribute parameter corresponding to the storage medium according to the numerical data, character string data, date data, and Boolean data corresponding to the stored data in the storage medium; Acquire the policy setting parameters and calculation mode parameters corresponding to the stream processing and batch processing of the stored data, and determine the data conversion policy corresponding to the attribute parameters by using the policy setting parameters and the calculation mode parameters; The data conversion strategy is used to perform formatting, reading and conversion operations on the stored data that has been read.
6. The general stream-batch integrated processing method based on different storage media according to claim 1 is characterized in that: The step of generating a semantic write strategy corresponding to the storage data based on the write parameters of the storage medium, and determining a storage database corresponding to the storage medium according to the semantic write strategy includes: Determining the writing parameters of the storage medium according to the batch data writing parameters and the line-by-line data writing parameters corresponding to the storage medium during the batch processing process; Determining a semantic write strategy corresponding to the stored data using the write parameters; A storage database corresponding to the storage medium is determined according to the semantic writing strategy.
7. The general stream-batch integrated processing method based on different storage media according to claim 1 is characterized in that: The step of saving the storage data that has completed the format conversion into the storage database according to the semantic writing strategy includes: Determining update parameters of the storage medium according to replacement parameters, appending parameters, and inserting update parameters corresponding to the storage medium during the stream processing process; Determining, according to the update parameters, equivalent transformation data corresponding to the stored data under the semantic write strategy; The equivalent transformation data is saved in the storage database corresponding to the storage medium.
8. The general stream-batch integrated processing method based on different storage media according to claim 7 is characterized in that: The step of determining, according to the update parameters, the equivalent transformation data corresponding to the stored data under the semantic write strategy comprises: According to the update parameters, determining that the stored data corresponds to one or more transformation strategies including data format transformation, data append transformation, merge update transformation and deletion transformation under the semantic write strategy; The equivalent transformation data corresponding to the stored data is obtained based on the transformation strategy.
9. A universal stream-batch integrated processing system based on different storage media, characterized in that: The system is applied to the stream processing and batch processing of data stored in different types of storage media, and the system includes: A data connection channel generation module, used to construct a data connection channel corresponding to the storage medium according to the type parameter of the storage medium; a stream-batch semantic reading strategy execution module, configured to generate a stream-batch semantic reading strategy corresponding to the stored data based on the reading parameters of the storage medium, and read the corresponding stored data in the data connection channel according to the stream-batch semantic reading strategy; A data conversion strategy execution module, used to determine a data conversion strategy for the stored data using the attribute parameters of the storage medium, and perform format conversion on the read stored data based on the data conversion strategy; A semantic write policy execution module, configured to generate a semantic write policy corresponding to the storage data based on the write parameters of the storage medium, and determine a storage database corresponding to the storage medium according to the semantic write policy; The data stream batch processing execution module is used to save the storage data that has completed the format conversion into the storage database according to the semantic writing strategy.
