Optimization method and system for data stream processing

Abstract

The invention relates to an optimization method and system for data stream processing, and the method comprises the steps: dividing an obtained flow code of a stream interface into N processing tasks, copying a to-be-processed data stream entering the stream interface into N copied data streams with the same number as the processing tasks, and distributing the N copied data streams to the N processing tasks one by one, n distributed processing tasks are obtained, and the N distributed processing tasks carry out task processing on the copied data streams of the N distributed processing tasks in a parallel mode so as to complete processing of the to-be-processed data streams. Therefore, parallel processing is carried out on the to-be-processed data stream in a stream copying and task processing combined mode, the to-be-processed data stream does not need to be stored in advance, meanwhile, the traditional mode that only serial processing can be carried out is broken through, the request time consumption and the processing time of the data stream are shortened, and the overall working efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to an optimization method and system for data stream processing. Background Art

[0002] Currently, most network requests are transmitted through data streams. Data is transmitted between various services or systems through various protocols. In essence, it is the transmission of various data streams through the network. Since the data stream can only be read once, it is usually written to a file or cache for storage after the data stream is transmitted. Subsequent functions read the data stream from the storage to perform corresponding functional operations. However, subsequent functional operations must be performed in a serial manner. This working method affects the overall work efficiency. Summary of the invention

[0003] The technical problem to be solved by the present invention is: the present invention provides an optimization method and system for data stream processing to improve overall work efficiency.

[0004] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0005] In a first aspect, the present invention provides a method for optimizing data stream processing, comprising:

[0006] Obtaining a process code of a flow interface, dividing the process code according to task rules, and obtaining N processing tasks;

[0007] Receiving a data stream to be processed entering the stream interface, copying the data stream to be processed into N copied data streams which are the same in number as the N processing tasks, and allocating the N copied data streams to the N processing tasks one by one to obtain N allocated processing tasks;

[0008] The N allocated processing tasks perform task processing on their own replicated data streams in a parallel manner to complete processing of the data stream to be processed.

[0009] The beneficial effects of the present invention are as follows: the process code of the stream interface is divided into N processing tasks according to the task rules, so that the data stream to be processed entering the stream interface is copied into N copied data streams which are the same number as the N processing tasks, and the copied data streams are assigned to the processing tasks one by one to perform task processing in a parallel manner, that is, the data stream to be processed is processed in parallel in a manner combining stream replication with processing tasks, and there is no need to store the data stream to be processed in advance. At the same time, the traditional method of only being able to process serially is broken, the request time and processing time of the data stream are reduced, and the overall work efficiency is improved.

[0010] Optionally, the processing tasks include ciphertext decryption tasks, signature verification tasks, MD5 verification tasks and business logic tasks.

[0011] According to the above description, the processing tasks are diverse and typical.

[0012] Optionally, the receiving the to-be-processed data stream entering the stream interface, and copying the to-be-processed data stream into N copied data streams having the same number as the N processing tasks comprises:

[0013] Reading the stream information of the data stream to be processed, and converting the stream information into a text string or byte array information;

[0014] The to-be-processed data stream is copied into N copied data streams which are equal in number to the N processing tasks according to the text string or the byte array information.

[0015] According to the above description, when the data stream to be processed is copied, the text string or byte array information converted from the stream information is copied to ensure the integrity and comprehensiveness of the copied data stream.

[0016] Optionally, the N allocated processing tasks perform task processing on their own replicated data streams in a parallel manner to complete processing of the data stream to be processed, including:

[0017] Determine whether all task processing is executed successfully. If not, generate a function abnormality alarm. If so, generate a function normal flag.

[0018] According to the above description, it can be seen that while performing task processing, it is also verified whether there are any abnormalities in the function. Only when all task processing is successfully executed will a normal function mark be generated, that is, it proves that the function is normal. Conversely, if any task processing fails to execute, it means that the function is abnormal and a function abnormality alarm is generated.

[0019] Optionally, generating a function abnormality alarm includes:

[0020] The process code corresponding to the task processing that failed to execute is marked as an exception code, and the exception code is thrown.

[0021] According to the above description, the process code corresponding to the failed task processing will be marked as an exception code and thrown out to facilitate subsequent search and maintenance.

[0022] In a second aspect, the present invention provides an optimization system for data stream processing, comprising:

[0023] A processing task division module is used to obtain the process code of the flow interface, divide the process code according to the task rules, and obtain N processing tasks;

[0024] a stream replication module, configured to receive a data stream to be processed entering the stream interface, replicate the data stream to be processed into N replicated data streams which are the same in number as the N processing tasks, and allocate the N replicated data streams one by one to the N processing tasks to obtain N allocated processing tasks;

[0025] The processing module is used for the N allocated processing tasks to perform task processing on their own replicated data streams in a parallel manner to complete the processing of the data stream to be processed.

[0026] The beneficial effects of the present invention are as follows: the process code of the stream interface is divided into N processing tasks according to the task rules, so that the data stream to be processed entering the stream interface is copied into N copied data streams which are the same number as the N processing tasks, and the copied data streams are assigned to the processing tasks one by one to perform task processing in a parallel manner, that is, the data stream to be processed is processed in parallel in a manner combining stream replication with processing tasks, and there is no need to store the data stream to be processed in advance. At the same time, the traditional method of only being able to process serially is broken, the request time and processing time of the data stream are reduced, and the overall work efficiency is improved.

[0027] Optionally, the processing tasks include ciphertext decryption tasks, signature verification tasks, MD5 verification tasks and business logic tasks.

[0028] According to the above description, the processing tasks are diverse and typical.

[0029] Optionally, the stream replication module is specifically:

[0030] Reading the stream information of the data stream to be processed, and converting the stream information into a text string or byte array information;

[0031] The to-be-processed data stream is copied into N copied data streams which are equal in number to the N processing tasks according to the text string or the byte array information.

[0032] According to the above description, when the data stream to be processed is copied, the text string or byte array information converted from the stream information is copied to ensure the integrity and comprehensiveness of the copied data stream.

[0033] Optionally, the processing module includes:

[0034] The judgment module is used to judge whether all task processing is executed successfully. If not, a function abnormality alarm is generated. If so, a function normal mark is generated.

[0035] According to the above description, it can be seen that while performing task processing, it is also verified whether there are any abnormalities in the function. Only when all task processing is successfully executed will a normal function mark be generated, that is, it proves that the function is normal. Conversely, if any task processing fails to execute, it means that the function is abnormal and a function abnormality alarm is generated.

[0036] Optionally, the judgment module is specifically:

[0037] The process code corresponding to the task processing that failed to execute is marked as an exception code, and the exception code is thrown.

[0038] According to the above description, the process code corresponding to the failed task processing will be marked as an exception code and thrown out to facilitate subsequent search and maintenance. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 A flow chart of a data processing optimization method provided in this embodiment;

[0040] Figure 2 A schematic diagram of the overall flow of a data processing optimization method provided in this embodiment;

[0041] Figure 3 A schematic diagram of the structure of a data processing optimization system provided in this embodiment.

[0042] [Description of Reference Numerals]

[0043] 1. An optimization system for data processing;

[0044] 2. Processing task division module;

[0045] 3. Stream replication module;

[0046] 4. Processing module; 41 judgment module. DETAILED DESCRIPTION

[0047] In order to better understand the above technical solution, exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to enable a clearer and more thorough understanding of the present invention and to fully convey the scope of the present invention to those skilled in the art.

[0048] Embodiment 1

[0049] Please refer to Figure 1 to Figure 2 The present invention provides a data processing optimization method, comprising the steps of:

[0050] S1. Obtain the process code of the flow interface, divide the process code according to the task rules, and obtain N processing tasks;

[0051] In this embodiment, if Figure 2 As shown, the process code of the flow interface is obtained, and the process code is divided according to the task rules, that is, the task logic of the code processing task, to obtain N processing tasks, where the processing tasks include but are not limited to ciphertext decryption tasks, signature verification tasks, md5 verification tasks and business logic tasks.

[0052] S2, receiving a data stream to be processed entering the stream interface, copying the data stream to be processed into N copied data streams which are the same in number as the N processing tasks, and assigning the N copied data streams to the N processing tasks one by one, to obtain N assigned processing tasks;

[0053] In this embodiment, if Figure 2 As shown, the data stream to be processed entering the stream interface is scheduled according to the processing task of step S1, and the data stream to be processed is copied into N copied data streams which are the same number as the N processing tasks, that is, the data stream to be processed is copied into N copies, and the N copied data streams are assigned one by one to the N processing tasks to obtain N assigned processing tasks.

[0054] In a specific embodiment, there are four processing tasks, namely: ciphertext decryption task, signature verification task, md5 verification task and business logic task. The data stream to be processed is copied into N copies to obtain: data stream to be processed 1, data stream to be processed 2, data stream to be processed 3 and data stream to be processed 4. They are allocated one by one, and the allocation results are as follows:

[0055] Ciphertext decryption task: data stream 1 to be processed;

[0056] Signature verification task: data stream 2 to be processed;

[0057] md5 verification task: data stream 3 to be processed;

[0058] Business logic task: data flow to be processed 4.

[0059] At this time, the step S2 of receiving the data stream to be processed entering the stream interface and copying the data stream to be processed into N copied data streams having the same number as the N processing tasks includes:

[0060] S21, reading the flow information of the data flow to be processed, and converting the flow information into a text string or byte array information;

[0061] S22. Copy the to-be-processed data stream into N copied data streams, which is the same number as the N processing tasks, according to the text string or the byte array information.

[0062] In this embodiment, if Figure 2 As shown, when copying the data stream to be processed, the stream information of the data stream to be processed is first read, the stream information is converted into a text string or byte array information, and then the data stream to be processed is copied into N copied data streams which are the same as the number of N processing tasks according to the text string or byte array information.

[0063] S3. The N allocated processing tasks perform task processing on their own replicated data streams in parallel to complete processing of the data stream to be processed.

[0064] In this embodiment, if Figure 2 As shown, the N assigned processing tasks process their own replicated data streams in parallel, as shown below:

[0065] Parallel 1: Ciphertext decryption task: data stream 1 to be processed;

[0066] Parallel 2: Signature verification task: data stream 2 to be processed;

[0067] Parallel 3: md5 verification task: data stream 3 to be processed;

[0068] Parallel 4: Business logic task: data stream 4 to be processed.

[0069] At this time, the N allocated processing tasks in step S3 perform task processing on their own replicated data streams in parallel to complete the processing of the data stream to be processed, including:

[0070] S31. Determine whether all task processing is successfully executed. If not, generate a function abnormality alarm. If so, generate a function normal flag.

[0071] At this time, generating a function abnormality alarm in step S31 includes:

[0072] S311. Mark the process code corresponding to the failed task processing as an exception code, and throw the exception code.

[0073] In this embodiment, if Figure 2 As shown, if all task processing is executed successfully, a normal function mark is generated. Otherwise, if there is an execution failure, a function abnormality alarm is generated, the process code corresponding to the failed task processing is marked as an abnormal code, and the abnormal code is thrown.

[0074] Embodiment 2

[0075] Please refer to Figure 3 The present invention provides a data processing optimization system 1, comprising: a processing task division module 2, a stream replication module 3, a processing module 4 and a judgment module 41.

[0076] The processing task division module 2 is used to obtain the process code of the flow interface, divide the process code according to the task rules, and obtain N processing tasks;

[0077] A stream replication module 3 is used to receive a data stream to be processed entering the stream interface, replicate the data stream to be processed into N replicated data streams which are the same in number as the N processing tasks, and allocate the N replicated data streams to the N processing tasks one by one to obtain N allocated processing tasks;

[0078] The processing module 4 is used for the N allocated processing tasks to perform task processing on their own replicated data streams in a parallel manner to complete the processing of the data stream to be processed.

[0079] Specifically, the processing tasks include ciphertext decryption tasks, signature verification tasks, md5 verification tasks and business logic tasks.

[0080] Specifically, the stream replication module 3 is specifically:

[0081] Reading the stream information of the data stream to be processed, and converting the stream information into a text string or byte array information;

[0082] The to-be-processed data stream is copied into N copied data streams which are equal in number to the N processing tasks according to the text string or the byte array information.

[0083] Specifically, the processing module 4 includes:

[0084] The judgment module 41 is used to judge whether all task processing is executed successfully. If not, a function abnormality alarm is generated. If so, a function normal flag is generated.

[0085] Specifically, the judging module 41 is:

[0086] The process code corresponding to the task processing that failed to execute is marked as an exception code, and the exception code is thrown.

[0087] Since the system / device described in the above embodiments of the present invention is a system / device used to implement the method of the above embodiments of the present invention, a person skilled in the art can understand the specific structure and deformation of the system / device based on the method described in the above embodiments of the present invention, and thus will not be described in detail here. All systems / devices used in the method of the above embodiments of the present invention belong to the scope of protection of the present invention.

[0088] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0089] The present invention is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present invention. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions.

[0090] It should be noted that in the claims, any reference numerals placed between brackets shall not be construed as limiting the claims. The word "comprising" does not exclude the presence of components or steps not listed in the claims. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. The invention may be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. In the claims enumerating several means, several of these means may be embodied by the same hardware. The use of the words first, second, third, etc., is for convenience of expression only and does not indicate any order. These words may be understood as part of the component name.

[0091] In addition, it should be noted that, in the description of this specification, the description of the terms "one embodiment", "some embodiments", "embodiment", "example", "specific example" or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, unless they are contradictory.

[0092] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments after knowing the basic creative concept. Therefore, the claims should be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present invention.

[0093] Obviously, those skilled in the art can make various modifications and variations to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention should also include these modifications and variations.

Claims

1. A data stream processing optimization method, characterized in that: include: Obtaining a process code of a flow interface, dividing the process code according to task rules, and obtaining N processing tasks; Receiving a data stream to be processed entering the stream interface, copying the data stream to be processed into N copied data streams which are the same in number as the N processing tasks, and allocating the N copied data streams to the N processing tasks one by one to obtain N allocated processing tasks; The N allocated processing tasks perform task processing on their own replicated data streams in a parallel manner to complete processing of the data stream to be processed.

2. A data stream processing optimization method as claimed in claim 1, characterized in that: The processing tasks include ciphertext decryption tasks, signature verification tasks, MD5 verification tasks and business logic tasks.

3. The method for optimizing data stream processing according to claim 1, characterized in that: The receiving of the to-be-processed data stream entering the stream interface, and copying the to-be-processed data stream into N copied data streams having the same number as the N processing tasks comprises: Reading the stream information of the data stream to be processed, and converting the stream information into a text string or byte array information; The to-be-processed data stream is copied into N copied data streams which are equal in number to the N processing tasks according to the text string or the byte array information.

4. The method for optimizing data stream processing according to claim 1, characterized in that: The N allocated processing tasks process their own replicated data streams in parallel to complete the processing of the data stream to be processed, including: Determine whether all task processing is executed successfully. If not, generate a function abnormality alarm. If so, generate a function normal flag.

5. A data stream processing optimization method as claimed in claim 4, characterized in that: Generating a function abnormality alarm includes: The process code corresponding to the task processing that failed to execute is marked as an exception code, and the exception code is thrown.

6. A data stream processing optimization system, characterized in that: include: A processing task division module is used to obtain the process code of the flow interface, divide the process code according to the task rules, and obtain N processing tasks; a stream replication module, configured to receive a data stream to be processed entering the stream interface, replicate the data stream to be processed into N replicated data streams which are the same in number as the N processing tasks, and allocate the N replicated data streams one by one to the N processing tasks to obtain N allocated processing tasks; The processing module is used for the N allocated processing tasks to perform task processing on their own replicated data streams in a parallel manner to complete the processing of the data stream to be processed.

7. A data stream processing optimization system as claimed in claim 6, characterized in that: The processing tasks include ciphertext decryption tasks, signature verification tasks, MD5 verification tasks and business logic tasks.

8. The data stream processing optimization system according to claim 6, characterized in that: The stream replication module is specifically: Reading the stream information of the data stream to be processed, and converting the stream information into a text string or byte array information; The to-be-processed data stream is copied into N copied data streams which are equal in number to the N processing tasks according to the text string or the byte array information.

9. The data stream processing optimization system according to claim 6, characterized in that: The processing module comprises: The judgment module is used to judge whether all task processing is executed successfully. If not, a function abnormality alarm is generated. If so, a function normal mark is generated.

10. The data stream processing optimization system according to claim 9, characterized in that: The judgment module is specifically: The process code corresponding to the task processing that failed to execute is marked as an exception code, and the exception code is thrown.

Images