Fault detection method and device for flow task processing

Abstract

The invention provides a fault detection method and device for stream task processing, and relates to the technical field of data processing. According to the method provided by the invention, a mapping relation table is established, wherein the mapping relation table is used for representing each piece of table data and a mapping relation among a database, a task queue and a data processing result corresponding to each piece of table data; then, faults occurring in the data processing process of the flow task can be quickly and accurately recognized according to each piece of table data included in the flow task and the mapping relation among the database, the task queue and the data processing result corresponding to each piece of table data, and the stability and accuracy in the flow task processing process are improved; and the use requirements of the user in different use scenes are met.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular, to a fault detection method and device for stream task processing. Background Art

[0002] With the development and application of big data technology, stream task processing has become an important part of the data processing field. However, due to the real-time and continuous nature of stream tasks, in the case of a fault occurring during the data processing process, the data processing systems in the related technologies cannot quickly detect the fault, seriously affecting the stability and accuracy of the data processing process and unable to meet the user's usage requirements.

[0003] Therefore, there is an urgent need for a fault detection method and device for stream task processing that can quickly and accurately identify the faults occurring during the data processing process of stream tasks, and improve the stability and accuracy during the stream task processing process. Summary of the Invention

[0004] Embodiments of the present invention provide a fault detection method and device for stream task processing, which can quickly and accurately identify the faults occurring during the data processing process of stream tasks, improve the stability and accuracy during the stream task processing process, and meet the user's usage requirements in different usage scenarios.

[0005] To achieve the above object, the embodiments of the present invention adopt the following technical solutions:

[0006] In a first aspect, a fault detection method for stream task processing is provided, which is applied to a data processing system. The data processing system includes a producer component, an intermediary component, and a consumer component; the producer component is used to obtain multiple pieces of to-be-processed data corresponding to a target stream task from multiple databases, and generate table data corresponding to each piece of to-be-processed data among the multiple pieces of to-be-processed data; the intermediary component is used to obtain the table data corresponding to each piece of to-be-processed data, and add the multiple pieces of table data into corresponding task queues according to the category of each piece of table data; the consumer component is used to obtain the table data from the corresponding task queue, and process the table data according to the category of each piece of table data to obtain a task execution result of the target stream task, where the task execution result includes a data processing result corresponding to each piece of table data;

[0007] The above method includes: establishing a mapping relation table, which is used to represent the mapping relation between each table data and the corresponding database, task queue, and data processing result; determining, according to the mapping relation table, the first quantity, second quantity, third quantity, fourth quantity, fifth quantity, and sixth quantity corresponding to the target flow task; wherein, the first quantity is the quantity of the to-be-processed data corresponding to the target flow task obtained by the producer component, and the second quantity is the quantity of the table data sent by the producer component to the mediator component; the third quantity is the quantity of the table data received by the mediator component, and the fourth quantity is the quantity of the table data added by the mediator component to the task queue; the fifth quantity is the quantity of the table data obtained by the consumer component from the task queue, and the sixth quantity is the quantity of the data processing results obtained by the consumer component by processing the table data according to the category of each table data; when the absolute difference between the first quantity and the second quantity is greater than or equal to a preset threshold, the absolute difference between the second quantity and the third quantity is greater than or equal to a preset threshold, the absolute difference between the third quantity and the fourth quantity is greater than or equal to a preset threshold, the absolute difference between the fourth quantity and the fifth quantity is greater than or equal to a preset threshold, or the absolute difference between the fifth quantity and the sixth quantity is greater than or equal to a preset threshold, generating a fault detection result, which is used to prompt that the execution of the target flow task is abnormal.

[0008] In a possible implementation manner of the first aspect, the above method further includes: when the absolute difference between the first quantity and the second quantity is greater than or equal to a preset threshold, determining that the cause of the fault in the fault detection result is the abnormality of the producer component; when the absolute difference between the second quantity and the third quantity is greater than or equal to a preset threshold or the absolute difference between the third quantity and the fourth quantity is greater than or equal to a preset threshold, determining that the cause of the fault in the fault detection result is the abnormality of the mediator component; when the absolute difference between the fourth quantity and the fifth quantity is greater than or equal to a preset threshold or the absolute difference between the fifth quantity and the sixth quantity is greater than or equal to a preset threshold, determining that the cause of the fault in the fault detection result is the abnormality of the consumer component.

[0009] In a possible implementation manner of the first aspect, after the consumer component processes the table data according to the task execution process corresponding to the task category of the table data to obtain the task processing result corresponding to each table data, the above method further includes: determining the status information of the task processing result of each table data included in the target flow task, where the status information includes correct or incorrect; determining the accuracy rate of the target flow task according to the status information of the task processing result of each table data included in the target flow task, where the accuracy rate is the ratio of the quantity of the task processing results with the status information being correct to the total quantity of the task processing results; when the accuracy rate is less than or equal to a preset threshold, generating a fault detection result, which is used to prompt that the multiple to-be-processed data corresponding to the target flow task obtained from multiple databases is abnormal.

[0010] In a possible implementation of the first aspect, determining the status information of the task processing result of each table data included in the target stream task includes: obtaining the verification value of the task processing result of each table data included in the target stream task; when the actual value of the task processing result of each table data included in the target stream task is consistent with the verification value, determining that the status information of the task processing result of the table data is correct; when the actual value of the task processing result of each table data included in the target stream task is inconsistent with the verification value, determining that the status information of the task processing result of the table data is incorrect.

[0011] In a possible implementation of the first aspect, after generating the fault detection result, the method further includes: sending an abort instruction to the producer component, the middleman component, and the consumer component, where the abort instruction is used to instruct the producer component, the middleman component, and the consumer component to abort the execution of the target stream task.

[0012] In a possible implementation of the first aspect, the method further includes: based on the mapping relation table, determining the database corresponding to each data processing result with an incorrect status information as an abnormal database; in response to the trusted data input by the user for replacing the multiple to-be-processed data included in the abnormal database, sending a task execution instruction to the producer component, the middleman component, and the consumer component, where the task execution instruction is used to instruct the producer component, the middleman component, and the consumer component to determine the task execution result of the target stream task according to the trusted data.

[0013] In a possible implementation of the first aspect, each database is configured with a corresponding identifier; establishing a mapping relation table includes: when the producer component generates the table data corresponding to each to-be-processed data among the multiple to-be-processed data, determining the identifier corresponding to each table data; generating a first mapping relation according to the identifier of each table data and the identifier of the database from which the to-be-processed data corresponding to each table data comes; when the middleman component adds the multiple table data into the corresponding task queue according to the category of each table data, determining the identifier corresponding to each task queue; generating a second mapping relation according to the identifier of each table data and the identifier of the task queue corresponding to each table data; when the consumer component processes the table data according to the category of each table data to obtain the data processing result corresponding to each table data, determining the identifier corresponding to the data processing result of each table data; generating a third mapping relation according to the identifier of each table data and the identifier corresponding to the data processing result of each table data; establishing a mapping relation table according to the first mapping relation, the second mapping relation, and the third mapping relation.

[0014] The beneficial effects of the present invention are as follows: By establishing a mapping relationship table, the method provided by the present invention can establish the association relationship between the data processed by each component during the processing of the flow task. Furthermore, it can determine the component with a running fault from multiple components according to the mapping relationship table, realizing the fast and accurate detection of faults during the processing of the flow task. Moreover, the method provided by the present invention determines the accuracy rate of the data processing results corresponding to multiple table data, and quickly and accurately determines the abnormal data in the database according to the mapping relationship table when the accuracy rate is less than or equal to the preset threshold. Then, it can quickly complete the correct execution of the flow task according to the reliable data input by the user for replacing the abnormal data, effectively reducing the data processing time and improving the data processing efficiency.

[0015] In a second aspect, the present invention provides a fault detection device for flow task processing, which is applied to a data processing system. The data processing system includes a producer component, an intermediary component, and a consumer component. The producer component is used to obtain multiple pieces of to-be-processed data corresponding to a target flow task from multiple databases and generate table data corresponding to each piece of to-be-processed data among the multiple pieces of to-be-processed data. The intermediary component is used to obtain the table data corresponding to each piece of to-be-processed data and add the multiple pieces of table data into corresponding task queues according to the category of each piece of table data. The consumer component is used to obtain the table data from the corresponding task queue, process the table data according to the category of each piece of table data, and obtain the task execution result of the target flow task. The task execution result includes the data processing result corresponding to each piece of table data.

[0016] The above device includes: a mapping establishment module, configured to establish a mapping relationship table, where the mapping relationship table is used to represent the mapping relationship between each table data and the corresponding database, task queue, and data processing result; a quantity determination module, configured to determine a first quantity, a second quantity, a third quantity, a fourth quantity, a fifth quantity, and a sixth quantity corresponding to the target flow task according to the mapping relationship table; where the first quantity is the quantity of the to-be-processed data corresponding to the target flow task obtained by the producer component, and the second quantity is the quantity of the table data sent by the producer component to the middleman component; the third quantity is the quantity of the table data received by the middleman component, and the fourth quantity is the quantity of the table data added by the middleman component to the task queue; the fifth quantity is the quantity of the table data obtained by the consumer component from the task queue, and the sixth quantity is the quantity of the data processing results obtained by the consumer component processing the table data according to the category of each table data; a result generation module, configured to generate a fault detection result when the absolute difference between the first quantity and the second quantity is greater than or equal to a preset threshold, the absolute difference between the second quantity and the third quantity is greater than or equal to a preset threshold, the absolute difference between the third quantity and the fourth quantity is greater than or equal to a preset threshold, the absolute difference between the fourth quantity and the fifth quantity is greater than or equal to a preset threshold, or the absolute difference between the fifth quantity and the sixth quantity is greater than or equal to a preset threshold, where the fault detection result is used to indicate that the execution of the target flow task is abnormal.

[0017] In a possible implementation manner of the second aspect, the device further includes a fault cause determination module, configured to: when the absolute difference between the first quantity and the second quantity is greater than or equal to a preset threshold, determine that the fault cause of the fault detection result is the abnormality of the producer component; when the absolute difference between the second quantity and the third quantity is greater than or equal to a preset threshold or the absolute difference between the third quantity and the fourth quantity is greater than or equal to a preset threshold, determine that the fault cause of the fault detection result is the abnormality of the middleman component; when the absolute difference between the fourth quantity and the fifth quantity is greater than or equal to a preset threshold or the absolute difference between the fifth quantity and the sixth quantity is greater than or equal to a preset threshold, determine that the fault cause of the fault detection result is the abnormality of the consumer component.

[0018] In a third aspect, an electronic device is provided, which includes a memory and one or more processors; the memory is coupled to the processor; where computer program code is stored in the memory, and the computer program code includes computer instructions, and when the computer instructions are executed by the processor, the electronic device is caused to execute the method in any implementation manner of the first aspect.

[0019] In a fourth aspect, a computer-readable storage medium is provided, including computer instructions, and when the computer instructions are run on an electronic device, the electronic device is caused to execute the method in any implementation manner of the first aspect.

[0020] In a fifth aspect, there is provided a computer program product which, when running on a computer, causes the computer to execute the method in any implementation manner of the first aspect.

[0021] It can be understood that for the beneficial effects that can be achieved by the system in the second aspect, the electronic device in the third aspect, the computer-readable storage medium in the fourth aspect, and the computer program product in the fifth aspect provided above, reference may be made to the beneficial effects in the first aspect and any of its possible design manners, which will not be elaborated herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 FIG. is a schematic hardware structure diagram of an electronic device shown in an embodiment of the present invention;

[0023] Figure 2 FIG. is a schematic hardware structure diagram of a data processing system shown in an embodiment of the present invention;

[0024] Figure 3 FIG. is a flowchart of a fault detection method for stream task processing provided by an embodiment of the present invention;

[0025] Figure 4 FIG. is a schematic diagram of the generation process of a mapping relationship table provided by an embodiment of the present invention;

[0026] Figure 5 FIG. is a flowchart of another fault detection method for stream task processing provided by an embodiment of the present invention;

[0027] Figure 6 FIG. is a flowchart of yet another fault detection method for stream task processing provided by an embodiment of the present invention;

[0028] Figure 7 FIG. is a schematic hardware structure diagram of a detection device shown in an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0029] Next, the technical solutions in the embodiments of the present invention will be described with reference to the accompanying drawings in the embodiments of the present invention. Among them, in the description of the present invention, unless otherwise specified, " / " indicates that the objects associated before and after are in an "or" relationship. For example, A / B may represent A or B; the "or" in the present invention is merely a description of the association relationship of the associated objects, indicating that there can be three relationships. For example, A or B may represent: A exists alone, A and B exist simultaneously, and B exists alone. Among them, A and B can be singular or plural. And, in the description of the present invention, unless otherwise specified, "a plurality of" means two or more than two. "At least one (item)" or its similar expression refers to any combination of these items, including any combination of single item (item) or plural items (items).

[0030] In addition, for the convenience of clearly describing the technical solutions of the embodiments of the present invention, in the embodiments of the present invention, terms such as "first" and "second" are used to distinguish identical or similar items with basically the same functions and roles. Those skilled in the art can understand that terms such as "first" and "second" do not limit the quantity and execution order, and terms such as "first" and "second" do not necessarily limit being different.

[0031] Meanwhile, in the embodiments of the present invention, words such as "exemplary" or "for example" are used to represent examples, illustrations or explanations. Any embodiment or design solution described as "exemplary" or "for example" in the embodiments of the present invention should not be construed as being more excellent or having more advantages than other embodiments or design solutions. Rather, the use of words such as "exemplary" or "for example" is intended to present relevant concepts in a specific manner for easy understanding.

[0032] With the development and application of big data technology, stream task processing has become an important part of the data processing field. In related technologies, stream tasks are usually processed by a data processing system, which includes a producer component, an intermediary component, and a consumer component. Among them, the producer component is used to obtain multiple data corresponding to stream task processing from multiple databases and send the multiple data to the intermediary component. The intermediary component is used to place the multiple data in different topics respectively, and each topic corresponds to a different task category. The consumer component is used to obtain multiple data from different topics and process them according to the corresponding task category to obtain a data processing result.

[0033] However, due to the real-time and continuous nature of stream tasks, in the case of a failure during the data processing process, the data processing system in related technologies cannot quickly detect and handle the failure, seriously affecting the stability and accuracy of the data processing process and unable to meet the user's usage requirements.

[0034] Therefore, there is an urgent need for a fault detection method and system based on stream task processing, which can accurately identify the faults that occur during the data processing process of stream tasks quickly and improve the stability and accuracy during the stream task processing process.

[0035] In view of this, an embodiment of the present invention provides a fault detection method for stream task processing, which is applied to a data processing system. The data processing system includes a producer component, an intermediary component, and a consumer component. The producer component is used to obtain multiple pieces of to-be-processed data corresponding to a target stream task from multiple databases, and generate table data corresponding to each piece of to-be-processed data among the multiple pieces of to-be-processed data. The intermediary component is used to obtain the table data corresponding to each piece of to-be-processed data, and add the multiple pieces of table data into corresponding task queues according to the category of each piece of table data. The consumer component is used to obtain the table data from the corresponding task queue, and process the table data according to the category of each piece of table data to obtain a task execution result of the target stream task. The task execution result includes a data processing result corresponding to each piece of table data. The above method includes: establishing a mapping relation table, where the mapping relation table is used to represent the mapping relation between each piece of table data and the database, task queue, and data processing result corresponding to each piece of table data; according to the mapping relation table, determining a first quantity, a second quantity, a third quantity, a fourth quantity, a fifth quantity, and a sixth quantity corresponding to the target stream task. Among them, the first quantity is the quantity of to-be-processed data corresponding to the target stream task obtained by the producer component, and the second quantity is the quantity of table data sent by the producer component to the intermediary component. The third quantity is the quantity of table data received by the intermediary component, and the fourth quantity is the quantity of table data added by the intermediary component to the task queue. The fifth quantity is the quantity of table data obtained by the consumer component from the task queue, and the sixth quantity is the quantity of data processing results obtained by the consumer component by processing the table data according to the category of each piece of table data. When the absolute difference between the first quantity and the second quantity is greater than or equal to a preset threshold, the absolute difference between the second quantity and the third quantity is greater than or equal to a preset threshold, the absolute difference between the third quantity and the fourth quantity is greater than or equal to a preset threshold, the absolute difference between the fourth quantity and the fifth quantity is greater than or equal to a preset threshold, or the absolute difference between the fifth quantity and the sixth quantity is greater than or equal to a preset threshold, a fault detection result is generated, and the fault detection result is used to prompt that the execution of the target stream task is abnormal.

[0036] The method provided by the present invention can establish the association relationship between the data processed by each component in the process of stream task processing by establishing a mapping relation table, and then can determine the component with a running fault from multiple components according to the mapping relation table, realizing the fast and accurate detection of faults in the process of stream task processing. Moreover, the method provided by the present invention can determine the accuracy rate of the data processing results corresponding to multiple pieces of table data, and quickly and accurately determine the abnormal data in the database according to the mapping relation table when the accuracy rate is less than or equal to a preset threshold. Furthermore, it can quickly complete the correct execution of the stream task according to the reliable data for replacing the abnormal data input by the user, effectively reducing the data processing time and improving the data processing efficiency.

[0037] In some embodiments, a fault detection method for stream task processing provided by an embodiment of the present invention can be executed by a fault detection device 100 for stream task processing (hereinafter referred to as the detection device 100). As an example, the detection device 100 can be any electronic device 200 with data processing capabilities, such as a general-purpose computer, a personal computer, a laptop computer, a switch, or a tablet computer, etc. The specific implementation manner of the detection device 100 is not limited herein.

[0038] Figure 1 FIG. shows a schematic hardware structure diagram of an electronic device provided by an embodiment of the present invention. The electronic device 200 includes a processor 210, a memory 220, and a communication interface 230.

[0039] The processor 210 may include one or more processing cores. The processor 210 connects various parts within the electronic device 200 through various interfaces and lines, and by running or executing instructions, programs, code sets, or instruction sets stored in the memory 220, and by calling data stored in the memory 220, executes various functions of the electronic device 200 and processes data. Optionally, the processor 210 may be implemented in at least one hardware form of a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processing (DSP), a field-programmable gate array (FPGA), or a programmable logic array (PLA).

[0040] The memory 220 may include a random access memory (RAM), and may also include a read-only memory (ROM). Optionally, the memory 220 includes a non-transitory computer-readable medium. The memory 220 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 220 may include a storage program area. Among them, the storage program area may store instructions for implementing an operating system, instructions for implementing at least one function (such as a mapping establishment function, a quantity determination function, etc.), and instructions for implementing the above-mentioned various method embodiments.

[0041] A communication interface 230 for communicating with other devices, apparatuses, or communication networks, such as data storage devices, image processing devices, or Ethernet, radio access networks (RANs), wireless local area networks (WLANs), etc.

[0042] In terms of physical implementation, the above-mentioned various devices (such as the processor 210, the memory 220, and the communication interface 230) can be components of the same device (such as a laptop computer) respectively. Alternatively, at least two of them can be disposed in the same device, that is, as different components in a device, similar to the deployment manner of devices or components in a distributed system.

[0043] It can be understood that the structure illustrated in this embodiment does not constitute a specific limitation on the electronic device 200. In other embodiments of the present invention, the electronic device 200 may include more or fewer components than those shown in the figure, or combine certain components, or split certain components, or have different component arrangements. The illustrated components can be implemented in hardware, software, or a combination of software and hardware.

[0044] See Figure 2 , Figure 2 FIG. is a schematic diagram of the hardware structure of a data processing system provided by an embodiment of the present invention. The data processing system 300 includes a producer component 310, an intermediary component 320, and a consumer component 330. The producer component 310 is used to obtain a plurality of data to be processed corresponding to a target stream task from a plurality of databases, and generate table data corresponding to each data to be processed among the plurality of data to be processed. The intermediary component 320 is used to obtain the table data corresponding to each data to be processed, and add the plurality of table data into corresponding task queues according to the category of each table data. The consumer component 330 is used to obtain the table data from the corresponding task queue, and process the table data according to the category of each table data to obtain a task execution result of the target stream task. The task execution result includes a data processing result corresponding to each table data.

[0045] In one example, the producer component 310 provided by the embodiment of the present invention is a Change Data Capture (CDC) component, the intermediary component 320 is a Kafka message queue component, and the consumer component 330 is a Hadoop Upserts Deletes and Incrementals (HUDI) component.

[0046] It should be understood that the above components are all for illustrative purposes. The data processing system 300 may include other producer components 310, mediator components 320, and consumer components 330 of different types or functions, and may also include a greater or smaller number of components. For example, the data processing system 300 includes 2 producer components and 3 consumer components. The embodiments of the present invention do not impose specific limitations on the types, functions, and numbers of components included in the data processing system 300.

[0047] The following will describe a fault detection method for stream task processing provided by an embodiment of the present invention with reference to the accompanying drawings of the specification.

[0048] Figure 3 The flowchart of a fault detection method for stream task processing provided by an embodiment of the present invention. Optionally, this method may be executed by Figure 1 the electronic device 200 shown, that is, executed by the detection device 100, and applied to Figure 3 the data processing system shown. This method may include the following steps:

[0049] S1. Establish a mapping relationship table, which is used to represent the mapping relationship between each table data and the corresponding database, task queue, and data processing result.

[0050] In some embodiments, each database is configured with a corresponding identifier. The above S1 specifically includes the following steps:

[0051] When the producer component generates the table data corresponding to each of the multiple data to be processed, determine the identifier corresponding to each table data; generate a first mapping relationship based on the identifier of each table data and the identifier of the database from which the corresponding data to be processed comes; when the mediator component adds multiple table data to the corresponding task queue according to the category of each table data, determine the identifier corresponding to each task queue; generate a second mapping relationship based on the identifier of each table data and the identifier of the task queue corresponding to each table data; when the consumer component processes the table data according to the category of each table data to obtain the data processing result corresponding to each table data, determine the identifier corresponding to the data processing result of each table data; generate a third mapping relationship based on the identifier of each table data and the identifier of the data processing result corresponding to each table data; establish a mapping relationship table according to the first mapping relationship, the second mapping relationship, and the third mapping relationship.

[0052] Exemplarily, refer to Figure 4 , Figure 4Schematic diagram of the establishment process of a mapping relationship table provided by an embodiment of the present invention. When the producer component generates table data corresponding to 5 pieces of data to be processed from 3 databases, the detection device 100 determines the identifiers corresponding to the 5 pieces of table data, namely table data A, table data B, table data C, table data D, and table data E, and the 3 databases are database 1, database 2, and database 3 respectively. The detection device 100 generates a first mapping relationship according to the identifier of each piece of table data and the identifier of the database from which the data to be processed corresponding to each piece of table data comes;

[0053] When the intermediary component adds the 5 pieces of table data to the corresponding 3 different task queues according to the category of each piece of table data, the detection device 100 determines the identifiers corresponding to each task queue, namely task queue 1, task queue 2, and task queue 3. The detection device 100 generates a second mapping relationship according to the identifier of each piece of table data and the identifier of the task queue corresponding to each piece of table data.

[0054] When the consumer component processes the table data according to the category of each piece of table data to obtain the data processing result corresponding to each piece of table data, the detection device 100 determines the identifier corresponding to the data processing result of each piece of table data among the 5 pieces of table data; namely data processing result A, data processing result B, data processing result C, data processing result D, and data processing result E. The detection device 100 generates a third mapping relationship according to the identifier of each piece of table data and the identifier of the data processing result corresponding to each piece of table data; finally, the detection device 100 establishes a mapping relationship table according to the first mapping relationship, the second mapping relationship, and the third mapping relationship.

[0055] S2. According to the mapping relationship table, determine the first quantity, second quantity, third quantity, fourth quantity, fifth quantity, and sixth quantity corresponding to the target flow task.

[0056] Among them, the first quantity is the quantity of the data to be processed corresponding to the target flow task obtained by the producer component, and the second quantity is the quantity of the table data sent by the producer component to the intermediary component; the third quantity is the quantity of the table data received by the intermediary component, and the fourth quantity is the quantity of the table data added by the intermediary component to the task queue; the fifth quantity is the quantity of the table data obtained by the consumer component from the task queue, and the sixth quantity is the quantity of the data processing results obtained by the consumer component processing the table data according to the category of each piece of table data;

[0057] S3. When the absolute difference between the first quantity and the second quantity is greater than or equal to a preset threshold, the absolute difference between the second quantity and the third quantity is greater than or equal to the preset threshold, the absolute difference between the third quantity and the fourth quantity is greater than or equal to the preset threshold, the absolute difference between the fourth quantity and the fifth quantity is greater than or equal to the preset threshold, or the absolute difference between the fifth quantity and the sixth quantity is greater than or equal to the preset threshold, a fault detection result is generated, and the fault detection result is used to indicate that the target flow task execution is abnormal.

[0058] Specifically, the absolute difference is the absolute value of the difference between two quantities. For example, the first quantity is 40, the second quantity is 60, and the third quantity is 30. The absolute difference between the first quantity and the second quantity is 20, and the absolute difference between the second quantity and the third quantity is 30.

[0059] As can be seen from the above S1 - S3, the method provided by the embodiment of the present invention can establish the association relationship between the data processed by each component in the process of processing the flow task by establishing a mapping relationship table, and then can determine the component with a running fault from multiple components according to the mapping relationship table, realizing the fast and accurate detection of faults in the process of processing the flow task.

[0060] In a possible implementation manner, the method provided by the embodiment of the present invention further includes:

[0061] When the absolute difference between the first quantity and the second quantity is greater than or equal to the preset threshold, it is determined that the cause of the fault in the fault detection result is the abnormality of the producer component; when the absolute difference between the second quantity and the third quantity is greater than or equal to the preset threshold or the absolute difference between the third quantity and the fourth quantity is greater than or equal to the preset threshold, it is determined that the cause of the fault in the fault detection result is the abnormality of the middle component; when the absolute difference between the fourth quantity and the fifth quantity is greater than or equal to the preset threshold or the absolute difference between the fifth quantity and the sixth quantity is greater than or equal to the preset threshold, it is determined that the cause of the fault in the fault detection result is the abnormality of the consumer component.

[0062] As can be seen from the above, the method provided by the embodiment of the present invention can determine the data quantities input and output by each component based on the mapping relationship table, and quickly and accurately determine the abnormal component based on the data quantities input and output by each component, realizing the fast and accurate fault detection in the process of processing the flow task.

[0063] In some embodiments, see Figure 5 , after the consumer component processes the table data according to the task execution process corresponding to the task category of the table data to obtain the task processing result corresponding to each table data, the method provided by the embodiment of the present invention further includes:

[0064] S510. Determine the status information of the task processing result of each table data included in the target flow task, and the status information includes correct or incorrect.

[0065] In a possible implementation, the above S510 specifically includes the following steps:

[0066] Obtain the verification value of the task processing result of each table data included in the target flow task; when the actual value of the task processing result of each table data included in the target flow task is consistent with the verification value, determine that the status information of the task processing result of the table data is correct; when the actual value of the task processing result of each table data included in the target flow task is inconsistent with the verification value, determine that the status information of the task processing result of the table data is incorrect.

[0067] Specifically, the verification value of the task processing result of each table data can be the true value of the task processing result of each table data, or can be obtained by processing the true value of the task processing result of each table data through a preset verification algorithm. For example, the verification algorithm can be a hash algorithm.

[0068] For example, the task processing result of table data A is that the order status is completed (actual value). The detection device 100 obtains the true order status (verification value) of the task processing result of table data A. When the true order status of the task processing result of table data A is completed, the actual value of the task processing result of table data A is consistent with the verification value, and it is determined that the status information of the task processing result of table data A is correct. When the true order status of the task processing result of table data A is not completed, the actual value of the task processing result of table data A is inconsistent with the verification value, and it is determined that the status information of the task processing result of table data A is incorrect.

[0069] S520. Determine the accuracy rate of the target flow task according to the status information of the task processing result of each table data included in the target flow task. The accuracy rate is the ratio of the number of task processing results with the status information being correct to the total number of task processing results.

[0070] Exemplarily, the number of task processing results with the status information being correct is 85, and the total number of task processing results is 100. Then the accuracy rate of the target flow task is 85%.

[0071] S530. When the accuracy rate is less than or equal to the preset threshold, generate a fault detection result, which is used to prompt that the multiple to-be-processed data corresponding to the target flow task obtained from multiple databases is abnormal.

[0072] Combined with the above example, when the preset threshold is 90%, the accuracy rate of the target flow task is less than the preset threshold, and a fault detection result is generated.

[0073] As can be seen from the above, the method provided by the embodiments of the present invention checks the data processing results corresponding to each table data to determine the accuracy rate of the target flow task, and can quickly and accurately detect faults when an error occurs in the process of executing the flow task, meeting the user's usage requirements in different usage scenarios.

[0074] In a possible implementation manner, after generating the fault detection result, the method provided by the embodiments of the present invention further includes:

[0075] Sending an abort instruction to the producer component, the mediator component, and the consumer component, where the abort instruction is used to instruct the producer component, the mediator component, and the consumer component to abort the execution of the target flow task.

[0076] In this way, the method provided by the embodiments of the present invention can abort the execution of the target flow task when a fault occurs in the process of executing the target flow task, effectively saving computing resources and avoiding waste of computing resources.

[0077] In some embodiments, referring to Figure 6 , the method provided by the embodiments of the present invention further includes:

[0078] S610. Based on the mapping relationship table, determine the database corresponding to each data processing result with an error status information as an abnormal database;

[0079] S620. In response to the trusted data input by the user for replacing the multiple data to be processed included in the abnormal database, send a task execution instruction to the producer component, the mediator component, and the consumer component, where the task execution instruction is used to instruct the producer component, the mediator component, and the consumer component to determine the task execution result of the target flow task according to the trusted data.

[0080] The method provided by the embodiments of the present invention can quickly and accurately determine the abnormal database according to the mapping relationship table, and after the user replaces the multiple data to be processed included in the abnormal database, instruct the data processing system to complete the execution of the target flow task according to the replaced trusted data, which can effectively improve the task execution efficiency and enhance the user's usage experience.

[0081] The above mainly introduces the solution of the embodiment of the present invention from the perspective of the method. It can be understood that in order for the detection device 100 to implement the above functions, it includes at least one of the corresponding hardware structures and software modules for executing each function. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the embodiments of the present invention can be implemented in the form of hardware or a combination of hardware and computer software. Whether a certain function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the embodiments of the present invention.

[0082] The embodiments of the present invention can divide the detection device 100 into functional units according to the above method examples. For example, the detection device 100 can be divided into respective functional units corresponding to each function, or two or more functions can be integrated into one processing unit. The above integrated units can be implemented in the form of hardware or in the form of software functional units. It should be noted that the division of units in the embodiments of the present invention is illustrative, only a logical function division, and there can be other division methods in actual implementation.

[0083] Exemplarily, Figure 7 FIG. shows a schematic hardware structure diagram of a detection device provided by an embodiment of the present invention. The detection device 100 includes: a mapping establishment module 110, configured to establish a mapping relationship table, where the mapping relationship table is used to represent the mapping relationship between each table data and the corresponding database, task queue, and data processing result; a quantity determination module 120, configured to determine a first quantity, a second quantity, a third quantity, a fourth quantity, a fifth quantity, and a sixth quantity corresponding to the target stream task according to the mapping relationship table; where the first quantity is the quantity of the to-be-processed data corresponding to the target stream task obtained by the producer component, the second quantity is the quantity of the table data sent by the producer component to the middleman component; the third quantity is the quantity of the table data received by the middleman component, the fourth quantity is the quantity of the table data added by the middleman component to the task queue; the fifth quantity is the quantity of the table data obtained by the consumer component from the task queue, and the sixth quantity is the quantity of the data processing result obtained by the consumer component processing the table data according to the category of each table data; a result generation module 130, configured to generate a fault detection result when the absolute difference between the first quantity and the second quantity is greater than or equal to a preset threshold, the absolute difference between the second quantity and the third quantity is greater than or equal to a preset threshold, the absolute difference between the third quantity and the fourth quantity is greater than or equal to a preset threshold, the absolute difference between the fourth quantity and the fifth quantity is greater than or equal to a preset threshold, or the absolute difference between the fifth quantity and the sixth quantity is greater than or equal to a preset threshold, where the fault detection result is used to prompt that the execution of the target stream task is abnormal.

[0084] Optionally, the detection device 100 further includes a fault cause determination module 140, configured to: when the absolute difference between the first quantity and the second quantity is greater than or equal to a preset threshold, determine that the fault cause of the fault detection result is an abnormality of the producer component; when the absolute difference between the second quantity and the third quantity is greater than or equal to the preset threshold or the absolute difference between the third quantity and the fourth quantity is greater than or equal to the preset threshold, determine that the fault cause of the fault detection result is an abnormality of the intermediary component; when the absolute difference between the fourth quantity and the fifth quantity is greater than or equal to the preset threshold or the absolute difference between the fifth quantity and the sixth quantity is greater than or equal to the preset threshold, determine that the fault cause of the fault detection result is an abnormality of the consumer component.

[0085] It should be understood that for the specific description of the above optional manner, reference may be made to the foregoing method embodiments, which will not be elaborated herein. In addition, for the explanation of any of the above detection devices 100 and the description of the beneficial effects, reference may be made to the corresponding method embodiments above, which will not be elaborated.

[0086] An embodiment of the present invention further provides a computer-readable storage medium, in which at least one computer instruction is stored, and the at least one computer instruction is loaded and executed by a processor to implement the methods of the above various embodiments. For the explanation of the relevant content and the description of the beneficial effects in any of the above computer-readable storage media, reference may be made to the corresponding embodiments above, which will not be elaborated herein.

[0087] An embodiment of the present invention further provides a chip. The chip integrates a control circuit for implementing the functions of the above detection device 100 and one or more ports. Optionally, the functions supported by the chip may refer to the above, which will not be elaborated herein.

[0088] Those of ordinary skill in the art can understand that all or part of the steps for implementing the above embodiments can be completed by a program instructing relevant hardware. The program can be stored in a computer-readable storage medium. The above-mentioned storage medium can be a read-only memory, a random access memory, etc. The above-mentioned processing unit or processor can be a central processing unit, a general-purpose processor, an application specific integrated circuit (ASIC), a digital signal processor (DSP), a field programmable gate array (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof.

[0089] Embodiments of the present invention also provide a computer program product containing instructions. When the instructions run on a computer, the computer is caused to execute any one of the methods in the above embodiments. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions according to the embodiments of the present invention are generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by wire (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wirelessly (such as infrared, wireless, microwave, etc.). The computer-readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server or a data center integrating one or more available media. The available medium may be a magnetic medium (such as a floppy disk, a hard disk, a magnetic tape), an optical medium (such as a DVD), or a semiconductor medium (such as an SSD), etc.

[0090] It should be noted that the devices for storing computer instructions or computer programs provided in the embodiments of the present invention, such as but not limited to, the above-mentioned memory, computer-readable storage medium, and communication chip, etc., are all non-transitory. Those skilled in the art should be able to realize that in the above one or more examples, the functions described in the embodiments of the present invention can be implemented by hardware, software, firmware, or any combination thereof. When implemented using software, these functions may be stored in a computer-readable storage medium or transmitted as one or more instructions or codes on a computer-readable storage medium. The computer-readable storage medium includes a computer storage medium and a communication medium, where the communication medium includes any medium facilitating the transmission of a computer program from one place to another. The storage medium may be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0091] Although the embodiments of the present invention have been shown and described above, it can be understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those of ordinary skill in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present invention.

Claims

1. A fault detection method for stream task processing, characterized in that, Applied to a data processing system, the data processing system includes a producer component, an intermediary component, and a consumer component; the producer component is used to obtain multiple pieces of to-be-processed data corresponding to a target flow task from multiple databases and generate table data corresponding to each piece of to-be-processed data among the multiple pieces of to-be-processed data; the intermediary component is used to obtain the table data corresponding to each piece of to-be-processed data and add the multiple pieces of table data into corresponding task queues according to the category of each piece of table data; The consumer component is used to obtain the table data from the corresponding task queue, process the table data according to the category of each piece of table data, and obtain the task execution result of the target flow task, where the task execution result includes the data processing result corresponding to each piece of table data; The method includes: Establish a mapping relation table, where the mapping relation table is used to represent the mapping relation between each piece of table data and the database, task queue, and data processing result corresponding to each piece of table data; According to the mapping relation table, determine the first quantity, second quantity, third quantity, fourth quantity, fifth quantity, and sixth quantity corresponding to the target flow task; where the first quantity is the quantity of to-be-processed data corresponding to the target flow task obtained by the producer component, the second quantity is the quantity of table data sent by the producer component to the intermediary component; the third quantity is the quantity of table data received by the intermediary component, the fourth quantity is the quantity of table data added by the intermediary component to the task queue; the fifth quantity is the quantity of table data obtained by the consumer component from the task queue, and the sixth quantity is the quantity of data processing results obtained by the consumer component by processing the table data according to the category of each piece of table data; When the absolute difference between the first quantity and the second quantity is greater than or equal to a preset threshold, the absolute difference between the second quantity and the third quantity is greater than or equal to a preset threshold, the absolute difference between the third quantity and the fourth quantity is greater than or equal to a preset threshold, the absolute difference between the fourth quantity and the fifth quantity is greater than or equal to a preset threshold, or the absolute difference between the fifth quantity and the sixth quantity is greater than or equal to a preset threshold, generate a fault detection result, where the fault detection result is used to prompt that the execution of the target flow task is abnormal.

2. The method according to claim 1, wherein The method further includes: When the absolute difference between the first quantity and the second quantity is greater than or equal to a preset threshold, determine that the cause of the fault in the fault detection result is an abnormality of the producer component; When the absolute difference between the second quantity and the third quantity is greater than or equal to a preset threshold or the absolute difference between the third quantity and the fourth quantity is greater than or equal to a preset threshold, determine that the cause of the fault in the fault detection result is an abnormality of the intermediary component; When the absolute difference between the fourth quantity and the fifth quantity is greater than or equal to a preset threshold or the absolute difference between the fifth quantity and the sixth quantity is greater than or equal to a preset threshold, determine that the cause of the fault in the fault detection result is an abnormality of the consumer component.

3. The method according to claim 2, wherein After the consumer component processes the table data according to the task execution process corresponding to the task category of the table data to obtain the task processing result corresponding to each piece of table data, the method further includes: Determine the status information of the task processing result of each table data included in the target flow task, where the status information includes correct or incorrect; Determine the accuracy rate of the target flow task according to the status information of the task processing result of each table data included in the target flow task, where the accuracy rate is the ratio of the number of task processing results with correct status information to the total number of task processing results; Generate a fault detection result when the accuracy rate is less than or equal to a preset threshold, where the fault detection result is used to prompt that the multiple pieces of to-be-processed data corresponding to the target flow task obtained from multiple databases are abnormal.

4. The method according to claim 3, characterized in that, Determine the status information of the task processing result of each table data included in the target flow task, including: Obtain the verification value of the task processing result of each table data included in the target flow task; When the actual value of the task processing result of each table data included in the target flow task is consistent with the verification value, determine that the status information of the task processing result of the table data is correct; When the actual value of the task processing result of each table data included in the target flow task is inconsistent with the verification value, determine that the status information of the task processing result of the table data is incorrect.

5. The method according to claim 4, wherein After generating the fault detection result, the method further includes: Send an abort instruction to the producer component, the middleman component, and the consumer component, where the abort instruction is used to instruct the producer component, the middleman component, and the consumer component to abort the execution of the target flow task.

6. The method according to claim 5, characterized in that, The method further includes: Based on the mapping relation table, determine the database corresponding to each data processing result with incorrect status information as an abnormal database; In response to the trusted data input by the user to replace the multiple pieces of to-be-processed data included in the abnormal database, send a task execution instruction to the producer component, the middleman component, and the consumer component, where the task execution instruction is used to instruct the producer component, the middleman component, and the consumer component to determine the task execution result of the target flow task according to the trusted data.

7. The method according to claim 6, characterized in that, Each database is configured with a corresponding identifier; Establishing the mapping relation table includes: When the producer component generates the table data corresponding to each piece of to-be-processed data in the multiple pieces of to-be-processed data, determine the identifier corresponding to each table data; Generate a first mapping relation according to the identifier of each table data and the identifier of the database from which the to-be-processed data corresponding to each table data comes; When the middleman component adds multiple table data to the corresponding task queue according to the category of each table data, determine the identifier corresponding to each task queue; Generate a second mapping relation according to the identifier of each table data and the identifier of the task queue corresponding to each table data; When the consumer component processes the table data according to the category of each table data to obtain the data processing result corresponding to each table data, determine the identifier corresponding to the data processing result of each table data; Generate a third mapping relation according to the identifier of each table data and the identifier corresponding to the data processing result of each table data; Establish a mapping relation table according to the first mapping relation, the second mapping relation, and the third mapping relation.

8. A fault detection device for stream task processing, characterized in that, Applied to a data processing system, the data processing system includes a producer component, an intermediary component, and a consumer component; the producer component is used to obtain multiple pieces of to-be-processed data corresponding to a target flow task from multiple databases, and generate table data corresponding to each piece of to-be-processed data among the multiple pieces of to-be-processed data; the intermediary component is used to obtain the table data corresponding to each piece of to-be-processed data, and add the multiple pieces of table data into corresponding task queues according to the category of each piece of table data; The consumer component is used to obtain the table data from the corresponding task queue, process the table data according to the category of each piece of table data, and obtain the task execution result of the target flow task, where the task execution result includes the data processing result corresponding to each piece of table data; The device includes: A mapping establishment module, configured to establish a mapping relation table, where the mapping relation table is used to represent the mapping relation between each piece of table data and the database, task queue, and data processing result corresponding to each piece of table data; A quantity determination module, configured to determine a first quantity, a second quantity, a third quantity, a fourth quantity, a fifth quantity, and a sixth quantity corresponding to the target flow task according to the mapping relation table; where the first quantity is the quantity of to-be-processed data corresponding to the target flow task obtained by the producer component, the second quantity is the quantity of table data sent by the producer component to the intermediary component; the third quantity is the quantity of table data received by the intermediary component, the fourth quantity is the quantity of table data added by the intermediary component to the task queue; the fifth quantity is the quantity of table data obtained by the consumer component from the task queue, and the sixth quantity is the quantity of data processing results obtained by the consumer component processing the table data according to the category of each piece of table data; A result generation module, configured to generate a fault detection result when the absolute difference between the first quantity and the second quantity is greater than or equal to a preset threshold, the absolute difference between the second quantity and the third quantity is greater than or equal to a preset threshold, the absolute difference between the third quantity and the fourth quantity is greater than or equal to a preset threshold, the absolute difference between the fourth quantity and the fifth quantity is greater than or equal to a preset threshold, or the absolute difference between the fifth quantity and the sixth quantity is greater than or equal to a preset threshold, where the fault detection result is used to prompt that the execution of the target flow task is abnormal.

9. The device according to claim 8, characterized in that, The device further includes a fault cause determination module, configured to: When the absolute difference between the first quantity and the second quantity is greater than or equal to a preset threshold, determine that the fault cause of the fault detection result is an abnormality of the producer component; When the absolute difference between the second quantity and the third quantity is greater than or equal to a preset threshold or the absolute difference between the third quantity and the fourth quantity is greater than or equal to a preset threshold, determine that the fault cause of the fault detection result is an abnormality of the intermediary component; When the absolute difference between the fourth quantity and the fifth quantity is greater than or equal to a preset threshold or the absolute difference between the fifth quantity and the sixth quantity is greater than or equal to a preset threshold, determine that the fault cause of the fault detection result is an abnormality of the consumer component.

10. An electronic device, characterized in that, Includes: A processor; A memory for storing executable instructions of the processor; Wherein, the processor is configured to execute the instructions to implement the fault detection method for stream task processing as described in any one of claims 1-7.

Images