Data processing method and device, electronic equipment and medium

Abstract

The invention provides a data processing method and device, electronic equipment and a medium, and can be applied to the technical field of big data and financial science and technology. The data processing method comprises the steps of determining a data partition used for executing a write operation in response to detection of the data write operation of a data storage platform; obtaining a message quantity difference before and after the write-in operation of the data partition through the first time linked list; determining a number loss detection result of the write-in operation according to the message number difference; and under the condition that the data loss detection result represents that data loss occurs in the write-in operation, determining a data loss reason according to an execution log of the write-in operation.

Description

Technical Field

[0001] The present disclosure relates to the technical field of big data and financial technology, and specifically designs a data processing method, device, electronic device, and medium. Background Art

[0002] With the continuous development of computer technology, enterprises provide more and more services to users, such as providing services to users through different functional modules, and providing services to multiple internal users through multiple office application software. The data of multiple application software and multiple functional modules can usually be stored in the enterprise's data lake with the help of an intermediate data storage platform.

[0003] However, in the process of processing data through the intermediate data storage platform, data loss, also known as data loss, may occur due to network failure, system failure, hardware failure, etc. If the data loss in this process cannot be discovered in time, it will affect the real-time performance of data processing, storage efficiency, and fault optimization efficiency. Summary of the invention

[0004] In view of the above problems, the present disclosure provides a data processing method, device, electronic device, and medium.

[0005] According to a first aspect of the present disclosure, a data processing method is provided, including: in response to detecting that a data write operation has occurred on a data storage platform, determining a data partition for performing the write operation; obtaining, through a first time linked list, a difference in the number of messages in the data partition before and after the write operation has occurred; determining a loss detection result of the write operation based on the difference in the number of messages; and in a case where the loss detection result indicates that a loss has occurred in the write operation, determining a cause of the loss based on an execution log of the write operation.

[0006] According to an embodiment of the present disclosure, a loss detection result of a write operation is determined based on the message quantity difference, including: when the message quantity difference is zero, determining that the loss detection result is all loss; when the message quantity difference is not zero, obtaining loss verification data; and determining the loss detection result based on the loss verification data.

[0007] According to an embodiment of the present disclosure, the loss number verification data includes the number of error messages and the number of ignored messages; based on the loss number verification data, the loss number detection result is determined, including: when it is determined that the sum of the number of error messages and the number of ignored messages is zero, determining that the loss number detection result is that no loss has occurred; when it is determined that the sum of the number of error messages and the number of ignored messages is not zero, determining that the loss number detection result is partial loss.

[0008] According to an embodiment of the present disclosure, obtaining lost data verification data includes: obtaining, through a second time chain table, the number of error messages of the data partition before and after the write operation occurs according to the timestamp information of the write operation occurring in the first time chain table; obtaining, through a third time chain table, the number of ignored messages of the data partition before and after the write operation occurs according to the timestamp information of the write operation occurring in the first time chain table; wherein the first time chain table, the second time chain table and the third time chain table share the same timestamp information; the first time chain table, the second time chain table and the third time chain table respectively display the message quantity difference, the number of error messages and the number of ignored messages through visual prompt information.

[0009] According to an embodiment of the present disclosure, a data storage platform includes multiple data storage sets, each data storage set includes multiple data partitions; when a loss detection result indicates that a loss occurs in a write operation, a cause of the loss is determined according to an execution log of the write operation, including: obtaining a message quantity difference for each of the multiple data partitions when the write operation occurs according to timestamp information of the write operation in a first time linked list; when the message quantity differences for each of the multiple data partitions are all zero, determining the cause of the loss according to the execution log of the write operation; when there is a non-zero message quantity difference among the message quantity differences for each of the multiple data partitions, obtaining partition registration information recorded in the data lake through a communication interface between the data storage platform and the data lake; and determining the cause of the loss according to the partition registration information.

[0010] According to an embodiment of the present disclosure, the cause of data loss is determined according to the partition registration information, including: determining the registered data partition registered in the data lake for the write operation according to the partition registration information; when the registered data partition and the data partition used to perform the write operation are the same, determining the cause of data loss according to the execution log of the write operation; when the registered data partition and the data partition used to perform the write operation are different, determining the cause of data loss to be an erroneous partition; and re-registering on the data storage platform based on the registered data partition.

[0011] According to an embodiment of the present disclosure, determining the cause of data loss according to an execution log of a write operation includes: determining an error keyword in the execution log; and determining the cause of data loss according to the error keyword.

[0012] According to an embodiment of the present disclosure, the cause of data loss is determined based on the error keyword, including: when the error keyword is a data source abnormality, obtaining the latest timestamp information of the data source; when the latest timestamp information of the data source is different from the timestamp information of the write operation in the first time linked list, determining that the cause of data loss is a time error.

[0013] A second aspect of the present disclosure provides a data processing device, characterized in that the device includes: a determination module, which is used to determine a data partition for performing a write operation in response to detecting that a data write operation has occurred on a data storage platform; an acquisition module, which is used to obtain the difference in the number of messages between the data partition before and after the write operation occurs through a first time linked list; a detection module, which is used to determine a loss detection result of the write operation based on the difference in the number of messages; and a processing module, which is used to determine the cause of the loss when the loss detection result indicates that a loss has occurred in the write operation.

[0014] A third aspect of the present disclosure provides an electronic device, comprising: one or more processors; a memory for storing one or more computer programs, and when the one or more processors execute the one or more computer programs, the steps of the above-mentioned data processing method are implemented.

[0015] The fourth aspect of the present disclosure further provides a computer-readable storage medium on which a computer program or instruction is stored. When the computer program or instruction is executed by a processor, the steps of the above-mentioned data processing method are implemented.

[0016] The fifth aspect of the present disclosure also provides a computer program product, including a computer program or instructions, which implement the steps of the above-mentioned data processing method when the computer program or instructions are executed by a processor.

[0017] The embodiments of the present disclosure determine the data partition where the write operation is performed, and obtain the difference in the number of messages before and after the write operation occurs in the dimension of the data partition, so that it can simply determine whether the current write operation has lost data based on the difference in the number of messages. Since the embodiments of the present disclosure only need to obtain the difference in the number of messages of the data partition of the current write operation, and no other data is required, the embodiments of the present disclosure can detect in real time and quickly whether the current write operation has lost data. In addition, in the event of data loss, the cause of the loss can be specifically determined through the execution log of the write operation, and fault maintenance can be performed based on the specific cause of the loss, thereby improving fault handling efficiency and storage efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The above contents and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0019] Figure 1 The application scenario of the data processing method according to the embodiment of the present disclosure is schematically shown;

[0020] Figure 2 A flowchart schematically shows a data processing method according to an embodiment of the present disclosure;

[0021] Figure 3A flowchart of determining a lost number detection result according to lost number verification data according to an embodiment of the present disclosure is schematically shown;

[0022] Figure 4 A schematic diagram of a time chain table according to a specific embodiment of the present disclosure is schematically shown;

[0023] Figure 5 A flowchart of determining a loss detection result according to a difference in the number of messages of each of a plurality of data partitions according to an embodiment of the present disclosure is schematically shown;

[0024] Figure 6 A structural block diagram schematically shows a data processing device according to an embodiment of the present disclosure; and

[0025] Figure 7 A block diagram of an electronic device suitable for a data processing method according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0026] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present disclosure. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0027] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0028] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification, and should not be interpreted in an idealized or overly rigid manner.

[0029] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0030] In the technical solution of the present disclosure, the user information (including but not limited to user personal information, user image information, user device information, such as location information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved are all information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with the relevant laws, regulations and standards of the relevant countries and regions, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0031] In the scenario of using personal information for automated decision-making, the methods, devices, and systems provided by the embodiments of the present disclosure provide users with corresponding operation portals for users to choose to agree or reject the automated decision-making results; if the user chooses to reject, the expert decision-making process will be entered. The expression "automated decision-making" here refers to the activity of automatically analyzing and evaluating a person's behavioral habits, interests and hobbies, or economic, health, credit status, etc. through computer programs, and making decisions. The expression "expert decision-making" here refers to the activity of making decisions by people who specialize in a certain field, have specialized experience, knowledge and skills, and have reached a certain level of professionalism.

[0032] In the related art, data storage platforms may experience data loss due to network failures, system failures, hardware failures, etc. Currently, when data loss occurs on a data storage platform, the data can only be rerun through the data storage platform. The specific data that has been lost cannot be determined in real time and in a targeted manner, nor can targeted remediation be performed.

[0033] Therefore, an embodiment of the present disclosure provides a data processing method: in response to detecting that a data write operation has occurred in a data storage platform, determining a data partition for performing the write operation; obtaining the difference in the number of messages in the data partition before and after the write operation occurs through a first time linked list; determining a loss detection result of the write operation based on the difference in the number of messages; and in a case where the loss detection result indicates that a loss has occurred in the write operation, determining the cause of the loss based on the execution log of the write operation.

[0034] Figure 1 The application scenario of the data processing method according to the embodiment of the present disclosure is schematically illustrated.

[0035] like Figure 1As shown, the application scenario 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 is used to provide a medium for a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.

[0036] The user may use at least one of the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications may be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only as examples).

[0037] The first terminal device 101, the second terminal device 102, and the third terminal device 103 may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0038] The server 105 may be a server that provides various services, such as a background management server (only as an example) that provides support for websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103. The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0039] In the embodiments of the present disclosure, the application software client installed in the first terminal device 101, the second terminal device 102, and the third terminal device 103 and the functional modules in the client can be used as the data source of the intermediate data storage platform. The server 105 may include an intermediate data storage platform for receiving data from the first terminal device 101, the second terminal device 102, and the third terminal device 103 through the network 104, processing data, and storing data in the data lake. The data lake can be set in the server 105, and can also be set in a server or server cluster that is different from the server 105 but can interact with the server 105.

[0040] It should be noted that the data processing method provided in the embodiment of the present disclosure can generally be executed by the server 105. Accordingly, the data processing device provided in the embodiment of the present disclosure can generally be set in the server 105. The data processing method provided in the embodiment of the present disclosure can also be executed by a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105. Accordingly, the data processing device provided in the embodiment of the present disclosure can also be set in a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105.

[0041] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0042] The following will be based on Figure 1 The scene described by Figure 2 to Figure 5 The data processing method of the disclosed embodiment is described in detail.

[0043] Figure 2 The flowchart of the data processing method according to the embodiment of the present disclosure is schematically shown.

[0044] like Figure 2 As shown, the method 200 includes operations S210 to S240.

[0045] In operation S210 , in response to detecting that a data write operation occurs on a data storage platform, a data partition for performing the write operation is determined.

[0046] In operation S220, the difference in the number of messages before and after the write operation occurs in the data partition is obtained through the first time linked table.

[0047] In operation S230, a loss detection result of the write operation is determined according to the message quantity difference.

[0048] In operation S240, when the data loss detection result indicates that data loss occurs in the write operation, a cause of the data loss is determined according to the execution log of the write operation.

[0049] According to an embodiment of the present disclosure, the data storage platform can serve as an intermediate data storage platform between producers and consumers, distributing the data written by producers in the data storage platform to corresponding consumers. For example, the data storage platform can be a distributed stream platform, such as Kafka.

[0050] The data storage platform can run as a cluster on one or more servers, or it can span multiple data centers, with each data center used to connect to at least one producer to implement message publishing.

[0051] In an embodiment of the present disclosure, when a producer publishes a message, the data storage platform will perform a data write operation accordingly. In a business system, the message of each thread can be written to a corresponding data partition in the data storage platform. The data partition for performing the write operation can be determined based on the producer that publishes the message.

[0052] According to an embodiment of the present disclosure, the data storage platform may include multiple data storage sets, such as Topics, each of which may include at least one data partition, such as a partition. Each message sent to the data storage platform has a data storage set.

[0053] For example, in Kafka, the data storage collection can be a Topic. Messages of different Topics are stored separately. Each Topic can have multiple producers sending messages to it, and multiple consumers consuming the messages. The messages are the data written or read by the data storage platform. Each Topic can be divided into multiple data partitions. The messages contained in different data partitions under the same Topic are different. When each message is added to a partition, it will be assigned a unique offset to ensure the order of the message in the data partition. The order of offsets does not cross partitions. Kafka only guarantees that the messages in the same data partition are in order.

[0054] According to an embodiment of the present disclosure, the first time linked list is a long linked list with time information, which is used to record the difference in the number of messages before and after a write operation occurs on the data storage platform.

[0055] In a specific embodiment, each data storage set corresponds to a first time linked list, that is, the first time linked list corresponding to a Topic includes the difference in the number of messages of multiple data partitions in the Topic before and after a certain write operation. Alternatively, the entire data storage platform corresponds to a unique first time linked list.

[0056] In the above two embodiments, the first time linked list arranges multiple Topics and multiple data partitions according to preset rules. Every time a write operation occurs, the difference in the number of messages in multiple data partitions in the first time linked list will change, that is, the entire first time linked list uniquely corresponds to one timestamp.

[0057] For example, in the case where each data storage set corresponds to a first time linked list, obtaining the difference in the number of messages between the data partition before and after the write operation occurs through the first time linked list includes: determining the data storage set Topic on which the write operation is performed and the first time linked list corresponding to the data storage set based on a first preset arrangement rule of the Topic; determining, in the first time linked list, the position information of the data partition on which the write operation is performed according to a second preset arrangement rule of the data storage set Topic; and taking out the difference in the number of messages at the position information, which is the difference in the number of messages before and after the write operation occurs.

[0058] Alternatively, in the case where the entire data storage platform corresponds to a unique first time linked list, obtaining the difference in the number of messages between the data partition before and after the write operation occurs through the first time linked list includes: determining the position information of the data partition where the write operation is performed in the first time linked list based on a third preset arrangement rule of the data storage platform; and taking out the difference in the number of messages at the position information, which is the difference in the number of messages before and after the write operation occurs.

[0059] Alternatively, each data partition may also uniquely correspond to a first time linked table, in which case the first time linked table may include a change in the difference in the number of messages in the data partition within a continuous time period. For example, the table name of the first time linked table may be topic-partition1, so that the first time linked table corresponding to the data partition can be found by the table name of the time linked table.

[0060] In the case where each data partition can also uniquely correspond to a first time linked list, obtaining the difference in the number of messages of the data partition before and after the write operation occurs through the first time linked list includes: determining the first time linked list corresponding to the data partition according to the partition identifier of the data partition used to perform the write operation; and using the difference in the number of messages at the latest moment in the first time linked list as the difference in the number of messages before and after the write operation is performed.

[0061] According to the embodiments of the present disclosure, since the write operation is used to write messages to the data storage platform, the difference in the number of messages before and after the write operation can be used to determine whether the write operation is successful, that is, whether data loss occurs. For example, the data loss detection result can be used to characterize whether data loss occurs in the write operation.

[0062] The loss detection results include the results that indicate that data loss occurred in the write operation, and the results that indicate that data loss did not occur in the write operation. When data loss did not occur in the write operation, the data storage platform can process data normally, and the message party can also obtain data from the data storage platform normally. When data loss occurs in the write operation, in order to ensure that the lost data can be promptly processed while the data loss is detected in real time, it is necessary to determine the cause of the data loss based on the data loss phenomenon.

[0063] According to the embodiments of the present disclosure, since the data writing process can be recorded in the execution log, when the data loss detection result indicates that data loss occurs in the writing operation, the data loss cause can be determined according to the execution log of the writing operation. Afterwards, the embodiments of the present disclosure can also execute a specific data loss processing strategy according to the specific data loss cause.

[0064] The embodiments of the present disclosure determine the data partition where the write operation is performed, and obtain the difference in the number of messages before and after the write operation occurs in the dimension of the data partition, so that it can simply determine whether the current write operation has lost data based on the difference in the number of messages. Since the embodiments of the present disclosure only need to obtain the difference in the number of messages of the data partition of the current write operation, and no other data is required, the embodiments of the present disclosure can detect in real time and quickly whether the current write operation has lost data. In addition, in the event of data loss, the cause of the loss can be specifically determined through the execution log of the write operation, and fault maintenance can be performed based on the specific cause of the loss, thereby improving fault handling efficiency and storage efficiency.

[0065] According to an embodiment of the present disclosure, a loss detection result of a write operation is determined based on the message quantity difference, including: when the message quantity difference is zero, determining that the loss detection result is all loss; when the message quantity difference is not zero, obtaining loss verification data; and determining the loss detection result based on the loss verification data.

[0066] According to an embodiment of the present disclosure, since the number of messages to be written into the data partition in each write operation cannot be zero, if the difference in the number of messages before and after the write operation is zero, it indicates that the message is not written into its corresponding data partition, and the loss detection result at this time is that all messages are lost.

[0067] The difference in the number of messages before and after the write operation is not zero only indicates that the number of messages written to the data partition by the write operation is not zero, and it cannot be determined that the write operation has not lost any messages. This is because when not all messages are lost but some messages are lost, the difference in the number of messages before and after the write operation is not zero, but the current situation is actually a fault.

[0068] When the message quantity difference is not zero, loss verification data can be obtained, and the loss verification data can be used to distinguish between partial loss and no loss, so as to determine the loss detection result in a finer granularity and more accurately.

[0069] According to an embodiment of the present disclosure, the loss verification data includes the number of error messages and the number of ignored messages.

[0070] The loss detection result is determined based on the loss verification data, including: when it is determined that the sum of the number of error messages and the number of ignored messages is zero, the loss detection result is determined to be no loss; when it is determined that the sum of the number of error messages and the number of ignored messages is not zero, the loss detection result is determined to be partial loss.

[0071] According to an embodiment of the present disclosure, the data storage platform may record the number of messages of the write operation, the number of ignored messages, the number of error messages, etc. The loss verification data includes the number of ignored messages and the number of error messages.

[0072] For example, the first database table includes the number of consumptions, the number of ignored messages, and the number of error messages; the second database table is a configuration table for configuring table information and table names that need to be counted; the third database table stores count records, and stores the verified tables and the verified number of consumptions, number of ignored messages, number of error messages, etc. The number of consumptions in the first database table + the number of ignored messages + the number of error messages = the total number of messages. The first database table, the second database table, and the third database table can be called table A, table B, and table C, respectively.

[0073] Therefore, it is possible to determine whether some data is lost based on whether the sum of the number of error messages and the number of ignored messages obtained is zero.

[0074] In the embodiment of the present disclosure, the sum of the number of error messages and the number of ignored messages is zero, indicating that there is a verification operation, that is, a message writing error or a message ignoring, and the part of the message is lost. The sum of the number of error messages and the number of ignored messages is not zero, indicating that there is no verification operation, that is, there is no message writing error and no message ignoring; therefore, it can be determined that the loss detection result corresponding to the difference in the number of messages is not zero is that no loss has occurred.

[0075] Figure 3 The flowchart of determining the lost number detection result according to the lost number verification data according to an embodiment of the present disclosure is schematically shown.

[0076] like Figure 3 As shown, embodiment 300 includes operations S301 to S306, schematically illustrating a process of determining a lost number detection result according to lost number verification data.

[0077] In operation S301, the difference in the number of messages before and after a write operation occurs in a data partition is obtained.

[0078] In operation S302, it is determined whether the message quantity difference is zero. If the message quantity difference is zero, the process proceeds to operation S307 to determine that the loss detection result is all loss; if the message quantity difference is not zero, the process proceeds to operation S303.

[0079] In operation S303, the number of error messages and the number of ignored messages are obtained.

[0080] In operation S304, the sum of the number of error messages and the number of ignored messages is zero. If the sum of the number of error messages and the number of ignored messages is zero, proceed to operation S306 to determine that the loss detection result is no loss; if the sum of the number of error messages and the number of ignored messages is not zero, proceed to operation S305.

[0081] In operation S305, it is determined that the number loss detection result is a partial number loss.

[0082] In operation S306, it is determined that the result of the data loss detection is that no data loss occurs.

[0083] In operation S307, it is determined that the result of the lost number detection is all lost numbers.

[0084] The embodiment of the present disclosure obtains the number of error messages and ignored messages recorded by the data storage platform, determines whether a verification operation has occurred based on the data verification record, and further distinguishes the situation where the difference in the number of messages is not zero into partial loss and total loss. The operation is simple and the accuracy of the loss detection results is improved.

[0085] In one embodiment, the number of error messages and the number of ignored messages can be obtained through the visualized second time chain table and the third time chain table.

[0086] According to an embodiment of the present disclosure, obtaining lost data verification data includes: obtaining, through a second time chain table, the number of error messages of the data partition before and after the write operation occurs according to the timestamp information of the write operation occurring in the first time chain table; obtaining, through a third time chain table, the number of ignored messages of the data partition before and after the write operation occurs according to the timestamp information of the write operation occurring in the first time chain table; wherein the first time chain table, the second time chain table and the third time chain table share the same timestamp information; the first time chain table, the second time chain table and the third time chain table respectively display the message quantity difference, the number of error messages and the number of ignored messages through visual prompt information.

[0087] According to an embodiment of the present disclosure, the number of error messages and the number of ignored messages in the second time chain table and the third time chain table can be obtained from the third database table; then, the second time chain table and the third time chain table are obtained by performing a visualization operation on the obtained number of error messages and the number of ignored messages. In this embodiment, the second time chain table and the third time chain table can facilitate the user to intuitively determine whether there are error messages and ignored messages.

[0088] According to the embodiments of the present disclosure, the second time linked list and the third time linked list are similar in form to the first time linked list. The entire database storage platform may correspond to one second time linked list and one third time linked list; each Topic may correspond to one second time linked list and one third time linked list; each data partition may correspond to one second time linked list and one third time linked list.

[0089] It should be noted that the timestamp information of the second time chain table, the third time chain table and the first time chain table is the same, that is, when the difference in the number of messages in the first time chain table changes, the data in the second time chain table and the third time chain table will also change.

[0090] In addition, visual prompt information may be provided in the visual first time chain list, the second time chain list and the third time chain list, respectively used to point to the message quantity difference, the number of error messages and the number of ignored messages of the current write operation.

[0091] Optionally, the difference in the number of messages, the number of error messages, and the number of ignored messages can be visualized in red and green to indicate an increase in number and a constant number, respectively. In addition, for the current data partition where data is lost, the difference in the number of messages can be visualized through arrows and other indicators, so that the user can intuitively see whether data is lost.

[0092] Figure 4 A schematic diagram of a time chain table according to a specific embodiment of the present disclosure is schematically shown.

[0093] like Figure 4 As shown, the first time linked table 401, the second time linked table 402 and the third time linked table 403 in the embodiment 400 are all time linked tables corresponding to the data partitions. At each moment, the value in the time linked table is the difference in the number of messages before and after the write operation.

[0094] When the current write operation occurs at time t1, the message quantity difference at time t1 is 10, and the number of error messages and the number of ignored messages are both 0. Therefore, the loss detection result of the write operation at time t1 is that no loss has occurred. At this time, the current time and the absence of loss at the current time can be prompted by a green visual prompt arrow.

[0095] When the current write operation occurs at time t2, the difference in the number of messages at time t2 is 7, the number of error messages is 2, and the number of ignored messages is 1. At this time, the difference in the number of messages at time t2 is not zero, and the sum of the number of error messages and the number of ignored messages needs to be calculated, that is, 3. Because the difference in the number of messages at time t2 is not zero, and the sum of the number of error messages and the number of ignored messages is not zero, the loss detection result of the write operation at time t2 is partial loss. At this time, the current time and the existence of partial loss at the current time can be prompted by a red visual prompt arrow.

[0096] According to an embodiment of the present disclosure, the lengths of the first time chain table, the second time chain table, and the third time chain table are equal and fixed. As time is continuously updated, the data at the previous moment in the first time chain table, the second time chain table, and the third time chain table will be cleared or transferred to other storage tables to avoid affecting the visualization effect.

[0097] The embodiments of the present disclosure use the visualized first time list, second time list and third time list, as well as the prompt information in the time list. The user only needs to call up the time list to intuitively view whether the current write operation is lost. The loss detection is simple, convenient and intuitive, and the user experience is better.

[0098] In another embodiment, the number of error messages and the number of ignored messages can also be directly obtained from the third database table. For example, if there is a core number record in Table C where the number of ignored messages and the number of error messages are not 0, there is partial loss. If there is partial loss, the cause of the loss is analyzed through logs. If there is no core number record where the number of ignored messages and the number of error messages are not 0, there is no partial loss.

[0099] According to an embodiment of the present disclosure, a data storage platform includes multiple data storage sets, each data storage set includes multiple data partitions; when a loss detection result indicates that a loss occurs in a write operation, a cause of the loss is determined according to an execution log of the write operation, including: obtaining a message quantity difference for each of the multiple data partitions when the write operation occurs according to timestamp information of the write operation in a first time linked list; when the message quantity differences for each of the multiple data partitions are all zero, determining the cause of the loss according to the execution log of the write operation; when there is a non-zero message quantity difference among the message quantity differences for each of the multiple data partitions, obtaining partition registration information recorded in the data lake through a communication interface between the data storage platform and the data lake; and determining the cause of the loss according to the partition registration information.

[0100] According to an embodiment of the present disclosure, for a scenario where the result of the data loss detection is total data loss or partial data loss, the cause of the data loss can be determined based on the execution log of the write operation.

[0101] Considering that the data loss may be caused by the difference between the registered data partition and the actually written data partition, and the data loss is all data loss, therefore, when the data loss detection result is determined to be all data loss, first determine whether the cause of the data loss is an error partition, and then, if the cause of the data loss is not an error partition, determine other reasons for the data loss based on the execution log of the write operation.

[0102] According to an embodiment of the present disclosure, according to the timestamp information of the write operation in the first time linked table, the difference in the number of messages of each of the multiple data partitions at the time corresponding to the timestamp information can be obtained. The multiple data partitions can be other data partitions except the data partition to be written. For the convenience of distinction, the data partition for performing the write operation can be the first data partition, and the multiple data partitions are the second data partitions.

[0103] The message quantity differences of the plurality of second data partitions are all zero, indicating that the plurality of second data partitions have no messages written at the current moment, and the error partition can be ruled out. Thus, other reasons for data loss can be determined based on the execution log of the write operation.

[0104] However, the data storage platform is usually processed in parallel, so when multiple second data partitions are working normally, the difference in the number of messages between them may not be zero. Therefore, when the difference in the number of messages between multiple second data partitions is not zero, it is possible to distinguish between normal operation and loss due to registration errors. Specifically, the partition registration information recorded in the data lake can be obtained through the communication interface between the data storage platform and the data lake; and the cause of the loss can be determined based on the partition registration information.

[0105] According to the embodiments of the present disclosure, the data lake side generally provides partition registration information such as tables, fields, topics, etc. The registration process is generally: use the verification service to verify the upstream writing process, service, hive table and permissions, and then register after verification.

[0106] However, the verification process may result in an erroneous partition due to human factors. Therefore, the embodiments of the present disclosure can determine whether the cause of the data loss is an erroneous partition based on the partition registration information.

[0107] The embodiments of the present disclosure can obtain partition registration information before determining the cause of data loss based on the execution log, and determine in a finer granularity whether the cause of data loss is an erroneous partition based on the partition registration information, thereby reducing the steps of scanning the execution log and analyzing the execution log as much as possible, reducing resource consumption and improving the efficiency of locating the cause of data loss.

[0108] According to an embodiment of the present disclosure, the cause of data loss is determined according to the partition registration information, including: determining the registered data partition registered in the data lake for the write operation according to the partition registration information; when the registered data partition and the data partition used to perform the write operation are the same, determining the cause of data loss according to the execution log of the write operation; when the registered data partition and the data partition used to perform the write operation are different, determining the cause of data loss to be an erroneous partition; and re-registering on the data storage platform based on the registered data partition.

[0109] According to an embodiment of the present disclosure, if the registration data partition and the data partition used to perform the write operation are the same, it indicates that the registration information is consistent with the actual execution information, and the reason for the loss is not an erroneous partition. Therefore, the reason for the loss can be determined based on the execution log of the write operation. If the registration data partition and the data partition used to perform the write operation are different, it indicates that the registration information is inconsistent with the actual execution information, and the reason for the loss is an erroneous partition. Therefore, the reason for the loss can be directly output as an erroneous partition without obtaining the execution log or performing keyword detection on the execution log.

[0110] Figure 5 A flowchart of determining a loss detection result according to a difference in the number of messages of each of a plurality of data partitions according to an embodiment of the present disclosure is schematically shown;

[0111] like Figure 5 As shown, embodiment 500 includes operations S501 to S508, schematically illustrating a process of determining a loss detection result according to a difference in the number of messages of each of a plurality of data partitions.

[0112] In operation S501, timestamp information of a write operation occurring in a first time linked list is obtained.

[0113] In operation S502, the message quantity difference of each of the plurality of data partitions when the write operation occurs is obtained according to the timestamp information.

[0114] In operation S503, the message quantity differences of the plurality of data partitions are all zero. If the message quantity differences of the plurality of data partitions are all zero, the process proceeds to operation S507 to determine that the loss detection result is all loss; if at least one of the message quantity differences of the plurality of data partitions is not zero, the process proceeds to operation S504.

[0115] In operation S504, partition registration information is acquired.

[0116] In operation S505 , the registered data partition for the write operation in the data lake is determined according to the partition registration information.

[0117] In operation S506, the data partition for registering and the data partition for performing the write operation are the same. If the data partition for registering and the data partition for performing the write operation are the same, proceed to operation S507; if the data partition for registering and the data partition for performing the write operation are different, proceed to operation S508.

[0118] In operation S507, the reason for data loss is determined according to the execution log of the write operation.

[0119] In operation S508, it is determined that the cause of data loss is an erroneous partition.

[0120] According to the embodiments of the present disclosure, whether the cause of data loss is an erroneous partition can be determined in a finer granularity based on the partition registration information, thereby reducing the steps of scanning the execution log and analyzing the execution log as much as possible, reducing resource consumption, and improving the efficiency of locating the cause of data loss.

[0121] According to an embodiment of the present disclosure, determining the cause of data loss according to an execution log of a write operation includes: determining an error keyword in the execution log; and determining the cause of data loss according to the error keyword.

[0122] According to the embodiment of the present disclosure, the error keywords for different reasons of data loss are different. For example, the error keywords include data source abnormality, configuration mode, consumption service, disk failure and other words.

[0123] According to an embodiment of the present disclosure, the execution log is in text format, and therefore, the error keyword can be obtained from the execution log by keyword matching.

[0124] According to an embodiment of the present disclosure, after determining the cause of data loss, different fault recovery operations may be performed for different causes of data loss.

[0125] For example, the configuration method may refer to table startup method configuration errors that lead to table data loss. There are three startup methods: normal startup method, earliest startup method, and latest startup method. Normal startup can ensure that no data is lost. The earliest startup means starting from the commit point of the last consumption, and the latest means starting from the end of the data. The error keyword is the configuration method. It may be due to the earliest startup method or the latest startup method that caused the data loss. In this case, reconfigure the correct startup method and restart consumption.

[0126] For another example, the error keyword is consumption service, and the reason for data loss is the consumer service problem. Taking Kafka as an example, Kafka's message publishing and consumption is an asynchronous operation. The consumer may lag behind the message queue's message publishing rate, resulting in message squeeze and data loss. Abnormal processing of messages by consumers, death, or restart may also cause message loss, that is, data loss. For the reason for data loss, you can confirm whether the consumption service is normal and re-consume when the consumption service is normal.

[0127] For another example, the error keywords are hardware anomalies such as network, cpu, disk, and memory. Since the nodes in the cluster of the data storage platform need to be interconnected through the network, when the network fails, the transmission speed is too slow, the link is disconnected, etc., it will cause message loss. Disk damage may also cause consumption loss, or even no data can be consumed. For example, hardware failure between nodes in the Kafka cluster. To determine the cause of the data loss, you can check the hardware facilities and data backup.

[0128] According to an embodiment of the present disclosure, the cause of data loss is determined based on the error keyword, including: when the error keyword is a data source abnormality, obtaining the latest timestamp information of the data source; when the latest timestamp information of the data source is different from the timestamp information of the write operation in the first time linked list, determining that the cause of data loss is a time error.

[0129] For example, the error keyword for data source exception can be "dbtime", where "db" is the data source and "time" is the time exception. When the error keyword is data source exception, you can scan the result log of the data source exception to obtain the latest timestamp information of the data source. Afterwards, based on the latest timestamp information and the timestamp information of the write operation in the first time linked list, confirm whether the latest partition message is stagnant and causes data loss due to the failure to update the time field. For the cause of the data loss, you can check whether the data source is abnormal. If some machine services of the data source are stuck, causing some operations entering the lake to be stuck, repair the drp data source service.

[0130] Figure 6 The structure block diagram of a data processing device according to an embodiment of the present disclosure is schematically shown.

[0131] like Figure 6 As shown, the data processing device 600 of this embodiment includes a determination module 610 , an acquisition module 620 , a detection module 630 , and a processing module 640 .

[0132] The determination module 610 is used to obtain the first identification information of the target code, and the first identification information includes the identification information of the stubbing position in the target code. In one embodiment, the determination module 610 can be used to perform the operation S210 described above, which will not be repeated here.

[0133] The acquisition module 620 is used to determine the first execution information corresponding to the test operation in response to the test operation within the preset time period, and the first execution information includes the execution record of the code corresponding to the stubbing position in the target code. In one embodiment, the acquisition module 620 can be used to perform the operation S220 described above, which will not be repeated here.

[0134] The detection module 630 is used to obtain the historical first execution information within a preset period of time. In one embodiment, the detection module 630 can be used to perform the operation S230 described above, which will not be described in detail here.

[0135] The processing module 640 is used to determine the second execution information according to the first execution information and the historical first execution information, wherein the second execution information includes the number of executions of the code corresponding to the stub position in the target code within a preset time period. In one embodiment, the processing module 640 can be used to perform the operation S240 described above, which will not be described in detail here.

[0136] According to an embodiment of the present disclosure, the detection module 630 includes a first detection submodule, a first acquisition submodule and a second detection submodule.

[0137] The first detection submodule is used to determine that the loss detection result is all loss when the message quantity difference is zero.

[0138] The first acquisition submodule is used to obtain the lost number verification data when the message quantity difference is not zero.

[0139] The second detection submodule is used to determine the loss number detection result according to the loss number verification data.

[0140] According to an embodiment of the present disclosure, the loss verification data includes the number of error messages and the number of ignored messages. The second detection submodule includes a first detection unit and a second detection unit.

[0141] The first detection unit is used to determine that the result of the data loss detection is that no data loss has occurred when it is determined that the sum of the number of error messages and the number of ignored messages is zero.

[0142] The second detection unit is used to determine that the loss number detection result is partial loss number when it is determined that the sum of the number of error messages and the number of ignored messages is not zero.

[0143] According to an embodiment of the present disclosure, the first acquisition submodule includes a first acquisition unit and a second acquisition unit.

[0144] The first acquisition unit is used to acquire the number of error messages of the data partition before and after the write operation occurs according to the timestamp information of the write operation occurring in the first time chain table through the second time chain table.

[0145] The second acquisition unit is used to acquire the number of ignored messages of the data partition before and after the write operation occurs according to the timestamp information of the write operation occurring in the first time chain table through the third time chain table.

[0146] Among them, the first time list, the second time list and the third time list share the same timestamp information; the first time list, the second time list and the third time list respectively display the message quantity difference, the number of error messages and the number of ignored messages through visual prompt information.

[0147] According to an embodiment of the present disclosure, the data storage platform includes multiple data storage sets, each of which includes multiple data partitions. The processing module 640 includes a second acquisition submodule, a first cause determination submodule, a third acquisition submodule and a second cause determination submodule.

[0148] The second acquisition submodule is used to acquire the message quantity difference of each of the multiple data partitions when the write operation occurs according to the timestamp information of the write operation in the first time linked list.

[0149] The first reason determination submodule is used to determine the reason for data loss according to the execution log of the write operation when the differences in the number of messages in the multiple data partitions are all zero.

[0150] The third acquisition submodule is used to obtain the partition registration information recorded in the data lake through the communication interface between the data storage platform and the data lake when there is a non-zero message quantity difference among the message quantity differences of the multiple data partitions.

[0151] The second reason determination submodule is used to determine the reason for data loss according to the partition registration information.

[0152] According to an embodiment of the present disclosure, the second cause determination submodule includes: a first determination unit, a second determination unit, a third determination unit and a registration unit.

[0153] The first determining unit is used to determine, according to the partition registration information, a registered data partition for a write operation that has been registered in the data lake.

[0154] The second determining unit is used to determine the reason for data loss according to the execution log of the write operation when the registration data partition and the data partition used to perform the write operation are the same.

[0155] The third determining unit is used to determine that the cause of data loss is an erroneous partition when the registration data partition and the data partition used to perform the write operation are different.

[0156] The registration unit is used to re-register on the data storage platform based on the registration data partition.

[0157] According to an embodiment of the present disclosure, the processing module 640 further includes a keyword determination submodule and a third cause determination submodule.

[0158] The keyword determination submodule is used to determine the error keywords in the execution log.

[0159] The third reason determination submodule is used to determine the reason for the data loss based on the error keyword.

[0160] According to an embodiment of the present disclosure, the third cause determination submodule includes a third acquisition unit and a fourth determination unit.

[0161] The third acquisition unit is used to acquire the latest timestamp information of the data source when the error keyword is that the data source is abnormal.

[0162] The fourth determining unit is used to determine that the reason for data loss is a time error when the latest timestamp information of the data source is different from the timestamp information of the write operation in the first time linked list.

[0163] According to an embodiment of the present disclosure, any multiple modules among the determination module 610, the acquisition module 620, the detection module 630, and the processing module 640 may be combined into one module for implementation, or any one of the modules may be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules may be combined with at least part of the functions of other modules and implemented in one module.

[0164] According to an embodiment of the present disclosure, at least one of the determination module 610, the acquisition module 620, the detection module 630, and the processing module 640 may be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application-specific integrated circuit (ASIC), or may be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or by any one of the three implementation methods of software, hardware, and firmware, or by an appropriate combination of any of them. Alternatively, at least one of the determination module 610, the acquisition module 620, the detection module 630, and the processing module 640 may be at least partially implemented as a computer program module, and when the computer program module is run, the corresponding function may be executed.

[0165] Figure 7 A block diagram of an electronic device suitable for a data processing method according to an embodiment of the present disclosure is schematically shown.

[0166] like Figure 7 As shown, the electronic device 700 according to an embodiment of the present disclosure includes a processor 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage part 708 into a random access memory (RAM) 703. The processor 701 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 701 may also include an onboard memory for caching purposes. The processor 701 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0167] In RAM 703, various programs and data required for the operation of electronic device 700 are stored. Processor 701, ROM 702 and RAM 703 are connected to each other via bus 704. Processor 701 performs various operations of the method flow according to the embodiment of the present disclosure by executing the program in ROM 702 and / or RAM 703. It should be noted that the program can also be stored in one or more memories other than ROM 702 and RAM 703. Processor 701 can also perform various operations of the method flow according to the embodiment of the present disclosure by executing the program stored in the one or more memories.

[0168] According to an embodiment of the present disclosure, the electronic device 700 may further include an input / output (I / O) interface 705, which is also connected to the bus 704. The electronic device 700 may further include one or more of the following components connected to the input / output I / O interface 705: an input portion 706 including a keyboard, a mouse, etc.; an output portion 707 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 708 including a hard disk, etc.; and a communication portion 709 including a network interface card such as a LAN card, a modem, etc. The communication portion 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. A removable medium 711, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 710 as needed, so that a computer program read therefrom is installed into the storage portion 708 as needed.

[0169] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present disclosure is implemented.

[0170] According to an embodiment of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus or a device. For example, according to an embodiment of the present disclosure, a computer-readable storage medium may include the ROM 702 and / or RAM 703 described above and / or one or more memories other than ROM 702 and RAM 703.

[0171] The embodiment of the present disclosure also includes a computer program product, which includes a computer program, and the computer program contains program code for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to enable the computer system to implement the above method provided by the embodiment of the present disclosure.

[0172] The above functions defined in the system / device of the embodiment of the present disclosure are performed when the computer program is executed by the processor 701. According to the embodiment of the present disclosure, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0173] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 709, and / or installed from the removable medium 711. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0174] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 709, and / or installed from the removable medium 711. When the computer program is executed by the processor 701, the above functions defined in the system of the embodiment of the present disclosure are performed. According to the embodiment of the present disclosure, the system, device, means, module, unit, etc. described above can be implemented by a computer program module.

[0175] According to an embodiment of the present disclosure, the program code for executing the computer program provided by the embodiment of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect through the Internet).

[0176] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0177] It will be appreciated by those skilled in the art that the features described in the various embodiments and / or claims of the present disclosure may be combined and / or combined in a variety of ways, even if such combinations and / or combinations are not explicitly described in the present disclosure. In particular, the features described in the various embodiments and / or claims of the present disclosure may be combined and / or combined in a variety of ways without departing from the spirit and teachings of the present disclosure. All of these combinations and / or combinations fall within the scope of the present disclosure.

[0178] The specific embodiments described above further illustrate the purpose, technical solutions and beneficial effects of the present disclosure. It should be understood that the above description is only a specific embodiment of the present disclosure and is not intended to limit the present disclosure. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present disclosure should be included in the protection scope of the present disclosure.

Claims

1. A data processing method, characterized in that: The method comprises: In response to detecting that a data write operation occurs on the data storage platform, determining a data partition for performing the write operation; Obtaining, through the first time linked table, a difference in the number of messages of the data partition before and after the write operation occurs; Determining a loss detection result of the write operation according to the message quantity difference; and When the data loss detection result indicates that data loss occurs in the write operation, the cause of the data loss is determined according to the execution log of the write operation.

2. The method according to claim 1, characterized in that The step of determining the loss detection result of the write operation according to the message quantity difference includes: When the message quantity difference is zero, determining that the loss number detection result is all loss numbers; When the message quantity difference is not zero, obtaining lost number verification data; and The lost number detection result is determined according to the lost number verification data.

3. The method according to claim 2, characterized in that The lost number verification data includes the number of error messages and the number of ignored messages; and determining the lost number detection result according to the lost number verification data includes: In the case where it is determined that the sum of the number of error messages and the number of ignored messages is zero, determining that the number loss detection result is that no number loss has occurred; When it is determined that the sum of the number of error messages and the number of ignored messages is not zero, it is determined that the number loss detection result is partial number loss.

4. The method according to claim 2 or 3, characterized in that: The obtaining of lost number verification data comprises: Obtain the number of error messages of the data partition before and after the write operation occurs, using the second time chain table and according to the timestamp information of the write operation occurring in the first time chain table; Obtain the number of ignored messages of the data partition before and after the write operation occurs, according to the timestamp information of the write operation occurring in the first time chained table, through the third time chained table; Among them, the first time list, the second time list and the third time list share the same timestamp information; the first time list, the second time list and the third time list respectively display the message quantity difference, the number of error messages and the number of ignored messages through visual prompt information.

5. The method according to claim 1, characterized in that The data storage platform includes a plurality of data storage sets, each of which includes a plurality of data partitions; when the data loss detection result indicates that data loss occurs in the write operation, determining the data loss cause according to the execution log of the write operation includes: According to the timestamp information of the write operation in the first time chain table, the message quantity difference of each of the multiple data partitions when the write operation occurs is obtained; When the message quantity differences of the plurality of data partitions are all zero, determining the reason for the data loss according to the execution log of the write operation; When there is a non-zero message quantity difference among the message quantity differences of the plurality of data partitions, obtaining the partition registration information recorded in the data lake through a communication interface between the data storage platform and the data lake; and The reason for the data loss is determined according to the partition registration information.

6. The method according to claim 5, characterized in that The determining the reason for the data loss according to the partition registration information includes: Determine, according to the partition registration information, a registered data partition for the write operation that has been registered in the data lake; In a case where the registration data partition is the same as the data partition used to perform the write operation, determining the reason for the data loss according to the execution log of the write operation; In a case where the registered data partition and the data partition used to perform the write operation are different, determining that the cause of the data loss is an erroneous partition; and Re-register on the data storage platform based on the registered data partition.

7. The method according to any one of claims 1, 5-6, characterized in that: The determining the reason for the data loss according to the execution log of the write operation includes: Determining error keywords in the execution log; and The reason for the data loss is determined based on the error keyword.

8. The method according to claim 7, characterized in that Determining the reason for the data loss according to the error keyword includes: When the error keyword is that the data source is abnormal, obtaining the latest timestamp information of the data source; When the latest timestamp information of the data source is different from the timestamp information of the write operation in the first time linked list, it is determined that the reason for the data loss is a time error.

9. A data processing device, characterized in that: The device comprises: A determination module, configured to determine a data partition for performing a data write operation in response to detecting that a data write operation occurs on the data storage platform; An acquisition module, used for acquiring, through a first time linked list, a difference in the number of messages of the data partition before and after the write operation occurs; a detection module, configured to determine a loss detection result of the write operation according to the message quantity difference; and The processing module is used to determine the cause of the data loss when the data loss detection result indicates that the write operation has caused data loss.

10. An electronic device comprising: one or more processors; a memory for storing one or more computer programs, The method is characterized in that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 8.

11. A computer-readable storage medium having a computer program or instruction stored thereon, characterized in that: When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

12. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images