Data writing task processing method and device

Abstract

The embodiments of the present specification provide a data writing task processing method and device, the data writing task processing method comprising: in response to a data writing task, receiving a sequence to be written, wherein the sequence to be written includes at least one data to be written; in the case where there is an abnormality in the target writing data in at least one data to be written, obtaining a reference state of the target writing data, and determining a predicted state corresponding to the reference state based on a target prediction model, wherein the reference state is the previous state of the target writing data corresponding to the target sending subject in the sequence to be written, the target sending subject is the sending subject of the target writing data, and the target prediction model is constructed based on the number of state transitions of the sample subsequences of each sending subject; updating the target writing data based on the predicted state, and executing the data writing task based on the updated target writing data. Predicting the current state based on the previous state of the target writing data with an abnormality and then writing improves the accuracy and execution efficiency.

Description

Technical Field

[0001] The embodiments of the present specification relate to the technical field of database and big data analysis, and in particular to a method and device for processing data writing tasks. Background Art

[0002] With the rapid development of computer technology, Internet technology and artificial intelligence technology, the amount of online data has exploded. A lot of data needs to be written into the database for storage. When writing into the database, dirty data may appear in the data sequence. Dirty data refers to data that is erroneous, incomplete, inconsistent or incorrectly formatted in the data sequence. These data will affect the accuracy and reliability of data analysis.

[0003] In the prior art, when executing a data writing task, if there is dirty data in the data sequence, such as missing values ​​or abnormal values, it is generally handled by interruption, skipping, transfer, etc. However, the data writing task will be interrupted after the interruption, which may have a greater impact on the user. After skipping, the data will be lost, and other field information of the data will also be discarded. Transferring is to transfer the dirty data field to an additional field, which requires an additional field in the database table, occupying additional storage space, and the dirty data field will also need manual analysis and operation and maintenance, which brings a large time cost. Summary of the invention

[0004] In view of this, an embodiment of this specification provides a data writing task processing method. One or more embodiments of this specification also relate to a prediction model construction method, a data writing task processing device, a prediction model construction device, a computing device, a computer-readable storage medium and a computer program product to solve the technical defects existing in the prior art.

[0005] According to a first aspect of an embodiment of this specification, a method for processing a data writing task is provided, including:

[0006] In response to a data writing task, receiving a sequence to be written, wherein the sequence to be written includes at least one data to be written;

[0007] In the case that there is an abnormality in the target write data in the at least one data to be written, obtaining a reference state of the target write data, and determining a predicted state corresponding to the reference state based on a target prediction model, wherein the reference state is a previous state of the target write data corresponding to a target sending subject in the sequence to be written, the target sending subject is a sending subject of the target write data, and the target prediction model is constructed based on the number of state transitions of a sample subsequence of each sending subject;

[0008] The target write data is updated based on the predicted state, and the data write task is executed based on the updated target write data.

[0009] According to a second aspect of the embodiments of this specification, a prediction model construction method is provided, including:

[0010] Acquire a sample sequence, wherein the sample sequence includes at least two sample data;

[0011] According to the sending entities of the at least two sample data, the at least two sample data are grouped to obtain sample subsequences of the sending entities;

[0012] Based on the state transition times of the sample subsequences of the sending subjects, the state transition models of the sending subjects are generated respectively, wherein the state transition times are the times that one state in the sample subsequence is transferred to another state, the state refers to the data content type of the sample data in the sample subsequence, and the times that one state in the sample subsequence is transferred to another state refers to the times that two data contents appear sequentially adjacently in the sample subsequence;

[0013] The state transition models of the sending entities are integrated to obtain a target prediction model, wherein the target prediction model is used to execute the above-mentioned data writing task processing method.

[0014] According to a third aspect of an embodiment of this specification, a data writing task processing device is provided, including:

[0015] A receiving module, configured to receive a sequence to be written in response to a data writing task, wherein the sequence to be written includes at least one data to be written;

[0016] A first acquisition module is configured to, when there is an abnormality in the target write data in the at least one data to be written, acquire a reference state of the target write data, and determine a predicted state corresponding to the reference state based on a target prediction model, wherein the reference state is a previous state of the target write data corresponding to the target sending subject in the sequence to be written, the target sending subject is a sending subject of the target write data, and the target prediction model is constructed based on the number of state transitions of a sample subsequence of each sending subject;

[0017] An execution module is configured to update the target write data based on the predicted state, and execute the data write task based on the updated target write data.

[0018] According to a fourth aspect of the embodiments of this specification, a prediction model construction device is provided, including:

[0019] A second acquisition module is configured to acquire a sample sequence, wherein the sample sequence includes at least two sample data;

[0020] A grouping module, configured to group the at least two sample data according to the sending subjects of the at least two sample data to obtain sample subsequences of each sending subject;

[0021] A generating module configured to generate a state transition model of each sending subject based on the state transition times of the sample subsequences of each sending subject, wherein the state transition times is the number of times one state transitions to another state in the sample subsequences;

[0022] The fusion module is configured to fuse the state transition models of the sending entities to obtain a target prediction model, wherein the target prediction model is used to execute the above-mentioned data writing task processing method.

[0023] According to a fifth aspect of an embodiment of this specification, a computing device is provided, including:

[0024] Memory and processor;

[0025] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the above-mentioned prediction model construction method or data writing task processing method are implemented.

[0026] According to the sixth aspect of the embodiments of this specification, a computer-readable storage medium is provided, which stores computer-executable instructions, which, when executed by a processor, implement the steps of the above-mentioned prediction model construction method or data writing task processing method.

[0027] According to the seventh aspect of the embodiments of this specification, a computer program product is provided, including a computer program / instruction, which, when executed by a processor, implements the steps of the above-mentioned prediction model construction method or data writing task processing method.

[0028] The embodiment of this specification provides a data writing task processing method, which realizes that when there is an abnormality in the target writing data in at least one of the to-be-written data corresponding to the data writing task, the previous state of the target writing data in the to-be-written sequence is obtained as a reference state, the predicted state corresponding to the reference state is determined using the target prediction model, and the target writing data is updated based on the predicted state, and then the writing task is executed based on the updated target writing data. The target prediction model is constructed based on the number of state transitions of the sample subsequences of each sending subject. The constructed target prediction model can predict the next state with a higher probability based on the previous state. When executing the data writing task, the current state can be predicted based on the previous state of the target writing data with an abnormality, so as to update the target writing data and then write it, thereby avoiding task interruption and data loss caused by abnormal data in the writing task. The processing of abnormal data is imperceptible, and no manual analysis and operation and maintenance are required, so that the abnormal data is free of operation and maintenance and can be replicated, saving time costs and improving the execution accuracy and efficiency of the writing task. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 is a flowchart of a prediction model construction method provided by an embodiment of this specification;

[0030] Figure 2 is a schematic diagram of a task creation page provided by an embodiment of this specification;

[0031] Figure 3 is a schematic diagram of a sample sequence provided by an embodiment of this specification;

[0032] Figure 4 is a schematic diagram of a sample subsequence of each sending subject provided by an embodiment of this specification;

[0033] Figure 5 This is a schematic diagram of a process of constructing a target prediction model provided by an embodiment of this specification;

[0034] Figure 6 is a schematic diagram of an iterative update process of a prediction model provided by an embodiment of this specification;

[0035] Figure 7 is a flow chart of a data writing task processing method provided by an embodiment of this specification;

[0036] Figure 8 is a processing flow chart of a data writing task processing method provided by an embodiment of this specification;

[0037] Fig. 9 is a processing flow chart of another data writing task processing method provided by an embodiment of this specification;

[0038] Fig.10 It is a structural schematic diagram of a prediction model building device provided by an embodiment of this specification;

[0039] Fig.11 It is a structural schematic diagram of a data writing task processing device provided by an embodiment of this specification;

[0040] Fig.12 It is a structural block diagram of a computing device provided by an embodiment of this specification. DETAILED DESCRIPTION

[0041] Many specific details are described in the following description to facilitate a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of this specification, so this specification is not limited to the specific implementation disclosed below.

[0042] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "a" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0043] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0044] In addition, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0045] First, the terms involved in one or more embodiments of this specification are explained.

[0046] Data lake: It is a system that centrally stores large amounts of raw data. It can store structured, semi-structured, and unstructured data. Data lakes are usually used for data exploration, advanced analysis, and machine learning. They have low storage costs, support multiple data formats, are suitable for big data processing and analysis, and can store historical data and real-time data.

[0047] Data warehouse: It is an integrated, subject-oriented, time-varying, non-volatile data collection used to support management and decision-making. Data warehouses are usually used for report generation and project intelligence. They are mainly based on structured data, optimize query performance, support complex analytical queries, and the data is cleaned and integrated.

[0048] Cloud Log Service (CLS): is a cloud-native log management service designed to help users efficiently collect, store, query, and analyze log data. Cloud Log Service generally provides a variety of functions, including log collection, log storage, log retrieval, log analysis, alarm notification, etc., suitable for various application scenarios, such as application monitoring, security auditing, and troubleshooting.

[0049] Dirty data: refers to data that is erroneous, incomplete, inconsistent, or in an incorrect format in a data set. This data will seriously affect the accuracy and reliability of data analysis. Identifying and processing dirty data is an important step in data preprocessing.

[0050] Markov model: is a statistical model widely used in natural language processing, bioinformatics, economic forecasting and other fields. It is based on an assumption that the probability distribution of a given state depends only on the previous state (for the first-order Markov model), and does not directly depend on earlier states or the entire history. In the Markov model, the frequency matrix and the probability matrix are two closely related but different concepts. They are both important tools for describing the transition relationship between states, but they have different meanings and uses in specific applications. The frequency matrix records the actual number of times a state transitions from one state to another in a series of data. This matrix is ​​usually generated in the first step of building a Markov model as the basis for calculating the probability matrix; the probability matrix (also called the transition matrix) represents the probability of transitioning from one state to another. Unlike the frequency matrix, each element in the probability matrix represents a conditional probability, that is, the probability of transitioning to the next state given the current state.

[0051] Information entropy: It is an important concept in information theory. It is an indicator used to measure the uncertainty of a random variable (such as a field or a variable). In data science and machine learning, field entropy is often used in feature selection, data compression, and information gain calculation.

[0052] It should be noted that the data warehouse / data lake can realize a one-stop analysis platform under the cloud log service, in which the missing value completion of log data is a common requirement in the preprocessing of analysis and training data. The database emphasizes that data synchronization is seamless and maintenance-free. Dirty data processing generally involves interrupting tasks, skipping, and transferring. After the interruption, the entire task will be interrupted. If it is delay-sensitive, it will have a greater impact on users; after skipping, the data will be lost, and other field information of this data will also be discarded; transferring is to transfer the dirty data field to an additional field. The disadvantage is that an additional field is required in the library table of the data warehouse / data lake, which takes up additional storage space, and the dirty data field will also require manual analysis and maintenance, which brings a large time cost to users.

[0053] For example, in a user's game marketing scenario project, the game event features provided by the user are time series data, which are stored in the cloud log service. Assuming that there are missing values ​​in the game event features, these data are discarded in the preprocessing stage. After being discarded, key event nodes in the time series data may be missing, affecting subsequent operations.

[0054] In addition, in actual implementation, you can also use a fixed value filling scheme to process missing data. Common fixed values ​​are 0, mean, median or mode. For example, for numerical data, you can use the mean or median of the column to fill missing values; for categorical data, you can use the mode. This method is simple, but it may introduce bias, especially when using special values ​​such as 0. Alternatively, you can also use AI (Artificial Intelligence) model predictions, such as regression models, classification models, K-nearest neighbors (KNN), etc. These AI models have high training costs and are not easy to iterate. They are usually used in offline scenarios and cannot better adapt to online log time series scenarios.

[0055] It can be seen from the above that both interruption tasks and transfers require operation and maintenance, and skipping strategies will result in data loss. Therefore, the embodiments of this specification provide a prediction model construction method and a data writing task processing method. Based on the constructed prediction model, an automatic update function of abnormal data in data writing tasks is provided, and dirty data is predicted as reasonable data, so that missing values ​​can be filled at a lower cost. This not only increases the analysis capability of automatically filling missing values, but also reduces the cost of filling data. It provides a method that is both maintenance-free and avoids data loss, and enhances the maintenance-free capability of writing log data into data warehouses / data lakes.

[0056] In this specification, a data writing task processing method is provided. This specification also involves a prediction model construction method, a prediction model construction device, a data writing task processing device, a computing device, and a computer-readable storage medium, which are described in detail one by one in the following embodiments.

[0057] See also Figure 1 , Figure 1 A flowchart of a prediction model construction method provided according to an embodiment of the present specification is shown, which specifically includes the following steps.

[0058] Step 102: Acquire a sample sequence, wherein the sample sequence includes at least two sample data.

[0059] Specifically, the sample sequence can be time series data used to build a prediction model. Time series data is a data set recorded in chronological order and used to describe the trend of data changes over time. The sample sequence includes a series of sample data. For example, in the scenario where log data is written into a data warehouse / data lake, the sample sequence can be a series of log data obtained and arranged in time. Alternatively, the sample sequence can be a time data series in other scenarios, such as weather data, product sales, etc.

[0060] In actual implementation, the client can display a task creation page to the user, which may include a configuration bar for multiple task options. The user can configure the corresponding task options in the corresponding configuration bar on the task creation page, and create the corresponding data writing task after the configuration is completed. The task options may be information items that need to be configured for the data writing task, such as task name, data source, database name, table name, user name, password, field mapping, task start time and other information.

[0061] Specifically, after creating a data writing task, M data can be obtained from the data source as sample data according to the data source and field mapping information configured in the data writing task. Assuming that the field mapping information of the data writing task indicates that there are N fields to be written to the database, an M*N matrix can be obtained as a sample sequence, in which a row of elements in the M*N matrix represents a sample data, and a column of elements represents a field.

[0062] For example, Figure 2This is a schematic diagram of a task creation page provided by an embodiment of this specification. Taking the writing of log data into a data warehouse / data lake as an example, the client can display a task creation page to the user. The task creation page may include multiple configuration columns for delivery options, such as Project Name, Logstore Name, Cloud Database Instance Id, Database Name, Table Name, User Name, Password, ColumnMapping, Start Time, etc. The user can configure the corresponding delivery options in the corresponding configuration column on the delivery page. After the configuration is completed, click the Submit control to create the warehouse / lake entry task.

[0063] Figure 3 is a schematic diagram of a sample sequence provided by an embodiment of this specification, such as Figure 3 As shown, M sample data, including sample data 1 (row1) - sample data M (rowM), are obtained, and each sample data includes N fields, including field 1 (column1) - field N (columnN), wherein the rectangular box represents the specific field content corresponding to the field.

[0064] Step 104: grouping the at least two sample data according to the sending entities of the at least two sample data to obtain sample subsequences of the sending entities.

[0065] The sending subject refers to the source user of the sample data, that is, the user who generates the sample data.

[0066] In actual implementation, at least two sample data can come from different sending entities, and thus at least two sample data can be grouped, and the sample data of the same sending entity can be divided into one group to obtain sample subsequences of each sending entity. A sample subsequence includes each sample data under the corresponding sending entity, that is, one sending entity corresponds to one sample subsequence.

[0067] In an optional implementation of this embodiment, the sample data includes a sending subject field; after obtaining the sample sequence, it also includes:

[0068] Determine candidate fields that meet the set field format from various fields of the sample data;

[0069] Based on the information entropy of the candidate fields and the set entropy value range, determining the sending subject field from the candidate fields, wherein the entropy value range is set based on the information entropy of the sending subject field;

[0070] Extract the sending subject ID from the sending subject field.

[0071] The sending subject field refers to a field in the sample data that stores a sending subject identifier, and the sending subject identifier is used to indicate the corresponding sending subject.

[0072] It should be noted that the sample data may include multiple fields, and the sending subject field needs to be identified from the multiple fields of the sample data, so as to extract the corresponding sending subject identifier.

[0073] In actual implementation, the format of the common sending subject identifier can be configured as the set field format in advance, and then determine whether the content of each field of the sample data conforms to the set field format, and filter out candidate fields that may be the sending subject fields. For example, if the sending subject identifier is UID (User ID or Unique Identifier), the format is usually a numeric type, and the length is usually around 16 bits. At this time, the field that is a number and has a length of 16 can be identified as a candidate field; if the sending subject identifier is UUID (Universally Unique Identifier), the format is usually 32 hexadecimal digits, usually displayed in the format of 8-4-4-4-12, such as 550e8400-e29b-41d4-a716-446655440000, then you can check whether each field contains 5 groups of characters, consisting of 8, 4, 4, 4 and 12 characters respectively, and separated by hyphens in the middle, to filter out candidate fields; if the sending subject identifier is MAC address (Media Access Control Address, the unique hardware address of a network device), the format is usually composed of 6 groups of two hexadecimal numbers, separated by colons or hyphens, such as 00:1A:2B:3C:4D:5E or 00-1A-2B-3C-4D-5E. At this time, you can check whether there are 6 groups of two-digit hexadecimal numbers. The separator can be a colon, a hyphen, or no separator, and filter out the candidate fields that meet the requirements. If the sender is identified as IMEI (International Mobile Equipment Identity), the format is usually 15 digits, such as 490154203237518. At this time, you can check whether the fields are only digits and the total length is 15 digits, and filter out the candidate fields that meet the requirements.

[0074] In addition, the entropy value range corresponding to the information entropy of the sending subject field can also be estimated in advance, and the entropy value range is the range corresponding to the information entropy of the sending subject field, and based on the entropy value range, it is determined which candidate field among the selected candidate fields is the sending subject field. Specifically, the information entropy of each candidate field can be calculated, and the candidate field whose information entropy is within the set entropy value range can be selected as the sending subject field.

[0075] Taking the game user event log scenario as an example, under the premise of user number, activity, and game events, by calculating the entropy of the UID field of the sending subject, the entropy value range for evaluating whether a certain field is the sending subject is given, and then the entropy value range is appropriately relaxed and extended to other scenarios.

[0076] Assume that a game has hundreds of thousands of active users every day. Each user has a unique UID field. The game events generated by these users will be stored in the log server in time series. There are hundreds of game events, and one user may generate multiple events. The event messages of the log server will be collected into the cloud log message system. Each message contains the UID of the user who generated the event. Assuming that 10,000 message data are consumed in time order, the entropy of the UID field is calculated below.

[0077] To estimate the entropy of UID in 10,000 messages, we need to consider the following factors: Number of users: If there are hundreds of thousands of active users every day, assume that the number of active users is 300,000 (i.e., 300,000); Frequency of event generation: Considering that a user may generate multiple events, if we assume that each user generates a certain number of events per day on average (for example, 10), then the number of users that may be involved in 10,000 messages can be calculated accordingly; Information entropy formula: Entropy can be calculated using the Shannon Entropy formula. Shannon Entropy is a basic concept in information theory that is used to quantify the uncertainty of information or the average amount of information. It can be used to measure the uncertainty of a random variable. The specific calculation formula can be shown in the following formula (1):

[0078] (1)

[0079] Among them, p(xi) is the probability of each sample data appearing.

[0080] Assuming that the events generated by each user are evenly distributed in these 10,000 messages, if the events are evenly distributed, the average number of events per user = 10,000 / 300,000≈0.0333, which means that on average, each active user will only appear once or not at all in these 10,000 messages, which will not affect the distribution of the entire user base. To simplify the calculation, we can consider the number of users involved in these 10,000 messages. Assuming that each user generates multiple events, there may be 2,000-3,000 unique UIDs.

[0081] Assume that there are 2500 different UIDs and they are evenly distributed. Then we can calculate the probability: p(xi)≈1 / 2500.

[0082] The corresponding information entropy is calculated by the above formula (1) as follows:

[0083] (2)

[0084] From the above, we can see that in 10,000 messages, the information entropy of the sending subject field (UID identifier) ​​is about 11.29 bits, which shows that in these 10,000 messages, the user's events are relatively scattered, but it is also possible that some users have generated a large number of events, so the actual information entropy may be different. The information entropy of the sending subject field will depend on the actual situation of the user generating the event, but under the above assumptions, the information entropy is roughly between 10 and 12 bits. Extended to other scenarios, 5 to 24 bits can be selected as the entropy value range.

[0085] After screening out candidate fields that conform to the set field format, select the candidate field whose information entropy is closest to 11 bits as the sending subject field, and extract the content of the field as the sending subject identifier.

[0086] In specific implementation, after extracting the sender identifier in the sender field, sample data with the same identifier can be divided into a group according to the sender identifier to obtain a sample subsequence corresponding to the sender identifier.

[0087] For example, Figure 4 is a schematic diagram of a sample subsequence of each sending subject provided by an embodiment of this specification, such as Figure 4As shown, taking the sample sequence including 9 sample data, namely sample data 1 (row1)-sample data 9 (row9), each sample data includes 4 fields (field 1 (column1)-field 3 (column3), and the sending subject field (user field)) as an example, that is, the sample sequence is a 9*4 matrix. By setting the field format and screening and identifying the entropy value range, it is automatically determined that the sending subject field in the sample sequence is the "user" field, that is, the "user" column corresponds to the sending subject column, and the field content of the "user" field is the sending subject identifier, indicating the corresponding sending subject, such as "subject 1 (user1)", "subject 2 (user2)", and "subject 3 (user3)" are sending subject identifiers, respectively indicating the corresponding sending subjects. The sample data 1 (row1), sample data 3 (row3), and sample data 5 (row5) corresponding to "subject 1 (user1)" are divided into 1 group, as the sample subsequence 1 corresponding to the sending subject "subject 1 (user1)"; the sample data 2 (row2), sample data 6 (row6), and sample data 7 (row7) corresponding to "subject 2 (user2)" are divided into 1 group, as the sample subsequence 2 corresponding to the sending subject "subject 2 (user2)"; the sample data 4 (row4), sample data 8 (row8), and sample data 9 (row9) corresponding to "subject 3 (user3)" are divided into 1 group, as the sample subsequence 3 corresponding to the sending subject "subject 3 (user3)".

[0088] In an embodiment of the present specification, after obtaining a sample sequence, each field value of each sample data of the sample sequence can be sampled, and whether the field format meets the requirements of the set field format can be automatically identified, candidate fields can be screened out, and then the information entropy of each candidate field can be calculated, and the candidate fields whose information entropy is within the entropy value range can be screened out as automatically identified sending subject fields, and the sending subject identifier in the sending subject field can be extracted to group each sample data according to the sending subject identifier, and obtain a sample subsequence of each sending subject. The sending subject field can be automatically identified quickly and accurately, thereby grouping each sample data according to different sending subjects, which is convenient for subsequent state transfer statistics for different sending subjects, thereby improving the pertinence and accuracy of the model.

[0089] Of course, in actual implementation, in addition to grouping each sample data according to the sending subject identification to obtain the sample subsequence of each sending subject, each sample data can also be grouped based on other methods to obtain the sample subsequence of each sending subject. The embodiments of this specification do not limit this. For example, based on the type of sending subject, the sample data collection time of the sending subject, etc., each sample data can be grouped to obtain the sample subsequence of each sending subject.

[0090] Step 106: Generate a state transition model for each sending subject based on the state transition times of the sample subsequences of each sending subject, wherein the state transition times is the number of times one state transitions to another state in the sample subsequence.

[0091] Specifically, the state refers to the type of data content of the sample data in the sample subsequence, the number of times a state in the sample subsequence is transferred to another state refers to the number of times two data contents in the sample subsequence appear sequentially adjacent to each other in the sample subsequence, that is, the number of times the first data content is located after the second data content and the two appear adjacent to each other, the first data content and the second data content are data contents of the same type or different types, and the two data contents are a combination of any two data contents in the sample subsequence. For example, a sample subsequence of a sending entity is: "a, b, c, a, a, b, a, d, a, b, c, b, a, a, b", where "a", "b", "c", and "d" are four types of data contents corresponding to the sample data of the sample subsequence. In the sample subsequence, the two data contents "a" and "b" are sequentially adjacent for 3 times, the two data contents "b" and "a" are sequentially adjacent for 2 times, and the two data contents "a" and "a" are sequentially adjacent for 2 times. That is, the sample subsequence transfers from the "a" state to the "b" state for 3 times, from the "b" state to the "a" state for 2 times, and from the "a" state to the "a" state for 2 times.

[0092] In actual implementation, the number of state transitions refers to the number of transitions from one state to another for a set of continuous states. The state transition model may refer to a situation in which each state in the sample subsequence of the sending subject transitions to other states. For example, the state transition model may be a frequency matrix or a probability matrix in a Markov model. The (i, j)th element in the state transition model represents the number of times or probability that the i-th state in the sample subsequence of the corresponding sending subject transitions to the j-th state.

[0093] It should be noted that in order to count the number of state transitions of the sample subsequence of each sending entity, it is necessary to first construct the corresponding state space, which includes all the states covered in the sample subsequence of each sending entity. The state refers to the field content of the sample data. The state transition means that for any field, the field content is transferred from the content in the current sample data to the content in the next sample data, that is, the content changes continuously in time sequence.

[0094] In specific implementation, after the state transfer model of each sending entity is generated, the state transfer model of each sending entity can be persistently stored, so that when the target prediction model is subsequently updated, the stored state transfer model can be used as the historical state transfer model, and the iteratively updated state transfer model can be superimposed to achieve the update of the target prediction model.

[0095] In an optional implementation of this embodiment, the sample data of the sample subsequence includes at least two fields, and the at least two fields include a sending subject field and at least one first field; based on the number of state transitions of the sample subsequences of each sending subject, a state transition model of each sending subject is generated respectively, including:

[0096] Constructing a first state space corresponding to a first field, wherein the first field is any field in a sample subsequence of each sending subject except a sending subject field;

[0097] The state transition times of the first sample subsequence are counted according to the first state space, and a state transition model corresponding to the first field of the first sending subject is generated, wherein the first sample subsequence is a sample subsequence of the first sending subject, and the first sending subject is any one of the sending subjects.

[0098] Among them, the state space refers to the set of all possible states. In the embodiments of this specification, the state space refers to the set of data content types. The first state space is the state space corresponding to the first field, that is, the first state space refers to the set of content types included in the field data of the first field. For example, the first field of the first sending subject is "b, a, c, a, a, c, a, b, c, b, d, b, a, a, b", the first field of the second sending subject is "a, d, d, a, b, b, a, b, d, b, a, d, c, a, b", and the first field of the third sending subject is "d, d, d, a, d, c, a, d, a, b, a, b, a, a, b", then the set of content types included in the field data of the first field is: "a", "b", "c", "d", and the first state space of the first field is [a, b, c, d].

[0099] It should be noted that the sample data includes at least two fields. Except for the sending subject field, different fields correspond to different types of field contents, which correspond to different state spaces. Therefore, it is possible to determine the field contents covered by the first field to construct the corresponding first state space. Each field can be used as the first field to construct the corresponding state space.

[0100] In actual implementation, the state transition times in the first sample subsequence can be counted in sequence according to each state included in the first state space to generate a state transition model corresponding to the first field of the first sending subject. In other words, each field of each sending subject can generate a corresponding state transition model. The state transition model can be a frequency matrix or a probability matrix.

[0101] For example, Figure 5 is a schematic diagram of a target prediction model construction process provided by an embodiment of this specification, such as Figure 5 As shown, taking the sample sequence including 9 sample data, namely sample data 1 (row1)-sample data (row9), each sample data includes 3 fields (field 1-field 3) in addition to the sending subject field as an example, the sample sequence includes 3 sending subjects, "subject 1 (user1)" corresponds to sample subsequence 1 (sample data 1 (row1), sample data 3 (row3), sample data 5 (row5)), "subject 2 (user2)" corresponds to sample subsequence 2 (sample data 2 (row2), sample data 6 (row6), sample data 7 (row7)), and "subject 3 (user3)" corresponds to sample subsequence 3 (sample data 4 (row4), sample data 8 (row8), sample data 9 (row9)).

[0102] For field 1, the field contents covered by each sample data of the sample sequence include: "e1", "e2", "e3", and the corresponding state space 1 [e1, e2, e3] is constructed; for field 2, the field contents covered by each sample data of the sample sequence include: "s1", "s2", "s3", and the corresponding state space 2 [s1, s2, s3] is constructed; for field 3, the field contents covered by each sample data of the sample sequence include: "a1", "a2", "a3", and the corresponding state space 3 [a1, a2, a3] is constructed. For field 1, based on state space 1 [e1, e2, e3], traverse the sample subsequence 1 corresponding to "subject 1 (user1)", count the number of state transitions, and generate the transition frequency matrix 11 corresponding to field 1 of "subject 1 (user1)"; traverse the sample subsequence 2 corresponding to "subject 2 (user2)", count the number of state transitions, and generate the transition frequency matrix 21 corresponding to field 1 of "subject 2 (user2)"; traverse the sample subsequence 3 corresponding to "subject 3 (user3)", count the number of state transitions, and generate the transition frequency matrix 31 corresponding to field 1 of "subject 3 (user3)". By analogy, the state transition model corresponding to each field of each sending subject can be obtained.

[0103] In the embodiments of the present specification, a corresponding state space is constructed for each field, and the number of state transitions of the corresponding sample subsequence is counted according to the state space of each field to generate a state transition model corresponding to each field of each sending subject. Different fields correspond to different state transition models. The state transition model can indicate the state transition situation at the field level, so that the target prediction model constructed subsequently can realize data prediction at the field level.

[0104] In an optional implementation of this embodiment, counting the number of state transitions of the first sample subsequence according to the first state space to generate a state transition model corresponding to the first field of the first sending subject includes:

[0105] Counting the number of transitions from the target state to each other state in the first sample subsequence, generating a state transition row vector of the target state, wherein the target state is any state in the first state space, and the other states are each state in the first state space except the target state;

[0106] Based on the state transition row vectors of each state in the first state space, a corresponding transition frequency matrix is ​​generated, and the transition frequency matrix is ​​used as a state transition model corresponding to the first field of the first sending subject.

[0107] In actual implementation, for the first field, any state is selected from the first state space as the target state, the first sample subsequence of the first sending subject is traversed, the number of transitions from the target state to each other state is counted, and the state transition row vector of the target state is generated. Other states are continuously selected from the first state space as the target state until the last state in the first state space. The state transition row vectors of each state in the first state space can be obtained, and the corresponding transition frequency matrix is ​​generated. The transition frequency matrix is ​​used as the state transition model corresponding to the first field of the first sending subject. In other words, the state transition model corresponding to any field of any sending subject is an n*n matrix, where n is the number of states included in the corresponding state space, and the (i, j)th element in the matrix represents the number of times the field in the corresponding sample subsequence is transferred from the i-th field content (i-th state) to the j-th field content (j-th state).

[0108] For example, assume that the state space 1 corresponding to field 1 is [e1, e2, e3]. Take "e1" as the target state, traverse the sample subsequence 1 of the sending subject "user1", and assume that the number of times "e1" transfers to "e1" is 0, the number of times it transfers to "e2" is 1, and the number of times it transfers to "e3" is 2. The state transition row vector corresponding to the target state "e1" can be obtained as [0, 1, 2]; and so on, "e2" can be taken as the target state in turn, and the corresponding state transition row vector is assumed to be [0, 3, 1]; take "e3" as the target state, and assume that the corresponding state transition row vector is [1, 1, 0]. At this time, the transition frequency matrix corresponding to field 1 of the sending subject "user1" can be obtained as [0, 1, 2; 0, 3, 1; 1, 1, 0]. For each field of each sending subject, the corresponding transition frequency matrix can be constructed according to the above process.

[0109] In one implementation, after the corresponding transfer frequency matrix is ​​generated based on the state transition row vectors of each state in the first state space, the transfer frequency matrix can be directly used as the state transition model corresponding to the first field of the first sending subject, and then converted into a transfer probability matrix after the transfer frequency matrices of each sending subject are fused. In another implementation, after the corresponding transfer frequency matrix is ​​generated based on the state transition row vectors of each state in the first state space, the transfer frequency matrix corresponding to the first field of the first sending subject can also be directly converted into a corresponding transfer probability matrix, and then the transfer probability matrices of each sending subject are directly fused.

[0110] Specifically, for the transition frequency matrix corresponding to the first field of the first sending entity, each element therein is divided by the sum of the elements in the corresponding row (that is, the sum of all transition times of the state) to obtain the transition probability corresponding to the element, and obtain the transition probability matrix, in which each element P (i, j) represents the probability of transitioning from state i to state j, and the sum of the elements in each row is equal to 1.

[0111] In the embodiments of the present specification, the number of transitions from the target state to each other state in the first sample subsequence can be counted to generate a corresponding transition frequency matrix as the state transition model corresponding to the first field, and the transition frequency matrix can also be converted into a transition probability matrix as the state transition model corresponding to the first field, that is, the state transition model corresponding to any field can indicate the number of times or probability that one state transitions to another state, so that the target prediction model obtained by subsequent construction has the function of predicting another state based on one state, and is applied to various data prediction scenarios to improve the accuracy and efficiency of task processing.

[0112] Step 108: The state transition models of each sending entity are integrated to obtain a target prediction model.

[0113] It should be noted that each sending entity can generate a corresponding state transition model. In actual implementation, the state transition models of each sending entity can be fused to obtain the corresponding target prediction model.

[0114] In an optional implementation of this embodiment, the state transition models of each sending subject are integrated to obtain the target prediction model, including:

[0115] The transfer frequency matrices corresponding to the first fields of the sending entities are integrated to obtain a total transfer frequency matrix of the first field, and a total transfer probability matrix of the first field is generated according to the total transfer frequency matrix of the first field;

[0116] The state space corresponding to each field and the total transition probability matrix of each field are used as the target prediction model.

[0117] In one implementation, the state transition model of each sending entity is a transition frequency matrix. The transition frequency matrices corresponding to the first fields of each sending entity can be fused to obtain the total transition frequency matrix of the first field, and then the total transition probability matrix of the first field can be generated based on the total transition frequency matrix of the first field.

[0118] In actual implementation, for the first field, the elements in the transfer frequency matrix of each sending subject can be superimposed to obtain the total transfer frequency matrix of the first field. Then, for each element in the total transfer frequency matrix, divide it by the sum of the corresponding row (that is, the sum of all transition times of the state) to obtain the probability of transitioning from the state to other states, and obtain the total transfer probability matrix, in which each element P (i, j) represents the probability of transitioning from state i to state j, and the sum of the elements in each row is equal to 1.

[0119] Using the above example, Figure 5 As shown, for field 1, the elements of each position in the transfer frequency matrix 11 corresponding to field 1 of the sending subject "user1", the transfer frequency matrix 21 corresponding to field 1 of the sending subject "user2", and the transfer frequency matrix 31 corresponding to field 1 of the sending subject "user3" can be superimposed to obtain the total transfer frequency matrix F1 corresponding to field 1; for field 2, the elements of each position in the transfer frequency matrix 12 corresponding to field 2 of the sending subject "user1", the transfer frequency matrix 22 corresponding to field 2 of the sending subject "user2", and the transfer frequency matrix 32 corresponding to field 2 of the sending subject "user3" can be superimposed to obtain the total transfer frequency matrix F2 corresponding to field 2; for field 3, the elements of each position in the transfer frequency matrix 13 corresponding to field 3 of the sending subject "user1", the transfer frequency matrix 23 corresponding to field 3 of the sending subject "user2", and the transfer frequency matrix 33 corresponding to field 3 of the sending subject "user3" can be superimposed to obtain the total transfer frequency matrix F3 corresponding to field 3.

[0120] For each element in the total transition frequency matrix F1, divide it by the sum of the corresponding rows to obtain the probability of transitioning from this state to other states, and obtain the total transition probability matrix P1 corresponding to field 1, which is assumed to be [0, 1 / 3, 2 / 3; 0, 3 / 4, 1 / 4; 1 / 2, 1 / 2, 0]. Similarly, obtain the total transition probability matrix P2 corresponding to field 2 and the total transition probability matrix P3 corresponding to field 3.

[0121] Of course, in another implementation, the state transition model of each sending subject can be a transition probability matrix. Each element in the transition probability matrix corresponding to the first field of each sending subject is superimposed according to the corresponding element position to obtain a probability fusion matrix. Each element in the probability fusion matrix is ​​divided by the number of sending subjects to obtain the total transition probability matrix of the first field. By analogy, the total transition probability matrix of each field can be obtained by superimposing. Among them, the sum of each element in any row of the probability fusion matrix is ​​1*the number of subjects, and the sum of each element in any row of the total transition probability matrix is ​​1.

[0122] In actual implementation, the state space corresponding to each field and the total transition probability matrix of each field can be used as the target prediction model. For example, the state space 1 corresponding to field 1, the state space 2 corresponding to field 2, the state space 3 corresponding to field 3, and the total transition probability matrix 1 corresponding to field 1, the total transition probability matrix 2 corresponding to field 2, and the total transition probability matrix 3 corresponding to field 3 are used as the constructed target prediction model.

[0123] In the embodiments of the present specification, for any field, the transfer frequency matrices of each sending subject can be fused, and the frequency elements can be converted into probability elements to obtain the total transfer probability matrix of the field. The state space of each field and the total transfer probability matrix are used to construct the obtained target prediction model. The obtained target prediction model includes the field-level transfer probability matrix and integrates each sending subject. It can achieve any data field of different sending subjects, and predict the next state with a higher probability based on the current state. Therefore, it can be widely used in various scenarios that require field-level data prediction, thereby improving the task execution accuracy and efficiency in various scenarios.

[0124] In an optional implementation of this embodiment, after the state transition models of each sending subject are integrated to obtain the target prediction model, the following further includes:

[0125] When the model iteration conditions are met, an update sample sequence is obtained, and the current state transition model of each sending subject is obtained according to the update sample sequence;

[0126] Generate a corresponding updated state transition model according to the historical state transition model and the current state transition model of each sending subject, wherein the historical state transition model is a state transition model obtained and persistently stored before the current time;

[0127] The update state transfer models of each sending entity are integrated to obtain the update prediction model.

[0128] The model iteration condition refers to the condition for iteratively updating the constructed target prediction model, such as the model iteration condition may include reaching the update time and / or detecting abnormal data.

[0129] In actual implementation, an updated sample sequence can be obtained when the model iteration conditions are met. The updated sample sequence is the time series data collected when the model iteration conditions are met. The updated sample sequence and the sample sequence of the above-mentioned target prediction model construction process belong to the same data writing task, thereby realizing continuous updating of the constructed target prediction model based on the data of the data writing task.

[0130] It should be noted that the specific implementation process of obtaining the current state transition model of each sending entity based on the updated sample sequence is similar to the implementation process of steps 104-106, and the implementation process of fusing the updated state transition models of each sending entity to obtain the updated prediction model is similar to the implementation process of the above-mentioned step 108, and the embodiments of this specification will not be repeated here.

[0131] In the embodiments of the present specification, when the model iteration conditions are met, an update sample sequence can be obtained, and the current state transition model of each sending subject can be obtained according to the update sample sequence. Then, the historical state transition model and the current state transition model of each sending subject are superimposed to generate a corresponding updated state transfer model, and the updated state transfer models of each sending subject are integrated to obtain an updated prediction model. When the model iteration conditions are met, the target prediction model obtained by constructing is actively iterated and updated, thereby improving the prediction accuracy of the target prediction model. The model can be iterated automatically without human intervention, thus saving labor costs. As the prediction model is used for a longer time, the prediction accuracy increases accordingly, and the model becomes more accurate with use.

[0132] In an optional implementation of this embodiment, generating a corresponding updated state transition model according to the historical state transition model and the current state transition model of each sending subject includes:

[0133] Superimpose the historical state transition model corresponding to the second field of the second sending subject and the current state transition model corresponding to the second field of the second sending subject to obtain an updated state transition model corresponding to the second field of the second sending subject, wherein the second sending subject is any one of the sending subjects and the second field is any field corresponding to the second sending subject.

[0134] In actual implementation, each field corresponds to a state transfer model. Therefore, for any field of any sending subject, the historical state transfer model of the field and the current state transfer model of the field can be superimposed one by one according to the element position to obtain the updated state transfer model of the field of the sending subject.

[0135] For example, Figure 6 is a schematic diagram of an iterative update process of a prediction model provided by an embodiment of this specification, such as Figure 6As shown, when the model iteration condition is met, an updated sample sequence is obtained, and the updated sample sequence is divided into sample subsequence 1 of the sending subject "user1", sample subsequence 2 of the sending subject "user2", and sample subsequence 3 of the sending subject "user3" according to the sending subject. For field 1, sample subsequence 1 is traversed to build the corresponding current state transition model 11; for field 2, sample subsequence 1 is traversed to build the corresponding current state transition model 12; for field 3, sample subsequence 1 is traversed to build the corresponding current state transition model 13. For field 1, sample subsequence 2 is traversed to build the corresponding current state transition model 21; for field 2, sample subsequence 2 is traversed to build the corresponding current state transition model 22; for field 3, sample subsequence 2 is traversed to build the corresponding current state transition model 23. For field 1, sample subsequence 3 is traversed to build the corresponding current state transition model 31; for field 2, sample subsequence 3 is traversed to build the corresponding current state transition model 32; for field 3, sample subsequence 3 is traversed to build the corresponding current state transition model 33.

[0136] Obtain historical state transfer model 11', historical state transfer model 12', historical state transfer model 13', historical state transfer model 21', historical state transfer model 22', historical state transfer model 23', historical state transfer model 31', historical state transfer model 32', historical state transfer model 33' generated and persistently stored before the current time that meets the iteration conditions.

[0137] Superimpose the elements in the current state transition model 11 and the historical state transition model 11' one by one to obtain the updated state transition model U11 corresponding to the sending subject "user1" and field 1, and so on to obtain the updated state transition models U12, U13, U21, U22, U23, U31, U32, U33 corresponding to each field of each sending subject.

[0138] Then, for each field, the update state transfer models of each sending subject are fused to obtain the update transfer probability matrix corresponding to each field, and the update transfer probability matrix and state space corresponding to each field are used as the update prediction model.

[0139] In the embodiments of the present specification, the historical state transition model of any sending subject or any field can be superimposed with the current state transition model of the sending subject or the field to obtain an updated state transition model of the sending subject or the field, and then merged to obtain an updated prediction model. When the prediction model is iteratively updated, superimposed updates are performed on different sending subjects and different fields respectively, and the prediction model is updated at the sending subject and field levels, thereby improving the accuracy of the model update and further improving the accuracy of data prediction.

[0140] In an optional implementation of this embodiment, the model iteration condition includes reaching the update time and / or detecting abnormal data; obtaining the update sample sequence includes:

[0141] When the current time reaches the update time, obtaining the sequence of the current cycle as the update sample sequence; and / or,

[0142] In the case of detecting abnormal data, the other data in the received current sequence except the abnormal data is used as an updated sample sequence.

[0143] In practical applications, model iteration conditions include reaching the update time and / or detecting abnormal data. In an optional implementation, the update interval or update time can be preset, such as an update interval of 2 hours, 4 hours, etc., and updates are performed every 2 hours or 4 hours. The update time is 12 o'clock and 16 o'clock every day. When the current time reaches the update time, the sequence of the current cycle is obtained as the update sample sequence, and the target prediction model is iteratively updated, so that the prediction model can learn the correct sample data of each time period, ensure the continuous iteration of the model, and then ensure the accuracy of the prediction model.

[0144] The current cycle refers to the period from the last update time to the current update time. For example, if the update interval is 2 hours, the last update time is 2 o'clock, and the current time is 4 o'clock, the update time is reached. At this time, the current cycle is the time period from 2 o'clock to 4 o'clock, and the sequence from 2 o'clock to 4 o'clock can be obtained as the update sample sequence.

[0145] In another optional implementation, when abnormal data is detected, the other data in the received current sequence except the abnormal data can be used as the update sample sequence to iteratively update the target prediction model. The abnormal data can be the dirty data in the received data sequence. In other words, the current batch of data can be obtained, and the rows where the abnormal data are located can be removed to prevent interference with the prediction model, while retaining the context information of the abnormal data, that is, other correct data rows, to iteratively update the prediction model.

[0146] It should be noted that when abnormal data is received, the model can be iterated based on the context information of the abnormal data, or the prediction model can be actively iterated and updated periodically to ensure that the correct data is sampled and learned while the context information of the abnormal data is learned efficiently, thereby ensuring the accuracy of the prediction model.

[0147] The embodiments of this specification provide a prediction model construction method, which can construct a state transition model corresponding to the sending subject based on the number of times a state is transferred to another state in the sample subsequence of the sending subject, and fuse the state transition models of each sending subject to construct a target prediction model. The obtained target prediction model fuses the state transition models of each sending subject, and can predict the next state with a higher probability of occurrence based on the current state for different sending subjects. The constructed target prediction model can be applied to the data writing task processing method. When executing the data writing task, the constructed target prediction model can be used to predict the current state based on the previous state of the target writing data with an exception, so as to update the target writing data and then write it, avoiding task interruption and data loss caused by abnormal data in the writing task. The processing of abnormal data is imperceptible, and no manual analysis and operation and maintenance are required, so that the abnormal data is free of operation and maintenance and can be replicated, saving time cost, improving the execution accuracy and efficiency of the data writing task, and making accurate and efficient predictions in the scenario where the sending subject exists. In addition, the target prediction model can be continuously updated and iterated to learn correct data and efficiently learn the contextual information of abnormal data. As the prediction model is used for a longer time, the prediction accuracy increases accordingly. The model becomes more and more accurate as it is used, providing a new option for maintenance-free data writing tasks and having the characteristics of replicability and promotion.

[0148] See also Figure 7 , Figure 7 A flow chart of a data writing task processing method provided according to an embodiment of the present specification is shown, which specifically includes the following steps.

[0149] Step 702: In response to a data writing task, a sequence to be written is received, wherein the sequence to be written includes at least one data to be written.

[0150] It should be noted that the data writing task and the data writing task corresponding to the sample sequence in step 102 are data writing tasks in the same scenario, such as log data storage / logging, and the data writing task can be created in the manner shown in the above step 102. After the user creates the data writing task, the data channel service can consume the data to be written from the data source according to the data source and field mapping information configured in the data writing task.

[0151] Step 704: When there is an abnormality in the target write data in at least one of the data to be written, obtain a reference state of the target write data, and determine a predicted state corresponding to the reference state based on a target prediction model, wherein the reference state is the previous state of the target write data corresponding to the target sending subject in the sequence to be written, the target sending subject is the sending subject of the target write data, and the target prediction model is constructed based on the number of state transitions of a sample subsequence of each sending subject.

[0152] It should be noted that in response to the data writing task, the field contents of all other fields except the sending subject field can be continuously stored and updated, that is, the status of each field can be continuously updated. If it is determined that the received target write data is stored abnormally, the previous data to be written of the target write data can be obtained, and the corresponding field content in the previous data to be written can be used as the reference state, and the target prediction model is used to determine the predicted state corresponding to the reference state, which is the more likely next state of the reference state determined by the target prediction model.

[0153] Specifically, abnormal data may refer to data that is erroneous, missing, inconsistent, or incorrectly formatted. Format checking, range checking, type checking, and other data verification methods can be used to determine whether erroneous data occurs; data integrity checking can be used to determine whether data loss occurs; and logical consistency checking can be used to determine whether inconsistent or incorrectly formatted data occurs.

[0154] In actual implementation, the reference state is the previous data to be written of the target write data, and the corresponding field content in the previous data to be written is used as a reference for predicting the next state. The reference state and the target write data should come from the same sending entity. The data channel service can continuously store the data to be written of each sending entity during the process of continuously consuming the data to be written. Therefore, in specific implementation, the target sending entity of the target write data can be determined, and the previous data to be written can be searched based on the target sending entity, so as to obtain the field content of the corresponding field therein as the reference state.

[0155] It should be noted that the target prediction model is constructed based on the state transition times of the sample subsequences of each sending subject. The specific construction process can be found in the above Figure 1 The contents described in the illustrated embodiment will not be repeated in this embodiment.

[0156] In an optional implementation of this embodiment, the target prediction model includes a state space corresponding to each field and a total transition probability matrix of each field; determining a prediction state corresponding to a reference state based on the target prediction model includes:

[0157] Determine the abnormal field of the target written data;

[0158] Determine a target state space corresponding to the abnormal field from the state space corresponding to each field, and determine a target transition probability matrix corresponding to the abnormal field from the total transition probability matrix of each field;

[0159] Based on the target state space and the target transition probability matrix, the predicted state corresponding to the reference state is determined.

[0160] In actual implementation, the target prediction model includes the state space corresponding to each field and the total transfer probability matrix of each field. It can determine the abnormal field of the target write data, and then determine the target state space and target transfer probability matrix corresponding to the abnormal field. Then, based on the target state space and target transfer probability matrix, the prediction state corresponding to the reference state is determined. The target state space and target transfer probability matrix corresponding to the abnormal field are used to accurately realize field-level data prediction, thereby updating the abnormal target write data and executing the data writing task, thereby ensuring the accuracy and efficiency of the data writing task.

[0161] Using the above example, Figure 5 As shown, assuming that data of missing field 1 "column1" in the target abnormal data is received, the state space 1 [e1, e2, e3] corresponding to field 1 can be determined as the target state space, and the total transition probability matrix 1 can be used as the target probability matrix. Assuming that the field content of field 1 "column1" in the previous data to be written of the target write data is "e2", that is, the reference state is "e2". Based on the state space 1 [e1, e2, e3] and the total transition probability matrix 1, the predicted state corresponding to the reference state "e2" is determined.

[0162] In the embodiments of the present specification, the corresponding target state space and target transition probability matrix are determined based on the abnormal field, and then the target state space and target transition probability matrix are used to predict the next state value of the reference state for the abnormal field, thereby realizing the prediction of the next state based on the most recent state of the abnormal field, realizing field-level prediction, facilitating the subsequent updating and processing of the abnormal fields in the abnormal data of the data writing task, and ensuring the execution accuracy and efficiency of the data writing task.

[0163] In an optional implementation of this embodiment, determining the predicted state corresponding to the reference state based on the target state space and the target transition probability matrix includes:

[0164] Determine the state index of the reference state in the target state space;

[0165] Obtaining a corresponding transition probability vector from the target transition probability matrix based on the state index, wherein the transition probability vector is used to indicate the transition probability from the reference state to each other state;

[0166] A predicted state is determined based on the transition probability vector.

[0167] In actual implementation, the reference state S can be determined first k The state index k in the target state space is then extracted from a row of transition probability vectors corresponding to the state index in the target transition probability matrix to obtain the reference state S kThe transition probability to other states is determined in the transition probability vector to determine the state S with the highest probability. j as a predicted state.

[0168] Continuing with the above example, the abnormal field is field 1 "column1", the reference state is "e2", and the state index of the reference state in the state space [e1, e2, e3] is "2". At this time, the elements in the second row of the total transition probability matrix 1 can be extracted as the transition probability vector, and the element with the highest probability in the transition probability vector is found, that is, the predicted state. Assuming that the probability of the third element in the second row is the highest, that is, the probability of transferring from "e2" to "e3" is the highest, it can be determined that the predicted state is "e3".

[0169] It should be noted that the state index of the reference state can be determined based on the target state space first, the probability of the reference state transferring to other states can be found, and then the state with the highest probability can be selected as the predicted state, thereby realizing the prediction of the next state value with the highest probability based on the most recent state of the abnormal field, realizing field-level prediction, facilitating the subsequent updating of the abnormal fields in the abnormal data of the data writing task, and ensuring the execution accuracy and efficiency of the data writing task.

[0170] Step 706: Update the target write data based on the predicted state, and execute the data write task based on the updated target write data.

[0171] In actual implementation, the predicted state can be used to replace the abnormal field in the target write data to obtain the completed updated write data, write the updated write data into the database, automatically predict the abnormal field as a reasonable value, and then write it, so that the data writing task is not interrupted, jumps, and data is not lost. No additional analysis and maintenance is required, and the writing effect is highly accurate. Since the prediction model provided in the embodiment of this specification has low construction cost, high iteration efficiency, and fast prediction speed, it is more suitable for application to online log time series scenarios than the fixed value filling solution and AI model prediction solution.

[0172] In addition, in the scenario of game customer payment prediction, the data prediction scheme in the data writing task process provided in the embodiment of this specification can be used as a comparison baseline for AI model evaluation to evaluate whether the accuracy of the AI ​​model is better than that of statistical methods.

[0173] It should be noted that in the process of writing data to the database in response to a data writing task, if abnormal target write data is detected, the abnormal field can be determined, and the next state can be predicted based on the previous state of the abnormal field using the transition probability matrix corresponding to the field to update and complete the abnormal field, and then write it into the database to ensure that data is written without delay. At the same time, the sequence where the abnormal data is located can be used to iteratively update the target prediction model, so that the target prediction model can learn the contextual information of the abnormal data, making the target prediction model more accurate with use.

[0174] The embodiment of this specification also provides a data writing task processing method, which realizes that when there is an abnormality in the target writing data in at least one of the to-be-written data corresponding to the data writing task, the previous state of the target writing data in the to-be-written sequence is obtained as a reference state, the predicted state corresponding to the reference state is determined using the target prediction model, and the target writing data is updated based on the predicted state, and then the data writing task is executed based on the updated target writing data. The target prediction model is constructed based on the number of state transitions of the sample subsequences of each sending subject. The constructed target prediction model can predict the next state with a higher probability based on the previous state. When executing the data writing task, the current state can be predicted based on the previous state of the target writing data with an abnormality, so as to update the target writing data and then write it, thereby avoiding task interruption and data loss caused by abnormal data in the data writing task. The processing of abnormal data is imperceptible, and no manual analysis and operation and maintenance are required, so that the abnormal data is free of operation and maintenance and can be replicated, saving time costs and improving the execution accuracy and efficiency of the data writing task.

[0175] The following combination Figure 8 and attached Fig. 9 , taking the application of the data writing task processing method provided in this specification in the log writing scenario as an example, the data writing task processing method is further explained. Among them, Figure 8 A process flow chart of a data writing task processing method provided by an embodiment of the present specification is shown. Fig. 9 A processing flow chart of another data writing task processing method provided by an embodiment of the present specification is shown.

[0176] like Figure 8 and Fig. 9 As shown, a log storage task is created. In response to the log storage task, the data channel service continuously consumes and obtains the log data in the initial stage. Based on the set field format and entropy value range, the sending subject field in the continuous log data within a period of time is automatically identified, and the continuous log data is grouped according to the sending subject to obtain the sub-log sequence corresponding to each sending subject.

[0177] Construct the state space corresponding to each field.

[0178] For any field, based on the state space of the arbitrary field, the number of transitions from the target state to each other state in each sub-log sequence is counted to generate a state transition row vector of the target state, where the target state is any state in the state space, and the other states are all states in the state space except the target state; based on the state transition row vectors of each state in the state space, a transition frequency matrix corresponding to each sending entity under the arbitrary field is generated.

[0179] For any field, the transfer frequency matrices corresponding to each sending subject are fused to obtain the total transfer frequency matrix of the arbitrary field, and the corresponding total transfer probability matrix is ​​generated according to the total transfer frequency matrix of the arbitrary field.

[0180] The state space corresponding to each field and the total transition probability matrix of each field are used as the target prediction model to be constructed.

[0181] In response to the log warehousing task, the data channel service continuously consumes the log data of each sending entity and writes it into the data warehouse.

[0182] When it is detected that there are missing values ​​in the target log data of the target sending subject, the missing fields are determined, and the previous field content of the missing field of the target sending subject is queried as a reference state; based on the reference state, the target state space corresponding to the missing field and the target probability model are used to determine the next state with the highest probability as the predicted state; based on the predicted state, the missing fields are supplemented to the target log data to obtain the updated log data and write it into the data warehouse.

[0183] Based on the log sequence corresponding to the target log data with missing values, the total transition probability matrix of each field is iteratively updated to realize model learning of abnormal log data samples.

[0184] In response to the log storage task, the data channel service continuously consumes the log data of each sender, and the log data is continuously written into the data warehouse. When the update time is reached, the log sequence of the current cycle is obtained as the update sample sequence, and the total transition probability matrix of each field is periodically updated. The model learns the correct log data samples.

[0185] The data channel service continuously consumes the log data of each sender, writes it into the data warehouse, and executes the log storage task until the task is completed.

[0186] The embodiment of this specification provides a data writing task processing method, which can build a state transition model corresponding to the sending subject based on the number of times a state is transferred to another state in the sub-log sequence of the sending subject, and integrate the state transition models of each sending subject to build a target prediction model. The obtained target prediction model integrates the state transition models of each sending subject, and can predict the next state with a higher probability of occurrence based on the current state for different sending subjects. The prediction is accurate and efficient in the scenario where the sending subject exists. Since the target prediction model has low construction cost, high iteration efficiency, and fast prediction speed, it is more suitable for application to online log time series scenarios, improving the execution accuracy and efficiency of log warehousing tasks. In addition, the target prediction model can be continuously iterated and updated, learns correct log data, and efficiently learns the context information of abnormal log data. As the use time of the prediction model increases, the prediction accuracy increases accordingly. The more the model is used, the more accurate it is, providing a new option for log warehousing tasks to be free of operation and maintenance, and has the characteristics of replicability and promotion.

[0187] Corresponding to the above method embodiment, this specification also provides a prediction model construction device embodiment, Fig.10 FIG. 1 is a schematic diagram showing a structure of a prediction model building device provided by an embodiment of the present specification. Fig.10 As shown, the device comprises:

[0188] The second acquisition module 1002 is configured to acquire a sample sequence, wherein the sample sequence includes at least two sample data;

[0189] The grouping module 1004 is configured to group at least two sample data according to the sending subjects of the at least two sample data to obtain sample subsequences of each sending subject;

[0190] The generating module 1006 is configured to generate a state transition model of each sending subject based on the state transition times of the sample subsequences of each sending subject, wherein the state transition times refers to the number of times one state in the sample subsequence is transferred to another state, the state refers to the data content type of the sample data in the sample subsequence, and the number of times one state in the sample subsequence is transferred to another state refers to the number of times two data contents appear sequentially adjacently in the sample subsequence;

[0191] The fusion module 1008 is configured to fuse the state transition models of each sending entity to obtain a target prediction model, wherein the target prediction model is used to execute the above-mentioned data writing task processing method.

[0192] Optionally, the sample data includes a sending subject field; the device also includes an extraction module configured to

[0193] Determine candidate fields that meet the set field format from various fields of the sample data;

[0194] Based on the information entropy of the candidate fields and the set entropy value range, determining the sending subject field from the candidate fields, wherein the entropy value range is set based on the information entropy of the sending subject field;

[0195] Extract the sending subject ID from the sending subject field.

[0196] Optionally, the sample data of the sample subsequence includes at least two fields, and the at least two fields include a sending body field and at least one first field; the generating module 1006 is further configured to:

[0197] Constructing a first state space corresponding to a first field, wherein the first field is any field in a sample subsequence of each sending subject except a sending subject field, and the first state space refers to a set of content types included in field data of the first field;

[0198] The state transition times of the first sample subsequence are counted according to the first state space, and a state transition model corresponding to the first field of the first sending subject is generated, wherein the first sample subsequence is a sample subsequence of the first sending subject, and the first sending subject is any one of the sending subjects.

[0199] Optionally, the generating module 1006 is further configured to:

[0200] Counting the number of transitions from the target state to each other state in the first sample subsequence, generating a state transition row vector of the target state, wherein the target state is any state in the first state space, and the other states are each state in the first state space except the target state;

[0201] Based on the state transition row vectors of each state in the first state space, a corresponding transition frequency matrix is ​​generated, and the transition frequency matrix is ​​used as a state transition model corresponding to the first field of the first sending subject.

[0202] Optionally, the fusion module 1008 is further configured to:

[0203] The transfer frequency matrices corresponding to the first fields of the sending entities are integrated to obtain a total transfer frequency matrix of the first field, and a total transfer probability matrix of the first field is generated according to the total transfer frequency matrix of the first field;

[0204] The state space corresponding to each field and the total transition probability matrix of each field are used as the target prediction model.

[0205] Optionally, the device further comprises an iteration module configured to:

[0206] When the model iteration conditions are met, an update sample sequence is obtained, and the current state transition model of each sending subject is obtained according to the update sample sequence;

[0207] Generate a corresponding updated state transition model according to the historical state transition model and the current state transition model of each sending subject, wherein the historical state transition model is a state transition model obtained and persistently stored before the current time;

[0208] The update state transfer models of each sending entity are integrated to obtain the update prediction model.

[0209] Optionally, the iteration module is further configured as follows:

[0210] Superimpose the historical state transition model corresponding to the second field of the second sending subject and the current state transition model corresponding to the second field of the second sending subject to obtain an updated state transition model corresponding to the second field of the second sending subject, wherein the second sending subject is any one of the sending subjects and the second field is any field corresponding to the second sending subject.

[0211] Optionally, the model iteration condition includes reaching the update time and / or detecting abnormal data; the iteration module is further configured to:

[0212] When the current time reaches the update time, obtaining the sequence of the current cycle as the update sample sequence; and / or,

[0213] In the case of detecting abnormal data, the other data in the received current sequence except the abnormal data is used as an updated sample sequence.

[0214] An embodiment of the present specification provides a prediction model construction device, which can construct a state transition model corresponding to a sending subject based on the number of times a state transitions to another state in a sample subsequence of the sending subject, and fuse the state transition models of each sending subject to construct a target prediction model. The obtained target prediction model fuses the state transition models of each sending subject, and can predict the next state with a higher probability of occurrence based on the current state for different sending subjects. The constructed target prediction model can be applied to the data writing task processing method, saving the time cost of the data writing task and improving the execution accuracy and efficiency of the data writing task.

[0215] The above is a schematic scheme of a prediction model construction device of this embodiment. It should be noted that the technical scheme of the prediction model construction device and the technical scheme of the prediction model construction method mentioned above belong to the same concept, and the details not described in detail in the technical scheme of the prediction model construction device can be found in the description of the technical scheme of the prediction model construction method mentioned above.

[0216] Corresponding to the above method embodiment, this specification also provides a data writing task processing device embodiment, Fig.11 FIG. 1 shows a schematic diagram of a data writing task processing device provided by an embodiment of the present specification. Fig.11 As shown, the device comprises:

[0217] The receiving module 1102 is configured to receive a sequence to be written in response to a data writing task, wherein the sequence to be written includes at least one data to be written;

[0218] The first acquisition module 1104 is configured to acquire a reference state of the target write data when there is an abnormality in the target write data in at least one of the data to be written, and determine a predicted state corresponding to the reference state based on a target prediction model, wherein the reference state is a previous state of the target write data corresponding to the target sending subject in the sequence to be written, the target sending subject is a sending subject of the target write data, and the target prediction model is constructed based on the state transition times of the sample subsequences of each sending subject;

[0219] The execution module 1106 is configured to update the target write data based on the predicted state, and execute the data write task based on the updated target write data.

[0220] Optionally, the target prediction model includes a state space corresponding to each field of each sending subject and a total transition probability matrix of each field; the first acquisition module 1104 is further configured to:

[0221] Determine the abnormal field of the target written data;

[0222] Determine a target state space corresponding to the abnormal field from the state space corresponding to each field, and determine a target transition probability matrix corresponding to the abnormal field from the total transition probability matrix of each field;

[0223] Based on the target state space and the target transition probability matrix, the predicted state corresponding to the reference state is determined.

[0224] Optionally, the first acquisition module 1104 is further configured to:

[0225] Determine the state index of the reference state in the target state space;

[0226] Obtaining a corresponding transition probability vector from the target transition probability matrix based on the state index, wherein the transition probability vector is used to indicate the transition probability from the reference state to each other state;

[0227] A predicted state is determined based on the transition probability vector.

[0228] The embodiment of the present specification also provides a data writing task processing device, which realizes that when there is an abnormality in the target writing data in at least one of the to-be-written data corresponding to the data writing task, the previous state of the target writing data in the to-be-written sequence is obtained as a reference state, the predicted state corresponding to the reference state is determined by using the target prediction model, and the target writing data is updated based on the predicted state, and then the data writing task is executed based on the updated target writing data. The target prediction model is constructed by the number of state transitions of the sample subsequences of each sending subject, and the target prediction model can predict the next state with a higher probability based on the previous state. When executing the data writing task, the current state can be predicted based on the previous state of the target writing data with an abnormality, so as to update the target writing data and then write it, thereby avoiding task interruption and data loss caused by abnormal data in the data writing task, and the processing of abnormal data is imperceptible, and no manual analysis and operation and maintenance are required, so that the abnormal data is free of operation and maintenance and can be replicated, saving time costs, and improving the execution accuracy and efficiency of the data writing task.

[0229] The above is a schematic scheme of a data writing task processing device of this embodiment. It should be noted that the technical scheme of the data writing task processing device and the technical scheme of the data writing task processing method described above belong to the same concept, and the details of the technical scheme of the data writing task processing device that are not described in detail can all be referred to the description of the technical scheme of the data writing task processing method described above.

[0230] Fig.12 The block diagram of a computing device according to an embodiment of the present specification is shown. The components of the computing device 1200 include but are not limited to a memory 1210 and a processor 1220. The processor 1220 is connected to the memory 1210 via a bus 1230, and the database 1250 is used to store data.

[0231] The computing device 1200 also includes an access device 1240 that enables the computing device 1200 to communicate via one or more networks 1260. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 1240 may include one or more of any type of network interface (e.g., a network interface card (NIC)) of wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a world-wide interoperability for microwave access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, and a near field communication (NFC).

[0232] In one embodiment of the present specification, the above components of the computing device 1200 and Fig.12 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Fig.12 The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0233] The computing device 1200 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or a personal computer (PC). The computing device 1200 may also be a mobile or stationary server.

[0234] Among them, the processor 1220 is used to execute the following computer executable instructions, which, when executed by the processor, implement the steps of the above-mentioned prediction model construction method or data writing task processing method.

[0235] The above is a schematic scheme of a computing device of this embodiment. It should be noted that the technical scheme of the computing device and the technical scheme of the above-mentioned prediction model construction method or data writing task processing method belong to the same concept, and the details not described in detail in the technical scheme of the computing device can be referred to the description of the technical scheme of the above-mentioned prediction model construction method or data writing task processing method.

[0236] An embodiment of the present specification also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the steps of the above-mentioned prediction model construction method or data writing task processing method.

[0237] The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of the storage medium and the technical scheme of the above-mentioned prediction model construction method or data writing task processing method belong to the same concept, and the details not described in detail in the technical scheme of the storage medium can be referred to the description of the technical scheme of the above-mentioned prediction model construction method or data writing task processing method.

[0238] An embodiment of the present specification also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the above-mentioned prediction model construction method or data writing task processing method.

[0239] The above is a schematic scheme of a computer program product of this embodiment. It should be noted that the technical scheme of the computer program product and the technical scheme of the above-mentioned prediction model construction method or data writing task processing method belong to the same concept, and the details not described in detail in the technical scheme of the computer program product can be referred to the description of the technical scheme of the above-mentioned prediction model construction method or data writing task processing method.

[0240] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0241] Computer instructions include computer program codes, which may be in source code form, object code form, executable files or some intermediate forms, etc. Computer readable media may include: any entity or device capable of carrying computer program codes, recording media, USB flash drives, mobile hard disks, magnetic disks, optical disks, computer memories, read-only memories (ROM), random access memories (RAM), electric carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the contents of computer readable media may be appropriately increased or decreased according to the requirements of patent practice. For example, in some regions, according to patent practice, computer readable media do not include electric carrier signals and telecommunication signals.

[0242] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of this specification are not limited by the order of the actions described, because according to the embodiments of this specification, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of this specification.

[0243] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0244] The preferred embodiments of this specification disclosed above are only used to help explain this specification. The optional embodiments do not describe all the details in detail, nor do they limit the invention to only the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that technicians in the relevant technical field can well understand and use this specification. This specification is only limited by the claims and their full scope and equivalents.

Claims

1. A data writing task processing method, comprising: In response to a data writing task, receiving a sequence to be written, wherein the sequence to be written includes at least one data to be written; In the case that there is an abnormality in the target write data in the at least one data to be written, obtaining a reference state of the target write data, and determining a predicted state corresponding to the reference state based on a target prediction model, wherein the reference state is a previous state of the target write data corresponding to a target sending subject in the sequence to be written, the target sending subject is a sending subject of the target write data, and the target prediction model is constructed based on the number of state transitions of a sample subsequence of each sending subject; updating the target write data based on the predicted state, and executing the data write task based on the updated target write data; The target prediction model is constructed based on the following method: Acquire a sample sequence, wherein the sample sequence includes at least two sample data; According to the sending entities of the at least two sample data, the at least two sample data are grouped to obtain sample subsequences of the sending entities; Based on the state transition times of the sample subsequences of the sending subjects, the state transition models of the sending subjects are generated respectively, wherein the state transition times are the times that one state in the sample subsequence is transferred to another state, the state refers to the data content type of the sample data in the sample subsequence, and the times that one state in the sample subsequence is transferred to another state refers to the times that two data contents appear sequentially adjacently in the sample subsequence; The state transition models of the sending entities are integrated to obtain a target prediction model.

2. The method according to claim 1, wherein the sample data includes a sending subject field; after obtaining the sample sequence, the method further includes: Determine, from each field of the sample data, a candidate field that conforms to a set field format; Based on the information entropy of the candidate field and a set entropy value range, determining a sending subject field from the candidate field, wherein the entropy value range is set based on the information entropy of the sending subject field; Extract the sending subject identifier in the sending subject field.

3. The method according to claim 1, wherein the sample data of the sample subsequence includes at least two fields, and the at least two fields include a sending subject field and at least one first field; and the state transition model of each sending subject is generated based on the state transition times of the sample subsequences of each sending subject, comprising: Constructing a first state space corresponding to a first field, wherein the first field is any field in the sample subsequence of each sending subject except the sending subject field, and the first state space refers to a set of content types included in the field data of the first field; A state transition model corresponding to a first field of a first sending subject is generated by counting the number of state transitions of a first sample subsequence according to the first state space, wherein the first sample subsequence is a sample subsequence of the first sending subject, and the first sending subject is any one of the sending subjects.

4. The method according to claim 3, wherein counting the number of state transitions of the first sample subsequence according to the first state space to generate a state transition model corresponding to the first field of the first sending entity comprises: Counting the number of transitions from a target state to each other state in the first sample subsequence, and generating a state transition row vector of the target state, wherein the target state is any state in the first state space, and the other states are each state in the first state space except the target state; Based on the state transition row vectors of each state in the first state space, generating a corresponding transition frequency matrix; The transfer frequency matrix is ​​used as a state transfer model corresponding to the first field of the first sending entity.

5. According to the method of claim 4, the step of fusing the state transition models of the sending entities to obtain the target prediction model comprises: Merging the transfer frequency matrices corresponding to the first fields of the sending entities to obtain a total transfer frequency matrix of the first field, and generating a total transfer probability matrix of the first field according to the total transfer frequency matrix of the first field; The state space corresponding to each field and the total transition probability matrix of each field are used as the target prediction model.

6. The method according to any one of claims 1 to 5, after fusing the state transition models of the sending entities to obtain the target prediction model, further comprising: When the model iteration condition is met, an update sample sequence is obtained, and a current state transition model of each sending subject is obtained according to the update sample sequence; Generate a corresponding updated state transition model according to the historical state transition model of each sending subject and the current state transition model, wherein the historical state transition model is a state transition model obtained and persistently stored before the current time; The update state transition models of the sending entities are integrated to obtain an update prediction model.

7. The method according to claim 6, wherein generating a corresponding updated state transition model according to the historical state transition model of each sending subject and the current state transition model comprises: Superimpose the historical state transition model corresponding to the second field of the second sending subject and the current state transition model corresponding to the second field of the second sending subject to obtain an updated state transition model corresponding to the second field of the second sending subject, wherein the second sending subject is any one of the sending subjects, and the second field is any field corresponding to the second sending subject.

8. The method according to claim 6, wherein the model iteration condition comprises reaching an update time and / or detecting abnormal data; The obtaining and updating the sample sequence comprises: When the current time reaches the update time, obtaining the sequence of the current cycle as the update sample sequence; and / or, In the case where abnormal data is detected, the other data in the received current sequence except the abnormal data is used as the updated sample sequence.

9. According to the method of claim 1, the target prediction model includes a state space corresponding to each field of each sending subject and a total transition probability matrix of each field; the step of determining the prediction state corresponding to the reference state based on the target prediction model includes: Determine the abnormal field of the target written data; Determine a target state space corresponding to the abnormal field from the state space corresponding to each field, and determine a target transition probability matrix corresponding to the abnormal field from the total transition probability matrix of each field; Based on the target state space and the target transition probability matrix, a predicted state corresponding to the reference state is determined.

10. The method according to claim 9, wherein determining the predicted state corresponding to the reference state based on the target state space and the target transition probability matrix comprises: Determine a state index of the reference state in the target state space; Acquire a corresponding transition probability vector from the target transition probability matrix based on the state index, wherein the transition probability vector is used to indicate the transition probability of the reference state to other states; The predicted state is determined based on the transition probability vector.

11. A data writing task processing device, comprising: A receiving module, configured to receive a sequence to be written in response to a data writing task, wherein the sequence to be written includes at least one data to be written; A first acquisition module is configured to, when there is an abnormality in the target write data in the at least one data to be written, acquire a reference state of the target write data, and determine a predicted state corresponding to the reference state based on a target prediction model, wherein the reference state is a previous state of the target write data corresponding to the target sending subject in the sequence to be written, the target sending subject is a sending subject of the target write data, and the target prediction model is constructed based on the number of state transitions of a sample subsequence of each sending subject; an execution module, configured to update the target write data based on the predicted state, and execute the data write task based on the updated target write data; The target prediction model is constructed based on the following modules: A second acquisition module is configured to acquire a sample sequence, wherein the sample sequence includes at least two sample data; A grouping module, configured to group at least two sample data according to the sending subjects of the at least two sample data, and obtain sample subsequences of each sending subject; A generating module is configured to generate a state transition model of each sending subject based on the state transition times of the sample subsequences of each sending subject, wherein the state transition times refers to the number of times one state in the sample subsequence is transferred to another state, the state refers to the data content type of the sample data in the sample subsequence, and the number of times one state in the sample subsequence is transferred to another state refers to the number of times two data contents appear sequentially adjacently in the sample subsequence; The fusion module is configured to fuse the state transition models of each sending subject to obtain a target prediction model.

12. A computing device comprising: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions. When the computer programs / instructions are executed by the processor, the steps of the method according to any one of claims 1 to 10 are implemented.

13. A computer-readable storage medium storing a computer program / instruction, wherein the computer program / instruction, when executed by a processor, implements the steps of the method according to any one of claims 1 to 10.

14. A computer program product, comprising a computer program / instruction, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 10.

Images