Data processing method, model training method, device, equipment and storage medium

Abstract

The embodiment of the present invention provides a data processing method, a model training method, an apparatus, a device and a storage medium, the method comprising: when monitoring a data change in a master database, obtaining first change data of the master database, and storing the first change data in a preset storage area; obtaining status information of the first change data; determining the validity of the first change data according to the status information; processing the first change data with valid validity in the preset storage area using a target neural network model to obtain a query result; and writing the first change data with valid validity in the preset storage area into a slave database corresponding to the master database. The embodiment of the present invention can realize efficient real-time query of the first change data, improve the accuracy of the query result, and improve the consistency of data between the master and slave databases.

Description

Technical Field

[0001] The present invention belongs to the field of computer technology, and in particular relates to a data processing method, a model training method, a device, a equipment and a storage medium. Background Art

[0002] The high availability of a database is crucial to the realization of the functions of a database system. Master-slave replication between the master database and the slave database in a database system is one of the important means to achieve high availability of a database.

[0003] In the related art, a long connection of a binary log dump (binlog dump) thread is established between a slave database and a master database. During the master-slave replication process, the slave database starts an input and output (I / O) thread to synchronize the binary log (binlog) of the master database from the master database in an asynchronous manner, and writes it to the local relay log (relaylog) of the slave database. The slave database realizes data synchronization with the master database based on the relay log.

[0004] However, in the process of asynchronous data synchronization, if the master database crashes and the data in the binlog has not been synchronized to the slave database, data will be lost in the master-slave replication process, leading to data inconsistency between the master and slave databases. Furthermore, in the scenario of high-concurrency data synchronization, there will be a certain degree of delay in the data synchronization process of the slave database, resulting in the failure to synchronize the changed data in the master database to the slave database in time, which in turn leads to the query results based on the slave database having poor real-time performance and low accuracy. Summary of the invention

[0005] The embodiments of the present invention provide a data processing method, a model training method, an apparatus, a device and a storage medium to solve the problems of data inconsistency between master and slave databases in the master-slave replication process in the related technology, and poor real-time performance and low accuracy of query results obtained based on data query from the database.

[0006] In order to solve the above problems, in a first aspect, an embodiment of the present invention discloses a data processing method, the method comprising:

[0007] When a data change in the master database is detected, first change data of the master database is acquired, and the first change data is stored in a preset storage area;

[0008] Acquire status information of the first change data;

[0009] determining the validity of the first change data according to the status information;

[0010] Using the target neural network model, the first change data with valid validity in the preset storage area is processed to obtain a query result;

[0011] The first changed data with valid validity in the preset storage area is written into the slave database corresponding to the master database.

[0012] In a second aspect, an embodiment of the present invention discloses a model training method, the method comprising:

[0013] Acquire a training data set; the training data set includes a changed data sample and a query result sample corresponding to the changed data sample;

[0014] In each round of training, the neural network model to be trained is trained using the changed data sample to obtain the prediction query result corresponding to the changed data sample;

[0015] Determine the loss value corresponding to the neural network model to be trained according to the predicted query result and the query result sample;

[0016] The model parameters of the neural network model to be trained are adjusted according to the loss value, and the next round of training is performed until the loss value meets the preset conditions, thereby obtaining the target neural network model as described in any one of claims 1 to 3.

[0017] In a third aspect, an embodiment of the present invention discloses a data processing device, the device comprising:

[0018] A first acquisition module is used to acquire first changed data of the master database when monitoring data changes in the master database, and store the first changed data in a preset storage area;

[0019] A second acquisition module, used to acquire status information of the first change data;

[0020] A first determining module, configured to determine the validity of the first change data according to the state information;

[0021] A query module, used for processing the first change data with valid validity in the preset storage area by using the target neural network model to obtain a query result;

[0022] The writing module is used to write the first change data with valid validity in the preset storage area into the slave database corresponding to the master database.

[0023] In a fourth aspect, an embodiment of the present invention discloses a model training device, the device comprising:

[0024] A third acquisition module is used to acquire a training data set; the training data set includes a changed data sample and a query result sample corresponding to the changed data sample;

[0025] A training module, used to train the neural network model to be trained using the changed data sample in each round of training to obtain the prediction query result corresponding to the changed data sample;

[0026] A second determination module, used to determine the loss value corresponding to the neural network model to be trained according to the predicted query result and the query result sample;

[0027] The training module is also used to adjust the model parameters of the neural network model to be trained according to the loss value, and perform the next round of training until the loss value meets the preset conditions to obtain the target neural network model as described above.

[0028] In a fifth aspect, an embodiment of the present invention further discloses an electronic device, comprising a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other through the communication bus; the memory is used to store executable instructions, and the executable instructions enable the processor to execute the data processing method as described above, or execute the model training method as described above.

[0029] An embodiment of the present invention also discloses a readable storage medium. When instructions in the readable storage medium are executed by a processor of an electronic device, the processor is enabled to execute the data processing method as described above, or to execute the model training method as described above.

[0030] The embodiments of the present invention include the following advantages:

[0031] The data processing method provided by the embodiment of the present invention obtains the first change data of the master database and stores the first change data in a preset storage area when monitoring a data change in the master database. Even if the master database fails or other faults occur, the slave database can still synchronize data based on the first change data in the preset storage area, avoiding the problem of data loss in the master-slave replication process, and improving the reliability and security of the first change data. Furthermore, the embodiment of the present invention also determines the validity of the first change data according to the status information of the first change data, and when data query is required, uses the target neural network model to process the first change data with valid validity in the preset storage area to obtain the query result, and implements adaptive streaming query based on the target neural network model, which can not only realize efficient real-time query of the first change data, but also the query result is obtained based on the valid first change data in the preset storage area, avoiding the influence of invalid first change data on the query result, and improving the accuracy of the query result. The embodiment of the present invention also writes the first change data with valid validity in the preset storage area into the slave database corresponding to the master database, so that the slave database only synchronizes data based on the valid first change data in the preset storage area, avoiding the introduction of invalid first change data into the slave database, and improving the consistency of data between the master and slave databases. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0033] Figure 1 is a flow chart of steps of a data processing method provided by an embodiment of the present invention;

[0034] Figure 2 It is a schematic diagram of a framework of a database system provided by an embodiment of the present invention;

[0035] Figure 3 is a flowchart of the steps of a model training method provided by an embodiment of the present invention;

[0036] Figure 4 is a flowchart of another data processing method provided by an embodiment of the present invention;

[0037] Figure 5 is a logic block diagram of a data processing device provided by an embodiment of the present invention;

[0038] Figure 6It is a logic block diagram of a model training device provided by an embodiment of the present invention;

[0039] Figure 7 It is a structural block diagram of an electronic device for data processing or model training provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0040] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0041] Method Embodiment

[0042] Reference Figure 1 , shows a flow chart of the steps of a data processing method provided by an embodiment of the present invention, and the method may specifically include the following steps:

[0043] Step S101: when a data change in a master database is detected, first change data of the master database is acquired, and the first change data is stored in a preset storage area.

[0044] Step S102: Obtain status information of the first changed data.

[0045] Step S103: Determine the validity of the first change data according to the status information.

[0046] Step S104: Use the target neural network model to process the first change data with valid validity in the preset storage area to obtain a query result.

[0047] Step S105: Write the first changed data with valid validity in the preset storage area into the slave database corresponding to the master database.

[0048] The data processing method provided in the embodiment of the present invention can be applied to a first electronic device, in which a target neural network model is installed, and the first electronic device includes a preset storage area. The first electronic device may include but is not limited to a smart terminal, a computer, a personal digital assistant (PDA), a tablet computer, a vehicle-mounted device, etc.

[0049] In some embodiments, the first electronic device may be an electronic device where a slave database corresponding to the master database is located.

[0050] In other embodiments, referring to Figure 2 , showing a framework schematic diagram of a database system provided by an embodiment of the present invention, the database system includes a first electronic device, a second electronic device where a master database is located, and a third electronic device where a slave database is located, wherein the number of the third electronic devices is the same as the number of the slave databases and is at least 1; the first electronic device is an electronic device independent of the second electronic device and the third electronic device, thereby improving the availability and robustness of the database system.

[0051] The database types of the master database and the slave database may include but are not limited to Oracle, MySQL, SQL Server, DB2, and PostgreSQL.

[0052] In an embodiment of the present invention, the master database includes a data change log (e.g., binlog), which is used to record the change operation statements corresponding to the data changes occurring in the master database, the types of change operation statements, etc. Among them, the change operation statements include data modification statements (e.g., INSERT, UPDATE, DELETE, etc.) and database structure change statements (e.g., CREATE, ALTER, TABLE, DROP, etc.); the type of change operation statement is used to identify whether the change operation statement is a data modification statement or a structure change statement.

[0053] Specifically, in step S101, the first electronic device can continuously monitor the data change log in the master database; when monitoring that the data change log is updated, determine that the master database has a data change, read the content of the updated data change entry in the data change log, and use the content of the updated data change entry in the data change log as the first change data of the current data change in the master database. The data change entry includes the change operation statement corresponding to the data change, the type of the change operation statement, etc.

[0054] As an example, the first electronic device includes a kafka message management platform and a Debezium plug-in, and the message queue in the kafka message management platform is used as a preset storage area; specifically, the first electronic device can use the Debezium plug-in to continuously monitor the data change log in the master database, and when it is monitored that the data change log is updated, obtain the first change data from the data change log, and transmit the first change data to the message queue in the kafka message management platform. Kafka has the characteristics of high throughput, persistence and fault tolerance, can reliably store the first change data of the master database, and allow multiple consumers to subscribe to and consume the first change data (for example, perform data query based on the first change data in the message queue, and write the first change data into at least one slave database corresponding to the master database).

[0055] When the embodiment of the present invention monitors that data changes occur in the primary database, the first change data can be obtained from the primary database in real time, and the first change data can be stored in a preset storage area. In the event of a failure such as a crash in the primary database, the first change data will not be lost, thereby improving the reliability and security of the first change data in the master-slave replication process of the database system.

[0056] After storing the first changed data in the preset storage area, the first electronic device may obtain state information of the first changed data in the main database in step S102.

[0057] The status information is used to characterize the execution status of the target transaction corresponding to the first change data in the primary database; specifically, the status information may include but is not limited to: the execution timestamp of the change operation statement in the first change data, the transaction identifier (transaction ID) corresponding to the first change data, etc.

[0058] When the state information of the first changed data is obtained, the first electronic device may execute step S103 to determine the validity of the first changed data in the preset storage area according to the state information. Specifically:

[0059] The first electronic device determines whether the target transaction corresponding to the first change data is a repeatedly submitted transaction based on the transaction identifier; the first electronic device determines the execution order of the change operation corresponding to the target transaction based on the execution timestamp of the change operation statement, and determines whether the execution process of the target transaction complies with the business rules based on the execution order; when the target transaction is repeatedly submitted and / or the execution process of the target transaction does not comply with the business rules, the first electronic device can determine that the validity of the first change data is invalid; when the target transaction is not submitted repeatedly and the execution process of the target transaction complies with the business rules, the first electronic device can determine that the validity of the first change data is valid.

[0060] After determining the validity of the first changed data in step S103, the first electronic device may add a validity tag to the first changed data in the preset storage area, where the validity tag is used to identify the validity determination result (invalid or valid) of the first changed data.

[0061] In an embodiment of the present invention, before executing step S104 and step S105, the first electronic device first executes step S103 to determine the validity of the first change data, which is beneficial to improving the accuracy and reliability of the query result obtained in step S104, and is also beneficial to improving the reliability of the first change data written to the slave database in step S105, and is further beneficial to achieving real-time synchronization of data in the master and slave databases, and improving the consistency of data between the master and slave databases.

[0062] When the first electronic device meets the data query condition, it can execute step S104 to obtain the query result and present the query result. The data query condition may include but is not limited to: receiving a data query instruction, the validity of the first change data in the preset storage area is updated, etc.

[0063] In step S104, the first electronic device may first read all first change data with valid validity from a preset storage area, and input the first change data into the target neural network model according to the timing relationship between the first change data; then, the first change data may be processed using the target neural network model to implement an adaptive streaming structured query language (SQL) query on the first change data using the target neural network model to obtain a query result.

[0064] The target neural network model is a neural network model obtained after model construction and model training according to prior knowledge such as query statements and query patterns of adaptive streaming SQL before step S104.

[0065] The target neural network model may be an incremental recurrent neural network (IRNN) model.

[0066] In step S105, the first electronic device can write the first change data with valid validity in the preset storage area into the relay log of the slave database corresponding to the master database, so that the slave database can parse the first change data in the relay log into specific operations and execute them serially to achieve data synchronization between the master and slave databases.

[0067] Specifically, in some embodiments, the first electronic device may periodically obtain first change data with valid validity from a preset storage area, and write the first change data with valid validity into a relay log of a slave database corresponding to the master database. In other embodiments, the first electronic device may write the first change data into a relay log of a slave database corresponding to the master database when the validity of the first change data in the preset storage area is updated to valid.

[0068] Among them, after the first electronic device executes step S105 to write the first change data with validity in the preset storage area into the slave database corresponding to the master database, the slave database parses the first change data into specific operations and executes them serially. After realizing data synchronization between the master and slave databases, a data synchronization completion signal can be sent to the first electronic device; when the first electronic device receives the data synchronization completion signal, it can delete the first change data in the preset storage area to release storage space in the preset storage area.

[0069] In the process of master-slave replication in the related art, the database system needs to establish a long connection of the binary log dump thread between the slave database and the master database, and the slave database synchronizes the binary log of the master database from the master database in an asynchronous manner, and writes it to the local relay log of the slave database, so that the slave database can realize data synchronization with the master database according to the relay log. However, in the process of asynchronous data synchronization, if the master database goes down, the data in the binary log has not been synchronized to the slave database, which will cause data loss in the master-slave replication process, and then cause data inconsistency between the master and slave databases; further, in the scenario of high-concurrency data synchronization, there will be a certain degree of delay in the data synchronization process of the slave database, resulting in the data that has been changed in the master database cannot be synchronized to the slave database in time, and then the query results obtained by querying the data based on the slave database have poor real-time performance and low accuracy. In addition, although the related art can overcome the data loss problem caused by the asynchronous method through a semi-synchronous method, in the process of data synchronization in a semi-synchronous method, the master database needs to wait for at least one slave database to confirm the receipt of the change data in the binary log before submitting the transaction, resulting in the operation blocking problem of the master database, which increases the operation overhead of the master database.

[0070] In an embodiment of the present invention, a thread connection may not be established between the master database and the slave database. When the first electronic device monitors that data changes occur in the master database, it obtains the first change data of the master database and stores the first change data in a preset storage area. Even if the master database crashes or other failures occur, the slave database can still synchronize data based on the first change data in the preset storage area, thereby avoiding data loss during the master-slave replication process and improving the reliability and security of the first change data. Furthermore, in an embodiment of the present invention, the validity of the first change data is determined based on the status information of the first change data, and when data query is required, the target neural network model is used to process the first change data with valid validity in the preset storage area to obtain a query result. Adaptive streaming query is implemented based on the target neural network model, which not only realizes efficient real-time query of the first change data, but also the query result is obtained based on the valid first change data in the preset storage area, thereby avoiding the influence of invalid first change data on the query result and improving the accuracy of the query result. In an embodiment of the present invention, the first change data with valid validity in the preset storage area is written into the slave database corresponding to the master database, so that the slave database only synchronizes data in real time based on the valid first change data in the preset storage area, thereby avoiding the introduction of invalid first change data into the slave database and improving the consistency of data between the master and slave databases.

[0071] Optionally, the state information includes a transaction identifier and a timestamp; and the step S103 of determining the validity of the first change data according to the state information includes:

[0072] Step S1031: Determine the target transaction corresponding to the first change data according to the transaction identifier.

[0073] Step S1032: Determine the commit status of the target transaction according to the timestamp.

[0074] Step S1033: When the submission status is submitted, determine that the validity of the first change data is valid.

[0075] Step S1034: when the submission status is unsubmitted, determine that the validity of the first change data is invalid.

[0076] In the embodiment of the present invention, the state information acquired by the first electronic device in step S102 includes a transaction identifier and a timestamp; wherein the transaction identifier is a transaction identifier that corresponds one-to-one to the target transaction corresponding to the first change data, and the timestamp is the execution timestamp of the change operation statement in the first change data. Specifically:

[0077] First, the first electronic device determines the target transaction of the first change data in the main database according to the transaction identifier.

[0078] Then, the first electronic device determines the execution state of the change operation corresponding to the target transaction according to the timestamp, and determines the commit state of the target transaction according to the execution state. The execution state includes states such as start, commit, and rollback; when the execution state is commit, the first electronic device determines the commit state of the target transaction as committed; when the execution state is non-commit (for example, start, rollback, etc.), the first electronic device determines the commit state of the target transaction as uncommitted.

[0079] Finally, when the submission status is submitted, the first electronic device determines that the validity of the first change data is valid; when the submission status is unsubmitted, the validity of the first change data is determined to be invalid. It can be understood that when the validity of the first change data is invalid, it indicates that the change operation corresponding to the target transaction is in an unfinished state. During the execution of the change operation corresponding to the target transaction, rollback, modification, etc. may occur. Data query based on the first change data with invalid validity will lead to inaccurate query results, and data synchronization operation of the slave database based on the first change data with invalid validity will introduce dirty data inconsistent with the master database in the slave database, resulting in inconsistent data between the master and slave databases. In an embodiment of the present invention, before executing step S104 and step S105, the first electronic device first executes steps S1031 to S1034 to determine the validity of the first change data, which is conducive to improving the accuracy and reliability of the query result obtained in step S104, and is also conducive to improving the reliability of the first change data written to the slave database in S105, thereby facilitating the real-time synchronization of data in the master and slave databases and improving the consistency of data between the master and slave databases.

[0080] In an embodiment of the present invention, the first electronic device may execute step S102 according to a preset period to obtain the state information of the first change data whose validity is invalid in the preset storage area, and determine the validity of the first change data again through steps S1031 to S1034, and update the validity of the first change data in the preset storage area. It can be understood that in the preset storage area, the validity of the first change data at the first moment is invalid, but the validity of the first change data at the second moment may be valid; the second moment is a moment after the first moment.

[0081] Optionally, in step S104, the first change data with valid validity in the preset storage area is processed by using the target neural network model to obtain a query result, including:

[0082] Step S1041: read first change data with valid validity from the preset storage area to obtain a first data stream.

[0083] Step S1042: input the first data stream into the target neural network model.

[0084] Step S1043: Use the target neural network model to extract the timing characteristics and data change trend characteristics of the first data stream.

[0085] Step S1044: using the target neural network model to output an initial processing result corresponding to the first data stream according to the timing characteristics and the data change trend characteristics.

[0086] Step S1045, obtaining the initial processing result output by the target neural network model, and generating a query result based on the initial processing result.

[0087] In an embodiment of the present invention, the first electronic device uses the target neural network model to process the first change data with validity in the preset storage area to obtain the query result, which can be achieved through steps S1041 to S1045.

[0088] Among them, a streaming processing engine is set in the first electronic device. In step S1041, the first electronic device can use the streaming processing engine to read the first change data with valid validity from the preset storage area to obtain the first data stream; the first data stream includes all the first change data with valid validity in the preset storage area, and each first change data is arranged in the time sequence obtained by the first electronic device through step S101.

[0089] The stream processing engine may include but is not limited to Flink, Spark Streaming, etc.

[0090] After acquiring the first data stream, the first electronic device inputs the first data stream into the target neural network model, and executes step S1043 to extract the timing characteristics and data change trend characteristics of the first data stream using the target neural network model, and executes step S1044 to generate an initial processing result corresponding to the first data stream based on the timing characteristics and data change trend characteristics using the target neural network model.

[0091] After obtaining the initial processing result output by the target neural network model, the first electronic device can perform format conversion processing on the initial processing result according to the preset presentation format of the query result to obtain the query result with the preset presentation format.

[0092] The data processing method provided by the embodiment of the present invention is that the target neural network model can gradually update its model parameters during the process of processing the first data stream, so as to adapt to the continuous changes of the first data stream, timely capture the new change trend characteristics and timing characteristics of the first data stream, and process the input first data stream in real time to improve the adaptability of the data query process. It can not only realize efficient real-time query of the first change data, but also improve the real-time performance of the query results obtained based on data query from the database; and the query results are obtained based on the valid first change data in the preset storage area, thereby avoiding the influence of invalid first change data on the query results and improving the accuracy of the query results.

[0093] As an optional implementation, after the first electronic device reads the first change data with valid validity from the preset storage area to obtain the first data stream through step S1041, before executing step S1042, it can also perform a preset data processing operation on the first data stream to meet specific business needs; wherein the preset data processing operation may include but is not limited to: data conversion, aggregation, filtering, window function, etc.

[0094] Reference Figure 3 , shows a flow chart of the steps of a model training method provided by an embodiment of the present invention, and the method may specifically include the following steps:

[0095] Step S201, obtaining a training data set; the training data set includes a changed data sample and a query result sample corresponding to the changed data sample.

[0096] Step S202: In each round of training, the neural network model to be trained is trained using the changed data sample to obtain a prediction query result corresponding to the changed data sample.

[0097] Step S203: Determine the loss value corresponding to the neural network model to be trained according to the predicted query result and the query result sample.

[0098] Step S204: adjust the model parameters of the neural network model to be trained according to the loss value, and perform the next round of training until the loss value meets the preset conditions to obtain the target neural network model.

[0099] The model training method provided in the embodiment of the present invention can be applied to a fourth electronic device, which includes but is not limited to a smart terminal, a computer, a personal digital assistant, a tablet computer, a vehicle-mounted device, etc. The fourth electronic device can be used alone to execute the model training method provided in the embodiment of the present invention.

[0100] The training data set includes change data samples and query result samples corresponding to the change data samples. The change data samples are determined based on the historical change data and real-time change data of the master database; the query results are the query results obtained by performing SQL data query on the change data in the change data samples according to the preset query mode.

[0101] The data change log in the master database includes data change entries and timestamps corresponding to the data change entries, wherein the data change entries include change operation statements corresponding to the data change, the types of change operation statements, etc. In step S201:

[0102] First, the fourth electronic device determines the content of the data change entries in the data change log within a first time period as real-time change data based on the timestamps corresponding to the data change entries in the data change log in the main database, and determines the content of the data change entries in the data change log within a second time period as historical change data; wherein the second time period is the time period before the first time period; and the right endpoint of the first time period is the current moment when the fourth electronic device executes step S201.

[0103] Then, the fourth electronic device arranges the historical change data and the real-time change data according to the time sequence relationship between the timestamps corresponding to the data change entries, and obtains a change data sample including the historical change data and the real-time change data of the main database.

[0104] Then, the fourth electronic device performs an SQL data query on the changed data in the changed data sample according to a preset query mode, and determines the query result as the query result sample corresponding to the changed data sample.

[0105] In an embodiment of the present invention, the neural network model to be trained is an IRNN model to be trained constructed based on prior knowledge (for example, change data distribution, query mode, change data change trend, etc.).

[0106] After acquiring the training data set, in each round of training, the fourth electronic device can execute step S202 to obtain the predicted query results corresponding to the changed data samples, and execute step S203 to determine the loss value corresponding to the neural network model to be trained based on the predicted query results and the query result samples; finally, based on the loss value, the model parameters and model structure of the neural network model to be trained are optimized through the back propagation algorithm, and the next round of training is performed until the loss value meets the preset conditions, so as to obtain the target neural network model as above, so as to use the target neural network model to realize the function of adaptive streaming SQL query.

[0107] The model training method provided by the embodiment of the present invention trains the neural network model to be trained so that the neural network model to be trained can adapt to the continuous changes in the data stream composed of the change data, timely capture the change trend and timing characteristics of the change data, dynamically adjust the execution plan and optimize the query, obtain the target neural network model, and implement the function of adaptive streaming SQL query based on the target neural network model. In the master-slave replication process of the database system, the target neural network model can be used to efficiently process the first change data with valid validity in the preset storage area of ​​the first electronic device to obtain the query result, thereby realizing efficient real-time query of the change data in the master-slave replication process and improving the real-time performance of the query result obtained based on the data query from the database.

[0108] Optionally, the step S201 of acquiring a training data set includes:

[0109] Step S2011: Obtain the data change log of the main database within a preset time period.

[0110] Step S2012: Acquire second change data of the primary database within the preset time period according to the data change log.

[0111] Step S2013: Obtain the target query result corresponding to the second change data.

[0112] Step S2014: determine the second changed data as a changed data sample.

[0113] Step S2015: Determine the target query result as the query result sample corresponding to the changed data sample.

[0114] In the embodiment of the present invention, the fourth electronic device may obtain a training data set through steps S2011 to S2015.

[0115] The preset time period includes a first time period and a second time period. In step S2011, the fourth electronic device may obtain the data change log of the main database within the preset time period according to the timestamp corresponding to the data change entry in the data change log of the main database.

[0116] In step S2012, first, the fourth electronic device can further determine the content of the data change entries in the data change log obtained in step S2011 within the first time period as real-time change data according to the timestamps corresponding to the data change entries in the data change log, and determine the content of the data change entries in the data change log obtained in step S2011 within the second time period as historical change data. Then, the fourth electronic device arranges the historical change data and the real-time change data according to the time sequence relationship between the timestamps corresponding to the data change entries, and obtains the second change data including the historical change data and the real-time change data of the master database.

[0117] When the fourth electronic device obtains the second change data through step S2012, it can determine the second change data as the change data sample in the training data set; when the fourth electronic device performs an SQL data query on the second change data according to a preset query mode through step S2013 and obtains the target query result corresponding to the second change data, it can determine the target query result as the query result sample corresponding to the change data sample in the training data set.

[0118] Optionally, the changed data sample includes a changed data sub-sample, and the query result sample includes a query result sub-sample corresponding to the changed data sub-sample. In step S202, in each round of training, the neural network model to be trained is trained using the changed data sample to obtain a predicted query result corresponding to the changed data sample, including:

[0119] The neural network model to be trained is trained using a first changed data subsample in the changed data sample to obtain a first prediction query result corresponding to the first changed data subsample.

[0120] The first changed data sub-sample is any changed data sub-sample in the changed data samples.

[0121] The step S203 of determining the loss value corresponding to the neural network model to be trained according to the predicted query result and the query result sample includes:

[0122] A loss value corresponding to the neural network model to be trained is determined according to the first predicted query result and the first query result subsample corresponding to the first changed data subsample.

[0123] The first query result sub-sample is a query result sub-sample in the query result sample that corresponds to the first changed data sub-sample.

[0124] Step S204 adjusts the model parameters of the neural network model to be trained according to the loss value, and performs the next round of training until the loss value meets the preset conditions to obtain the target neural network model, including:

[0125] The model parameters of the neural network model to be trained are adjusted according to the loss value, and the next round of training is performed using the second changed data subsample in the changed data sample until the loss value meets the preset conditions to obtain the target neural network model.

[0126] The second changed data sub-sample is a changed data sub-sample in the changed data sample except the first changed data sub-sample.

[0127] In an embodiment of the present invention, the changed data samples in the training data set are divided into several batches of changed data sub-samples, and the query result samples in the training data set are divided into query result sub-samples corresponding to the changed data sub-samples, and the neural network model to be trained is incrementally iteratively trained. Each round of iterative training only updates part of the model parameters of the neural network model to be trained. Through multiple rounds of iterative training, the model parameters of the neural network model to be trained can be gradually updated to adapt to the continuous changes of the changed data, reduce the model calculation burden, and improve the model training efficiency.

[0128] After the model training and optimization are completed, the function of adaptive streaming SQL query is implemented based on the target neural network model. During the master-slave replication process of the database system, the target neural network model can be used to efficiently process the first change data with valid validity in the preset storage area of ​​the first electronic device to obtain the query result, thereby realizing efficient real-time query of the changed data during the master-slave replication process and improving the real-time performance of the query results obtained based on data query from the database.

[0129] As an example, the database system includes a first electronic device, a second electronic device where a master database is located, and a third electronic device where a slave database is located, wherein the number of the third electronic devices is the same as the number of the slave databases and is at least 1; the first electronic device is an electronic device independent of the second electronic device and the third electronic device. The first electronic device includes a kafka message management platform and a Debezium plug-in, and the message queue in the kafka message management platform is used as a preset storage area.

[0130] Reference Figure 4 , shows a flowchart of another data processing method provided by an embodiment of the present invention, the method comprising steps S301 to S308:

[0131] Step S301, configure master-slave replication and enable binlog.

[0132] Specifically, master-slave replication is configured in the database system and the binlog function is enabled in the master data to record data change logs.

[0133] Step S302: Debezium connects to the main database.

[0134] Specifically, the Debezium plug-in is connected to the master data to continuously monitor the data change log in the master database using the Debezium plug-in.

[0135] Step S303: monitor data changes and send the first changed data to Kafka.

[0136] Specifically, when the first electronic device monitors the data change log for update using the Debezium plug-in, it obtains the first change data from the data change log and transmits the first change data to the message queue in the Kafka message management platform.

[0137] Step S304: Determine the validity of the first change data.

[0138] Specifically, the first electronic device obtains state information of the first changed data in the main database, and determines the validity of the first changed data in the preset storage area according to the state information.

[0139] Step S305: IRNN model training.

[0140] Specifically, before step S306, the fourth electronic device may execute operations corresponding to steps S201 to S204 to perform model training on the IRNN model to be trained to obtain a target IRNN model.

[0141] Step S306: the first data stream of the valid first changed data in Kafka.

[0142] Specifically, when it is necessary to query or process the first data stream consisting of valid first change data in the Kafka message management platform, the first electronic device can read the valid first change data in the Kafka message management platform in real time through the streaming processing engine to obtain the first data stream.

[0143] Step S307: Obtain real-time query results or real-time data processing results.

[0144] Specifically, the first electronic device executes operations corresponding to steps S1042 to S1045 to obtain query results corresponding to the first data stream; and / or the first electronic device performs preset data processing operations on the first data stream to obtain processing results to meet specific business needs.

[0145] Step S308: Write the first changed data into the slave database.

[0146] Specifically, the first electronic device writes the first change data with valid validity in the preset storage area into the relay log of the slave database corresponding to the master database, so that the slave database can parse the first change data in the relay log into specific operations and execute them serially to achieve data synchronization between the master and slave databases.

[0147] In summary, the data processing method provided by the embodiment of the present invention obtains the first change data of the master database when monitoring the data change of the master database, and stores the first change data in the preset storage area. Even if the master database fails or other failures occur, the slave database can still achieve data synchronization based on the first change data in the preset storage area, avoiding the problem of data loss in the master-slave replication process, and improving the reliability and security of the first change data; further, the embodiment of the present invention also determines the validity of the first change data according to the status information of the first change data, and when data query is required, uses the target neural network model to process the first change data with valid validity in the preset storage area to obtain the query result, and implements adaptive streaming query based on the target neural network model, which can not only realize efficient real-time query of the first change data, but also the query result is obtained based on the valid first change data in the preset storage area, avoiding the influence of invalid first change data on the query result, and improving the accuracy of the query result; the embodiment of the present invention also writes the first change data with valid validity in the preset storage area into the slave database corresponding to the master database, so that the slave database only performs data synchronization based on the valid first change data in the preset storage area, avoiding the introduction of invalid first change data into the slave database, and improving the consistency of data between the master and slave databases.

[0148] Device Embodiment

[0149] Reference Figure 5 , shows a logic block diagram of a data processing device provided by an embodiment of the present invention, and the device may include:

[0150] The first acquisition module 501 is used to acquire first changed data of the master database when monitoring data changes in the master database, and store the first changed data in a preset storage area;

[0151] A second acquisition module 502 is used to acquire the state information of the first change data;

[0152] A first determining module 503, configured to determine the validity of the first change data according to the state information;

[0153] A query module 504 is used to process the first change data with valid validity in the preset storage area by using the target neural network model to obtain a query result;

[0154] The writing module 505 is used to write the first change data with valid validity in the preset storage area into the slave database corresponding to the master database.

[0155] Optionally, the state information includes a transaction identifier and a timestamp; and the first determining module includes:

[0156] A first determination submodule, configured to determine, according to the transaction identifier, a target transaction corresponding to the first change data;

[0157] A second determination submodule, configured to determine a commit status of the target transaction according to the timestamp;

[0158] A third determination submodule, configured to determine, when the submission status is submitted, that the validity of the first change data is valid;

[0159] The fourth determining submodule is used to determine that the validity of the first change data is invalid when the submission status is unsubmitted.

[0160] Optionally, the query module includes:

[0161] A reading submodule, configured to read first change data with valid validity from the preset storage area to obtain a first data stream;

[0162] An input submodule, used for inputting the first data stream into a target neural network model;

[0163] An extraction submodule, used for extracting the time series characteristics and data change trend characteristics of the first data stream by using the target neural network model;

[0164] An output submodule, configured to output an initial processing result corresponding to the first data stream according to the time series characteristics and the data change trend characteristics using the target neural network model;

[0165] The first acquisition submodule is used to obtain the initial processing result output by the target neural network model and generate a query result based on the initial processing result.

[0166] Reference Figure 6 , shows a logic block diagram of a model training device provided by an embodiment of the present invention, and the device may include:

[0167] The third acquisition module 601 is used to acquire a training data set; the training data set includes a changed data sample and a query result sample corresponding to the changed data sample;

[0168] A training module 602 is used to train the neural network model to be trained using the changed data sample in each round of training to obtain a prediction query result corresponding to the changed data sample;

[0169] A second determination module 603 is used to determine the loss value corresponding to the neural network model to be trained according to the predicted query result and the query result sample;

[0170] The training module 602 is also used to adjust the model parameters of the neural network model to be trained according to the loss value, and perform the next round of training until the loss value meets the preset conditions to obtain the target neural network model as described above.

[0171] Optionally, the third acquisition module includes:

[0172] The second acquisition submodule is used to obtain the data change log of the main database within a preset time period;

[0173] A third acquisition submodule is used to acquire second change data of the main database within the preset time period according to the data change log;

[0174] A fourth acquisition submodule, used to acquire a target query result corresponding to the second change data;

[0175] a fifth determining submodule, configured to determine the second changed data as a changed data sample;

[0176] The sixth determination submodule is used to determine the target query result as the query result sample corresponding to the changed data sample.

[0177] Optionally, the changed data sample includes a changed data sub-sample, and the query result sample includes a query result sub-sample corresponding to the changed data sub-sample; and the training module includes:

[0178] A first training submodule is configured to train the neural network model to be trained using a first changed data subsample in the changed data samples to obtain a first prediction query result corresponding to the first changed data subsample; the first changed data subsample is any changed data subsample in the changed data samples;

[0179] The second determining module includes:

[0180] a seventh determination submodule, configured to determine a loss value corresponding to the neural network model to be trained according to the first predicted query result and a first query result subsample corresponding to the first changed data subsample; the first query result subsample is a query result subsample in the query result sample corresponding to the first changed data subsample;

[0181] The training submodule further includes:

[0182] The second training submodule is used to adjust the model parameters of the neural network model to be trained according to the loss value, and use the second change data subsample in the change data sample to perform the next round of training until the loss value meets the preset conditions to obtain the target neural network model as described above; the second change data subsample is the change data subsample in the change data sample except the first change data subsample.

[0183] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0184] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0185] Reference Figure 7 , is a structural block diagram of an electronic device for data processing or model training provided by an embodiment of the present invention. Figure 7 As shown, the electronic device includes: a processor, a memory, a communication interface and a communication bus, and the processor, the memory and the communication interface communicate with each other through the communication bus; the memory is used to store executable instructions, and the executable instructions enable the processor to execute the data processing method of the aforementioned embodiment, or execute the model training method of the aforementioned embodiment.

[0186] The processor may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other editable devices, transistor logic devices, hardware components or any combination thereof. The processor may also be a combination that implements a computing function, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0187] The communication bus may include a path for transmitting information between the memory and the communication interface. The communication bus may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The communication bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 7 The fact that only one line is used in the diagram does not mean that there is only one bus or only one type of bus.

[0188] The memory may be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable ReadOnly), a CD-ROM (Compact Disa Read Only), a magnetic tape, a floppy disk, an optical data storage device, etc.

[0189] The embodiment of the present invention also provides a non-temporary computer-readable storage medium, when the instructions in the storage medium are executed by a processor of an electronic device (server or terminal), the processor is able to execute Figure 1 The data processing method shown, or execute Figure 3 The model training method shown.

[0190] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0191] It will be appreciated by those skilled in the art that the embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the embodiments of the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0192] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0193] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing terminal device to operate in a predictable manner, so that the instructions stored in the computer readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0194] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0195] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0196] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0197] The data processing method, model training method, device, equipment and storage medium provided by the present invention are introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A data processing method, characterized in that: The method comprises: When a data change in the master database is detected, first change data of the master database is acquired, and the first change data is stored in a preset storage area; Acquire status information of the first change data; determining the validity of the first change data according to the status information; Using the target neural network model, the first change data with valid validity in the preset storage area is processed to obtain a query result; The first changed data with valid validity in the preset storage area is written into the slave database corresponding to the master database.

2. The method according to claim 1, characterized in that The state information includes a transaction identifier and a timestamp; The determining the validity of the first change data according to the state information includes: Determining, according to the transaction identifier, a target transaction corresponding to the first change data; Determining the commit status of the target transaction according to the timestamp; When the submission status is submitted, determining the validity of the first change data is valid; When the submission status is unsubmitted, the validity of the first change data is determined to be invalid.

3. The method according to claim 1, characterized in that The using the target neural network model to process the first change data with valid validity in the preset storage area to obtain the query result includes: Reading first change data with valid validity from the preset storage area to obtain a first data stream; Inputting the first data stream into a target neural network model; Extracting the time series characteristics and data change trend characteristics of the first data stream by using the target neural network model; Outputting an initial processing result corresponding to the first data stream using the target neural network model according to the time series characteristics and the data change trend characteristics; Obtain an initial processing result output by the target neural network model, and generate a query result based on the initial processing result.

4. A model training method, characterized in that: The method comprises: Acquire a training data set; the training data set includes a changed data sample and a query result sample corresponding to the changed data sample; In each round of training, the neural network model to be trained is trained using the changed data sample to obtain the prediction query result corresponding to the changed data sample; Determine the loss value corresponding to the neural network model to be trained according to the predicted query result and the query result sample; The model parameters of the neural network model to be trained are adjusted according to the loss value, and the next round of training is performed until the loss value meets the preset conditions, thereby obtaining the target neural network model as described in any one of claims 1 to 3.

5. The method according to claim 4, characterized in that The step of obtaining a training data set includes: Obtain the data change log of the master database within a preset time period; According to the data change log, obtaining second change data of the primary database within the preset time period; Obtaining a target query result corresponding to the second change data; determining the second change data as a change data sample; The target query result is determined as the query result sample corresponding to the changed data sample.

6. The method according to claim 4, characterized in that The changed data sample includes a changed data sub-sample, and the query result sample includes a query result sub-sample corresponding to the changed data sub-sample; In each round of training, the neural network model to be trained is trained using the changed data sample to obtain the prediction query result corresponding to the changed data sample, including: Using a first changed data subsample in the changed data samples to train the neural network model to be trained, to obtain a first prediction query result corresponding to the first changed data subsample; the first changed data subsample is any changed data subsample in the changed data samples; The step of determining the loss value corresponding to the neural network model to be trained according to the predicted query result and the query result sample includes: Determine a loss value corresponding to the neural network model to be trained according to the first predicted query result and a first query result subsample corresponding to the first changed data subsample; the first query result subsample is a query result subsample in the query result sample corresponding to the first changed data subsample; The step of adjusting the model parameters of the neural network model to be trained according to the loss value and performing the next round of training until the loss value meets a preset condition to obtain the target neural network model according to any one of claims 1 to 3 comprises: The model parameters of the neural network model to be trained are adjusted according to the loss value, and the next round of training is carried out using the second change data subsample in the change data sample until the loss value meets the preset conditions, thereby obtaining the target neural network model as described in any one of claims 1 to 3; the second change data subsample is the change data subsample in the change data sample except the first change data subsample.

7. A data processing device, characterized in that: The device comprises: A first acquisition module is used to acquire first changed data of the master database when monitoring data changes in the master database, and store the first changed data in a preset storage area; A second acquisition module, used to acquire status information of the first change data; A first determining module, configured to determine the validity of the first change data according to the state information; A query module, used for processing the first change data with valid validity in the preset storage area by using the target neural network model to obtain a query result; The writing module is used to write the first change data with valid validity in the preset storage area into the slave database corresponding to the master database.

8. A model training device, characterized in that: The device comprises: A third acquisition module is used to acquire a training data set; the training data set includes a changed data sample and a query result sample corresponding to the changed data sample; A training module, used to train the neural network model to be trained using the changed data sample in each round of training to obtain the prediction query result corresponding to the changed data sample; A second determination module, used to determine the loss value corresponding to the neural network model to be trained according to the predicted query result and the query result sample; The training module is also used to adjust the model parameters of the neural network model to be trained according to the loss value, and perform the next round of training until the loss value meets the preset conditions, thereby obtaining the target neural network model as described in any one of claims 1 to 3.

9. An electronic device, characterized in that: include: The electronic device comprises a processor, a memory, a communication interface and a communication bus, and the processor, the memory and the communication interface communicate with each other via the communication bus; The memory is used to store executable instructions, and the executable instructions enable the processor to execute the data processing method as described in any one of claims 1 to 3, or execute the model training method as described in any one of claims 4 to 6.

10. A readable storage medium, characterized in that: When the instructions in the readable storage medium are executed by a processor of an electronic device, the processor is enabled to execute the data processing method as described in any one of claims 1 to 3, or to execute the model training method as described in any one of claims 4 to 6.

Images