A method and device for analyzing user behavior based on underlying data, and related media

Abstract

The application discloses a kind of based on bottom layer data analysis user behavior method, device and related medium, the method includes: data record is generated when transaction is executed, and the data record is marked, obtains global transaction ID in mysql database;The data record and global transaction ID are stored to time series database;Log file in mysql database is listened to, and corresponding log event and object information are generated to the log file by data flow conversion service, then the log event and object information are stored to time series database;The storage data in the time series database is read regularly by data analysis service, and operation log data is obtained from the storage data according to rule engine;The operation log data is stored to time series database, and operation log data in the time series database is inquired and shown.The application realizes the analysis user behavior while reducing business code coupling, guarantees business system performance.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer software, and particularly relates to a method and device for analyzing user behavior based on underlying data and related media. BACKGROUND

[0002] In the prior art, there are generally two methods for collecting real user operation business system behavior operation logs:

[0003] 1. A large number of interceptors are added in the routing layer to intercept preRequest and postRequest of most interfaces, and then the operations are parsed into user operation logs;

[0004] 2. An interceptor is added in the model layer to parse the operations of adding, deleting and modifying.

[0005] However, both of the two methods have a large invasion to the original business code and cause a decline in overall performance. SUMMARY

[0006] Embodiments of the present application provide a method and device for analyzing user behavior based on underlying data, a computer device and a storage medium, aiming to analyze user behavior while reducing business code coupling and ensuring business system performance.

[0007] In a first aspect, the embodiments of the present application provide a method for analyzing user behavior based on underlying data, comprising:

[0008] In response to user operations on a business system, data records are generated during transaction execution in a mysql database, and the data records are marked;

[0009] After transaction execution, a global transaction ID in the mysql database is obtained;

[0010] The data records and the global transaction ID are stored in a time series database;

[0011] Log files in the mysql database are listened to, and corresponding log events and object information are generated for the log files through a data flow service, and then the log events and object information are stored in the time series database;

[0012] The stored data in the time series database is read at a fixed time through a data analysis service, and operation log data about user operations on the business system is parsed from the stored data according to a rule engine; wherein the stored data includes the data records, the global transaction ID, the log events and the object information;

[0013] The operation log data is stored in the time series database.

[0014] In response to the operation of the user on the business system, the operation log data in the time series database is queried and displayed.

[0015] In a second aspect, an embodiment of the present application provides a device for analyzing user behavior based on underlying data, comprising:

[0016] A response unit is configured to, in response to the operation of the user on the business system, generate a data record during transaction execution in the MySQL database, and mark the data record.

[0017] A transaction acquisition unit is configured to acquire a global transaction ID in the MySQL database after transaction execution.

[0018] A first storage unit is configured to store the data record and the global transaction ID into a time series database.

[0019] A listening unit is configured to listen to a log file in the MySQL database, generate corresponding log events and object information for the log file through a data stream conversion service, and then store the log events and object information into the time series database.

[0020] An analysis unit is configured to perform timed reading on stored data in the time series database through a data analysis service, and analyze operation log data about the operation of the user on the business system from the stored data according to a rule engine, wherein the stored data comprises the data record, the global transaction ID, the log events and the object information.

[0021] A second storage unit is configured to store the operation log data into the time series database.

[0022] A query unit is configured to, in response to the operation of the user on the business system, query and display the operation log data in the time series database.

[0023] In a third aspect, an embodiment of the present application provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program capable of running on the processor, and the processor implements the method for analyzing user behavior based on underlying data according to the first aspect when executing the computer program.

[0024] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the method for analyzing user behavior based on underlying data according to the first aspect.

[0025] The embodiment of the present application provides a method, device, computer equipment and storage medium for analyzing user behavior based on underlying data, which comprises the following steps: in response to user operation on a business system, generating a data record during transaction execution of a mysql database and marking the data record; obtaining a global transaction ID in the mysql database after transaction execution; storing the data record and the global transaction ID into a time series database; listening to a log file in the mysql database and generating corresponding log events and object information through a data flow service, and then storing the log events and object information into the time series database; reading stored data in the time series database through a data analysis service, and analyzing operation log data of the user on the business system from the stored data according to a rule engine; wherein the stored data comprises the data record, the global transaction ID, the log events and the object information; storing the operation log data into the time series database; and in response to user operation on the business system, querying and displaying the operation log data in the time series database. The embodiment of the present application realizes the analysis of user behavior while reducing business code coupling and ensuring the performance of the business system by analyzing and storing underlying data of the mysql database. BRIEF DESCRIPTION OF DRAWINGS

[0026] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiment description. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0027] Figure 1 A flowchart of a method for analyzing user behavior based on underlying data provided by the embodiment of the present application;

[0028] Figure 2 A network flowchart of a method for analyzing user behavior based on underlying data provided by the embodiment of the present application;

[0029] Figure 3 A schematic block diagram of a device for analyzing user behavior based on underlying data provided by the embodiment of the present application. DETAILED DESCRIPTION

[0030] The technical solutions in the embodiments of the present application will be described clearly and completely in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are some embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the protection scope of the present application.

[0031] It should be understood that the terms "comprises" and "comprising," when used in this specification and the following claims, indicate the presence of the described features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0032] It should also be understood that the terms used in the specification and the following claims are intended to describe particular embodiments and do not intend to limit the present application. As used in the specification and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.

[0033] It should further be understood that the term "and / or" used in the specification and the following claims means one or more of the associated listed items as well as all possible combinations of the items and includes the combinations.

[0034] Please see the following Figure 1 , Figure 1 A flowchart of a method for analyzing user behavior based on underlying data is provided for an embodiment of the present application. The method is applied to a server or a service terminal, and specifically includes steps S101-S105.

[0035] S101, in response to user operation on a business system, data records are generated during transaction execution in a mysql database, and the data records are marked;

[0036] S102, after transaction execution, global transaction IDs in the mysql database are obtained;

[0037] S103, the data records and the global transaction IDs are stored in a time series database;

[0038] S104, log files in the mysql database are listened to, and corresponding log events and object information are generated for the log files through a data stream service, and then the log events and object information are stored in the time series database;

[0039] S105, the stored data in the time series database is read at a fixed time through a data analysis service, and operation log data about the user operation on the business system is parsed from the stored data according to a rule engine; wherein the stored data includes the data records, global transaction IDs, log events and object information;

[0040] S106, the operation log data is stored in the time series database;

[0041] S107, in response to the user's operation on the business system, query and display the operation log data in the time series database.

[0042] In this embodiment, in combination with Figure 2 When the operation is performed on the business system, the mysql database generates a data record, records the data record, obtains a global transaction ID after the transaction execution of the mysql database, and then writes the data record and the global transaction ID into the time series database. Subsequently, the log file in the mysql database is listened to and transferred through the data flow service to generate corresponding log events and object information, and the log events and object information are also stored into the time series database. Then, the data stored in the time series database is read by the data analysis service at regular intervals, and is analyzed according to the rule engine to obtain the operation log data of the user. When the user subsequently operates the business system, the operation log data corresponding to the user can be queried from the time series database by the business system, and is displayed to the user.

[0043] The embodiment realizes the analysis of user behavior while reducing the coupling of business code and ensuring the performance of the business system by analyzing and storing the underlying data of the mysql database.

[0044] In an embodiment, when the data record is generated during the transaction execution of the mysql database and the data record is marked, a transaction identifier can be generated by the mysql database, and when the transaction is executed, the transaction identifier is used to establish a mapping relationship with the data record to realize the marking of the data record.

[0045] In an embodiment, the step S102 comprises:

[0046] After the transaction is executed, the global transaction ID is obtained from the ok package in the mysql database by using the reflection technology.

[0047] In the embodiment, in the business system, when a user operates the business system, most of the time, data records are generated from the mysql database, and the data records are specifically generated through transactions of the mysql database. The transactions are marked when they are executed, and the marking content is: organization / team, user, marking the data records can accurately identify in the subsequent data analysis service that some operations are generated by which team and which user. The GTID (Global Transaction ID) in the mysql database is obtained after the transaction is executed. Since there is no way to directly obtain the GTID in the mysql driver of the go language, the code of the mysql driver is adjusted, and the GTID is obtained from the ok package of the mysql database by using the reflection technology. The GTID is obtained in the transaction by using the reflection, which can reduce the understanding of the underlying of the developer, and only the business layer needs to be concerned.

[0048] After the transaction is executed, the data (i.e., the data records and the global transaction ID) is written into the mysql database, and the operator information (including organization / team, user, time, etc.) of the transaction and the GTID are recorded into the pgsql database (i.e., the time series database). Pgsql (PostgreSQL) is an object relational database management system (ORDBMS), which is mainly used as a time series database in the embodiment. It should be noted that during the interaction with the mysql database, when a command is completed, the ok package is returned by the mysql database.

[0049] In a specific embodiment, in the step of obtaining the global transaction ID from the ok package of the mysql database by using the reflection technology after the transaction is executed, the following steps are included:

[0050] A mapping relationship between the global transaction IDs of different systems is established in the mysql database based on the elastic configuration mode;

[0051] A formatting tool written by using getter and setter methods is adopted, and the global transaction ID in the mysql database is formatted by using the formatting tool, so as to meet the getter and setter method calls;

[0052] The global transaction ID is automatically identified and automatically converted;

[0053] According to the mapping relationship, the global transaction ID is extracted from the mysql database by using the reflection technology.

[0054] In an embodiment, the step S104 includes:

[0055] The canal technology is used to simulate the master-slave synchronization of the MySQL database to stream the binlog data in the MySQL database to the data streaming service, wherein the binlog data is the log file.

[0056] Further, the canal technology is used to simulate the master-slave synchronization of the MySQL database to stream the binlog data in the MySQL database to the data streaming service, wherein the binlog data is the log file, including:

[0057] The binlog data is streamed to the data streaming service in a streaming manner.

[0058] In the data streaming service binlog-event-sync, the canal technology is used to listen to the binlog data in the MySQL database, and the listening process actually simulates the master-slave synchronization of the MySQL database to stream the binlog data in the MySQL database to the target in the binlog_format=ROW manner.

[0059] The data streaming service binlog-event-sync based on the canal technology is developed to stream the binlog data in the MySQL database to the pgsql database, and in addition to the common data streaming, the service additionally provides the pre-processing data generated according to the configuration file.

[0060] In addition, the canal is a technology for streaming the binlog data in the MySQL database to the target, which is mainly used for the incremental log analysis of the MySQL database to provide the incremental data subscription and consumption. The binlog data is the log file in the binary format of the MySQL database. The binlog is used to record the changes of the database in the MySQL (only the modification operation of the data), which is mainly used for the master-slave replication and incremental recovery of the database.

[0061] In a specific embodiment, the canal technology is used to simulate the master-slave synchronization of the MySQL database to stream the binlog data in the MySQL database to the data streaming service, and further includes:

[0062] The canal cluster is used to acquire the binlog data in the MySQL database in real time.

[0063] The table association strategy pre-stored in the canal cluster is used to construct the target table.

[0064] writing the target table into a data flow transfer service.

[0065] The table association strategy is specifically: if two data tables in the mysql database are modified in the same transaction, the newly added binlog data in the two data tables is associated in the canal cluster, and the target table is written into the data flow transfer service; if there is one data table in the mysql database with newly added binlog data, the fields associated with the binlog data are obtained through the data flow transfer service, and the target table is written into the data flow transfer service. In addition, the canal cluster adopts a key-value mode, and the data of the associated fields is used as the key, and the corresponding value is matched.

[0066] In an embodiment, the step S104 further includes:

[0067] The binlog data is pre-processed by a configuration file in the data flow transfer service to generate corresponding log events;

[0068] The data type of the binlog data is determined, and whether the binlog data is directly serialized into the data structure of the time series database and synchronized to the time series database is determined according to the determination result.

[0069] In this embodiment, when the data flow transfer service binlog-event-sync receives binlog data, there are two processes of dump and stream to process the received data, so as to ensure that the source data of the transaction operation is not lost. The specific process is as follows:

[0070] dump: refers to when the binlog-event-sync service is just started, actually receiving the dump mysql full data, at this time, the data will be pre-processed according to the configuration file defined in the data flow transfer service, and the corresponding <uuid:name>data of the user and some attributes of <alias:name>The data is used to resolve some corresponding names when parsing the log data.

[0071] For example, the obj_mapping and field_setting are defined in the service, and the format is similar to:

[0072]

[0073] According to the above format, the corresponding <uuid:name>and <alias:name>Data.

[0074] stream: judging the type of binlog data. When judging that the type of binlog data is not GTID or XID, it can be considered that the data is the mysql data generated by the user using the business system at that time, at this time, the binlog data can be directly serialized into the data structure defined in the pgsql database, and synchronized to the pgsql, so as to ensure that the data will not be lost.

[0075] In an embodiment, the step S105 comprises:

[0076] According to the characteristics of the underlying data structure, a log is written to generate a rule engine method set;

[0077] The storage data is divided by using the rule engine method set, and the corresponding operation log data is generated.

[0078] Further, the log is written according to the characteristics of the underlying data structure to generate a rule engine method set, comprising:

[0079] The Action interface is set to record or access the global transaction ID;

[0080] The DbHint interface and the TablesHint interface are set to monitor the mysql database and / or the time series database;

[0081] The Verify interface is set to verify each data in the parsing process;

[0082] The BuildLogs interface is set to generate storage data of a predetermined structure according to the verification data provided by the Verify interface, and store the storage data into the time series database for the business system to query.

[0083] In this embodiment, the data parsing service reads the binlog data in the pgsql database, the pre-parsing data (such as the log event, object information) and the user operation information at regular intervals, and then parses the operation log data of the real user operating the business system through the rule engine, so that new business rules can be quickly added without affecting the original log generation.

[0084] The rule engine according to the characteristics of the underlying data structure generates a method set of the log in this embodiment. After the data stream comes in, it is judged by the database, the data table and the feature verification which rule to enter, and after entering the rule, the log data is generated according to the unified way in the rule.

[0085] The following interfaces need to be implemented when writing rules:

[0086]

[0087]

[0088] Action(): represents the ID of the operation, for example, the global transaction ID;

[0089] DbHint(): represents the database that needs to be monitored, and the pre-checking is performed to speed up the parsing speed. For example, the operation of user A is operated in the business system B, and the database B of the business system needs to be filled in.

[0090] TablesHint(): represents the data table that needs to be monitored, for example, the mysql database and the time series database, and the pre-checking is performed to speed up the parsing speed. For example, user A invites user B to join the team, and the action can add a data in the user table, and therefore the user table needs to be filled in.

[0091] Verify(): represents the specific verification rule method. If only the DbHint and TablesHint are not enough to achieve the corresponding effect, because different operations may operate the same data table, and therefore the more specific field is needed to judge. In addition, the step also generates some necessary data, such as the user UUID.

[0092] BuildLogs(): represents that the data of the predetermined structure is generated according to the data provided by Verify, and is stored in the pgsql database for the business system to query and show to the user.

[0093] Figure 3 A schematic block diagram of an apparatus 300 for analyzing user behavior based on underlying data provided by an embodiment of the application is provided, and the apparatus 300 comprises:

[0094] A response unit 301 is configured to respond to the operation of the user on the business system, generate a data record during the transaction execution of the mysql database, and mark the data record;

[0095] A transaction acquisition unit 302 is configured to acquire the global transaction ID in the mysql database after the transaction execution; and a first storage unit 303 is configured to store the data record and the global transaction ID into the time series database;

[0096] A monitoring unit 304 is configured to monitor the log file in the mysql database, and generate corresponding log events and object information through the data flow service, and then store the log events and object information into the time series database;

[0097] The analysis unit 305 is configured to perform timed reading on the stored data in the time series database by a data analysis service, and parse operation log data about a user on a business system from the stored data according to a rule engine; wherein the stored data includes the data record, the global transaction ID, the log event and the object information.

[0098] The second storage unit 306 is configured to store the operation log data into the time series database.

[0099] The query unit 307 is configured to query and display the operation log data in the time series database in response to the operation of the user on the business system.

[0100] In the embodiment, the operation log data is obtained by analyzing the data in the mysql database. Figure 2 When the business system is operated, the mysql database generates a data record, the data record is recorded, a global transaction ID is obtained after the transaction execution of the mysql database, and then the data record and the global transaction ID are written into the time series database. Subsequently, the log file in the mysql database is listened to and transferred by a data transfer service to generate corresponding log events and object information, and the log events and object information are also stored into the time series database. Then, the data stored in the time series database is read by a data analysis service at a fixed time, and is parsed according to a rule engine to obtain the operation log data of the user. When the user subsequently operates the business system, the operation log data corresponding to the user can be queried from the time series database by the business system, and is displayed to the user.

[0101] The embodiment realizes the parsing of the user behavior, reduces the business code coupling, and guarantees the performance of the business system by analyzing and storing the underlying data of the mysql database.

[0102] In an embodiment, when the data record is generated during the transaction execution of the mysql database and the data record is marked, the transaction identifier of the mysql database can be generated, and the mapping relationship between the transaction identifier and the data record is established when the transaction is executed, so as to realize the marking of the data record.

[0103] In an embodiment, the transaction obtaining unit 302 includes:

[0104] The reflection obtaining unit is configured to obtain the global transaction ID from the ok package in the mysql database by using the reflection technology after the transaction execution.

[0105] In the embodiment, in the business system, when a user operates the business system, most of the time, data records are generated from the mysql database, and the data records are specifically generated through transactions of the mysql database. The transactions are marked when they are executed, and the marking content is: organization / team, user. The data records can accurately identify in the subsequent data analysis service that some operations are generated by which team and which user. The GTID (Global Transaction ID) in the mysql database is obtained after the transaction is executed. Since there is no way to directly obtain the GTID in the mysql driver of the go language, the code of the mysql driver is adjusted, and the GTID is obtained from the ok package of the mysql database by using the reflection technology. The GTID is obtained in the transaction by using the reflection, which can reduce the understanding of the underlying of the developer, and only the business layer needs to be concerned.

[0106] After the transaction is executed, the data (i.e., the data records and the global transaction ID) is written into the mysql database, and the operator information (including organization / team, user, time, etc.) of the transaction and the GTID are recorded into the pgsql database (i.e., the time series database). Pgsql (PostgreSQL) is an object relational database management system (ORDBMS), which is mainly used as a time series database in the embodiment.

[0107] In a specific embodiment, in the step of obtaining the global transaction ID from the ok package of the mysql database by using the reflection technology after the transaction is executed, the following steps are included:

[0108] A mapping relationship of the global transaction ID between different systems is established in the mysql database based on an elastic configuration mode;

[0109] A formatting tool written by using getter and setter methods is adopted, and the global transaction ID in the mysql database is formatted by using the formatting tool, so as to meet the getter and setter method calls;

[0110] The global transaction ID is automatically identified and automatically converted;

[0111] According to the mapping relationship, the global transaction ID is extracted from the mysql database by using the reflection technology.

[0112] In an embodiment, the listening unit 304 includes:

[0113] The first data flow transfer unit is configured to simulate the manner of master-slave synchronization of the MySQL database to transfer binlog data in the MySQL database to the data flow transfer service by using the canal technology, wherein the binlog data is the log file.

[0114] In an embodiment, the first data flow transfer unit comprises:

[0115] The second data flow transfer unit is configured to transfer the binlog data to the data flow transfer service in a streaming manner.

[0116] In the data flow transfer service binlog-event-sync, the binlog data in the MySQL database is listened to by using the canal technology. The listening process actually simulates the manner of master-slave synchronization of the MySQL database, so that the binlog data in the MySQL database is transferred to the target in the manner of binlog_format=ROW.

[0117] The data flow transfer service binlog-event-sync disclosed in the embodiment is a service developed on the basis of the canal technology, and is configured to transfer the binlog data of the MySQL database to the pgsql database. In addition to the common data transfer, the service additionally provides pre-processing data generated according to a configuration file.

[0118] In addition, the canal is a technology for transferring the binlog data in the MySQL database to the target in a streaming manner. The canal is mainly used for incremental log analysis of the MySQL database, and provides incremental data subscription and consumption. The binlog data is a log file in the binary format of the MySQL database. The binlog is used to record the changes of the database in the MySQL (only the modification operation of the data is recorded), and is mainly used for master-slave replication and incremental recovery of the database.

[0119] In a specific embodiment, the canal technology is used to simulate the manner of master-slave synchronization of the MySQL database to transfer the binlog data in the MySQL database to the data flow transfer service, and further comprises:

[0120] The binlog data in the MySQL database is acquired in real time by using the canal cluster;

[0121] The target table is constructed by using the table association strategy pre-stored in the canal cluster.

[0122] The target table is written into the data flow transfer service.

[0123] The table association strategy is specifically: if two data tables in the mysql database are modified in the same transaction, the newly added binlog data in the two data tables is associated in the canal cluster, and the target table is written into the data flow service; if there is a data table in the mysql database with newly added binlog data, the fields associated with the binlog data are obtained through the data flow service, and the target table is written into the data flow service. In addition, the canal cluster adopts a key-value mode, and the data of the associated field is used as the key, and the corresponding value is matched.

[0124] In an embodiment, the monitoring unit 304 further includes:

[0125] The parsing processing unit is configured to perform pre-parsing processing on the binlog data through a configuration file in the data flow service to generate corresponding log events.

[0126] The type judgment unit is configured to judge the data type of the binlog data, and according to the judgment result, to confirm whether to directly serialize the binlog data into the data structure of the time series database and synchronize to the time series database.

[0127] In this embodiment, when the data flow service binlog-event-sync receives binlog data, there are two processes of dump and stream to process the received data, so as to ensure that the source data of the transaction operation is not lost. The specific process is as follows:

[0128] dump: refers to when the binlog-event-sync service is just started, actually receiving the dump mysql full data, at this time, the data will be pre-processed according to the configuration file defined in the data flow service, and the corresponding <uuid:name>data of the user and some attributes of <alias:name>The data is used to resolve the log data and associate some corresponding names, etc.

[0129] For example, the obj_mapping and field_setting are defined in the service, and the format is similar to:

[0130]

[0131]

[0132] According to the above format, the corresponding <uuid:name>and <alias:name>data.

[0133] stream: Determines the type of binlog data. If the binlog data type is not GTID or XID, it is considered to be MySQL data generated by the user when using the business system. In this case, the binlog data can be directly serialized into the data structure defined in the PGSQL database and synchronized to PGSQL to ensure data loss.

[0134] In one embodiment, the parsing unit 305 includes:

[0135] A log writing unit, used to write logs based on the characteristics of the underlying data structure to generate a set of rule engine methods;

[0136] A rule division unit is used to divide the stored data into rules using the rule engine method set and generate corresponding operation log data.

[0137] In one embodiment, the log writing unit includes:

[0138] A first interface setting unit, configured to set an Action interface to record or access a global transaction ID;

[0139] The second interface setting unit is used to set the DbHint interface and the TablesHint interface to monitor the MySQL database and / or the time series database;

[0140] The third interface setting unit is used to set the Verify interface to verify each data in the parsing process;

[0141] The fourth interface setting unit is used to set the BuildLogs interface to generate storage data of a predetermined structure according to the verification data provided by the Verify interface, and store it in the time series database for query by the business system.

[0142] In this embodiment, the data parsing service will regularly read the binlog data, pre-parsed data (such as the log events, object information) and user operation information in the pgsql database, and then use the rule engine to parse the operation log data when the real user operates the business system. In this way, new business rules can be added quickly without affecting the original log generation.

[0143] The rule engine described in this embodiment writes a set of log generation methods based on the characteristics of the underlying data structure. After the data flows in, it will determine which rule it enters through the database, data table and feature verification, and after entering the rule, it will generate log data in a unified manner within the rule.

[0144] The rule needs to implement the following interfaces:

[0145]

[0146]

[0147] Action(): indicates the ID of the operation, for example, a global transaction ID;

[0148] DbHint(): needs to pay attention to the monitored database, checks in advance, and accelerates the parsing speed. For example, a certain operation of user A is operated in business system B, and the database B of the business system needs to be filled in here.

[0149] TablesHint(): indicates the data table that needs to be monitored, for example, a mysql database and a time series database, and the data table is checked in advance to accelerate the parsing speed. For example, user A invites user B to join the team, and the action can add a data in the user table, so the user table needs to be filled in here.

[0150] Verify(): indicates a specific verification rule method. If only the preceding DbHint and TablesHint are not enough to achieve the corresponding effect, because different operations may operate the same data table, a more specific field is needed to judge. In addition, this step also generates some necessary data, such as user UUID.

[0151] BuildLogs(): indicates that the data of a predetermined structure is directly generated according to the data provided by Verify, and is stored in a pgsql database for business system query and user display.

[0152] Since the embodiments of the device part correspond to the embodiments of the method part, the embodiments of the device part are described in the description of the embodiments of the method part, and will not be described here.

[0153] The embodiment of the application further provides a computer readable storage medium, which has a computer program stored thereon, and the computer program can implement the steps provided by the above embodiment when executed. The storage medium can include: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0154] The embodiment of the application provides a method for analyzing user behavior based on underlying data, which comprises the following steps:

[0155] In response to the operation of the user on the business system, data records are generated during transaction execution in the mysql database, and the data records are marked;

[0156] After transaction execution, the global transaction ID in the mysql database is obtained;

[0157] The data records and the global transaction ID are stored in the time series database;

[0158] The log files in the mysql database are listened to, and corresponding log events and object information are generated for the log files through a data flow conversion service, and then the log events and object information are stored in the time series database;

[0159] The stored data in the time series database is read at a fixed time through a data analysis service, and operation log data about the user on the business system is parsed from the stored data according to a rule engine; wherein the stored data includes the data records, the global transaction ID, the log events and the object information;

[0160] The operation log data is stored in the time series database;

[0161] In response to the operation of the user on the business system, the operation log data in the time series database is queried and displayed.

[0162] The embodiment of the application also provides a computer device, which can include a memory and a processor, the memory has a computer program stored therein, and when the processor calls the computer program in the memory, the steps provided by the above embodiment can be realized. Of course, the computer device can also include various network interfaces, power supplies and other components.

[0163] The embodiment of the application provides a method for analyzing user behavior based on underlying data, which comprises:

[0164] In response to the operation of the user on the business system, data records are generated during transaction execution in the mysql database, and the data records are marked;

[0165] After transaction execution, the global transaction ID in the mysql database is obtained;

[0166] The data records and the global transaction ID are stored in the time series database;

[0167] The log files in the mysql database are listened to, and corresponding log events and object information are generated for the log files through a data flow conversion service, and then the log events and object information are stored in the time series database;

[0168] The stored data in the time series database is read in time by a data analysis service, and operation log data about a user's operation on a business system is parsed from the stored data according to a rule engine; wherein the stored data comprises the data record, a global transaction ID, log events and object information;

[0169] The operation log data is stored into the time series database;

[0170] In response to a user's operation on a business system, the operation log data in the time series database is queried and displayed.

[0171] The various embodiments described in the specification are described in progressive manner, and each embodiment focuses on the difference from other embodiments, and the same or similar parts between the various embodiments can be referred to each other. For the system disclosed by the embodiments, since it corresponds to the method disclosed by the embodiments, the description is relatively simple, and the related parts can be referred to the method part. It should be pointed out that for ordinary skilled in the art, without departing from the principles of the present application, the present application can be improved and modified, and these improvements and modifications also fall within the scope of protection of the claims of the present application.

[0172] It should also be noted that in this specification, relationship terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between the entities or operations. Moreover, the terms "include", "contain" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. Without further limitation, the element defined by the statement "including a" does not exclude the presence of another identical element in the process, method, article or device including the element.< / alias:name> < / uuid:name> < / alias:name> < / uuid:name> < / alias:name> < / uuid:name> < / alias:name> < / uuid:name>

Claims

1. A method for analyzing user behavior based on underlying data, characterized in that: include: In response to user operations on the business system, data records are generated when transactions are executed in the MySQL database, and the data records are marked; Get the global transaction ID in the MySQL database after the transaction is executed; Storing the data record and the global transaction ID in a time series database; Monitor the log files in the MySQL database, generate corresponding log events and object information for the log files through the data flow service, and then store the log events and object information in the time series database; The data stored in the time series database is read periodically through a data parsing service, and the user's operation log data on the business system is parsed from the stored data according to a rule engine; wherein the stored data includes the data record, global transaction ID, log event and object information; storing the operation log data in the time series database; In response to user operations on the business system, query and display the operation log data in the time series database; The data parsing service periodically reads the stored data in the time series database, and parses the stored data according to the rule engine to obtain the user's operation log data on the business system, including: Write logs based on the characteristics of the underlying data structure to generate a set of rule engine methods; Using the rule engine method set to divide the stored data into rules and generate corresponding operation log data; The log is written according to the characteristics of the underlying data structure to generate a set of rule engine methods, including: Set the Action interface to record or access the global transaction ID; Set the DbHint interface and TablesHint interface to monitor the MySQL database and / or time series database; Set up the Verify interface to verify each data in the parsing process; Set up the BuildLogs interface to generate storage data with a predetermined structure based on the verification data provided by the Verify interface, and store it in the time series database for business system queries.

2. The method for analyzing user behavior based on underlying data according to claim 1, characterized in that: Obtaining the global transaction ID in the database after the transaction is executed includes: After the transaction is executed, the global transaction ID is obtained from the ok package in the MySQL database using reflection technology.

3. The method for analyzing user behavior based on underlying data according to claim 1, characterized in that: The monitoring of the log files in the MySQL database, generating corresponding log events and object information for the log files through the data flow service, and then storing the log events and object information in the time series database includes: The master-slave synchronization mode of the MySQL database is simulated by using the canal technology to transfer the binlog data in the MySQL database to the data transfer service; wherein, the binlog data is the log file.

4. The method for analyzing user behavior based on underlying data according to claim 3, characterized in that: The method of simulating the master-slave synchronization of the MySQL database by using the canal technology is used to transfer the binlog data in the MySQL database to the data transfer service; wherein, the binlog data is the log file, including: The binlog data is transferred to the data transfer service in a streaming manner.

5. The method for analyzing user behavior based on underlying data according to claim 3, characterized in that: The method further includes monitoring the log files in the MySQL database, generating corresponding log events and object information for the log files through the data flow service, and then storing the log events and object information in the time series database: Pre-parse the binlog data using the configuration file in the data flow service to generate corresponding log events; Determine the data type of the binlog data, and confirm whether to directly serialize the binlog data into the data structure of the time series database and synchronize it to the time series database based on the determination result.

6. A device for analyzing user behavior based on underlying data, characterized in that: include: A response unit, configured to generate data records when executing transactions in the MySQL database in response to user operations on the business system, and mark the data records; Transaction acquisition unit, used to obtain the global transaction ID in the MySQL database after the transaction is executed; A first storage unit, configured to store the data record and the global transaction ID in a time series database; A monitoring unit is used to monitor the log files in the MySQL database, generate corresponding log events and object information for the log files through the data flow service, and then store the log events and object information in the time series database; A parsing unit, configured to periodically read the stored data in the time series database through a data parsing service, and parse the stored data according to a rule engine to obtain log data about user operations on the business system; wherein the stored data includes the data record, global transaction ID, log event, and object information; A second storage unit is used to store the operation log data in a time series database; a query unit, configured to query and display the operation log data in the time series database in response to a user's operation on the business system; The parsing unit includes: A log writing unit, used to write logs based on the characteristics of the underlying data structure to generate a set of rule engine methods; A rule division unit, configured to divide the stored data into rules using the rule engine method set and generate corresponding operation log data; The log writing unit includes: A first interface setting unit, configured to set an Action interface to record or access a global transaction ID; The second interface setting unit is used to set the DbHint interface and the TablesHint interface to monitor the MySQL database and / or the time series database; The third interface setting unit is used to set the Verify interface to verify each data in the parsing process; The fourth interface setting unit is used to set the BuildLogs interface to generate storage data of a predetermined structure according to the verification data provided by the Verify interface, and store it in the time series database for query by the business system.

7. A computer device, characterized in that: It includes a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the computer program, it implements the method for parsing user behavior based on underlying data as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the method for analyzing user behavior based on underlying data according to any one of claims 1 to 5.

Images