Data processing method and device

Abstract

The present specification provides a data processing method and device, wherein the data processing method is applied to a data processing system, the system comprising a data acquisition device applied to a target server and a data collection device applied to a data processing terminal different from the target server, wherein the data acquisition device obtains initial operation data of a target application and corresponding data content, determines target operation data based on the data content, and sends the target operation data to the data collection device; the data collection device performs syntax analysis on the received target operation data, determines characteristic parameters in the target operation data, and processes the target operation data based on the characteristic parameters to obtain the target data.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and more particularly to a data processing method. The present invention also relates to a data processing system, a computing device, a computer-readable storage medium, and a computer program. Background Art

[0002] With the continuous development of science and technology, many companies have begun to migrate their existing databases to cloud databases to achieve digital transformation or industrial upgrading. However, during the database migration process, due to the structural differences between the existing database and the cloud database, the different grammatical characteristics of the two, and the partial incompatibility of database objects and application codes, database execution errors or semantic inconsistencies may occur. In order to ensure the normal operation of the database after replacement, it is necessary to collect the execution statements generated by the application (such as SQL statements) and find the incompatibilities between the database and the application based on the SQL statements.

[0003] However, the existing technology uses a database periodic collection method to directly monitor the database, thereby collecting the SQL list of the database. However, this method, on the one hand, since the database will generate SQL statements during operation, it is impossible to distinguish whether the SQL statements collected from the database are initiated by the application. On the other hand, the process of monitoring the database will have a significant impact on the performance of the database and the application, reducing the operating efficiency of the database and the application. Summary of the invention

[0004] In view of this, the embodiments of this specification provide a data processing method. This specification also relates to a data processing system, a computing device, a computer-readable storage medium, and a computer program to solve the technical defects existing in the prior art.

[0005] According to a first aspect of an embodiment of this specification, a data processing method is provided, which is applied to a data processing system, wherein the system includes a data acquisition device applied to a target server and a data collection device applied to a data processing terminal different from the target server, wherein:

[0006] The data acquisition device acquires initial operation data of the target application and corresponding data content, determines target operation data based on the data content, and sends the target operation data to the data collection device;

[0007] The data collection device performs syntax analysis on the received target operation data, determines characteristic parameters in the target operation data, and processes the target operation data based on the characteristic parameters to obtain target data.

[0008] According to a second aspect of the embodiments of this specification, a data processing system is provided, comprising a data acquisition device applied to a target server and a data collection device applied to a data processing terminal different from the target server, wherein:

[0009] The data acquisition device is configured to obtain initial operation data of the target application and corresponding data content, determine target operation data based on the data content, and send the target operation data to the data collection device;

[0010] The data collection device is configured to perform syntax analysis on the received target operation data, determine characteristic parameters in the target operation data, and process the target operation data based on the characteristic parameters to obtain target data.

[0011] According to a third aspect of an embodiment of this specification, a computing device is provided, including:

[0012] Memory and processor;

[0013] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of any of the data processing methods are implemented.

[0014] According to a fourth aspect of the embodiments of this specification, a computer-readable storage medium is provided, which stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, the steps of any of the data processing methods are implemented.

[0015] According to a fifth aspect of the embodiments of this specification, a computer program is provided, wherein when the computer program is executed in a computer, the computer is caused to execute any step of the data processing method.

[0016] The data processing method provided in this specification includes a data acquisition device applied to a target server and a data collection device applied to a data processing terminal different from the target server, wherein the data acquisition device is configured to obtain the initial operation data of the target application and the corresponding data content, and determine the target operation data based on the data content, and send the target operation data to the data collection device; the data collection device is configured to perform syntax analysis on the received target operation data, determine the characteristic parameters in the target operation data, and process the target operation data based on the characteristic parameters to obtain the target data.

[0017] Specifically, the data processing method obtains target operation data based on the data content of the initial operation data generated by the target application in the target server during operation, by applying a data collection device to the target server, thereby effectively avoiding the problem of being unable to distinguish whether the execution statement collected from the database is an execution statement initiated by the application, and can accurately and quickly obtain the execution statement issued by the application. After obtaining the target operation data, the target operation data is heterogeneously processed by a data collection device applied to a data processing end different from the target server to obtain the target data; thereby reducing the impact on the application and the database, and effectively improving the operation efficiency of the application and the database. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 is a flow chart of a data processing method provided by an embodiment of this specification;

[0019] Figure 2 It is a flowchart of a data processing method for obtaining application operation data provided by an embodiment of this specification;

[0020] Figure 3 It is a structural diagram of a data processing system provided in an embodiment of this specification;

[0021] Figure 4 It is a structural block diagram of a computing device provided in one embodiment of this specification. DETAILED DESCRIPTION

[0022] Many specific details are described in the following description to facilitate a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of this specification, so this specification is not limited to the specific implementation disclosed below.

[0023] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "a", "said" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0024] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0025] First, the terms involved in one or more embodiments of this specification are explained.

[0026] Database: A warehouse that organizes, stores, and manages data according to data structures.

[0027] ADAM: Advanced Database & Application Migration, referred to as ADAM, is a product used to help heterogeneous databases and applications to evaluate and migrate their structures. ADAM can be used to migrate databases and applications to cloud databases (public or private clouds), significantly reducing the technical difficulty and cost of cloud migration. ADAM can comprehensively evaluate the feasibility, cost, and cloud storage selection of cloud migration, and has built-in implementation assistance, data, application migration and other tools to ensure reliable and fast cloud migration.

[0028] Dynamic collection: Data is collected while the application service is running in real time.

[0029] Source database: the customer's original database.

[0030] Target database: The target database that the customer wants to migrate to.

[0031] Heterogeneous database migration: Customers migrate from the original database to the target database.

[0032] Syntax incompatibility: The original database is not compatible with the new target database due to different syntax characteristics, which manifests as execution errors or semantic errors.

[0033] Sensitive information: refers to all information that may be used improperly, accessed without authorization, or be detrimental to the user's privacy rights.

[0034] Desensitization: Potentially sensitive data (sensitive information) is replaced and deleted with special characters to ensure that the output content does not contain the customer's actual data and query conditions.

[0035] APM: Application Performance Management, referred to as APM.

[0036] SQL: Structured Query Language, referred to as SQL.

[0037] JVM: refers to Java Virtual Machine (Java Virtual Machine).

[0038] Java: is an object-oriented programming language.

[0039] With the development of digital transformation and technology, many companies are currently undergoing digital transformation or industrial upgrading, hoping to migrate their original databases to cloud databases. Heterogeneous database migration will definitely involve application transformation. This is because the syntax, semantics, and usage of the original database of the company and the cloud database are different, and there will be some incompatibility issues between the original database objects and application codes and the cloud database, resulting in database execution errors or semantic inconsistencies. Therefore, in order to ensure the normal operation of the database after replacement, it is necessary to find and correct the incompatibility points between the database and the application, so as to achieve the transformation of the application. This is also the most time-consuming part in the heterogeneous database migration process.

[0040] However, since application SQL is particularly scattered and composed of multiple scenarios, it is difficult to sort out all SQL. Based on the above problems, we can manually write test cases to gradually identify the areas that need to be modified, test and verify each function one by one, and then gradually modify them one by one, but this will lead to a very long modification cycle, which is very labor-intensive and time-consuming.

[0041] Based on the above problems, one way is to obtain the SQL list by directly monitoring the database for collection. Although this method can obtain SQL, it cannot distinguish whether the obtained SQL is initiated by the application, cannot distinguish which application sent the obtained SQL, and cannot locate the specific location of the obtained SQL in the application.

[0042] Another way is to obtain the application operation data through APM tools, but this method is only used to help customers monitor the health of applications, has a great impact on application performance, and cannot desensitize the collected SQL.

[0043] Based on this, in this specification, a data processing method is provided. This specification also involves a data processing system, a computing device, and a computer-readable storage medium, which are described in detail one by one in the following embodiments.

[0044] See also Figure 1 , Figure 1A flow chart of a data processing method provided according to an embodiment of the present specification is shown, wherein the data processing method is applied to a data processing system, the system includes a data acquisition device applied to a target server and a data collection device applied to a data processing terminal different from the target server, and specifically includes the following steps.

[0045] Step 102: The data acquisition device obtains initial operation data of the target application and corresponding data content, determines target operation data based on the data content, and sends the target operation data to the data collection device.

[0046] Among them, the data processing system can be understood as a system that collects the target operation information generated by the target application in the target server during operation, and obtains the target data by heterogeneously processing the target operation information. In practical applications, the data processing system can be an application dynamic collector deployed on ADAM. The application dynamic collector can help customers sort out all real SQL and call stack locations and other operation information at one time without modifying the application code, and perform data desensitization and information processing on the operation information to provide basic data for the workload evaluation of subsequent heterogeneous database migration. The application dynamic collector includes two models: an agent module and a collector module; the agent module is also called a dynamic collector agent, which is deployed in the target server and is used to collect the operation data of the application in the target server. In practical applications, the dynamic collector agent includes at least one; the collector module is also called a dynamic collector collector, which is deployed on a data processing terminal different from the target server and is used to process the collected target operation data. The target server can be an application server; the data processing terminal different from the target server includes but is not limited to a client and a server different from the application server.

[0047] The data collection device can be understood as a device that collects the target operation data of the target application by periodic collection; the data collection device is deployed on the application server, and the data collection device includes at least one; in actual application, the data collection device can be a dynamic collector agent of the application dynamic collector, and the target operation data of the target application in the application server can be dynamically collected by starting the collector agent; wherein, in the embodiment of this specification, two ways of starting the collector agent can be provided according to the actual operation of the client application:

[0048] When the customer application is relatively stable, the collector agent can be started by entering a command;

[0049] In the case that the client application is frequently started / shutdown, the method of starting the collector agent by inputting commands will cause the client to frequently input commands. Based on this, during the process of the client starting the application, a parameter is added to the application startup command document, and the collector agent is started by the parameter, so that during the process of starting or stopping the client application, the collector agent can be started or stopped along with the client application.

[0050] The data collection device can be understood as a device for performing heterogeneous processing on the received target operation data. In actual applications, the data collection device can be a dynamic collector collector that applies a dynamic collector; by starting the dynamic collector collector, the received target operation data can be heterogeneously processed to obtain the target data. In the embodiment of this specification, the dynamic collector collector can be started by entering a command.

[0051] The target application can be understood as the application that needs to be transformed during the heterogeneous database migration process. In actual applications, due to incompatibilities between the source database and the target database, it is necessary to correct the incompatible points in the application to ensure the normal operation of the replaced database.

[0052] The initial operation data may be understood as all operation data generated during the operation of the target application.

[0053] The data content can be understood as data identification, parameter information, function and other data of the initial operation data.

[0054] The target operation data can be understood as the specific operation data generated during the actual operation of the target application. The target operation data includes but is not limited to URL information, SQL statements, call stack information, SQL-dbinfo information and SQL response information. In actual applications, the data acquisition device can collect any one or more of the target operation data such as URL information, SQL statements, call stack information, SQL-dbinfo information and SQL response information, and process any one or more of the data through the data collection device.

[0055] Specifically, the data acquisition device applied to the target server obtains the initial operation data of multiple target applications through periodic acquisition, and determines the data content corresponding to the initial operation data; based on the data content, one or more target operation data are determined from the initial operation data, and the one or more target operation data are sent to the data collection device.

[0056] For example, in the case where the data processing system is used as an application dynamic collector, the data processing system is further described in detail, wherein the target running data may be an SQL statement.

[0057] The application dynamic collector includes two parts: a dynamic collector agent and a dynamic collector collector. For the dynamic collector agent, the execution logic of the dynamic collector agent is added to the JVM of multiple client applications through bytecode enhancement technology and the dynamic collector agent is started. After being started, the dynamic collector agent can monitor the main link of the client application in real time, and dynamically collect the running data of multiple client applications through periodic collection, and obtain the SQL statements generated by multiple client applications from the running data through the data content corresponding to the running data.

[0058] The dynamic collector agent records the collected SQL statements into the asynchronous thread pool on the main link without processing the SQL statements; and sends the SQL statements in the asynchronous thread pool to the dynamic collector collector for processing through the processing thread in an asynchronous manner.

[0059] In actual applications, the application dynamic collector also has a rich self-monitoring mechanism. When the customer application is at its peak operation, it automatically suspends work and performs off-peak work after the peak operation drops.

[0060] In the embodiments of the present specification, the target operation data collected by the dynamic collector includes call stack information. By collecting the call stack information, the specific location of the application code that calls the SQL can be recorded in reverse when the SQL is obtained, which is convenient for directly locating the specific location of the application code that calls the SQL when the customer application is subsequently modified, thereby improving the efficiency of the modification.

[0061] Furthermore, the data generated by the target application during its operation is diverse, which means that the data acquisition device needs to collect all the operation data generated by the target application during the process of acquiring the target operation data, detect and identify the collected operation data to determine the data content, and acquire the target operation data based on the data content. However, in the process of the data acquisition device acquiring the target operation data from the initial operation data based on the data content, since the types of the initial operation data are diverse and large in number, the efficiency of acquiring the target operation data from the initial operation data based on the data content is low, which wastes a lot of computer processing resources. In order to solve the above problem, an embodiment of the present specification acquires the target operation data by determining the data format of the initial operation data, and the specific implementation method is as follows.

[0062] The data acquisition device determines target operation data based on the data content, including:

[0063] The data acquisition device determines a data format of the initial operation data based on the data content, and acquires target operation data from the initial operation data based on the data format.

[0064] The data format can be understood as the format of the operation data generated by the target application. In actual applications, the types of operation data generated by the target application are various, such as URL data and SQL data. Each type of operation data has a specific data format.

[0065] Specifically, after determining the data content in the initial operating data, the data acquisition device determines the data format of the initial operating data based on the data content, and based on the data format, filters out the initial operating data that conforms to the target data format from the initial operating data, and determines the initial operating data that conforms to the target data format as the target operating data, thereby completing the acquisition of the target operating data.

[0066] Continuing with the above example, taking the scenario where the data processing method is applied to obtain the SQL generated by the application as an example, obtaining the target running data from the initial running data based on the data format is explained in detail.

[0067] After determining the data content of all the operation data, the dynamic collector agent determines the data format of each operation data based on the data content, and matches the data format of each operation data with the target data format, which may be a SQL statement format.

[0068] When the data format of the operation data is consistent with the SQL statement format, the operation data is determined as the operation data to be acquired, thereby completing the acquisition of the target operation data;

[0069] In the case where the data format of the running data is inconsistent with the SQL statement format, it means that the running data is not the running data that needs to be obtained, and the running data is discarded.

[0070] In the embodiment of the present specification, the data acquisition device determines the data format of the initial operation data based on the data content, and obtains the target operation data from the initial operation data based on the data format. Thus, the acquisition of the target operation data is completed efficiently, avoiding the problem of low efficiency in obtaining the target operation data from the initial operation data, and effectively improving the accuracy of the target operation data.

[0071] Further, in another case of the embodiment of this specification, the data content includes a data identifier;

[0072] Accordingly, the data acquisition device determines the target operation data based on the data content, including:

[0073] The data acquisition device determines the initial operation data as the target operation data when the data identifier of the initial operation data is consistent with the preset data identifier.

[0074] The data identifier of the initial operation data can be understood as an identifier used to represent the initial operation data, and each data identifier uniquely represents an identifier of the initial operation data.

[0075] The preset data identifier can be set according to different actual application scenarios. In the embodiments of this specification, no specific restrictions are imposed on the preset data identifier. For example, the preset data identifier can be an identifier of a SQL statement.

[0076] Specifically, after determining the data identifier of the initial operation data, the data acquisition device matches the data identifier of the initial operation data with a preset data identifier, and when the data identifier of the initial operation data is consistent with the preset data identifier, the initial operation data is determined as the target operation data.

[0077] Using the above example, the determination of target operation data through the data identifier of the initial operation data is further described in detail. The operation data may include SQL statements, URL information, http information, etc. generated by the client application during operation.

[0078] After obtaining the operation data generated by the client application, the dynamic collector agent determines the data identifier of the operation data and matches the data identifier with the identifier of the SQL statement. If the data identifier is consistent with the identifier of the SQL statement, the operation data corresponding to the data identifier is determined as the SQL statement.

[0079] In the embodiments of the present specification, when the data identifier of the initial operating data is consistent with the preset data identifier, the initial operating data is determined as the target operating data, thereby efficiently completing the acquisition of the target operating data, avoiding the problem of low efficiency in acquiring the target operating data from the initial operating data, and effectively improving the accuracy of the target operating data, thereby further improving the precision of the target data.

[0080] Step 104: The data collection device performs syntax analysis on the received target operation data, determines characteristic parameters in the target operation data, and processes the target operation data based on the characteristic parameters to obtain target data.

[0081] Among them, the characteristic parameters can be understood as sensitive information contained in the target operation data. In actual applications, the characteristic parameters can be set according to the actual application, and the embodiments of this specification do not impose any restrictions on this. For example, when the target operation data is SQL information, the characteristic parameters can be "table_name", "username=139****", "password=123******"; for example, when the target operation data is a URL statement, the characteristic parameters can be "iPhone=182*****".

[0082] Processing the target operation data includes but is not limited to desensitizing or deduplicating the target operation data; correspondingly, the target data can be understood as the processed target operation data.

[0083] Specifically, after receiving the target operation data sent by the data acquisition device, the data collection device performs syntax analysis on the target operation data through a syntax parser deployed in the data collection device, thereby obtaining characteristic parameters in the target operation data, and processes the target operation data based on the characteristic parameters to obtain the target data.

[0084] Using the above example, the method of obtaining characteristic parameters by parsing the target running data is further explained in detail.

[0085] After the dynamic collector collector is decompressed, it is started by a command. After the dynamic collector collector is started, it can receive the SQL statement sent by the dynamic collector agent.

[0086] After receiving the SQL statement sent by the dynamic collector agent, the dynamic collector collector inputs the SQL statement into the syntax parser deployed on the dynamic collector collector. The syntax parser can parse the SQL statement to obtain the sensitive information in the SQL statement, and desensitize the SQL statement to obtain an SQL statement that does not contain any application sensitive information.

[0087] In the embodiments of the present specification, the data collection device performs syntax analysis on the received target operation data, thereby quickly and accurately obtaining characteristic parameters in the target operation data, improving the efficiency of acquiring the target object and saving computer processing resources.

[0088] Furthermore, in the embodiments of the specification, the data collection device processes the target operation data based on the characteristic parameters to obtain the target data. At the same time, the target data can also be obtained through the characteristic parameters and the identification information corresponding to the characteristic parameters. The specific implementation method is as follows.

[0089] The data collection device processes the target operation data based on the characteristic parameters to obtain target data, including:

[0090] The data collection device determines the identification information corresponding to the characteristic parameter, and processes the target operation data based on the characteristic parameter and the identification information corresponding to the characteristic parameter to obtain target data.

[0091] Among them, identification information can be understood as information that can uniquely represent a characteristic parameter. In the embodiments of this specification, different characteristic information corresponds to different identification information. For example, when the characteristic information is "username=139****", the identification information corresponding to the characteristic parameter can be the user name; when the characteristic information is "iPhone=182*****", the identification information corresponding to the characteristic parameter is the mobile phone number.

[0092] Specifically, after determining the characteristic parameters in the target operation data, the data collection device determines the identification information corresponding to the characteristic parameters, and processes the target operation data based on the characteristic parameters and the identification information corresponding to the characteristic parameters, thereby obtaining the target data.

[0093] Continuing with the above example, the processing of target operation data based on characteristic parameters and identification information corresponding to the characteristic parameters is further described in detail.

[0094] After determining the characteristic parameters in the SQL statement through the syntax parser, the dynamic collector identifies the identification information corresponding to each characteristic parameter based on the syntax parser.

[0095] The dynamic collector processes the SQL statement based on the characteristic parameters and the identification information corresponding to the characteristic parameters, thereby obtaining the target SQL statement.

[0096] Furthermore, the data collection device processes the target operation data based on the characteristic parameters and the identification information to obtain target data, including:

[0097] The data collection device processes the target operation data based on the characteristic parameters and the identification information to obtain pre-processed data;

[0098] The data collection device performs deduplication processing on the pre-processed data to obtain target data.

[0099] Among them, when the target running data is an SQL statement, the pre-processed data can be understood as the SQL statement after desensitization processing.

[0100] Specifically, after determining the identification information corresponding to the characteristic parameters, the data collection device processes the target operation data based on the characteristic parameters and the identification information to obtain pre-processed data; and obtains the target data by deduplicating the pre-processed data.

[0101] Continuing with the above example, obtaining preprocessed data through feature parameters and identification information and performing deduplication processing on the preprocessed data to obtain target data is further described in detail.

[0102] After the dynamic collector determines the characteristic parameters and the identification information corresponding to the characteristic parameters based on the syntax parser, it determines whether the identification information meets the preset identification condition, where the preset identification condition can be whether the identification information is the target identification information, and the target identification information can be the user name, password, and mobile phone number. If the identification information is consistent with the preset target identification, it means that the identification information meets the preset identification condition, and the characteristic parameters corresponding to the identification information are determined as sensitive characteristic parameters that need to be desensitized, and the desensitized SQL statement is obtained by desensitizing the sensitive characteristic parameters.

[0103] After obtaining the desensitized SQL statements, the dynamic collector performs deduplication processing on the desensitized SQL statements to obtain the target SQL statements.

[0104] In the embodiment of this specification, the data collection device obtains pre-processed data based on characteristic parameters and identification information; and obtains target data by performing deduplication processing on the pre-processed data. The desensitization process avoids the leakage of the customer's privacy information, and the deduplication process avoids the acquisition of a large amount of duplicate data, saves computer processing resources, and thus improves the efficiency of subsequent database migration.

[0105] Furthermore, the data collection device processes the target operation data based on the characteristic parameters and the identification information to obtain pre-processed data, including:

[0106] The data collection device determines the characteristic parameter corresponding to the identification information as the target characteristic parameter when the identification information corresponding to the characteristic parameter meets the preset identification condition;

[0107] The data collection device replaces the target characteristic parameters in the target operation data based on preset target characters to obtain preprocessed data.

[0108] The preset identification condition can be set according to actual application, and the embodiments of this specification do not limit this. For example, the preset identification condition can be whether the identification information is target identification information; the target identification information can be a user name, password, and mobile phone number.

[0109] The preset target characters can be set according to actual applications, and the embodiments of this specification do not impose any limitation on this. For example, special characters include text and / or symbols.

[0110] Specifically, when the identification information corresponding to the characteristic parameter meets the preset identification condition, the data collection device determines the characteristic parameter corresponding to the identification information as the target characteristic parameter; and replaces the target characteristic parameter in the target operation data with the preset target character, thereby obtaining the preprocessed data.

[0111] Continuing with the above example, the target feature parameters determined based on the preset target characters are replaced to obtain preprocessed data for further detailed description.

[0112] After the dynamic collector determines the feature parameters and the identification information corresponding to the feature parameters based on the syntax parser, it determines whether the identification information meets the preset identification condition, which may be whether the identification information is target identification information. When the identification information is consistent with the preset target identification, it means that the identification information meets the preset identification condition, and determines the feature parameters corresponding to the identification information as sensitive feature parameters that need to be desensitized. The sensitive feature parameters that need to be desensitized are replaced by preset target characters, thereby completing the desensitization operation on the SQL statement and obtaining the desensitized SQL statement.

[0113] In the embodiment of this specification, when the identification information corresponding to the characteristic parameter meets the preset identification condition, the data collection device determines the characteristic parameter corresponding to the identification information as the target characteristic parameter; and replaces the target characteristic parameter in the target operation data with the preset target character, thereby obtaining pre-processed data. Thus, the target operation data is effectively desensitized, thereby ensuring the data security of the customer application and avoiding the occurrence of data leakage problems.

[0114] In the embodiments of this specification, the target operation data obtained in multiple formats can be adjusted to a unified format and the adjusted target operation data can be stored in the customer's application database, thereby facilitating the customer to process the target operation data. The specific implementation method is as follows.

[0115] After the data collection device determines the characteristic parameters in the target operation data, the data collection device further includes:

[0116] The data collection device performs format processing on the target operation data to obtain the processed target operation data, and stores the processed target operation data in an application database.

[0117] The application database may be understood as a database for storing customer application information. In actual applications, the application database is a local database of the customer.

[0118] Specifically, the data collection device adjusts the acquired target operation data into a unified format, obtains the processed target operation data, and stores the processed target operation data in the application database.

[0119] Continuing with the above example, obtaining the processed target operation data and storing it in the application database is further described in detail.

[0120] After acquiring SQL statements of multiple customer applications, the dynamic collector converts the data formats of the SQL statements of multiple customer applications into a unified data format by formatting the SQL statements, and stores the converted SQL statements in the customer's local database.

[0121] In the embodiment of this specification, the data collection device formats the acquired target operation data and stores the processed target operation data in the application database, so as to facilitate the customer to process the target operation data, thereby improving the customer's use experience.

[0122] In order to obtain the operating status information of the data acquisition device and the data collection device in real time and facilitate users to monitor the data acquisition device and the data collection device, in the embodiments of this specification, the status data of the data acquisition device and the data collection device are determined through the attribute information of the target data, so as to obtain the operating status of the data acquisition device and the data collection device in real time. The specific implementation method is as follows.

[0123] The data collection device processes the target operation data based on the characteristic parameters to obtain the target data, and further includes:

[0124] The data collection device acquires attribute information of the target data, and determines status data of the data acquisition device and the data collection device based on the attribute information.

[0125] The attribute information of the target data can be set according to different practical applications, and the embodiments of this specification do not impose any limitation on this. For example, the attribute information can be information such as the number of target data and the generation time of the target data.

[0126] The status data can be understood as the dynamic collector and the collection time of the dynamic collector, the number of collections within a preset time period and the average collection time, and the success rate of the dynamic collector.

[0127] Specifically, the data collection device determines the attribute information of the target data, and thereby determines the status data indicating the operating status of the data acquisition device and the data collection device based on the attribute information.

[0128] Continuing with the above example, taking the scenario where the data processing method is applied to obtain SQL generated by an application as an example, determining the status data of the data acquisition device and the data collection device is further described in detail.

[0129] After obtaining the target SQL statement, the dynamic collector determines the attribute information such as the number and generation time of the target SQL statement, and obtains the running status of the dynamic collector and the dynamic collector agent based on the attribute information, so that the user can monitor the running status of the dynamic collector and the dynamic collector agent.

[0130] In the embodiments of this specification, attribute information of the target data is obtained through a data collection device, and the status data of the data collection device and the data collection device are determined based on the attribute information. The operating status information of the data collection device and the data collection device can be accurately obtained, which is convenient for users to monitor the data collection device and the data collection device.

[0131] In one embodiment of the present specification, after obtaining target data and status data of the data acquisition device and the data collection device through the data acquisition device and the data collection device, the data processing method further includes:

[0132] The data collection device uploads the target data and the status data to a target database, and performs compatibility detection on the target data in the target database.

[0133] The target database can be understood as the database to which the customer application needs to be migrated during the heterogeneous database migration process, such as a cloud database.

[0134] Specifically, the data collection device uploads the target data and the status data to the target database, and performs compatibility detection on the target data in the target database.

[0135] Using the above example, the compatibility check of the target data in the target database is further explained in detail.

[0136] After the dynamic collector uploads the acquired target SQL statements and status information to the cloud database, it can perform compatibility detection on the target SQL in the target database based on the application evaluation platform, thereby determining the workload assessment of heterogeneous database migration.

[0137] In the embodiments of this specification, the target data and status data are uploaded to the target database through a data collection device to implement compatibility detection of the target data in the target database, facilitate architectural topology analysis of the customer application cluster, and effectively evaluate the workload of heterogeneous database migration.

[0138] In one embodiment of the present specification, the data processing method further includes:

[0139] The data acquisition device acquires dynamic operation information of the target application and sends the dynamic operation information to the data collection device;

[0140] The data collection device processes the received dynamic operation information and sends the processed dynamic operation information to a target database, thereby implementing performance detection of the target application in the target database.

[0141] The dynamic operation information can be understood as the performance information of the target application during operation, such as application server disk information, JVM performance information, and application server memory status information.

[0142] Specifically, the data acquisition device obtains dynamic operation information of the target application and sends the dynamic operation information to the data collection device; after receiving the dynamic operation information, the data collection device processes the dynamic operation information and sends the processed dynamic operation information to the target database, thereby realizing performance detection of the target application in the target database.

[0143] Using the above example, further explanation is given on obtaining the dynamic operation information of the target application and sending the processed dynamic operation information to the target database to implement performance detection of the target application in the target database.

[0144] The dynamic collector agent obtains the dynamic operation information of the client application during operation by periodic collection, and sends the operation dynamic information to the dynamic collector collector for processing by asynchronous sending.

[0145] After receiving the dynamic operation information, the dynamic collector processes the format of the dynamic operation information, converts the dynamic operation information of multiple client applications into dynamic operation information in a unified format, and sends the processed dynamic operation information to the cloud database for storage.

[0146] After the dynamic operation information is sent to the cloud database, the performance of the target application is tested in the target database through the application evaluation platform to determine the performance of the target application.

[0147] In the embodiments of this specification, the dynamic operation information of the target application is obtained through a data acquisition device, the dynamic operation information is processed by the data collection device, and the processed dynamic operation information is sent to the target database, so as to implement performance detection of the target application in the target database, thereby accurately determining the performance of the target application and improving the customer's user experience.

[0148] The data processing method provided in the embodiments of this specification obtains the target operation data of the target application through a data collection device applied to the target server, thereby effectively avoiding the problem of being unable to distinguish whether the obtained SQL is initiated by the application. The SQL issued by the application can be accurately and quickly obtained, and the target operation data is heterogeneously processed by a data collection device applied to a data processing end different from the target server, thereby obtaining the target data; the impact on the application and the database is reduced, and the operation efficiency of the application and the database is effectively improved.

[0149] At the same time, by adopting a heterogeneous processing solution that separates data collection and data processing, there is almost no impact on the request link, application memory and CPU; and the acquired target operation data is desensitized to ensure that the output target data does not contain any sensitive information.

[0150] The following combination Figure 2 , taking the application of the data processing method provided in this specification in the scenario of obtaining the running data of an application as an example, the data processing method is further described. Figure 2 A flow chart of a data processing method for a scenario of obtaining the operating data of an application provided in an embodiment of the present specification is shown. Based on the flow chart, it can be seen that the data processing method for a scenario of obtaining the operating data of an application provided in the present specification includes two devices: a dynamic collector and a dynamic collector.

[0151] The dynamic collector is deployed on the application server. The dynamic collector can collect the operation information generated by the client application during operation and the system dynamic information of the client application through periodic collection. The client application can be deployed in multiple application servers. After obtaining the operation information and system dynamic information, the dynamic collector sends the collected information to the dynamic collector for processing through asynchronous sending.

[0152] Among them, the dynamic collector is deployed on a data processing end that is different from the application server. After receiving the information sent by the dynamic collector, the dynamic collector performs semantic analysis on the running information generated by the client application during the running process through a syntax parser, identifies the parameters, functions and other data in the running information, and replaces the potentially sensitive data with special characters to desensitize the running information, thereby obtaining running data without sensitive information; after obtaining the running data without sensitive information, deduplicate the running data, and then send the deduplicated running data to the cloud database, so as to evaluate the workload of heterogeneous database migration through the running data in the cloud database.

[0153] The dynamic collector standardizes the system dynamic information of customer applications, converts it into a unified data format, and sends the processed system dynamic information to the cloud database, thereby realizing performance testing of customer applications through the system dynamic information in the cloud data.

[0154] Based on the above introduction, the data processing method provided in this specification for obtaining the running data of an application specifically includes the following steps:

[0155] Step 220: The dynamic collector periodically collects the operation information generated by the client application in the server during operation, and sends the operation information to the dynamic collector.

[0156] The dynamic collector can be understood as the data collection device in the above embodiments.

[0157] The operation information can be understood as the target operation data in the above embodiment, and the operation information can be URL information, SQL-dbinfo information, SQL response information, and call stack information.

[0158] The dynamic collector can be understood as the data collection device in the above embodiments.

[0159] After receiving the operation information sent by the dynamic collector, the dynamic collector can process the operation information and send the processed operation information to the cloud database and the local database. The specific implementation steps include:

[0160] Step 2202: The dynamic collector performs syntax analysis on the received operation information through the syntax analyzer, and sends the analyzed operation information to the formatting processing module for processing.

[0161] Step 2204: The formatting processing module formats the received operation information and sends the processed operation information to the local database for storage.

[0162] The local database may be understood as the application database in the above embodiment.

[0163] Step 2206: The dynamic collector performs syntax analysis on the received operation information through the syntax analyzer, and sends the analyzed operation information to the data desensitization module for desensitization.

[0164] Step 2208: The data desensitizing module performs desensitization processing on the parsed operation information, and sends the desensitized operation data to the data deduplication module for deduplication processing.

[0165] Specifically, after receiving the parsed operation information, the data desensitization module identifies the possible sensitive data in the operation information and replaces the possible sensitive data with characteristic characters, thereby desensitizing the operation information. The characteristic characters can be understood as the preset target characters in the above embodiment.

[0166] After completing the desensitization processing of the operation information, the data desensitization module sends the desensitized operation data to the data deduplication module.

[0167] Step 2210: The data deduplication module performs deduplication processing on the received operation data, and sends the deduplication-processed operation data to the cloud database.

[0168] Step 2212: The data deduplication module performs deduplication processing on the received operation data, and also sends the deduplication-processed operation data to the data processing module for processing.

[0169] Step 2214: The data processing module processes the received operation data, determines the operation status data of the dynamic collector and the dynamic collector, and sends the operation status data to the cloud data.

[0170] The operation status data includes the dynamic collector and the collection time of the dynamic collector, the number of collections in a preset time period and the average collection time, and the success rate of the dynamic collector. The preset time period can be set according to the actual application, and the embodiment of this specification does not make any limitation on this, for example, one day or one week.

[0171] Step 240: The dynamic collector obtains the system dynamic information of the client application in a periodic collection manner, and sends the system dynamic information to the dynamic collector.

[0172] The system dynamic information can be understood as the dynamic operation information in the above embodiment, and can be server disk information, JVM performance information, and server memory status information.

[0173] After receiving the system dynamic information sent by the dynamic collector, the dynamic collector can process the system dynamic information and send the processed system dynamic information to the cloud database. The specific implementation steps include:

[0174] Step 2402: The dynamic collector processes the system dynamic information received from the dynamic collector through the system dynamic information processing module, and sends the processed system dynamic information to the cloud database.

[0175] Specifically, the dynamic collector formats the system dynamic information through the system dynamic information processing module, converts the format of the system dynamic information into a unified data format, and sends the processed system dynamic information to the cloud database for storage.

[0176] Step 260: The cloud database sends the operation information and system dynamic information to the application evaluation platform for evaluation processing.

[0177] Specifically, the cloud database sends the operation information and system dynamic information to the application evaluation platform, and the application evaluation platform evaluates and processes the operation information, thereby evaluating the workload of heterogeneous database migration.

[0178] The application evaluation platform performs performance detection and processing on system dynamic information, thereby detecting the performance information of the customer application.

[0179] In the embodiments of the present specification, the operation data of the customer application is obtained through a dynamic collector, and the operation data is parsed, desensitized and deduplicated by the dynamic collector to obtain the processed operation data; thereby, it is clearly determined that the obtained SQL statement is the real and valid SQL sent by the application, and it is known which specific application initiated it, thereby improving the efficiency of heterogeneous database migration.

[0180] At the same time, by adopting a heterogeneous processing solution that separates data collection and data processing, there is almost no impact on the request link, application memory and CPU; and the acquired target operation data is desensitized to ensure that the output target data does not contain any sensitive information.

[0181] Corresponding to the above method embodiment, this specification also provides a data processing system embodiment. Figure 3 FIG. 1 shows a schematic diagram of a data processing system provided by an embodiment of the present specification. Figure 3 As shown, the system includes a data acquisition device 302 applied to a target server and a data collection device 304 applied to a data processing terminal different from the target server, wherein:

[0182] The data acquisition device 302 is configured to obtain initial operation data of the target application and corresponding data content, determine target operation data based on the data content, and send the target operation data to the data collection device;

[0183] The data collection device 304 is configured to perform syntax analysis on the received target operation data, determine characteristic parameters in the target operation data, and process the target operation data based on the characteristic parameters to obtain target data.

[0184] Optionally, the data content includes a data identifier;

[0185] Correspondingly, the data acquisition device 302 is further configured to determine the initial operation data as the target operation data when the data identifier of the initial operation data is consistent with the preset data identifier.

[0186] Optionally, the data acquisition device 302 is further configured to determine a data format of the initial operation data based on the data content, and acquire target operation data from the initial operation data based on the data format.

[0187] Optionally, the data collection device 304 is further configured to determine identification information corresponding to the characteristic parameter, and process the target operation data based on the characteristic parameter and the identification information corresponding to the characteristic parameter to obtain target data.

[0188] Optionally, the data collection device 304 is further configured to:

[0189] Processing the target operation data based on the characteristic parameters and the identification information to obtain preprocessed data;

[0190] The pre-processed data is deduplicated to obtain target data.

[0191] Optionally, the data collection device 304 is further configured to:

[0192] In the case where the identification information corresponding to the characteristic parameter meets the preset identification condition, determining the characteristic parameter corresponding to the identification information as the target characteristic parameter;

[0193] The target characteristic parameters in the target operation data are replaced based on the preset target characters to obtain preprocessed data.

[0194] Optionally, the data collection device 304 is further configured to perform format processing on the target operation data, obtain the processed target operation data, and store the processed target operation data in an application database.

[0195] Optionally, the data collection device 304 is further configured to obtain attribute information of the target data, and determine status data of the data acquisition device and the data collection device based on the attribute information.

[0196] Optionally, the data collection device 304 is further configured to upload the target data and the status data to a target database, so as to perform compatibility detection on the target data in the target database.

[0197] Optionally, the data collection device 304 is further configured to:

[0198] Acquire dynamic operation information of the target application, and send the dynamic operation information to the data collection device;

[0199] The received dynamic operation information is processed, and the processed dynamic operation information is sent to a target database, so as to implement performance detection of the target application in the target database.

[0200] The data processing system provided in the present specification obtains target operation data based on the data content of the initial operation data generated by the target application in the target server during operation, through a data collection device applied to the target server, effectively avoiding the problem of being unable to distinguish whether the execution statement collected from the database is an execution statement initiated by the application, and can accurately and quickly obtain the execution statement issued by the application. After obtaining the target operation data, the target operation data is heterogeneously processed by a data collection device applied to a data processing end different from the target server to obtain the target data; thereby reducing the impact on the application and the database, and effectively improving the operation efficiency of the application and the database.

[0201] The above is a schematic scheme of a data processing system of this embodiment. It should be noted that the technical scheme of the data processing system and the technical scheme of the above data processing method belong to the same concept, and the details not described in detail in the technical scheme of the data processing system can be referred to the description of the technical scheme of the above data processing method.

[0202] Figure 4 The structure block diagram of a computing device 400 provided according to an embodiment of the present specification is shown. The components of the computing device 400 include but are not limited to a memory 410 and a processor 420. The processor 420 is connected to the memory 410 via a bus 430, and the database 450 is used to store data.

[0203] The computing device 400 also includes an access device 440 that enables the computing device 400 to communicate via one or more networks 460. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 440 may include one or more of any type of network interface (e.g., a network interface card (NIC)) wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a World Wide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, a near field communication (NFC) interface, and the like.

[0204] In one embodiment of the present specification, the above components of the computing device 400 and Figure 4 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Figure 4 The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0205] Computing device 400 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or PC. Computing device 400 may also be a mobile or stationary server.

[0206] The processor 420 is used to execute computer executable instructions, which, when executed by the processor 420, implement the steps of any of the data processing methods.

[0207] The above is a schematic scheme of a computing device of this embodiment. It should be noted that the technical scheme of the computing device and the technical scheme of the above data processing method belong to the same concept, and the details not described in detail in the technical scheme of the computing device can be referred to the description of the technical scheme of the above data processing method.

[0208] An embodiment of the present specification further provides a computer-readable storage medium storing computer-executable instructions, which can implement any step of the data processing method when executed by a processor.

[0209] The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of the storage medium and the technical scheme of the above data processing method belong to the same concept, and the details not described in detail in the technical scheme of the storage medium can be referred to the description of the technical scheme of the above data processing method.

[0210] An embodiment of the present specification further provides a computer program, wherein when the computer program is executed in a computer, the computer is caused to execute any step of the data processing method.

[0211] The above is a schematic scheme of a computer program of this embodiment. It should be noted that the schematic scheme of the computer program and the technical scheme of the above data processing method belong to the same concept, and the details not described in detail in the schematic scheme of the computer program can be referred to the description of the technical scheme of the above data processing method.

[0212] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0213] The computer instructions include computer program codes, which may be in source code form, object code form, executable files or some intermediate forms, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0214] It should be noted that, for the convenience of description, the aforementioned method embodiments are all described as a series of action combinations, but those skilled in the art should be aware that this specification is not limited by the order of the actions described, because according to this specification, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by this specification.

[0215] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0216] The preferred embodiments of this specification disclosed above are only used to help explain this specification. The optional embodiments do not describe all the details in detail, nor do they limit the invention to only the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of this specification, so that technicians in the relevant technical field can well understand and use this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. A data processing method, applied to a data processing system, the system comprising a data acquisition device applied to a target server and a data collection device applied to a data processing terminal different from the target server, wherein: The data acquisition device acquires the initial operation data of the target application and the corresponding data content, determines the target operation data based on the data content, and sends the target operation data to the data collection device, wherein the determining the target operation data based on the data content includes: the data acquisition device determines the data format of the initial operation data based on the data content, and acquires the target operation data from the initial operation data based on the data format; The data collection device performs syntax analysis on the received target operation data, determines characteristic parameters in the target operation data, and determines identification information corresponding to the characteristic parameters, processes the target operation data based on the characteristic parameters and the identification information to obtain preprocessed data, and deduplicates the preprocessed data to obtain target data.

2. The data processing method according to claim 1, wherein the data content includes a data identifier; Accordingly, the data acquisition device determines the target operation data based on the data content, including: The data acquisition device determines the initial operation data as the target operation data when the data identifier of the initial operation data is consistent with the preset data identifier.

3. The data processing method according to claim 1, wherein the data collection device processes the target operation data based on the characteristic parameter and the identification information to obtain pre-processed data, comprising: The data collection device determines the characteristic parameter corresponding to the identification information as the target characteristic parameter when the identification information corresponding to the characteristic parameter meets the preset identification condition; The data collection device replaces the target characteristic parameters in the target operation data based on preset target characters to obtain preprocessed data.

4. The data processing method according to claim 1, after the data collection device determines the characteristic parameters in the target operation data, further comprising: The data collection device performs format processing on the target operation data to obtain the processed target operation data, and stores the processed target operation data in an application database.

5. The data processing method according to claim 1, wherein the data collection device processes the target operation data based on the characteristic parameter, and after obtaining the target data, further comprises: The data collection device acquires attribute information of the target data, and determines status data of the data acquisition device and the data collection device based on the attribute information.

6. The data processing method according to claim 5, further comprising: The data collection device uploads the target data and the status data to a target database, and performs compatibility detection on the target data in the target database.

7. The data processing method according to claim 1, further comprising: The data acquisition device acquires dynamic operation information of the target application and sends the dynamic operation information to the data collection device; The data collection device processes the received dynamic operation information and sends the processed dynamic operation information to a target database, thereby implementing performance detection of the target application in the target database.

8. A data processing system, comprising a data acquisition device applied to a target server and a data collection device applied to a data processing terminal different from the target server, wherein: The data acquisition device is configured to acquire initial operation data of a target application and corresponding data content, determine target operation data based on the data content, and send the target operation data to the data collection device, wherein the determining the target operation data based on the data content comprises: the data acquisition device determines a data format of the initial operation data based on the data content, and acquires the target operation data from the initial operation data based on the data format; The data collection device is configured to perform syntax parsing on the received target operation data, determine characteristic parameters in the target operation data, and determine identification information corresponding to the characteristic parameters, process the target operation data based on the characteristic parameters and the identification information to obtain preprocessed data, and deduplicate the preprocessed data to obtain target data.

9. A computing device comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the data processing method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions, when executed by a processor, implement the steps of the data processing method according to any one of claims 1 to 7.

11. A computer program product, wherein: When the computer program product is executed in a computer, it enables the computer to execute the steps of the data processing method according to any one of claims 1 to 7.

Images