Data processing system, method, device and storage medium

Abstract

The invention provides a data processing system, method and device and a storage medium. The system comprises a terminal device used for generating a window unique identifier for a currently operated target window, generating an operation request carrying the window unique identifier and a database operation instruction for a database operation instruction in the target window, and sending the operation request to an intermediate proxy server; the intermediate proxy server is used for screening out a target middle station corresponding to the window unique identifier from a plurality of middle stations of the B / S architecture based on the window unique identifier in the operation request, and sending the database operation instruction to the target middle station for processing; the middle station is used for executing a database operation instruction on the database based on the long connection with the database under the condition that the middle station is the target middle station; furthermore, through a cross-thread variable transmission strategy, recording the execution process of the database operation instruction into a first log file step by step; and performing information monitoring and statistical analysis based on the first log file.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technology, and in particular to the field of database technology. Background Art

[0002] In recent years, with the advent of the digital age, the size of data has increased dozens of times. Therefore, enterprises have invested more and more in data storage and management. In actual database management, due to the different management details, personnel quality and familiarity of each enterprise, enterprises need to introduce operation and maintenance tools to help with data management. Summary of the invention

[0003] The present disclosure provides a data processing system, method, device, and storage medium to solve or alleviate one or more technical problems in the prior art.

[0004] In a first aspect, the present disclosure provides a data processing system, comprising:

[0005] The terminal device is used to generate a window unique identifier for the target window of the current operation, and generate an operation request carrying the window unique identifier and the database operation instruction for the database operation instruction in the target window, and send the operation request to the intermediate proxy server;

[0006] The intermediate proxy server is used to filter out a target middle station corresponding to the window unique identifier from multiple middle stations of the B / S architecture based on the window unique identifier in the operation request, and send the database operation instruction to the target middle station for processing;

[0007] The middle station is used to execute the database operation instructions on the database based on the long connection between the database when the middle station is the target middle station; and, through the cross-thread variable transfer strategy, record the execution process of the database operation instructions step by step into the first log file; and perform information monitoring and statistical analysis based on the first log file.

[0008] In a second aspect, the present disclosure provides a data processing method, comprising:

[0009] Based on the long connection with the database, execute database operation instructions on the database;

[0010] By using a cross-thread variable transfer strategy, the execution process of the database operation instruction is recorded step by step into a log file;

[0011] Information monitoring and statistical analysis are performed based on the log files.

[0012] In a third aspect, the present disclosure provides a data processing device, including:

[0013] An execution module, used to execute database operation instructions on the database based on a long connection with the database;

[0014] A recording module, used to record the execution process of the database operation instruction in a log file step by step through a cross-thread variable transfer strategy;

[0015] A processing module is used to perform information monitoring and statistical analysis based on the log file.

[0016] In a fourth aspect, an electronic device is provided, including:

[0017] at least one processor; and

[0018] a memory communicatively connected to the at least one processor; wherein,

[0019] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute any method in the embodiments of the present disclosure.

[0020] In a fifth aspect, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute any method according to the embodiments of the present disclosure.

[0021] In a sixth aspect, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements any method according to the embodiments of the present disclosure.

[0022] The beneficial effects of the technical solution provided by the present disclosure include at least: based on the window unique identifier in the operation request, the target middle station corresponding to the window unique identifier is screened out from multiple middle stations. Based on this method, multiple operation requests with contextual relationships can be sent to the same middle station for processing, which can ensure that the operation instructions with contextual relationships can be aggregated to the same middle station, and the edge monitors and counts the operation data. In addition, the same middle station can receive and process the operation instructions with contextual relationships based on the long link with the database, effectively ensuring the normal execution of the operation instructions with contextual relationships, reducing the possibility of execution failure, and thus improving the resource utilization of the entire B / S system. At the same time, through the cross-thread variable transfer strategy, the execution process of the database operation instruction is recorded step by step in the log file, which realizes the effective monitoring of the execution process of the operation instruction, so as to improve the system performance.

[0023] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the multiple drawings represent the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings only depict some embodiments provided according to the present disclosure and should not be regarded as limiting the scope of the present disclosure.

[0025] Figure 1 is a schematic diagram of the architecture of a data processing system according to an embodiment of the present disclosure;

[0026] Figure 2 is a first flow chart of a data processing system according to an embodiment of the present disclosure;

[0027] Figure 3 is a second flow chart of a data processing system according to an embodiment of the present disclosure;

[0028] Figure 4 is a third flow chart of a data processing system according to an embodiment of the present disclosure;

[0029] Figure 5 is a schematic diagram of creating a child thread according to an embodiment of the present disclosure;

[0030] Figure 6 is a schematic diagram of processing log files according to an embodiment of the present disclosure;

[0031] Figure 7 is a schematic diagram of statistical analysis according to an embodiment of the present disclosure;

[0032] Figure 8 is a flowchart of a data processing method according to an embodiment of the present disclosure;

[0033] Fig. 9 is a structural schematic diagram of a data processing device according to an embodiment of the present disclosure;

[0034] Fig.10 It is a block diagram of an electronic device used to implement the data processing method of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0035] The present disclosure will be further described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0036] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the specific embodiments below. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some embodiments, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the main purpose of the present disclosure.

[0037] In the description of the embodiments of the present disclosure, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present disclosure, unless otherwise specified, "plurality" means two or more.

[0038] In the Internet era, the way of processing data has gradually changed from the early C / S (Client / Server) mode to the B / S (Browser / Server) mode. For most database products, session maintenance and operation are undoubtedly the biggest challenges for B / S.

[0039] Among them, under the BS architecture, the client can control the database through the terminal device (exemplarily, a browser, APP (APPlication, application service), etc.). However, the context information is not retained under this architecture, and each operation (such as different operations on the same page, switching from one page to another) will trigger a new operation request. However, the request of the same terminal device may be distributed to different middle stations through the reverse proxy Nginx between the client and the middle station through the load balancing strategy. In this case, operations with contextual relationships may cause operation failures.

[0040] In view of this, a data processing system is proposed in the embodiment of the present disclosure, and the specific architecture diagram is as follows: Figure 1 As shown, it includes a terminal device 101, an intermediate proxy server 102, a middle station 103 and a database 104. Among them, the terminal device 101 includes but is not limited to electronic devices such as desktop computers, mobile phones, mobile computers, tablet computers, media players, smart wearable devices, smart TVs, etc. The middle station 103 can be implemented as a server, etc. The intermediate proxy server 102 is located between the terminal device 101 and the middle station 103, and is responsible for forwarding the operation request of the terminal device 101 and sending it to the corresponding target middle station 103 for processing. The middle station 103 realizes the control of the database through a long link with the database 104.

[0041] The number of terminal devices 101, the number of intermediate proxy servers 102, the number of middle stations 103, and the number of databases 104 can be the same or different, and can be determined based on actual conditions. The embodiments of the present disclosure are not limited to this. Exemplarily, the number of intermediate proxy servers 102 can be n, and the number of middle stations 103 is m, where n can be less than m, and both n and m are positive integers greater than 0.

[0042] Based on this system architecture, each device performs the following steps: Figure 2 As shown, including:

[0043] The terminal device is used to execute S201, generate a window unique identifier for the target window of the current operation, and generate an operation request carrying the window unique identifier and the database operation instruction for the database operation instruction in the target window, and send it to the intermediate proxy server.

[0044] The intermediate proxy server is used to execute S202. Based on the window unique identifier in the operation request, it selects the target middle station corresponding to the window unique identifier from multiple middle stations in the B / S architecture, and sends the database operation instruction to the target middle station for processing.

[0045] The middle station is used to execute S203. When the middle station is the target middle station, the database operation instruction is executed on the database based on the long connection between the database; and, through the cross-thread variable transfer strategy, the execution process of the database operation instruction is recorded step by step in the first log file; and information monitoring and statistical analysis are performed based on the first log file.

[0046] The steps that need to be recorded during the execution process can be determined according to the actual situation, which can be all steps or core key steps. The information of each step that needs to be recorded can be realized through tracking points.

[0047] The database may be a SQL (Structured Query Language) database or a NOSQL (Not Only Structured Query Language) database, which is not limited in the embodiments of the present disclosure.

[0048] Among them, in order to facilitate data monitoring of database operations, each middle station in the embodiment of the present disclosure has a corresponding first log file, and the first log file stores data information of all threads corresponding to the middle station.

[0049] In the disclosed embodiment, based on the window unique identifier in the operation request, the target middle station corresponding to the window unique identifier is screened out from multiple middle stations. Based on this method, multiple database operation instructions with contextual relationships can be sent to the same middle station for processing, ensuring the successful execution of the database operation instructions. In addition, based on this method, it is also convenient to monitor and count the operation process. Moreover, the same middle station can process the operation instructions with contextual relationships based on the long link with the database, effectively ensuring the normal execution of the operation instructions with contextual relationships, further reducing the possibility of execution failure, thereby improving the performance of the entire B / S system. At the same time, through the cross-thread variable transfer strategy, the execution process of the database operation instructions is recorded step by step in the first log file, which realizes the effective monitoring of the execution process of the database operation instructions, and at the same time, based on the first log file, it is also convenient to improve the monitoring flexibility of the system.

[0050] In some embodiments, the operation request is performed based on the window unique identifier in the operation request, and a target middle station corresponding to the window unique identifier is screened out from multiple middle stations of the B / S architecture, and the intermediate proxy server is, for example, Figure 3 The following are specifically used for:

[0051] S301, when the database operation instruction is the first operation instruction in the operation instruction set with a contextual relationship, a middle station is selected from multiple middle stations of the B / S architecture based on the load balancing strategy as the target middle station corresponding to the window unique identifier.

[0052] Specifically, the contextual relationship in the embodiments of the present disclosure can be understood as the need for multiple operation instructions to be completed in sequence to achieve a task, and the next database operation instruction can only be executed when the previous operation instruction is executed successfully. That is, the execution of database operation instructions is dependent. For example, taking the modification of the price of a product as an example, the first step is to send an update request for pre-execution through the terminal device, that is, to pre-execute the price modification first. The price corresponding to the product in the database is not modified under this operation. After the pre-execution is successful, the second step is to execute the commit operation to complete the price update in the database. In the B / S architecture, the database operation instructions of the first and second steps in the related technology will be assigned to different middle stations. Then the middle station that receives the second step database operation instruction will fail to execute because it cannot find the message of the successful pre-execution of the previous instruction.

[0053] During implementation, since there may be multiple database operation instructions with contextual relationships, these multiple operation instructions can be regarded as an operation instruction set. The first database operation instruction executed in sequence in the operation instruction set is the first operation instruction, and the remaining database operation instructions are subsequent operation instructions. In the embodiment of the present disclosure, the operation instructions belonging to the same window unique identifier will be assigned to the same target middle station for processing, which will enable the database operation instructions with contextual relationships to be successfully executed.

[0054] As in S302, when the database operation instruction is a subsequent operation instruction other than the first operation instruction in the operation instruction set having a contextual relationship, the target middle station corresponding to the first operation instruction is determined as the target middle station corresponding to the subsequent operation instruction.

[0055] During implementation, you can use a key-value pair to store the correspondence between the window unique identifier and the target middle station, so that you can quickly find the corresponding target middle station when processing subsequent operation instructions. You can use the window unique identifier as the key and the target middle station number as the value to build the first correspondence. Determine the target middle station by finding the first correspondence.

[0056] In addition, a load balancing strategy can be used to distribute database operation instructions with the same window unique identifier to the same middle station. For example, the same target middle station can be determined by obtaining the hash value of the window unique identifier.

[0057] In the disclosed embodiment, the first operation instruction is distributed based on the load balancing strategy, and the target middle station corresponding to the first operation instruction is determined, which is conducive to screening the appropriate middle station in the B / S architecture to process the database operation instruction and improve the resource utilization of the middle station. In addition, by distributing all the operation instructions that have a contextual relationship with the first operation instruction to the same target middle station, it can be ensured that the following database operation instructions can be executed based on the execution status of the above database operation instructions, ensuring that the tasks can be smoothly transferred step by step, and improving the success rate of task execution.

[0058] In some embodiments, the database operation instruction is executed on the database based on the long connection with the database, such as Figure 4 As shown, the target middle station is specifically used for:

[0059] S401, when the database operation instruction is the first operation instruction in an operation instruction set having a contextual relationship, record a long connection used to execute the database operation instruction.

[0060] S402: When a subsequent operation instruction having a contextual relationship with the database operation instruction is received, a recorded long connection is searched, and the subsequent operation instruction is executed based on the found long connection.

[0061] During implementation, when the middle station receives multiple target tasks corresponding to the window unique identifier, the correspondence between the middle station and the window unique identifier can be screened out from the first correspondence, and the number of the long link is used as another value to construct a second correspondence. The second correspondence is recorded in the middle station so that subsequent operation instructions in the operation instruction set can determine the long link by searching the second correspondence.

[0062] The second corresponding relationship is shown in Table 1:

[0063] Table 1

[0064] Window unique identifier The number of the middle station Long link number A Middle Platform A Long Link A B Middle Platform A Long link B … … …

[0065] Among them, the B / S architecture is essentially non-connection-oriented. Since multiple operations of the database may have contextual relationships, operations facing the database require long connections. The following operation instructions can only be executed if the previous operation instructions are executed successfully. Therefore, the instructions with contextual relationships need to be executed in sequence, and the operation instructions with contextual relationships need to be completed in the same long connection (session). During implementation, the operation instructions with contextual relationships can be forwarded based on the same long link to ensure the success rate of subsequent operation instruction execution.

[0066] In the disclosed embodiment, a long link is used as a forwarding rule so that operation instructions with context association are forwarded based on the same long link, thereby realizing that the operation instructions of the context can be executed in sequence and ensuring the smooth progress of the entire operation.

[0067] In some embodiments, the variable transfer strategy through cross-threads is executed, and the execution process of the database operation instruction is recorded step by step in the first log file, and the target middle station is specifically used for:

[0068] Step A1: distribute the database operation instruction to the main thread, so that the main thread records at least part of the variables to be recorded in the target field based on the database operation instruction.

[0069] During implementation, you can determine which variables to be recorded in the log according to business needs. Since the database operation instructions are passed from the main thread to the child thread during execution, these variables to be recorded may be obtained by different threads. During implementation, you need to first assign the database operation instructions to the main thread, and the main thread will collect the variables to be recorded that the main thread can obtain according to the task requirements, and record and store them through ThreadLocal (thread variables). The remaining unobtained variables to be recorded can be collected and recorded by the child thread that executes the database operation instructions according to business needs. The child thread is also implemented through ThreadLocal, so that the variables to be recorded can be passed across threads and can be decoupled from the business code to reduce business intrusion.

[0070] Step A2, using the main thread to create a child thread based on the at least part of the variables to be recorded, and after passing the at least part of the variables to be recorded to the child thread for storage, the main thread clears the target field of the main thread so that the main thread processes the next database operation instruction.

[0071] When implementing, the specific Figure 5 As shown, after the main thread creates a child thread based on the variable to be recorded, the variable to be recorded is passed from the main thread to the child thread, and the child thread is used to record the variable to be recorded. In addition, the child thread obtains other variables to be recorded based on the business tracking method. When the main thread passes the variable to be recorded to the child thread, the ThreadLocal of the main thread is cleared so that the main thread can process the next database operation instruction.

[0072] The get method can be called to obtain the variables to be recorded, and the set method can be called to record the variables to be recorded.

[0073] Step A3, in the process of using the child thread to execute the database operation instruction to complete the business operation, the variables to be recorded corresponding to the target step are added to the target field of the child thread through the embedding point until the required multiple variables to be recorded are collected, and then the multiple variables to be recorded are spliced ​​according to the preset format and written into the first log file.

[0074] In the disclosed embodiment, the variable to be recorded is passed from the main thread to the child thread, which is executed by the child thread, thereby realizing cross-thread variable transfer, so that the required information can be written into the log and saved. Moreover, in the disclosed embodiment, the storage resources can be effectively utilized by variable transfer to minimize the memory usage. The main thread and the child thread only need to focus on recording the variables to be recorded at the location of the buried point, and the main thread and the child thread do not need to perform redundant and complex information collection processes, so that the acquisition of monitoring information can be decoupled from the business, ensuring the execution efficiency of the business. In addition, recording information in the form of log files can make this solution easy to implement, so that information monitoring can be lightweight and convenient to a certain extent.

[0075] In some embodiments, the sub-thread of the target middle station is specifically used to execute the splicing of the multiple variables to be recorded according to a preset format and write them into the first log file: when there is a need to concurrently write the first log file with other sub-threads in the target middle station, the sub-thread competes for a write lock; when the write lock is obtained through competition, the multiple variables to be recorded spliced ​​according to the preset format are written into the first log file as a record of the first log file.

[0076] The read-write lock is divided into two parts: the read lock and the write lock. The read lock allows multiple threads to obtain it at the same time, because the read operation itself is thread-safe, while the write lock is a mutex lock, which does not allow multiple threads to obtain the write lock at the same time, and the write operation and the read operation are also mutually exclusive. This ensures the safety of the thread.

[0077] During implementation, since each middle station corresponds to a first log file, the variables to be recorded obtained by multiple child threads can be written into the same first log file. In this case, there may be multiple child threads writing content to the first log file at the same time, so the child threads need to compete for the write lock. When the write lock is obtained through competition, the multiple variables to be recorded spliced ​​in a preset format are written into the first log file as a record of the first log file.

[0078] The preset format can be exemplarily connected with special characters or in a fixed order, as long as different fields can be distinguished, and the embodiments of the present disclosure are not limited to this.

[0079] The process of burying points is as follows Figure 6As shown, the example defines two types, including a first log file for burying points and a second log file for statistics. Among them, the first log file and the second log file can be recorded using one log file, or they can be recorded using their own files. However, both need to be executed using their own corresponding threads. In the burying point thread, it is necessary to first obtain the variables to be recorded in order to collect a variety of burying point information (i.e., various required variables to be recorded). The various burying point information is spliced ​​according to the preset format to obtain the splicing information and write it into the first log file. The statistical thread needs to read the first log file and split it to obtain the required burying point information. During implementation, the statistical thread parses the burying point information based on the preset format, filters out the required burying point information, and performs statistics. Then, the statistical results can be stored in the second log file and returned to the control layer, so that when the user needs to view the monitoring information, the statistical results can be obtained from the second log file for viewing, and even the information of the second log files of multiple middle stations can be summarized for secondary statistical analysis and displayed to the user.

[0080] The first log file may store multiple lines of records, each of which includes the required variables to be recorded. When there are different operations, each database operation may correspond to a line of records. For example, for multiple database operation instructions with a contextual relationship, each database operation instruction may correspond to a line of records. Figure 6 As shown, after a row of records is statistically analyzed to obtain statistical results, the row of records can be deleted from the first log file.

[0081] In the disclosed embodiment, the use of a write lock can improve the execution performance of a thread. Since multiple read locks can be executed simultaneously, compared to ordinary locks that must be queued for execution under any circumstances, the write lock improves the execution performance of the thread. In addition, since the read lock and the write lock are mutually exclusive and queued for execution, it can be ensured that the read operation will not read half-written temporary data. This ensures that the storage space of the child thread is not wasted.

[0082] In some embodiments, the sub-thread of the target middle station is also used to: splice the multiple variables to be recorded according to a preset format, write them into the first log file, and then clear the target field of the sub-thread to release the storage resources of the target field of the sub-thread.

[0083] The preset formats have been described above, and the embodiments of the present disclosure will not be described in detail here.

[0084] After the child thread writes the variables to be recorded into the first log file, the content of the child thread can be cleared to release the storage resources of the child thread so that the child thread can prepare for counting the variables to be recorded corresponding to the next operation instruction.

[0085] In the disclosed embodiment, after the variables to be recorded are written into the log file, the target field of the child thread can be cleared to release the storage resources of the child thread in time to cope with a large number of operation instructions.

[0086] In some embodiments, statistical analysis is performed based on the first log file, and the target middle station is specifically used to: use a statistical thread to group multiple records in the first log file according to preset grouping rules; wherein multiple database operation instructions with contextual relationships are divided into the same group; for multiple records in the same group, statistical analysis is performed according to preset statistical rules to obtain statistical analysis results.

[0087] When implementing, in the statistics thread, such as Figure 7 As shown, the split buried point information (i.e., each field in the variable to be recorded) is processed, and the processing process includes sorting and grouping. Among them, each record in the first log file can be sorted based on the time order during sorting, and grouping means that each record in the first log file is grouped according to the grouping method required for monitoring. For log records of multiple database operation instructions with contextual relationships, they need to be divided into the same group. Statistics are taken for the operation instructions of each group. During implementation, the flag type of each database operation instruction can be defined to help the statistical algorithm perform statistical analysis. Exemplarily, for transaction-type operations, three types can be provided, including: start type, end type, and intermediate type.

[0088] The start type indicates that the business task has started to execute, the intermediate type indicates that the business task has not yet been completed, and the end type indicates that the business task has been completed. In the log, each record of each database operation instruction will record the execution time and / or end time, and even detailed information such as the operator and the operation object.

[0089] For the same preset group, if the current log record in the first log file is judged to be the start type, a new statistical record can be initialized based on the current log record and written into the second log file. In the exemplary order process, in the case of the start type of the order, information such as the product name and the user number can be obtained, and a new statistical record is created for the product name and the user number and written into the second log file. If the last record in the preset group sorted by execution time is the intermediate type, it means that the execution has not been completed, the group of log records is saved, and the group of log records is deleted to release the cache, waiting to obtain the log records in other groups for statistical analysis. Thus, the start type and the intermediate type represent that the task has not been completely completed, and the created log record can still be supplemented based on the log record obtained later. In the case that the current log record is the end type, all the information required for statistics of the group can be obtained, so a complete statistical record can be generated and written into the second log file, and the log records corresponding to the start type to the end type of the group are deleted.

[0090] In some embodiments, taking the operation instruction in the group as the process of placing an order as an example, the variables to be recorded may include the following fields: operation SQL, operation object database, user number, request time, operation type, etc. A Map may be created first to store the fields in the variables to be recorded, and the operation instructions in the group are traversed to obtain the current timestamp, and the corresponding operation is performed based on the type of the operation instruction. T _START indicates the start of execution, indicates the start type, records the time when SQL execution starts, obtains the operation object database and user number, initializes a new statistical record and writes it to the second log file. T _E XCUTE Indicates that SQL is being executed, indicating the intermediate type, OPT_OVER indicates that the execution is completed, indicating the end type, and records the end time, and records it in the second log file. In this way, the request time can be counted, and the request time can be understood as the execution end time minus the start time. Finally, the operation instructions for the start and end of SQL execution are removed from the first log file, and the relevant content is recorded in the second log file.

[0091] It should be noted that no matter what type, the disclosed embodiments will update the task status in a timely manner based on the type. For example, assuming that multiple log records in the same group are sorted in chronological order, in order to obtain the task status as quickly as possible, the flag type of the log record can be obtained from a log record with the largest timestamp (i.e., the closest to the current time). In the case where the flag type is an end type, the execution status is determined to be ended; in the case where the flag type is an intermediate type, the execution status is determined to be in progress; in the case where the flag type is a start type, the execution status is determined to be started. In this way, the execution status can be quickly captured based on the log records in the same group and sorted by time.

[0092] In the disclosed embodiment, the operation instructions are first grouped so that the operation instructions in the same group have a contextual relationship. Based on this method, data monitoring can be accurately achieved.

[0093] Based on the same technical concept, a data processing method is proposed in the embodiment of the present disclosure, which is applied to the aforementioned middle platform, such as Figure 8 As shown, including:

[0094] S801, executing a database operation instruction on the database based on the long connection with the database.

[0095] The database operation instruction is carried in the operation request sent by the terminal device based on the target window. The operation request also carries the window unique identifier of the target window. The window unique identifier is used by the intermediate proxy server to distribute the database operation instruction in the target window to the same middle station.

[0096] S802, recording the execution process of the database operation instruction step by step into a log file through a cross-thread variable transfer strategy.

[0097] S803, perform information monitoring and statistical analysis based on the log file.

[0098] The log file may include a first log file for embedding points and a second log file for statistics. The first log file and the second log file may be the same log file or two log files.

[0099] In the disclosed embodiment, by executing database operation instructions on the database based on long links, operation instructions with contextual relationships can be distributed to the same middle station. In addition, the cross-thread variable transfer strategy ensures that each thread is effectively utilized, and information monitoring and statistical analysis can be achieved efficiently and accurately.

[0100] In summary, in some embodiments, the fields used for tracking in the first log file may exemplarily include at least one of the following:

[0101] "token" indicates the unique identifier of the window;

[0102] "methodName" indicates the operation type;

[0103] "operator" means the operator;

[0104] "userId" indicates the user ID.

[0105] In some embodiments, part of the content of the first log file for tracking points constructed using the aforementioned information is as follows:

[0106] 1724141850194 / / Timestamp

[0107] SQL / / Type

[0108] 2374161e-9113-449a-b9ae-1ab07e41939a / / Long link number --- [http-nio-8088-exec-7] / / Thread number

[0109] (EXECUTE_START) / / / The type of the operation instruction

[0110] <-=->{"containerId":"SYSTEM@d3e5bfa6fcb24b1e94de3c0bacb23ae6

[0111] / helowin", / / / Container number in the database

[0112] "token":"sql_console_a239018c-9201-4292-820f-8c420873f245", / / window unique identifier

[0113] "sql":"ALTER SESSION SET CURRENT_SCHEMA\u003dBOBO", / / Corresponding operations of the database

[0114] "methodName":"execute", / / Operation type

[0115] "operator":"admin (super administrator)", / / operator

[0116] "userId":"7a63105a56a3408088ca0277d089d375", / / User ID

[0117] "connectionPattern":2} / / Connection mode

[0118] In some embodiments, the fields used for statistics in the second log file may exemplarily include at least one of the following:

[0119] "private string sqL" indicates the executed sql statement;

[0120] "private Long executeCount" indicates the number of executions;

[0121] "private Long executeTime" indicates the execution time;

[0122] “private Long slowest” indicates the longest execution time;

[0123] "private Integer transactionExecution" indicates the number of transactions;

[0124] "private Integer errorcount" indicates the number of errors;

[0125] "private Long renewRowCount" indicates an update operation;

[0126] "private Long readRowCount" indicates the number of rows read;

[0127] "private Integer ongoing" indicates the number of execution states;

[0128] "private int maxConcurrent" indicates the maximum number of concurrent executions;

[0129] "private long[]executeTimePart" indicates the detailed list of execution time;

[0130] "private long[]executeRsTimePart" indicates a detailed list of time consumption;

[0131] "private long[]readRowPart" indicates reading time list.

[0132] In some embodiments, the second log file for statistics constructed using the aforementioned information may also include:

[0133] 1724141741982 / / Timestamp

[0134] STAT_SQL null / / type

[0135] ---[daemon-Thread-3]() / / Thread number

[0136] <-=->

[0137] {"sql":"select id as id1 from ddd ddd", / / Corresponding operations of the database

[0138] "slowest":0, / / delay duration

[0139] "errorCount":1, / / Number of errors

[0140] "ongoing":0, / / Number of execution states

[0141] "maxConcurrent": 0, / / Maximum number of concurrent executions

[0142] "executeTimePart":[0,0,0,0,0,0,0,0], / / Detailed list of execution time

[0143] "executeRsTimePart":[0,0,0,0,0,0,0,0], / / Detailed list of time consumption

[0144] "readRowPart":[0,0,0,0,0,0], / / Read time list

[0145] "renewRowPart":[0,0,0,0,0,0]} / / Update time list

[0146] Based on the same technical concept, a data processing device 900 is proposed in the embodiment of the present disclosure, such as Fig. 9 As shown, including:

[0147] An execution module 901 is used to execute a database operation instruction on a database based on a long connection with the database;

[0148] The recording module 902 is used to record the execution process of the database operation instruction in a log file step by step through a cross-thread variable transfer strategy;

[0149] The processing module 903 is used to perform information monitoring and statistical analysis based on the log file.

[0150] For the description of specific functions and examples of each module and submodule of the device in the embodiment of the present disclosure, reference can be made to the relevant description of the corresponding steps in the above method embodiment, which will not be repeated here.

[0151] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0152] Fig.10 FIG. 1 is a structural block diagram of an electronic device according to an embodiment of the present disclosure. Fig.10 As shown, the electronic device includes: a memory 1010 and a processor 1020, and the memory 1010 stores a computer program that can be run on the processor 1020. The number of memories 1010 and processors 1020 can be one or more. The memory 1010 can store one or more computer programs. When the one or more computer programs are executed by the electronic device, the electronic device executes the method provided by the above method embodiment. The electronic device may also include: a communication interface 1030, which is used to communicate with external devices and perform data exchange transmission.

[0153] If the memory 1010, the processor 1020 and the communication interface 1030 are implemented independently, the memory 1010, the processor 1020 and the communication interface 1030 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.10 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0154] Optionally, in a specific implementation, if the memory 1010, the processor 1020 and the communication interface 1030 are integrated on a chip, the memory 1010, the processor 1020 and the communication interface 1030 can communicate with each other through an internal interface.

[0155] It should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. It is worth noting that the processor may be a processor supporting the Advanced RISC Machines (ARM) architecture.

[0156] Further, optionally, the above-mentioned memory may include a read-only memory and a random access memory, and may also include a non-volatile random access memory. The memory may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may include a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may include a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available. For example, static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct memory bus random access memory (DR RAM).

[0157] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, the process or function described in the embodiment of the present disclosure is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center by wired (for example: coaxial cable, optical fiber, data subscriber line (Digital Subscriber Line, DSL)) or wireless (for example: infrared, Bluetooth, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access, or a data storage device such as a server or data center that includes one or more available media integrations. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a digital versatile disc (DVD)), or a semiconductor medium (e.g., a solid state disk (SSD)), etc. It is worth noting that the computer-readable storage medium mentioned in the present disclosure may be a non-volatile storage medium, in other words, a non-transient storage medium.

[0158] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0159] In the description of the embodiments of the present disclosure, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present disclosure. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, unless they are contradictory.

[0160] In the description of the embodiments of the present disclosure, unless otherwise specified, " / " means or, for example, A / B can mean A or B. "And / or" in this article is only a way to describe the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0161] The above description is only an exemplary embodiment of the present disclosure and is not intended to limit the present disclosure. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A data processing system, comprising: The terminal device is used to generate a window unique identifier for the target window of the current operation, and generate an operation request carrying the window unique identifier and the database operation instruction for the database operation instruction in the target window, and send the operation request to the intermediate proxy server; The intermediate proxy server is used to filter out a target middle station corresponding to the window unique identifier from multiple middle stations of the B / S architecture based on the window unique identifier in the operation request, and send the database operation instruction to the target middle station for processing; The middle station is used to execute the database operation instruction on the database based on the long connection between the middle station and the database when the middle station is the target middle station; And, through a cross-thread variable transfer strategy, the execution process of the database operation instruction is recorded step by step into a first log file; And information monitoring and statistical analysis are performed based on the first log file.

2. The system according to claim 1, wherein: Executing the operation request based on the window unique identifier, filtering out a target middle station corresponding to the window unique identifier from multiple middle stations of the B / S architecture, the intermediate proxy server is specifically used to: In the case where the database operation instruction is the first operation instruction in the operation instruction set with a contextual relationship, a middle station is selected from multiple middle stations of the B / S architecture based on a load balancing strategy as the target middle station corresponding to the window unique identifier; In the case where the database operation instruction is a subsequent operation instruction other than the first operation instruction in the operation instruction set having a contextual relationship, the target middle station corresponding to the first operation instruction is determined as the target middle station corresponding to the subsequent operation instruction.

3. The system according to claim 1, wherein: Executing the database operation instruction based on the long connection with the database, the target middle station is specifically used for: In the case where the database operation instruction is the first operation instruction in the operation instruction set having a contextual relationship, recording the long connection used to execute the database operation instruction; When a subsequent operation instruction having a contextual relationship with the database operation instruction is received, the recorded long connection is searched, and the subsequent operation instruction is executed based on the found long connection.

4. The system according to claim 1, wherein: Execute the variable transfer strategy across threads, and record the execution process of the database operation instruction step by step into the first log file. The target middle station is specifically used for: Distributing the database operation instruction to the main thread, so that the main thread records at least part of the variables to be recorded in the target field based on the database operation instruction; The main thread is used to create a child thread based on at least part of the variables to be recorded, and after the at least part of the variables to be recorded is passed to the child thread for storage, the main thread clears the target field of the main thread so that the main thread processes the next database operation instruction; In the process of using the child thread to execute the database operation instruction to complete the business operation, the variables to be recorded corresponding to the target step are added to the target field of the child thread through the embedding point until the required multiple variables to be recorded are collected, and then the multiple variables to be recorded are spliced ​​according to the preset format and written into the first log file.

5. The system according to claim 4, wherein: Executing the step of splicing the multiple variables to be recorded according to a preset format and writing them into the first log file, the sub-thread of the target middle station is specifically used for: When there is a need to concurrently write the first log file with other sub-threads in the target middleware, the sub-thread competes for a write lock; When the write lock is obtained through competition, the plurality of variables to be recorded that are spliced ​​according to a preset format are written into the first log file as a record of the first log file.

6. According to the system of claim 4, the sub-thread of the target middle station is also used for: After the multiple variables to be recorded are spliced ​​according to a preset format and written into the first log file, the target field of the child thread is cleared to release storage resources of the target field of the child thread.

7. The system according to claim 1, wherein: Performing statistical analysis based on the first log file, the target middle station is specifically used for: Using a statistical thread, multiple records in the first log file are grouped according to a preset grouping rule; wherein multiple database operation instructions having a contextual relationship are grouped into the same group; For multiple records in the same group, statistical analysis is performed according to preset statistical rules to obtain statistical analysis results.

8. A data processing method, comprising: Based on the long connection with the database, execute database operation instructions on the database; By using a cross-thread variable transfer strategy, the execution process of the database operation instruction is recorded step by step into a log file; Information monitoring and statistical analysis are performed based on the log files.

9. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of claim 8.

10. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to claim 8.

Images