Data processing method and device, storage medium and electronic equipment

Abstract

The invention discloses a data processing method and device, a storage medium and electronic equipment. The method comprises the steps that a data request is sent to each application in a group of applications, a group of log information corresponding to the group of applications is obtained, and one piece of log information in the group of log information is used for indicating the running state of each application in a preset time period; a target application is determined from the group of applications according to a target time period and a target service indicated by target log information in the group of log information, application information of the target application is obtained, and the target log information is used for indicating the running state of the target application in the target time period. The application information comprises a code package file needing to be called when the target application runs in the target time period and a target service code generated by the target application in the process of executing the target service, and the target service code is used for indicating the execution state of the target service; and displaying the target log information and the application information. The technical problem that the data processing efficiency is low is solved.

Description

Technical Field

[0001] The present application relates to the field of computers, and in particular to a data processing method and device, a storage medium, and an electronic device. Background Art

[0002] At present, in the relevant technologies, the traditional information collection method is mainly used to obtain application information, which is not only inefficient but also prone to errors. For example, when it is necessary to troubleshoot a specific business or detect the application running status, it is often necessary to spend a lot of manpower to manually search for relevant information in the log files of each application, resulting in technical problems such as low data processing efficiency in the relevant technologies.

[0003] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0004] The embodiments of the present application provide a data processing method and device, a storage medium, and an electronic device to at least solve the technical problem of low efficiency of data processing.

[0005] According to one aspect of an embodiment of the present application, a data processing method is provided, comprising: obtaining a data request; sending the data request to each application in a group of applications, obtaining a group of log information corresponding to the group of applications, wherein one log information in the group of log information is used to indicate the running status of each application within a preset time period; determining a target application from the group of applications according to a target time period and a target business indicated by a target log information in the group of log information, and obtaining application information of the target application, wherein the target log information is used to indicate the running status of the target application within the target time period, and the application information includes a code package file that needs to be called when the target application runs in the target time period and a target business code generated by the target application in the process of executing the target business, and the target business code is used to indicate the execution status of the target business; and displaying the target log information and the application information.

[0006] According to another aspect of an embodiment of the present application, a data processing device is also provided, including: an acquisition module, used to obtain a data request; a transmission module, used to send the data request to each application in a group of applications, and obtain a group of log information corresponding to the group of applications, wherein one log information in the group of log information is used to indicate the running status of each application within a preset time period; a determination module, used to determine a target application from the group of applications according to the target time period and target business indicated by the target log information in the group of log information, and obtain application information of the target application, wherein the target log information is used to indicate the running status of the target application within the target time period, and the application information includes a code package file that needs to be called when the target application runs in the target time period and a target business code generated by the target application in the process of executing the target business, and the target business code is used to indicate the execution status of the target business; a display module, used to display the target log information and the application information.

[0007] Optionally, the device is used to determine the target application from the group of applications according to the target time period and target business indicated by the target log information in the group of log information, and obtain the application information of the target application in the following manner: determine the time period log data from the target log information, wherein the time period log data is used to indicate the target time period; determine the target application from the group of applications based on the time period log data, wherein the target application is set to run within the target time period; obtain target code framework data associated with the target application from a group of application framework set data, wherein the group of application framework data set includes code framework data of each application, and the target code framework data is used to indicate the code package file; determine the application information based on the target code framework data.

[0008] Optionally, the device is also used to: before obtaining the target code framework data associated with the target application from a set of application framework set data, obtain the application name, startup time, and configuration file of the target application; when the configuration file includes a framework configuration item, query the set of application framework data sets according to the framework configuration item to obtain the target code framework data, wherein the framework configuration item is used to indicate a code framework pre-set for the target application; and store the target code framework data in the set of application framework data sets.

[0009] Optionally, the device is used to determine the target application from the group of applications according to the target time period and target business indicated by the target log information in the group of log information, and obtain the application information of the target application in the following manner: determine business log data from the target log information, wherein the business log data is used to indicate the target business; determine the target application from the group of applications based on the business log data, wherein the target application can execute the target business; obtain the target business code associated with the target application from a set of business code sets, wherein the set of business code sets includes the execution status of the target business in each of the applications that can execute the target business; determine the application information based on the target business code.

[0010] Optionally, the device is also used to: before obtaining the target business code associated with the target application from a set of business codes, obtain the application name, startup time and configuration file of the target application; when the configuration file includes a business configuration item, query the set of business codes according to the business configuration item to obtain the target business code, wherein the business configuration item is used to indicate the business that the target application can execute; and store the target business code in the set of business codes.

[0011] Optionally, the device is also used to: deploy a target data plug-in in the target application, wherein the target data plug-in is used to collect the application information; and use the target data plug-in to obtain the code package file and the target business code.

[0012] Optionally, the device is also used to: in response to the start of the target application, obtain first code framework data collected by the target data plug-in, wherein the first code framework data is used to indicate the code package file called when the target application is initialized and started; in response to a framework update request, obtain second code framework data collected by the target data plug-in, wherein the second code framework data is used to indicate the code package file called when the target application needs to update the code framework; in response to the start of the target application, obtain a first business code collected by the target data plug-in, wherein the first business code is used to indicate the execution status of the target business when the target application executes the target business for the first time; in response to a business update request, obtain a second business code collected by the target data plug-in, wherein the second business code is used to indicate the execution status of the target business when the target application executes the target business again when the target business is updated.

[0013] According to another aspect of the embodiments of the present application, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the above-mentioned data processing method when running.

[0014] According to another aspect of the embodiments of the present application, a computer program product or a computer program is provided, the computer program product or the computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the above data processing method.

[0015] According to another aspect of the embodiments of the present application, there is further provided an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the data processing method through the computer program.

[0016] In an embodiment of the present application, a data request is adopted; a data request is sent to each application in a group of applications, and a group of log information corresponding to the group of applications is obtained, wherein one log information in a group of log information is used to indicate the running status of each application within a preset time period; a target application is determined from a group of applications according to a target time period and a target business indicated by a target log information in a group of log information, and application information of the target application is obtained, wherein the target log information is used to indicate the running status of the target application within the target time period, and the application information includes a code package file that needs to be called when the target application runs in the target time period and a target business code generated by the target application in the process of executing the target business, and the target business code is used to indicate the execution status of the target business; the target log information and application information are displayed in a way that the log information is intelligently screened, and the log information and application information associated with the log information are displayed through a visual interface, so as to achieve the purpose of quickly locating and efficiently analyzing the running status of the application, thereby achieving the technical effect of refined information supervision and collection, and further solving the technical problem of low data processing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0018] Figure 1 is a schematic diagram of an application environment of an optional data processing method according to an embodiment of the present application;

[0019] Figure 2 is a flowchart of an optional data processing method according to an embodiment of the present application;

[0020] Figure 3 is a schematic diagram of an optional data processing method according to an embodiment of the present application;

[0021] Figure 4 is a schematic structural diagram of an optional data processing device according to an embodiment of the present application;

[0022] Figure 5 is a schematic diagram of the structure of an optional data processing product according to an embodiment of the present application;

[0023] Figure 6 It is a schematic diagram of the structure of an optional electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0024] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0025] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0026] The present application is described below in conjunction with embodiments:

[0027] According to one aspect of an embodiment of the present application, a data processing method is provided. Optionally, in this embodiment, the data processing method can be applied to Figure 1 In the hardware environment composed of the server 101 and the terminal device 103 shown in FIG. Figure 1As shown, the server 101 is connected to the terminal 103 via a network, and can be used to provide services for the terminal device or an application 107 installed on the terminal device. The application can be a video application, an instant messaging application, a browser application, an educational application, a game application, etc. A database 105 may be set up on the server or independently of the server to provide data storage services for the server 101, for example, a game data storage server. The above-mentioned network may include, but is not limited to, a wired network and a wireless network, wherein the wired network includes, for example, a local area network, a metropolitan area network, and a wide area network; the wireless network includes, for example, Bluetooth, WIFI, and other networks that implement wireless communication; the terminal device 103 may be a terminal configured with an application program, and may include, but is not limited to, at least one of the following: a mobile phone (such as an Android phone, an iOS phone, etc.), a laptop computer, a tablet computer, a PDA, a MID (Mobile Internet Devices), a PAD, a desktop computer, a smart TV, an intelligent voice interaction device, a smart home appliance, a vehicle-mounted terminal, an aircraft, a virtual reality (VR) terminal, an augmented reality (AR) terminal, a mixed reality (MR) terminal, and other computer devices; the above-mentioned server may be a single server, a server cluster consisting of multiple servers, or a cloud server.

[0028] Combination Figure 1 As shown, the above data processing method can be executed by an electronic device, which can be a terminal device or a server. The above data processing method can be implemented by the terminal device or the server respectively, or by the terminal device and the server together.

[0029] The above is only an example and is not specifically limited in this embodiment.

[0030] Optionally, as an optional implementation, as Figure 2 As shown, the above data processing method includes:

[0031] S202, obtaining data request;

[0032] S204, sending a data request to each application in a group of applications, obtaining a group of log information corresponding to the group of applications, wherein one log information in the group of log information is used to indicate the running status of each application within a preset period of time;

[0033] Optionally, in an embodiment of the present application, the above-mentioned data request includes but is not limited to request instructions for requesting to obtain and summarize static code information and dynamic running status data of the application, for example, requesting application name, basic framework version, code package file, transaction code (the above-mentioned business code) and log information, etc.; the above-mentioned set of log information may include but is not limited to collecting and summarizing log information of multiple applications through the ELK platform (log analysis and visualization platform), including but not limited to filebeat collecting logs from each application, Logstash formatting and filtering logs, Elasticsearch storing logs, and Kibana displaying log content, etc.

[0034] For example, in the application scenario of real-time monitoring and troubleshooting of transaction applications, when the application operation management platform needs to understand the operating status of all transaction applications within a preset time period, it will send a data request to each application in the transaction application cluster. After each of the above applications receives the data request, its built-in filebeat module starts to collect log information for that period, and the Logstash module can format and filter the received logs by keywords.

[0035] It should be noted that the preset time period can be flexibly set, such as the last hour, the last day, or the last week, etc. At the same time, in the above example, the operation management platform can not only collect log information, but also obtain static code information of the application, such as the framework version and business code used, so as to achieve more comprehensive and refined monitoring of the application.

[0036] S206, determining a target application from a group of applications according to a target period and a target service indicated by target log information in a group of log information, and obtaining application information of the target application, wherein the target log information is used to indicate the running state of the target application in the target period, and the application information includes a code package file that needs to be called when the target application runs in the target period and a target service code generated by the target application in the process of executing the target service, and the target service code is used to indicate the execution state of the target service;

[0037] S208, display target log information and application information.

[0038] Optionally, in an embodiment of the present application, the above-mentioned target log information refers to a log record that is selected from a group of log information according to specific query conditions (such as target time period, target business) and can reflect the running status of the target application, including but not limited to error information, execution time, etc.; the above-mentioned application information refers to data obtained from the target application, which can reflect the basic framework information used in the target time period, the called code package files, and the business codes generated in the process of executing the target business, including but not limited to the application name, version information, framework dependency, transaction code (one of the above-mentioned target business codes) and execution status, etc.

[0039] It should be noted that in different application scenarios, the above-mentioned target business, the above-mentioned target application and the target business code can be used to indicate different contents. For example, in a banking transaction scenario, the above-mentioned target application may include but is not limited to "online banking transaction system", "ATM withdrawal system" or "credit card payment system", etc., and the above-mentioned target business code may include but is not limited to "transfer transaction code", "balance inquiry transaction code" or "payment transaction code", etc. The target business code can be understood as a code identifier used to identify and execute specific business functions within the target application. The above-mentioned target business may include but is not limited to "user account balance inquiry", "inter-bank transfer" or "credit card repayment", etc.; for another example, in a smart home operation and management scenario, the above-mentioned target application may include but is not limited to "smart lighting control system application", "smart security monitoring application" or "smart temperature control system application", etc., the above-mentioned target business code may include but is not limited to "lighting instruction code", "security alarm processing code" or "temperature adjustment code", etc., and the above-mentioned target business may include but is not limited to "smart home lighting control", "intrusion detection and alarm" or "smart environment temperature and humidity adjustment", etc.

[0040] It should also be noted that, among the above group of applications, several applications may be determined as the above target applications, and the type and number of target service codes are not limited.

[0041] For example, in the application scenario of bank transaction troubleshooting, suppose the bank's operation management platform receives a user complaint about a transaction anomaly, and the system administrator needs to quickly locate the target application corresponding to the transaction and understand its operation during the abnormal period. At this time, first obtain the above set of log information, then determine the target log information based on the time period and location of the transaction anomaly in a set of log information, and obtain the target time period and target business, specifically:

[0042] Relevant target log information can be filtered out based on keywords of the target time period (e.g., the time period when the target application anomaly occurs) and the target business (e.g., a specific transaction reported by a user). The target log information may include, but is not limited to, transaction codes and error information to help administrators make preliminary judgments on business applications that may have errors.

[0043] Furthermore, through the service list function of the operation management platform, the detailed application information of the target application is queried, including the code package files called within the target time period and the target business code generated, to further confirm the root cause of the fault.

[0044] In an exemplary embodiment, Figure 3 is a schematic diagram of an optional data processing method according to an embodiment of the present application. Taking the banking business scenario as an example, the system architecture is as follows Figure 3 Specifically, the data processing method proposed in this application may include but is not limited to:

[0045] The ELK (Elasticsearch, Logstash, Kibana) open source platform is introduced into the banking application to centrally manage and analyze the above set of log information in real time. At the same time, a data collection module is defined in the basic framework of the banking application (for example, an SDK plug-in for automatic data collection, that is, the above target data plug-in). When the application is started, the module automatically scans and collects the basic framework information of the application (including management framework, transaction framework, etc.), version package, key transaction code (the above target business code), etc., sends the data to the operation management platform through the HTTP protocol and stores it, and finally displays it in the service list of the operation management platform in the banking system.

[0046] Through the embodiment of the present application, a data request is obtained; a data request is sent to each application in a group of applications, and a group of log information corresponding to the group of applications is obtained, wherein one log information in the group of log information is used to indicate the running status of each application within a preset time period; a target application is determined from a group of applications according to the target time period and target business indicated by the target log information in the group of log information, and application information of the target application is obtained, wherein the target log information is used to indicate the running status of the target application within the target time period, and the application information includes a code package file that needs to be called when the target application runs in the target time period and a target business code generated by the target application in the process of executing the target business, and the target business code is used to indicate the execution status of the target business; the target log information and application information are displayed in a way that the log information is intelligently screened, and the log information and application information associated with the log information are displayed through a visual interface, so as to achieve the purpose of quickly locating and efficiently analyzing the running status of the application, thereby achieving the technical effect of refined information supervision and collection, and further solving the technical problem of low data processing efficiency.

[0047] As an optional solution, the target application is determined from the above-mentioned group of applications according to the target time period and target business indicated by the target log information in the above-mentioned group of log information, and the application information of the above-mentioned target application is obtained, including: determining the time period log data from the above-mentioned target log information, wherein the above-mentioned time period log data is used to indicate the above-mentioned target time period; determining the above-mentioned target application from the above-mentioned group of applications based on the above-mentioned time period log data, wherein the above-mentioned target application is set to run within the above-mentioned target time period; obtaining the target code framework data associated with the above-mentioned target application from a group of application framework set data, wherein the above-mentioned group of application framework data set includes the code framework data of the above-mentioned each application, and the above-mentioned target code framework data is used to indicate the above-mentioned code package file; and determining the above-mentioned application information based on the above-mentioned target code framework data.

[0048] Optionally, in an embodiment of the present application, the above-mentioned time period log data refers to log data extracted from the target log information and associated with a preset target time period, including but not limited to all log entries generated during the time period, and the above-mentioned time period log data is used to reflect the operating status and business activities of the target application during a specific time period; the above-mentioned target code framework data refers to code framework information associated with the target application that is screened out from a set of application framework data sets, including but not limited to the basic framework name, version number and code package file used by the application, etc., which is used to indicate and describe the code architecture and dependency environment of the application.

[0049] For example, in the application scenario of bank system upgrade testing, the operation management platform needs to determine all active applications within a specific target period, as well as the basic framework versions used by these applications, in order to evaluate the feasibility and risks of the upgrade plan. First, the log data within the target period is queried through the ELK platform, that is, the above-mentioned period log data, and the applications with activity records within the target period are screened out, that is, the above-mentioned target applications; then, the code framework data corresponding to these target applications is obtained from the service list, that is, the above-mentioned target code framework data, to determine which code package files and framework versions are specifically used by the target application.

[0050] Through the embodiments of the present application, log information is collected and analyzed in real time, and an automated scanning operation of the application framework information is performed, thereby achieving the purpose of accurately tracking the running status of the target application and the code framework used, and achieving the technical effect of efficiently and accurately performing application supervision and troubleshooting.

[0051] As an optional solution, before obtaining the target code framework data associated with the target application from a set of application framework set data, the method further includes: obtaining the application name, startup time and configuration file of the target application; when the configuration file includes a framework configuration item, querying the set of application framework data according to the framework configuration item to obtain the target code framework data, wherein the framework configuration item is used to indicate a code framework pre-set for the target application; and storing the target code framework data in the set of application framework data.

[0052] Optionally, in an embodiment of the present application, the above-mentioned framework configuration items refer to configuration items pre-set in the configuration file of the above-mentioned target application, which are used to specify the code framework used by the application, including but not limited to framework name, version requirements, dependency path and other information, providing a query basis for the data processing module.

[0053] For example, in the application scenario of remote control of smart home devices, assuming that user feedback is received that the smart curtain control response is slow, it is necessary to determine the above target application that processes the curtain control request during the user operation period, and check whether the code framework used by it is the latest version or has known performance issues, specifically:

[0054] First, obtain the target application's "application name", "startup time" and "configuration file" and other information, then determine whether the configuration file includes framework configuration items. If so, query the application framework data set according to the framework configuration items to obtain the above-mentioned target code framework data.

[0055] It should be noted that the existence of the above framework configuration items can be determined based on the development environment and configuration management strategy of the target application, and there is no limitation on the specific form of the configuration file (such as XML, JSON, YAML, etc.).

[0056] Through the embodiments of the present application, the configuration file is read and an automated framework information scanning operation is performed, thereby achieving accurate acquisition and version update of the above-mentioned code framework data, and achieving the technical effect of improving application performance and quickly responding to operation and maintenance needs.

[0057] As an optional scheme, the target application is determined from the above-mentioned group of applications according to the target time period and target business indicated by the target log information in the above-mentioned group of log information, and the application information of the above-mentioned target application is obtained, including: determining business log data from the above-mentioned target log information, wherein the above-mentioned business log data is used to indicate the above-mentioned target business; determining the above-mentioned target application from the above-mentioned group of applications based on the above-mentioned business log data, wherein the above-mentioned target application can execute the above-mentioned target business; obtaining the above-mentioned target business code associated with the above-mentioned target application from a set of business codes, wherein the above-mentioned set of business codes includes the execution status of the above-mentioned target business in the above-mentioned each application that can execute the above-mentioned target business; and determining the above-mentioned application information based on the above-mentioned target business code.

[0058] Optionally, in an embodiment of the present application, the above-mentioned business log data refers to log data related to the execution of specific business functions extracted from the above-mentioned target log information, including but not limited to the start time, end time, transaction code, execution status and exception information of the business execution, etc., which is used to reflect the execution status of the target business in the target application. The above-mentioned business code set refers to the collection data of business codes generated when each application executes different businesses.

[0059] For example, in the application scenario of bank business anomaly troubleshooting, if a user account balance query anomaly occurs, it is necessary to quickly locate the application responsible for processing this business and analyze the cause of the problem, specifically:

[0060] First, determine the above-mentioned business log data related to balance query from the target log information, which may include but is not limited to the timestamp of the query operation, the user number, the account information involved and the exception stack, etc.;

[0061] Next, determine the target application that executes the target business based on the business log data; obtain the above-mentioned target business code associated with the target application from the business code set, for example, the execution status and version information of the balance query transaction code, and quickly take corresponding repair measures through analysis of the target business code.

[0062] Through the embodiments of the present application, automated business log data extraction and business code set query are used to achieve accurate detection of the execution status of the target business, thereby achieving the purpose of quickly locating business anomalies and optimizing them.

[0063] As an optional solution, before obtaining the target business code associated with the target application from a set of business codes, the method further includes: obtaining the application name, startup time and configuration file of the target application; when the configuration file includes a business configuration item, querying the set of business codes according to the business configuration item to obtain the target business code, wherein the business configuration item is used to indicate the business that the target application can execute; and storing the target business code in the set of business codes.

[0064] Optionally, in an embodiment of the present application, the above-mentioned business configuration items refer to configuration items pre-set in the configuration file of the target application, which are used to indicate the business functions that the application can execute, including but not limited to the name of the business function, business code, execution path and parameter configuration, etc., to provide a query basis for the data processing module; the above-mentioned business code set refers to the collection data of business codes generated when each application executes different businesses, including but not limited to the execution status of the business code, version information of the business code and the specific business function corresponding to the business code, etc.

[0065] In an exemplary embodiment, in an application scenario of banking business system operation and maintenance, it is assumed that the operation center has received a user complaint about a delay in a transfer business, and needs to determine the above-mentioned target application that executes the business and check whether the business configuration is normal. First, the name, startup time and configuration file of the application are obtained from the operation management platform, and then, it is checked whether the configuration file contains business configuration items related to the transfer business; further, for the target application containing business configuration items, the business code set can be queried according to the configuration items to obtain the above-mentioned target business code, that is, the business code information related to the transfer business.

[0066] It should be noted that the specific implementation of the above-mentioned business configuration items and business code sets of different target applications may be the same or different, depending on the architecture, business logic and logging strategy of the above-mentioned applications. In addition, the reading of the above-mentioned business configuration items and the query of the business code set may be static (such as reading configuration files, querying databases) or dynamic (such as obtaining real-time status through remote calls), and this application does not impose any restrictions on this.

[0067] Through the embodiments of the present application, automated reading of business configuration items and query of business code sets are realized, achieving the purpose of quickly responding to business anomalies and ensuring stable operation of the system.

[0068] As an optional solution, the method further includes: deploying a target data plug-in in the target application, wherein the target data plug-in is used to collect the application information; and obtaining the code package file and the target business code using the target data plug-in.

[0069] As an optional solution, the method further includes: in response to the start of the target application, obtaining first code framework data collected by the target data plug-in, wherein the first code framework data is used to indicate the code package file called when the target application is initialized and started; in response to a framework update request, obtaining second code framework data collected by the target data plug-in, wherein the second code framework data is used to indicate the code package file called when the target application needs to update the code framework; in response to the start of the target application, obtaining a first business code collected by the target data plug-in, wherein the first business code is used to indicate the execution status of the target business when the target application executes the target business for the first time; in response to a business update request, obtaining a second business code collected by the target data plug-in, wherein the second business code is used to indicate the execution status of the target business when the target application executes the target business again when the target business is updated.

[0070] Optionally, in an embodiment of the present application, the above-mentioned target data plug-in refers to a software component specially designed to monitor and collect application information, including but not limited to a data collection SDK, a code scanning tool or a dynamic proxy plug-in, etc., whose function is to extract data such as the execution status of code package files and business codes in real time during the operation of the target application; the above-mentioned first code framework data and the above-mentioned second code framework data respectively refer to the relevant information of the code package files collected by the target data plug-in when the target application is initialized and started and the framework is updated, including but not limited to the framework name, version number, dependency path, etc., which are used to track the framework configuration status of the application.

[0071] In an exemplary embodiment, in an application scenario of banking system maintenance, the target data plug-in is pre-deployed in the target application, and responds when the application is started to collect the first code framework data for verifying the framework configuration when the application is initialized.

[0072] In yet another exemplary embodiment, in response to a framework update request, the second code framework data is obtained through a target data plug-in to check whether the updated code package file has been correctly loaded and used.

[0073] Similarly, in response to application startup, the first service code can be obtained to ensure that the execution status of the target service meets expectations during the first execution; after the service is updated, the second service code is obtained to evaluate the impact of the service update on application execution.

[0074] It should be noted that the specific implementation of the above-mentioned target data plug-in depends on the development environment and technology stack of the above-mentioned target application, and this application does not make any specific limitations. In addition, the collection of the above-mentioned first code framework data and the above-mentioned second code framework data, the above-mentioned first business code and the above-mentioned second business code can be triggered by specific events of the target application, such as application startup, framework update or business update.

[0075] Through the embodiments of the present application, automated target data plug-in deployment and data collection are adopted to achieve real-time monitoring of code package files and business code execution status during application startup and framework update, thereby achieving the purpose of improving system security and maintenance efficiency.

[0076] In an exemplary embodiment, the data processing method proposed in this application can be applied to the field of application management. First, some terms in the embodiments of this application are explained:

[0077] Microservices: Divide a single application into small services, which use lightweight communication mechanisms to coordinate and cooperate with each other, and each runs in an independent process;

[0078] Operation management: The operation management platform is built based on a distributed framework and is a platform for monitoring, analyzing, diagnosing, and managing each node in operation;

[0079] ELK: An open source real-time log analysis platform.

[0080] Exemplarily, in an embodiment of the present application, the ELK platform is used to centrally manage application logs, and the framework data processing module is used to monitor the key frameworks and transaction codes used by the applications, thereby realizing centralized monitoring and management of the basic frameworks, version packages, key transaction codes and application running status used by each application.

[0081] It should be noted that, taking bank application management as an example, the bank system can be divided into transaction type and management type from the functional level. The transaction type is a customer-oriented system involving account information, and the management type tends to be a management-oriented system that does not involve account information. The two types of systems respectively use a unified framework as the basic framework. The embodiment of the present application utilizes the unity of the framework to design a general framework data processing module to collect the basic framework, version package, and key transaction code information of the application from various applications, and at the same time deploys the ELK platform to collect the logs of each application, so as to achieve the purpose of application monitoring and management. The log information of each application is collected through ELK and the relevant information of each application code framework is collected through the framework data processing module, and the two types of information are integrated on the operation management platform to realize automated data collection and processing and monitoring of the running status and code level of various applications, thereby helping system administrators to more conveniently understand the status of each application and improve the efficiency and accuracy of troubleshooting of each application.

[0082] In an exemplary embodiment, the service log and service list are divided into two parts according to the functional modules, as shown below:

[0083] S1, Service log: The main function of the service log module is to collect the log information of each application in real time, and to query the log according to the date, application name, keyword log level, and log content on the system front end; the front end can display the time, service name, level, class, and log content of each collected log. The system administrator can locate the location of the relevant code through the above log information to improve the efficiency of troubleshooting. Among them, the collection of log information can include but is not limited to the use of the ELK open source real-time log analysis platform; Filebeat is a lightweight data collection engine, which is configured in each application to collect application log file information; Logstash is mainly used for log collection, analysis and filtering, and is a tool for managing logs and events; Elasticsearch is a search server based on Lucene. It provides three major functions: search, analysis, and storage of data; Kibana: It is a front-end log display framework that can search and display data stored in Elasticsearch and provide powerful data visualization support;

[0084] The specific implementation process is as follows, including but not limited to:

[0085] S1-1, deploy filebeat (a lightweight log file collector for forwarding and centralizing log data) on the server where the application is located, and configure the log path for filebeat to collect logs in the application configuration file;

[0086] S1-2, the log information collected by filebeat is forwarded to Logstash for aggregation and formatting, and then output to Elasticsearch;

[0087] S1-3, Kibana queries logs from Elasticsearch and displays them to users on the operation management platform.

[0088] S2, Service List: The function implemented by the service list module is to collect, aggregate, process and display information at the code level of each application. The collection of application code level information can grasp the current framework version used by the application and the status of the transaction code, which is conducive to the centralized management and upgrading of each application. Among them, the code level information collection uses the method of defining the data collection module in the framework. Take the collection of framework information of transaction applications as an example to illustrate:

[0089] First, define an SDK plug-in (the target data plug-in mentioned above) for automatic data collection in the transaction class framework, which encapsulates a service class SapiClientSideService as a data processing module for receiving, processing and forwarding the collected application framework class information. As long as the application references this SDK plug-in, the data processing module can obtain the application name of the application and the basic framework information currently referenced by the application; then, after obtaining the basic information of the application, the module sends the information to the operation management platform through domain name addressing; the operation management platform processes the information and stores it in the data table of the data dictionary. The final data is displayed in the service list of the operation management platform.

[0090] Specifically, the above data processing module is the core function of the service list module, and the specific working principle is as follows:

[0091] S2-1, initialization phase: When the application starts, the system automatically loads the pre-integrated automatic data collection SDK plug-in. The Sapi Runner method of the plug-in implements the Application Runner interface and calls the start Pkg Report method to trigger the start of the data processing process;

[0092] S2-2, data acquisition phase: the data processing module first obtains the basic information of the currently running application, including but not limited to the application name, startup time, etc., to ensure that the collected data has clear ownership and timestamp, thereby ensuring the validity of the data;

[0093] S2-3, framework information scanning stage: according to the name of the framework information to be collected set in the configuration file, the data processing module will scan the specified package file and identify the various frameworks that the application depends on, so as to collect the detailed version information of these frameworks later;

[0094] S2-4, data encapsulation and sending stage: Once the required framework information version is found and recorded, the data processing module will encapsulate this information together with other basic application information into a request, and then send this request to the operation management platform through the HTTP protocol to complete the data upload;

[0095] S2-5, if data acquisition fails, data will be acquired again after 1000s.

[0096] It should be noted that the data scanning and collection work of this service is designed in the startup item configuration of the application project. As long as the project uses this framework, the framework will automatically collect, process and upload application-related data when the application is started, thereby achieving the purpose of automated statistical data. In addition, the dependency package information can be obtained by specifying the dependency package to be scanned according to actual needs. When the application is started, the application code can be scanned to obtain the specified dependency package information used by the application. This can be expanded to the screening of commonly used version packages, which significantly improves the efficiency and accuracy of application information management and reduces the cost of manual intervention. It is suitable for operation and management in large-scale application environments and has certain practical value.

[0097] Through the embodiments of the present application, the above-mentioned data collection module is encapsulated in the SDK of the basic framework. As long as the application references the SDK, it can automatically realize the refined collection and processing of data when the application starts. The ELK platform can be used to realize the real-time collection, processing and display of each application log. At the same time, keywords can be specified in the display to achieve accurate monitoring; at the same time, it realizes the monitoring of each application framework information and log information, and achieves the purpose of comprehensive and refined monitoring of the application from the data level.

[0098] It is understandable that in the specific implementation of this application, related data such as user information is involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0099] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0100] According to another aspect of the embodiment of the present application, a data processing device for implementing the above data processing method is also provided. Figure 4 As shown, the device comprises:

[0101] The acquisition module 402 is used to acquire a data request;

[0102] The transmission module 404 is used to send a data request to each application in a group of applications, and obtain a group of log information corresponding to the group of applications, wherein one log information in the group of log information is used to indicate the running status of each application within a preset period of time;

[0103] A determination module 406 is used to determine a target application from a group of applications according to a target period and a target service indicated by target log information in a group of log information, and obtain application information of the target application, wherein the target log information is used to indicate the running state of the target application in the target period, and the application information includes a code package file that needs to be called when the target application runs in the target period and a target service code generated by the target application in the process of executing the target service, and the target service code is used to indicate the execution state of the target service;

[0104] The display module 408 is used to display target log information and application information.

[0105] As an optional solution, the above-mentioned device is used to determine the target application from a group of applications according to the target time period and target business indicated by the target log information in a group of log information, and obtain the application information of the target application in the following manner: determine the time period log data from the target log information, wherein the time period log data is used to indicate the target time period; determine the target application from a group of applications based on the time period log data, wherein the target application is set to run within the target time period; obtain target code framework data associated with the target application from a group of application framework set data, wherein a group of application framework data sets includes code framework data of each application, and the target code framework data is used to indicate a code package file; determine the application information based on the target code framework data.

[0106] As an optional solution, the above-mentioned device is also used to: obtain the application name, startup time and configuration file of the target application before obtaining the target code framework data associated with the target application from a set of application framework set data; when the configuration file includes a framework configuration item, query a set of application framework data sets according to the framework configuration item to obtain the target code framework data, wherein the framework configuration item is used to indicate the code framework pre-set for the target application; and store the target code framework data in a set of application framework data sets.

[0107] As an optional solution, the above-mentioned device is used to determine the target application from a group of applications according to the target time period and target business indicated by the target log information in a group of log information, and obtain the application information of the target application in the following manner: determine the business log data from the target log information, wherein the business log data is used to indicate the target business; determine the target application from a group of applications based on the business log data, wherein the target application can execute the target business; obtain the target business code associated with the target application from a set of business code sets, wherein the set of business code sets includes the execution status of the target business in each application that can execute the target business; determine the application information based on the target business code.

[0108] As an optional solution, the above-mentioned device is also used to: obtain the application name, startup time and configuration file of the target application before obtaining the target business code associated with the target application from a set of business code sets; when the configuration file includes business configuration items, query a set of business code sets according to the business configuration items to obtain the target business code, wherein the business configuration items are used to indicate the business that the target application can execute; and store the target business code in a set of business code sets.

[0109] As an optional solution, the above device is also used to: deploy a target data plug-in in a target application, wherein the target data plug-in is used to collect application information; and use the target data plug-in to obtain a code package file and a target business code.

[0110] As an optional solution, the above-mentioned device is also used to: in response to the start of the target application, obtain first code framework data collected by the target data plug-in, wherein the first code framework data is used to indicate the code package file called when the target application is initialized and started; in response to the framework update request, obtain second code framework data collected by the target data plug-in, wherein the second code framework data is used to indicate the code package file called when the target application needs to update the code framework; in response to the start of the target application, obtain a first business code collected by the target data plug-in, wherein the first business code is used to indicate the execution status of the target business when the target application executes the target business for the first time; in response to the business update request, obtain a second business code collected by the target data plug-in, wherein the second business code is used to indicate the execution status of the target business when the target application executes the target business again in the case of an update of the target business.

[0111] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0112] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0113] According to one aspect of the present application, a computer program product is provided. The computer program product includes a computer program.

[0114] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0115] Figure 5 The structure block diagram of a computer system for implementing an electronic device according to an embodiment of the present application is schematically shown.

[0116] It should be noted that Figure 5 The computer system 500 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0117] like Figure 5 As shown, the computer system 500 includes a central processing unit 501 (CPU), which can perform various appropriate actions and processes according to the program stored in the read-only memory 502 (ROM) or the program loaded from the storage part 508 to the random access memory 503 (RAM). Various programs and data required for system operation are also stored in the random access memory 503. The central processing unit 501, the read-only memory 502 and the random access memory 503 are connected to each other through a bus 504. The input / output interface 505 (Input / Output interface, i.e., I / O interface) is also connected to the bus 504.

[0118] The following components are connected to the input / output interface 505: an input section 506 including a keyboard, a mouse, etc.; an output section 507 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker; a storage section 508 including a hard disk, etc.; and a communication section 509 including a network interface card such as a local area network card, a modem, etc. The communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the input / output interface 505 as needed. A removable medium 511, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 510 as needed so that a computer program read therefrom is installed into the storage section 508 as needed.

[0119] In particular, according to an embodiment of the present application, the process described in each method flow chart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer readable medium, and the computer program contains a program code for executing the method shown in the flow chart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 509, and / or installed from the removable medium 511. When the computer program is executed by the central processor 501, various functions defined in the system of the present application are executed.

[0120] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 509, and / or installed from the removable medium 511. When the computer program is executed by the central processor 501, various functions provided by the embodiment of the present application are performed.

[0121] According to another aspect of the embodiment of the present application, an electronic device for implementing the above data processing method is also provided. The electronic device may be Figure 1 The terminal device or server shown in the figure. This embodiment is described by taking the electronic device as a terminal device as an example. Figure 6 As shown, the electronic device includes a memory 602 and a processor 604. The memory 602 stores a computer program, and the processor 604 is configured to execute the steps in any of the above method embodiments through the computer program.

[0122] Optionally, in this embodiment, the electronic device may be located in at least one network device among a plurality of network devices of a computer network.

[0123] Optionally, in this embodiment, the above-mentioned processor can be configured to execute the methods in each embodiment of the present application through a computer program.

[0124] Alternatively, a person skilled in the art may understand that: Figure 6 The structure shown is for illustration only. Figure 6 The structure of the electronic device is not limited. Figure 6 More or fewer components (such as network interfaces, etc.) as shown in, or with Figure 6 Different configurations are shown.

[0125] Among them, the memory 602 can be used to store software programs and modules, such as program instructions / modules corresponding to the data processing method and device in the embodiments of the present application. The processor 604 executes various functional applications and data processing by running the software programs and modules stored in the memory 602, that is, realizing the above-mentioned data processing method. The memory 602 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 602 may further include a memory remotely located relative to the processor 604, and these remote memories may be connected to the terminal via a network. Examples of the above-mentioned networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof. Among them, the memory 602 can be specifically, but not limited to, used to store information such as the above-mentioned log information and application information. As an example, such as Figure 6As shown, the memory 602 may include but is not limited to the acquisition module 402, transmission module 404, determination module 406 and display module 408 in the data processing device. In addition, it may also include but is not limited to other module units in the data processing device, which will not be repeated in this example.

[0126] Optionally, the transmission device 606 is used to receive or send data via a network. Specific examples of the network may include a wired network and a wireless network. In one example, the transmission device 606 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices and routers via a network cable so as to communicate with the Internet or a local area network. In one example, the transmission device 606 is a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0127] In addition, the electronic device further includes: a display 608 for displaying the log information and application information, etc.; and a connection bus 610 for connecting various module components in the electronic device.

[0128] In other embodiments, the terminal device or server may be a node in a distributed system, wherein the distributed system may be a blockchain system, and the blockchain system may be a distributed system formed by connecting the multiple nodes through network communication. The nodes may form a peer-to-peer network, and any form of computing device, such as a server, terminal or other electronic device, may become a node in the blockchain system by joining the peer-to-peer network.

[0129] According to one aspect of the present application, a computer-readable storage medium is provided, and a processor of an electronic device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the electronic device executes the data processing method provided in various optional implementations of the above-mentioned data processing aspects.

[0130] Optionally, in this embodiment, the above-mentioned computer-readable storage medium can be configured to store data for executing the methods in various embodiments of the present application.

[0131] Optionally, in this embodiment, a person of ordinary skill in the art may understand that all or part of the steps in the various methods of the above embodiments may be completed by instructing hardware related to the terminal device through a program, and the program may be stored in a computer-readable storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc.

[0132] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0133] If the integrated units in the above embodiments are implemented in the form of software functional units and sold or used as independent products, they can be stored in the above computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for one or more electronic devices to execute all or part of the steps of the methods described in each embodiment of the present application.

[0134] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0135] In the several embodiments provided in the present application, it should be understood that the disclosed application can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0136] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0137] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0138] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A data processing method, characterized in that: include: Get data request; Sending the data request to each application in a group of applications, obtaining a group of log information corresponding to the group of applications, wherein one log information in the group of log information is used to indicate the running status of each application within a preset time period; Determine a target application from the group of applications according to the target time period and the target service indicated by the target log information in the group of log information, and obtain application information of the target application, wherein the target log information is used to indicate the running state of the target application in the target time period, and the application information includes a code package file that needs to be called when the target application runs in the target time period and a target service code generated by the target application in the process of executing the target service, and the target service code is used to indicate the execution state of the target service; The target log information and the application information are displayed.

2. The method according to claim 1, characterized in that The determining a target application from the group of applications according to the target time period and the target service indicated by the target log information in the group of log information, and obtaining the application information of the target application includes: Determining time period log data from the target log information, wherein the time period log data is used to indicate the target time period; determining the target application from the group of applications based on the period log data, wherein the target application is set to run within the target period; Acquire target code framework data associated with the target application from a set of application framework data, wherein the set of application framework data includes code framework data of each application, and the target code framework data is used to indicate the code package file; The application information is determined based on the object code framework data.

3. The method according to claim 2, characterized in that Before acquiring the target code framework data associated with the target application from a set of application framework set data, the method further includes: Obtain the application name, startup time and configuration file of the target application; In the case where the configuration file includes a framework configuration item, querying the set of application framework data according to the framework configuration item to obtain the target code framework data, wherein the framework configuration item is used to indicate a code framework pre-set for the target application; The object code framework data is stored in the set of application framework data.

4. The method according to claim 1, characterized in that: The determining a target application from the group of applications according to the target time period and the target service indicated by the target log information in the group of log information, and obtaining the application information of the target application includes: Determining service log data from the target log information, wherein the service log data is used to indicate the target service; Determining the target application from the group of applications based on the service log data, wherein the target application can execute the target service; Acquire the target service code associated with the target application from a set of service code sets, wherein the set of service code sets includes the execution status of the target service in each of the applications capable of executing the target service; The application information is determined based on the target service code.

5. The method according to claim 4, characterized in that Before acquiring the target service code associated with the target application from a set of service codes, the method further includes: Obtain the application name, startup time and configuration file of the target application; In the case where the configuration file includes a service configuration item, querying the set of service codes according to the service configuration item to obtain the target service code, wherein the service configuration item is used to indicate the service that the target application can execute; The target service code is stored in the set of service codes.

6. The method according to claim 1, characterized in that The method further comprises: deploying a target data plug-in in the target application, wherein the target data plug-in is used to collect the application information; The target data plug-in is used to obtain the code package file and the target business code.

7. The method according to claim 6, characterized in that The method further comprises: In response to the target application being started, obtaining first code framework data collected by the target data plug-in, wherein the first code framework data is used to indicate the code package file called when the target application is initialized and started; In response to the framework update request, obtaining second code framework data collected by the target data plug-in, wherein the second code framework data is used to indicate the code package file called when the target application needs to update the code framework; In response to the target application being started, obtaining a first service code collected by the target data plug-in, wherein the first service code is used to indicate an execution state of the target service when the target application executes the target service for the first time; In response to a service update request, a second service code collected by the target data plug-in is obtained, wherein the second service code is used to indicate the execution status of the target service when the target application executes the target service again when the target service is updated.

8. A data processing device, characterized in that: include: The acquisition module is used to obtain data requests; A transmission module, used to send the data request to each application in a group of applications, and obtain a group of log information corresponding to the group of applications, wherein one log information in the group of log information is used to indicate the running status of each application within a preset time period; a determination module, configured to determine a target application from the group of applications according to a target period and a target service indicated by target log information in the group of log information, and obtain application information of the target application, wherein the target log information is used to indicate a running state of the target application in the target period, and the application information includes a code package file that needs to be called when the target application runs in the target period and a target service code generated by the target application in the process of executing the target service, and the target service code is used to indicate an execution state of the target service; The display module is used to display the target log information and the application information.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored computer program, wherein the computer program can be executed by an electronic device to perform the method described in any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method described in any one of claims 1 to 7 are implemented.

11. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to execute the method according to any one of claims 1 to 7 through the computer program.

Images