Data processing method and device, storage medium and electronic equipment

Abstract

The application discloses a data processing method and device, a storage medium and electronic equipment. It relates to the technical field of financial technology, and the method comprises the following steps: collecting data records of a target host through real-time interaction services of the target host, wherein the data records comprise at least a plurality of data records; analyzing the data records of the target host to obtain analyzed data records, and extracting target performance data of the target host from the analyzed data records according to the data structure of the data records; storing the target performance data in a first preset message queue of the target host; and storing the target performance data in the first preset message queue of the target host in a second preset message queue of a target platform through a preset interface. The application solves the technical problem of low efficiency in analyzing and obtaining product performance data in the related art, which adopts a report generation mode on the host side to obtain the performance data of the CICS product of the large host.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of financial technology, in particular to a data processing method and device, a storage medium and an electronic device. BACKGROUND

[0002] At present, IBM mainframes are widely used in core business systems of large financial institutions due to their stability and security. Due to their high security and reliability, the system is relatively closed, so many business operations need to be completed on the host side, and it is difficult and complex to connect with an open platform. In daily operation and maintenance, CICS product performance is usually checked by generating a report on the host side. The report generation process is complex and time-consuming, resulting in a complex and time-consuming SMF data acquisition method. Analyzing the performance of each product on the host requires a series of complex steps, making it difficult to visually display the performance data of each subsystem on the host.

[0003] To address the above problems, no effective solutions have been proposed so far. SUMMARY

[0004] The embodiments of the present application provide a data processing method and device, a storage medium and an electronic device to at least solve the technical problem of low efficiency in analyzing and obtaining product performance data in the related art, which uses a report generation method on the host side to obtain CICS product performance data in a large mainframe.

[0005] According to an aspect of an embodiment of the present application, a data processing method is provided, including: collecting data records of a target host through a real-time interactive service of the target host, wherein the data records include at least a plurality of data records, and each data record includes at least one or more performance data; parsing the data records of the target host to obtain parsed data records, and extracting target performance data of the target host from the parsed data records according to the data structure of the data records; storing the target performance data in a first preset message queue of the target host; and storing the target performance data in the first preset message queue of the target host in a second preset message queue of a target platform through a preset interface, wherein the target platform is used to store and display the target performance data.

[0006] Further, the target performance data of the target host is extracted from the parsed data records according to the data structure of the data records, including: target data records belonging to a target data type are extracted from the parsed data records according to a plurality of data fields in each data record in the parsed data records, wherein the plurality of data fields are determined by the data structure of the data records; and performance data included in the target data records is extracted from target data fields in the target data records to obtain the target performance data, wherein the target data fields at least include data values of performance data.

[0007] Further, the target data type at least includes a first data type and a second data type, and the target data records belonging to the target data type are extracted from the parsed data records according to the plurality of data fields in each data record in the parsed data records, including: data records belonging to the first data type are extracted from the parsed data records according to first data fields of each data record in the parsed data records to obtain first data records, wherein the first data type is a software product type corresponding to the data records; and data records belonging to the second data type are extracted from the first data records according to second data fields of each data record in the first data records to obtain the target data records, wherein the second data type is a subsystem type corresponding to the data records, and the second data type is a sub-type of the first data type.

[0008] Further, the target performance data is stored in a first preset message queue of the target host, including: a data structure body is defined according to a data structure and a data length of the data records of the first data type; and the target performance data is stored in the data structure body, and the data structure body storing the target performance data is stored in the first preset message queue.

[0009] Further, after the target performance data in the first preset message queue of the target host is stored in a second preset message queue of a target platform through a preset interface, the target performance data in the second preset message queue is further classified and stored in a time series database of the target platform, and the target performance data in the time series database is displayed.

[0010] Further, after the target performance data in the time series database is displayed, the target performance data is obtained, the target performance data is analyzed and processed to obtain a processing result, and the processing result is sent to a target object.

[0011] Further, the data record of the target host is collected through the real-time interactive service of the target host, including: reading the data record in a target storage unit of the target host through a data interface in the real-time interactive service to obtain the data record of the target host, wherein the target storage unit at least includes all data records in a running time of the target host.

[0012] According to another aspect of the embodiments of the present application, a data processing device is further provided, including: a collection unit, configured to collect a data record of a target host through a real-time interactive service of the target host, wherein the data record at least includes a plurality of data records, and each data record at least includes one or more performance data; a first processing unit, configured to parse the data record of the target host to obtain a parsed data record, and extract target performance data of the target host from the parsed data record according to a data structure of the data record; a second processing unit, configured to store the target performance data into a first preset message queue of the target host; and a third processing unit, configured to store the target performance data in the first preset message queue of the target host into a second preset message queue of a target platform through a preset interface, wherein the target platform is configured to store and display the target performance data.

[0013] Further, the first processing unit includes: a first extraction subunit, configured to extract target data records of a target data type from the parsed data record according to a plurality of data fields in each data record in the parsed data record, wherein the plurality of data fields are determined by the data structure of the data record; and a second extraction subunit, configured to extract performance data included in the target data record from a target data field in the target data record to obtain the target performance data, wherein the target data field at least includes a data value of the performance data.

[0014] Further, the target data type at least includes a first data type and a second data type, and the first extraction subunit includes: a first extraction module, configured to extract data records of the first data type from the parsed data record according to a first data field in each data record in the parsed data record to obtain first data records, wherein the first data type is a software product type corresponding to the data record; and a second extraction module, configured to extract data records of the second data type from a second data field in each data record in the first data record to obtain the target data record, wherein the second data type is a subsystem type corresponding to the data record, and the second data type is a sub-type of the first data type.

[0015] Further, the second processing unit comprises a defining subunit configured to define a data structure body according to a data structure and a data length of the data record of the first data type, and a processing subunit configured to store the target performance data into the data structure body and store the data structure body storing the target performance data into the first preset message queue.

[0016] Further, the data processing apparatus further comprises a classified storage unit configured to, after storing the target performance data of the target host in the first preset message queue into the second preset message queue of the target platform through the preset interface, store the target performance data in the second preset message queue into the time sequence database of the target platform in a classified manner, and a display unit configured to display the target performance data in the time sequence database.

[0017] Further, the data processing apparatus further comprises an obtaining unit configured to obtain the target performance data after displaying the target performance data in the time sequence database, an analyzing unit configured to analyze and process the target performance data to obtain a processing result, and a sending unit configured to send the processing result to a target object.

[0018] Further, the collecting unit comprises a reading subunit configured to read data records in a target storage unit of the target host through a data interface in the real-time interaction service to obtain the data records of the target host, wherein the target storage unit at least comprises all data records in a running time of the target host.

[0019] According to another aspect of the embodiments of the present application, an electronic device is also provided, which comprises a processor and a memory for storing executable instructions of the processor, wherein the processor is configured to execute the data processing method of any one of the above through executing the executable instructions.

[0020] According to another aspect of the embodiments of the present application, a computer readable storage medium is also provided, which stores a computer program, wherein the computer readable storage medium controls a device where the computer readable storage medium is located to execute the data processing method of any one of the above when the computer program runs.

[0021] In the present application, the data record of the target host is collected through the real-time interactive service of the target host, wherein the data record at least includes a plurality of data records, and each data record at least includes one or more performance data; the data record of the target host is parsed to obtain a parsed data record, and in the parsed data record, the target performance data of the target host is extracted according to the data structure of the data record; the target performance data is stored in the first preset message queue of the target host; the target performance data in the first preset message queue of the target host is stored in the second preset message queue of the target platform through the preset interface, wherein the target platform is used to store and display the target performance data. Further, the technical problem of low efficiency in analyzing and obtaining product performance data in the related art is solved by generating a report on the host side to obtain the performance data of the CICS product in the large host. In the present application, the data record of the host side is collected in real time, and data parsing and data extraction are performed, and the extracted target performance data is imported from the message queue of the host side to the target platform, thereby avoiding the low efficiency of obtaining performance data by generating a data report on the host side, and achieving the technical effect of improving the efficiency of obtaining performance data of the large host. BRIEF DESCRIPTION OF DRAWINGS

[0022] The accompanying drawings, which are included to provide a further understanding of the present application and are incorporated in and constitute a part of this application, illustrate embodiments of the present application and serve to explain the present application, and do not constitute improper limitations on the present application. In the drawings:

[0023] Figure 1 is a flowchart of an optional data processing method according to an embodiment of the present application;

[0024] Figure 2 is a schematic diagram of an optional SMF data structure according to an embodiment of the present application;

[0025] Figure 3 is a flowchart of an optional data parsing according to an embodiment of the present application;

[0026] Figure 4 is a flowchart of an optional data processing according to an embodiment of the present application;

[0027] Figure 5 is a schematic diagram of an optional data processing device according to an embodiment of the present application;

[0028] Figure 6 is a schematic diagram of an optional electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0029] In the following, the technical solutions in the embodiments of the present application will be described clearly and completely with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all the other embodiments obtained by a person of ordinary skill in the art without creative work should belong to the protection scope of the present application.

[0030] It should be noted that the terms "first", "second" and the like in the description and claims of the present application and the above drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device that includes a series of steps or units does not necessarily have to include only those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to the process, method, product or device.

[0031] For the convenience of description, part of the nouns or terms involved in the present application are described below.

[0032] CICS: a product for providing online transaction management for applications on a mainframe platform, which is middleware connecting applications and databases, the structure design of CICS can be oriented to transaction processing, and CICS constructs a hierarchical application system, which can effectively distinguish the expression logic layer, transaction logic layer and data logic layer in the application system.

[0033] SMF: used for collecting and recording various events and resource usage of a data collection and recording system, commonly used for problem analysis, performance analysis, performance tuning, auditing, etc.

[0034] It should be noted that the related information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for display, analyzed data, etc.) involved in the present disclosure are all information and data authorized by the user or authorized by all parties. For example, an interface is provided between the system and the related user or institution, and before obtaining the related information, the interface needs to send a request for obtaining the information to the aforementioned user or institution, and after receiving the consent information fed back by the aforementioned user or institution, the related information is obtained.

[0035] The present application will be further described below in conjunction with the embodiments.

[0036] Embodiment one

[0037] According to an embodiment of the present application, an optional method for processing data is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0038] Figure 1 is a flowchart of an optional method for processing data according to an embodiment of the present application, as shown in Figure 1 the method comprises the following steps:

[0039] In step S101, data records of a target host are collected through a real-time interactive service of the target host, wherein the data records comprise at least a plurality of data records, and each data record comprises at least one or more performance data.

[0040] The target host described above can be an IBM mainframe, the data records described above can be SMF data, and the core of the real-time interactive service described above comprises a set of IFAMxxx interfaces (a kind of software interface) and data cache, which can realize obtaining host SMF real-time records from host SMF Buffer (used for storing SMF data of the target host in a period of time). The collection program is written in C language, and the collected data is stored in a defined structure after being obtained.

[0041] In step S102, the data records of the target host are parsed to obtain parsed data records, and in the parsed data records, target performance data of the target host is extracted according to the data structure of the data records.

[0042] In this embodiment, the data records of the target host are parsed to obtain parsed data records, and in the parsed data records, performance data contained in each record (record, and each data record in the data records) is extracted according to the data structure of the data records. Parsing host performance data is a prerequisite for host real-time performance analysis. Since the data structures of SMF data of various subsystems of the target host are quite different, the full amount of host SMF real-time data (corresponding to the data records of the target host described above) collected can be parsed according to each subsystem to be distinguished, and then the performance indicators required according to the SMF data structure of each subsystem are parsed to extract the target performance data of the target host.

[0043] In step S103, the target performance data is stored in a first preset message queue of the target host.

[0044] The first preset message queue can be an MQ message queue of the host side. By storing the target performance data in the first preset message queue, the performance data in the target host can be exported in real time, and the situation of low analysis efficiency caused by generating a report on the host side and analyzing the performance data is avoided.

[0045] In step S104, the target performance data in the first preset message queue of the target host is stored in the second preset message queue of the target platform through a preset interface, and the target platform is used to store and display the target performance data.

[0046] In the embodiment, the extracted data of the SMF 110 (corresponding to the target performance data) can be downloaded to the target platform MQ using the IBM MQ (the MQ message queue of the host side, corresponding to the first preset message queue).

[0047] The preset interface can be an encapsulated MQ interface called on the target host side, for example, MQPUT(HConn, HObj, &MsgDesc, &PutMsgOpts, msgLength, msgBuffer, &CompCode, &Reason). Through the preset interface, the parsed target performance data can be downloaded to the target platform MQ message queue (corresponding to the second preset message queue) connected to the target host MQ. In actual deployment, two channels of an MQ can be used independently for each product type of performance data, so as to reduce the phenomenon that data cannot be downloaded due to failure of one channel and improve the tool availability.

[0048] The platform MQ is used to configure a set of connection channels with the host MQ, and a corresponding MQ GET interface is called to receive the SMF performance data (i.e., the target performance data) parsed by the host side from the open platform MQ Recv queue using the readStringOfByteLength method (a program method).

[0049] Through the above steps, in the embodiment, the data records of the host side are collected in real time, and data parsing and data extraction are performed. The extracted target performance data is imported into the target platform from the message queue of the host side, and the situation of low performance data efficiency caused by generating a data report on the host side is avoided, so as to realize the technical effect of improving the performance data efficiency of the large host, and further solve the technical problem of low efficiency of analyzing and obtaining the performance data of the CICS product in the large host caused by generating a report on the host side.

[0050] In order to accurately extract the performance data of the target data type in the plurality of data records, in the embodiment, in the parsed data records, the target performance data of the target host is extracted according to the data structure of the data records, and further includes the following contents: according to the plurality of data fields in each data record in the parsed data records, the target data record belonging to the target data type is extracted from the parsed data records, wherein the plurality of data fields are determined by the data structure of the data records; and the performance data included in the target data record is extracted from the target data field in the target data record, so as to obtain the target performance data, wherein the target data field at least includes the data value of the performance data.

[0051] From the full amount of SMF data records to a certain performance indicator of a certain subsystem, four to five layers of nested structure need to be parsed. Taking the CICS subsystem as an example, the CICS data TYPE is 110 from the full amount of data, the CICS record data is filtered, Figure 2 is a schematic diagram of a data structure of an optional SMF data according to an embodiment of the application, as Figure 2 shown, the CICS SMF record data includes three parts of SMF Header, SMF Product Section and Dictionary Data Section.

[0052] Among them, the Header is definition information; the Product section is subsystem information (the subtype is monitoring (real-time monitoring data), and the Product section is CICS region (address space) information); the Dictionary Data Section part includes Dictionary Entry and Field two parts, and is used to store actual performance data, the plurality of data fields described above at least include: the Header field and the Product section field, and the target data field described above corresponds to the Dictionary Data Section data field.

[0053] The subtype of the record (data record) can be obtained from the SMFSTY field in the SMF Header data section. The CICS subtype includes CICS journaling (log data), CICS monitoring, CICS statistics, Shared temporary storage queue server (shared storage data), Coupling facility data table server statistics (statistics of the host coupling facility), etc., and contains six categories. The data stored in each category of subtype is different, for example, statistics is used to store statistical data, monitoring is used to store real-time monitoring data, etc. The SMFMNCL field in the SMF Product Section is used to indicate that the record belongs to a certain class (subclass of subtype) under the subclass. The class is divided into multiple types. The subtype and class can be selected and specified according to the analysis needs, or all the data of the subtype and class can be extracted to extract the target performance data, so as to achieve the technical effect of improving the efficiency of obtaining the target performance data.

[0054] In order to avoid the case that it is difficult to extract the performance data of the target data type in a large number of data records, in the embodiment, the target data type at least includes a first data type and a second data type. According to the multiple data fields in each data record in the parsed data record, the target data record belonging to the target data type is extracted from the parsed data record. The method further includes the following contents: according to the first data field of each data record in the parsed data record, the data record belonging to the first data type is extracted from the parsed data record to obtain the first data record, wherein the first data type is the software product type corresponding to the data record; and according to the second data field of each data record in the first data record, the data record belonging to the second data type is extracted from the first data record to obtain the target data record, wherein the second data type is the subsystem type corresponding to the data record, and the second data type is a subtype of the first data type.

[0055] In the embodiment, Figure 3 is a flowchart of optional data analysis according to an embodiment of the application, as Figure 3As shown, after the Real-time Service service is called to collect the SMF record (corresponding to the data record described above), it can be determined which TYPE (corresponding to the first data type described above, TYPE110 in this example) according to the SMF header part (corresponding to the first data field described above) of each data record after analysis, and then the first data record belonging to the first data type is extracted, and the number of product sections (the number of second data fields to be extracted) is determined according to Smf110Nps (the number of second data records contained in the first data type) of the extracted first data record. The number of product sections can be used to determine how many times the second data field needs to be judged to determine whether the first data record belongs to the second data type. By entering the product section block (and the product section field) of each first data record, the product name (parameter name) is parsed, which is mostly applid (address space name), and the subtype (subsystem type) and class (subtype type) are parsed. The second data type described above includes at least subtype and class types. By determining whether each first data record belongs to the second data type, the target data record of the second data type can be extracted from the plurality of first data records, that is, the target data record belonging to the target data type is extracted. The target performance data in each data record of the target data record can also be extracted, achieving the technical effect of quickly and accurately extracting the target performance data.

[0056] In order to perform the performance data of the host end to the target platform in each case, in this embodiment, the target performance data is stored in the first preset message queue of the target host, and further includes the following contents: defining a data structure body according to the data structure and data length of the data record of the first data type; storing the target performance data in the data structure body, and storing the data structure body storing the target performance data in the first preset message queue.

[0057] In this embodiment, as shown in FIG. 6, the target performance data is stored in the first preset message queue of the target host, and further includes the following contents: defining a data structure body according to the data structure and data length of the data record of the first data type; storing the target performance data in the data structure body, and storing the data structure body storing the target performance data in the first preset message queue. Figure 3As shown, according to the 110TYPE (corresponding to the data record of the first data type) data structure and the data length, the data structure body can be defined, the data (corresponding to the target performance data described above) collected from the buffer is matched with the data structure body, and the data structure body is stored in the data structure body. In this embodiment, the data structure body can also be time converted (the source data record time is UTC (Coordinated Universal Time) time zone and the host time is hexadecimal and has a special format), and the data format in the data structure body can also be converted to the required format. By storing the performance data generated in real time at the host end into the first message queue, the technical effect of facilitating the host end to import the target performance data into the target platform is realized.

[0058] Optionally, after storing the target performance data in the first preset message queue of the target host into the second preset message queue of the target platform through the preset interface, the method further comprises: storing the target performance data in the second preset message queue into the time sequence database of the target platform in a classified manner; and displaying the target performance data in the time sequence database.

[0059] In this embodiment, the target performance data can also be data screened and spliced and written into the stream database Influxdb (corresponding to the time sequence database described above). Due to the characteristics of large SMF data volume and record collection according to time sequence, based on the characteristics of time sequence, measurement, and unstructured, the storage requirements of SMF performance data are met. In this embodiment, the open source distributed time sequence database Influxdb is selected for the data storage part of the target platform. The Influxdb database supports RESTful interface (a kind of data interface) connection, warehousing and other operations. The data warehousing program can also store the processed target performance data according to the subsystem type respectively, so as to realize the technical effect of facilitating the query during data display.

[0060] In order to improve the efficiency of analyzing the performance data of the target host, in this embodiment, after displaying the target performance data in the time sequence database, the following contents are further included: obtaining the target performance data; analyzing and processing the target performance data to obtain a processing result; and sending the processing result to a target object.

[0061] In this embodiment, when analyzing the performance of the target host, the target performance data can be obtained from the target platform, the target performance data can be analyzed and processed to obtain a processing result, and the processing result can be sent to a target object. This avoids the situation that the data report is generated at the host end for performance analysis, and the analysis efficiency is low. The technical effect of improving the efficiency of analyzing the performance data of the target host is realized.

[0062] In order to improve the efficiency of obtaining performance data of the large host, in the embodiment, the data record of the target host is collected through the real-time interactive service of the target host, and the following is further included: the data record of the target host is read through a data interface in the real-time interactive service, and the data record of the target host is obtained, wherein the target storage unit at least includes all data records of the target host within a period of running time.

[0063] The target host described above can be an IBM mainframe, the data record described above can be SMF data, and the core of the real-time interactive service described above includes a set of IFAMxxx interfaces and data cache, and can realize obtaining the real-time record of the host SMF from the host SMF buffer. The collection program is written in C language, and after the collected data is stored in the defined structure, the technical effect of facilitating the extraction of the performance data of the target host is realized.

[0064] Figure 4 An optional flowchart of data processing according to an embodiment of the present application is shown in FIG. 1. Figure 4 As shown in FIG. 1, the embodiment can collect SMF data (corresponding to the data record described above) through a real-time data collection program on the host side, then a host real-time analysis program performs real-time analysis on the collected data, extracts target performance data, and stores the target performance data into an MQ (corresponding to the first preset message queue described above) of the host, then the target performance data in the host MQ can be transmitted to an MQ (corresponding to the second preset message queue described above) of a platform (corresponding to the target platform described above), and the target performance data in the platform MQ can be further processed and stored in a memory database influxdb according to the type of the subsystem, so as to be called by an application data model and analyzed in performance.

[0065] Through the embodiment, the long time period for obtaining performance comparison data by traditional SMF collection and report processing is shortened to near real time, the timeliness of performance problem analysis and processing is effectively improved, and it is possible to introduce analysis algorithms and intelligent performance problem analysis in combination with practical experience in the future. Through the data collection system of the embodiment, SMF data of all products of the host can be obtained, and therefore, performance data of the host in various demand scenarios can be analyzed.

[0066] Embodiment two

[0067] The embodiment provides an optional data processing device, and each implementation unit in the data processing device corresponds to each implementation step of the first embodiment.

[0068] Figure 5is a schematic diagram of an optional data processing device according to an embodiment of the application, as shown, comprising: an acquisition unit 51, a first processing unit 52, a second processing unit 53, a third processing unit 54. Figure 5

[0069] Specifically, the acquisition unit 51 is configured to acquire data records of a target host through real-time interactive services of the target host, wherein the data records comprise at least a plurality of data records, and each data record comprises at least one or more performance data;

[0070] The first processing unit 52 is configured to parse the data records of the target host to obtain parsed data records, and extract target performance data of the target host from the parsed data records according to a data structure of the data records;

[0071] The second processing unit 53 is configured to store the target performance data in a first preset message queue of the target host;

[0072] The third processing unit 54 is configured to store the target performance data in the first preset message queue of the target host in a second preset message queue of a target platform through a preset interface, wherein the target platform is configured to store and display the target performance data.

[0073] In the data processing device provided in Embodiment Two, the acquisition unit 51 acquires data records of a target host through real-time interactive services of the target host, wherein the data records comprise at least a plurality of data records, and each data record comprises at least one or more performance data. The first processing unit 52 parses the data records of the target host to obtain parsed data records, and extracts target performance data of the target host from the parsed data records according to a data structure of the data records. The second processing unit 53 stores the target performance data in a first preset message queue of the target host. The third processing unit 54 stores the target performance data in the first preset message queue of the target host in a second preset message queue of a target platform through a preset interface, wherein the target platform is configured to store and display the target performance data. Thus, the technical problem of low efficiency in obtaining performance data of a CICS product in a large host by generating a report on the host is solved. In this embodiment, data records on the host are acquired in real time, and data parsing and data extraction are performed. The extracted target performance data is imported from a message queue of the host into the target platform, thereby avoiding the low efficiency in obtaining performance data by generating a data report on the host, and achieving the technical effect of improving the efficiency of obtaining performance data of a large host.

[0074] ​Optionally, in the data processing apparatus provided in Embodiment Two of the present application, the first processing unit 51 comprises: a first extraction subunit, configured to extract target data records belonging to a target data type from the parsed data records according to a plurality of data fields in each data record in the parsed data records, wherein the plurality of data fields are determined by a data structure of the data records; and a second extraction subunit, configured to extract performance data included in the target data records from target data fields in the target data records, to obtain target performance data, wherein the target data fields at least include data values of the performance data.

[0075] Optionally, in the data processing apparatus provided in Embodiment Two of the present application, the target data type at least includes a first data type and a second data type, the first extraction subunit comprises: a first extraction module, configured to extract data records belonging to the first data type from the parsed data records according to first data fields in each data record in the parsed data records, to obtain first data records, wherein the first data type is a software product type corresponding to the data records; and a second extraction module, configured to extract data records belonging to the second data type from the first data records according to second data fields in each data record in the first data records, to obtain the target data records, wherein the second data type is a subsystem type corresponding to the data records, and the second data type is a sub-type of the first data type.

[0076] Optionally, in the data processing apparatus provided in Embodiment Two of the present application, the second processing unit 53 comprises: a definition subunit, configured to define a data structure body according to a data structure and a data length of the data records of the first data type; and a processing subunit, configured to store the target performance data into the data structure body, and store the data structure body storing the target performance data into the first preset message queue.

[0077] Optionally, in the data processing apparatus provided in Embodiment Two of the present application, the data processing apparatus further comprises: a classified storage unit, configured to, after storing the target performance data in the first preset message queue of the target host into the second preset message queue of the target platform, store the target performance data in the second preset message queue into a time sequence database of the target platform in a classified manner; and a display unit, configured to display the target performance data in the time sequence database.

[0078] Optionally, in the data processing apparatus provided in Embodiment Two of the present application, the data processing apparatus further comprises: an acquisition unit, configured to acquire the target performance data after displaying the target performance data in the time sequence database; an analysis unit, configured to analyze and process the target performance data to obtain a processing result; and a sending unit, configured to send the processing result to a target object.

[0079] Optionally, in the data processing apparatus provided in Embodiment Two of the present application, the acquisition unit 51 comprises a reading subunit configured to read the data records in the target storage unit of the target host through a data interface in the real-time interaction service to obtain the data records of the target host, wherein the target storage unit comprises at least all the data records of the target host within a period of running time.

[0080] The data processing apparatus described above can further comprise a processor and a memory, and the acquisition unit 51, the first processing unit 52, the second processing unit 53, the third processing unit 54, etc. are all stored in the memory as program units, and the processor executes the program units stored in the memory to realize the corresponding functions.

[0081] The processor described above comprises a core, and the core retrieves the corresponding program units from the memory. The core can be one or more, and the core parameters are adjusted to collect the data records of the host end in real time, perform data analysis and data extraction, import the target performance data from the message queue of the host end into the target platform, and avoid the low efficiency of obtaining performance data by generating a data report on the host end, thereby realizing the technical effect of improving the efficiency of obtaining performance data of the mainframe.

[0082] The memory described above can comprise a non-permanent memory in a computer readable medium, a random access memory (RAM) and / or a non-volatile memory such as a read-only memory (ROM) or a flash memory (flash RAM), and the memory comprises at least one memory chip.

[0083] According to another aspect of the embodiments of the present application, an electronic device is further provided, comprising a processor and a memory for storing executable instructions of the processor, wherein the processor is configured to execute the data processing method of any one of the above by executing the executable instructions.

[0084] According to another aspect of the embodiments of the present application, a computer readable storage medium is further provided, which stores a computer program, wherein the computer readable storage medium controls the device where the computer readable storage medium is located to execute the data processing method of any one of the above when the computer program is running.

[0085] Figure 6 is a schematic diagram of an optional electronic device according to the embodiments of the present application, as shown in Figure 6 The embodiments of the present application provide an electronic device 60, which comprises a processor, a memory and a program stored in the memory and executable on the processor, and the processor realizes the data processing method of any one of the above when executing the program.

[0086] The above-mentioned embodiment numbers of the present application are only for description, and do not represent the advantages or disadvantages of the embodiments.

[0087] In the above-mentioned embodiments of the present application, the description of each embodiment has its own focus, and the parts not described in detail in a certain embodiment can be referred to the relevant description of other embodiments.

[0088] In several embodiments provided in the present application, it should be understood that the disclosed technical contents can be implemented by other ways. Among them, the above-mentioned device embodiments are only schematic, for example, the division of the units can be a logical function division, and actual implementation can have another division way, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units or modules shown or discussed can be indirect coupling or communication connection through some interfaces, units or modules, which can be electrical or other forms.

[0089] The units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed to multiple units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0090] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The above integrated unit can be realized in the form of hardware or in the form of software functional unit.

[0091] The integrated unit, if realized in the form of software functional unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application or the whole or part of the technical solutions which essentially contribute to the prior art can be embodied in the form of software product, which is stored in a storage medium and includes a plurality of instructions for making a computer device (which can be a personal computer, a server or a network device, etc.) execute all or part of the steps of the method described in each embodiment of the present application. The foregoing storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and various program code storage media.

[0092] The above merely describes the preferred embodiments of the present application, and it should be pointed out that, for those skilled in the art, several improvements and refinements can be made without departing from the principles of the present application, and these improvements and refinements should also be considered as falling within the protection scope of the present application.

Claims

1. A method of processing data, characterized by, The method comprises: collecting data records of a target host through real-time interactive services of the target host, wherein the data records comprise at least a plurality of data records, and each data record comprises at least one or more performance data; parsing the data records of the target host to obtain parsed data records, and extracting target performance data of the target host from the parsed data records according to a data structure of the data records; storing the target performance data in a first preset message queue of the target host; storing the target performance data in a second preset message queue of a target platform through a preset interface, wherein the target platform is configured to store and display the target performance data; wherein the target performance data is extracted from the parsed data records according to the data structure of the data records, comprising: extracting target data records of a target data type from the parsed data records according to a plurality of data fields in each data record in the parsed data records; the target data type comprises at least a first data type and a second data type; the target data records of the target data type are extracted from the parsed data records according to the plurality of data fields in each data record in the parsed data records, comprising: extracting data records of the first data type from the parsed data records according to a first data field of each data record in the parsed data records to obtain first data records; storing the target performance data in the first preset message queue of the target host, comprising: defining a data structure body according to a data structure and a data length of the data records of the first data type; storing the target performance data in the data structure body, and storing the data structure body storing the target performance data in the first preset message queue; after storing the target performance data in the second preset message queue of the target platform through the preset interface, further comprising: storing the target performance data in the second preset message queue in a time series database of the target platform in a classified manner; displaying the target performance data in the time series database.

2. The treatment method according to claim 1, characterized in that, In the parsed data records, the target performance data of the target host is extracted according to the data structure of the data records, comprising: the plurality of data fields are determined by the data structure of the data records; extracting performance data included in the target data records from target data fields in the target data records to obtain the target performance data, wherein the target data fields comprise at least data values of performance data.

3. The treatment method of claim 1, wherein According to the plurality of data fields in each data record in the parsed data records, the target data records of the target data type are extracted from the parsed data records, comprising: the first data type is a software product type corresponding to the data records; The second data field of each data record in the first data record is used to extract data records of a second data type in the first data record to obtain the target data record, wherein the second data type is a subsystem type corresponding to the data record, and the second data type is a subtype of the first data type.

4. The treatment method of claim 1, wherein After the target performance data in the time series database is displayed, the method further includes: obtaining the target performance data; analyzing and processing the target performance data to obtain a processing result; sending the processing result to a target object.

5. The treatment method of claim 1, wherein The data records of the target host are collected through the real-time interactive service of the target host, including: reading the data records in a target storage unit of the target host through a data interface in the real-time interactive service to obtain the data records of the target host, wherein the target storage unit at least includes all data records of the target host within a period of running time.

6. A data processing device, characterized by comprising: including: a collection unit configured to collect data records of a target host through a real-time interactive service of the target host, wherein the data records at least include a plurality of data records, and each data record at least includes one or more performance data; a first processing unit configured to parse the data records of the target host to obtain parsed data records, and extract target performance data of the target host from the parsed data records according to a data structure of the data records; a second processing unit configured to store the target performance data in a first preset message queue of the target host; a third processing unit configured to store the target performance data in the first preset message queue of the target host in a second preset message queue of a target platform through a preset interface, wherein the target platform is configured to store and display the target performance data; wherein the first processing unit includes a first extraction subunit configured to extract target data records of a target data type from the parsed data records according to a plurality of data fields in each data record in the parsed data records; the target data type at least includes a first data type and a second data type, and the first extraction subunit includes a first extraction module configured to extract data records of the first data type from the parsed data records according to a first data field of each data record in the parsed data records to obtain first data records; the second processing unit includes a definition subunit configured to define a data structure body according to a data structure and a data length of the data records of the first data type, and a processing subunit configured to store the target performance data in the data structure body and store the data structure body storing the target performance data in the first preset message queue. The device further comprises a classification storage unit, configured to store target performance data in a first preset message queue of a target host into a second preset message queue of a target platform, and then store the target performance data in the second preset message queue into a time sequence database of the target platform through a preset interface; and a display unit, configured to display the target performance data in the time sequence database.

7. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and when the computer program runs, controls a device where the computer readable storage medium is located to execute the data processing method in any one of claims 1 to 5.

8. An electronic device, comprising: The device comprises one or more processors and a memory, and the memory is configured to store one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors implement the data processing method in any one of claims 1 to 5.

Images