Data processing method, device, electronic device and storage medium
By generating data index parser and message queue decoupling data processing, the problem of high data index update cost and serious coupling of interface analysis modules is solved, resource optimization and traffic management are realized, and system risks are reduced.
Patent Information
- Application Number
- CN202210851420.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-07-19
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2042-07-19
AI Technical Summary
In the prior art, manual code is required to be written when updating data indicators, resulting in high time and labor costs, and serious coupling between the data reporting interface and the parsing module may lead to excessive system resource consumption and loss of requests.
The corresponding data index parser is generated through the data index configuration information, and the data is decoupled using the message queue to realize traffic peak-cutting and valley filling, avoiding downstream services from large traffic requests, and reducing the use of development and testing resources.
There is no need for developers to manually write code to update data metrics, reduce time and labor costs, improve system resource utilization, avoid request loss, and realize on-demand consumption and traffic management.
Smart Images

Figure CN115080150B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of computer technology, and more specifically, to a data processing method, device, electronic device, and storage medium. Background Art
[0002] During the operation of the application, some dot data can be recorded at different times to reflect the operation quality of the application at the corresponding time. By analyzing the operation quality of the application, problems in the operation of the application can be found. In addition, in order to more quickly and accurately troubleshoot problems in the operation of the application, it may be necessary to continuously add or modify some data indicators.
[0003] In the related art, when updating data indicators, staff are required to manually add codes for collecting data. Each time a data indicator is updated, corresponding codes need to be written, which will take up more development and testing resources, and the time and manpower costs are high. Summary of the invention
[0004] The present disclosure provides a data processing method, device, electronic device and storage medium to at least solve the problem in the above-mentioned related technologies that when updating data indicators, staff need to manually add codes for collecting data, which consumes high time and labor costs.
[0005] According to a first aspect of an embodiment of the present disclosure, there is provided a data processing method, which is applied to a parsing server, and the data processing method comprises: obtaining request data to be parsed, wherein the request data to be parsed is client request data received and processed by an interface server; parsing the request data to be parsed using a data indicator parser to obtain parsed data, wherein the data indicator parser is a data indicator parser corresponding to each data indicator generated according to the configuration information of each data indicator after the parsing server obtains the configuration information of at least one data indicator.
[0006] Optionally, the configuration information of each data indicator includes text type information, and before the step of obtaining the request data to be parsed, it also includes: creating a parsing manager; creating at least one text type manager in the parsing manager; and generating a data indicator parser corresponding to each data indicator in the text type manager indicated by the text type information included in the configuration information of each data indicator according to the configuration information of each data indicator.
[0007] Optionally, the configuration information of each data indicator includes type information and code information of at least one function processor, and generating a data indicator parser corresponding to each data indicator in the text type manager indicated by the text type information included in the configuration information of each data indicator based on the configuration information of each data indicator, including: obtaining the function processor indicated by the code information of each function processor from a function processor library indicated by the type information of each function processor based on the type information of each function processor in at least one function processor included in the configuration information of each data indicator, wherein the function processor library includes multiple function processors; and generating a data indicator parser corresponding to each data indicator in the text type manager indicated by the text type information included in the configuration information of each data indicator using the obtained at least one function processor corresponding to each data indicator.
[0008] Optionally, the use of a data indicator parser to parse the request data to be parsed to obtain parsed data includes: determining a target text type contained in the request data to be parsed; searching for a target text type manager whose corresponding text type is the target text type in at least one text type manager; and using the data indicator parser generated in the target text type manager to traverse and parse the request data to be parsed to obtain the parsed data.
[0009] Optionally, the traversal and parsing processing of the request data to be parsed using the data indicator parser generated in the target text type manager includes: determining the data type contained in the request data to be parsed; determining the data type that can be parsed by each data indicator parser in at least one data indicator parser generated in the target text type manager; and traversal and parsing processing of the request data to be parsed using the data indicator parser generated in the target text type manager and whose parsable data type matches the data type contained in the request data to be parsed to obtain the parsed data.
[0010] Optionally, the data processing method also includes: creating a parameter template object in each text type manager in the at least one text type manager; extracting the application identifier and the account identifier contained in the request data to be parsed; parsing the request data to be parsed using the data indicator parser, including: parsing the data in the request data to be parsed except for the application identifier and the account identifier using the data indicator parser to obtain the parsed data; the data processing method also includes: storing the application identifier, the account identifier and the parsed data in the parameter template object in the text type manager where the data indicator parser is located.
[0011] Optionally, before the step of obtaining the requested data to be parsed, it also includes: receiving a data indicator setting instruction; displaying a data indicator configuration interface in response to the data indicator setting instruction; and setting the configuration information of the at least one data indicator in the data indicator configuration interface in response to the user's setting input.
[0012] Optionally, the obtaining the request data to be parsed includes: pulling the request data to be parsed from a first message queue, wherein the request data to be parsed is uploaded to the first message queue by the interface server.
[0013] Optionally, the data processing method further includes: sending the parsed data to a second message queue, so as to store the parsed data in a storage server through the second message queue.
[0014] According to a second aspect of an embodiment of the present disclosure, there is provided a data processing device, which is applied to a parsing server and includes: an acquisition module, configured to acquire request data to be parsed, wherein the request data to be parsed is client request data received and processed by an interface server; a parsing module, configured to parse and process the request data to be parsed using a data indicator parser to obtain parsed data, wherein the data indicator parser is a data indicator parser corresponding to each data indicator generated according to the configuration information of each data indicator after the parsing server acquires the configuration information of at least one data indicator.
[0015] Optionally, the configuration information of each data indicator includes text type information, and the data processing device further includes: a parsing manager creation module configured to create a parsing manager; a text type manager creation module configured to create at least one text type manager in the parsing manager; and a generation module configured to generate a data indicator parser corresponding to each data indicator in the text type manager indicated by the text type information included in the configuration information of each data indicator according to the configuration information of each data indicator.
[0016] Optionally, the configuration information of each data indicator includes type information and code information of at least one function processor, and the generation module is configured to: obtain the function processor indicated by the code information of each function processor from the function processor library indicated by the type information of each function processor according to the type information of each function processor in the at least one function processor included in the configuration information of each data indicator, wherein the function processor library includes multiple function processors; and generate a data indicator parser corresponding to each data indicator in the text type manager indicated by the text type information included in the configuration information of each data indicator using the at least one function processor corresponding to each data indicator obtained.
[0017] Optionally, the parsing module is configured to: determine the target text type contained in the request data to be parsed; search for a target text type manager whose corresponding text type is the target text type in the at least one text type manager; and use the data indicator parser generated in the target text type manager to traverse and parse the request data to be parsed to obtain the parsed data.
[0018] Optionally, the parsing module is configured to: determine the data type contained in the request data to be parsed; determine the data type that can be parsed by each data indicator parser in at least one data indicator parser generated in the target text type manager; use the data indicator parser generated in the target text type manager and whose parsable data type matches the data type contained in the request data to be parsed to traverse and parse the request data to be parsed to obtain the parsed data.
[0019] Optionally, the data processing device also includes: a parameter template object creation module, configured to create a parameter template object in each text type manager in the at least one text type manager; an extraction module, configured to extract the application identifier and the account identifier contained in the request data to be parsed; the parsing module is configured to use the data indicator parser to parse the data in the request data to be parsed except for the application identifier and the account identifier, and obtain the parsed data; the data processing device also includes: a storage module, configured to store the application identifier, the account identifier and the parsed data in the parameter template object in the text type manager where the data indicator parser is located.
[0020] Optionally, the data processing device also includes: a receiving module configured to receive a data indicator setting instruction; a display module configured to display a data indicator configuration interface in response to the data indicator setting instruction; and a setting module configured to set the configuration information of the at least one data indicator in the data indicator configuration interface in response to the user's setting input.
[0021] Optionally, the acquisition module is configured to pull the request data to be parsed from a first message queue, wherein the request data to be parsed is uploaded to the first message queue by the interface server.
[0022] Optionally, the data processing device further includes: a sending module configured to send the parsed data to a second message queue, so as to store the parsed data in a storage server through the second message queue.
[0023] According to a third aspect of an embodiment of the present disclosure, there is provided an electronic device, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions to implement a data processing method according to the present disclosure.
[0024] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform a data processing method according to the present disclosure.
[0025] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the data processing method according to the present disclosure is implemented.
[0026] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects:
[0027] The demander can configure the configuration information of the data indicator according to its own needs, and then generate the data indicator parser corresponding to the data indicator through the configuration information of the data indicator. When updating the data indicator, there is no need for developers to manually write the corresponding code, avoiding the occupation of development and testing resources, and reducing time and labor costs.
[0028] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS
[0029] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.
[0030] Figure 1 is a schematic diagram showing a data processing architecture according to an exemplary embodiment of the present disclosure;
[0031] Figure 2 is a flow chart illustrating a data processing method according to an exemplary embodiment of the present disclosure;
[0032] Figure 3 is a schematic diagram showing a text type manager according to an exemplary embodiment of the present disclosure;
[0033] Figure 4 is a schematic diagram showing a method of obtaining a function processor in a function processor library according to an exemplary embodiment of the present disclosure;
[0034] Figure 5 is a schematic diagram illustrating a data indicator parser according to an exemplary embodiment of the present disclosure;
[0035] Figure 6 is a schematic diagram showing a hierarchical relationship among a parsing manager, a text type manager, and a data indicator parser according to an exemplary embodiment of the present disclosure;
[0036] Figure 7 is a schematic diagram showing a parameter template object according to an exemplary embodiment of the present disclosure;
[0037] Figure 8 is a specific implementation flow chart showing a data processing method according to an exemplary embodiment of the present disclosure;
[0038] Fig. 9 is a block diagram illustrating a data processing apparatus according to an exemplary embodiment of the present disclosure;
[0039] Fig.10 is a block diagram illustrating an electronic device according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION
[0040] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.
[0041] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The implementation methods described in the following examples do not represent all implementation methods consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the attached claims.
[0042] It should be noted that the phrase "at least one of the items" in the present disclosure includes three types of parallel situations: "any one of the items", "a combination of any number of the items", and "all of the items". For example, "including at least one of A and B" includes the following three parallel situations: (1) including A; (2) including B; (3) including A and B. Another example is "executing at least one of step 1 and step 2" which means the following three parallel situations: (1) executing step 1; (2) executing step 2; (3) executing step 1 and step 2.
[0043] At present, when using audio and video applications, the client will record some dot data at a certain moment to reflect the running quality of the audio and video applications at that moment. By collecting, grouping and analyzing these quality data, possible problems in the running of audio and video applications can be found. With the development of audio and video technology, the data indicators that need to be collected have become more and more complex. How to add and update data indicators (Counters) has become a problem that needs to be solved.
[0044] In the related art, the code logic for parsing the corresponding counter is directly written in the interface for receiving quality data reporting, and the data format conversion and other operations are performed on the counter, and finally the converted results are stored in the database. However, in this way, the data reporting interface and the counter parsing module are seriously coupled. For a reporting request, it is necessary to receive the reported data and parse the data, which will consume more system resources and reduce the query rate per second (Query Per Second, QPS) of the interface. When a large number of requests arrive at the server, some requests may be lost. In addition, this method requires developers to manually add data collection code to the program, and then go through the test release process. It takes at least several days to add a new counter demand, which will incur additional communication costs, and it will take up more development and testing resources, which will cost more time and manpower costs.
[0045] In order to solve the problem of high time cost and manpower cost consumed in updating data indicators in related technologies, the data processing method proposed in the present invention allows the demander to configure the configuration information of the data indicator according to its own needs, and then generate a data indicator parser corresponding to the data indicator through the configuration information of the data indicator. When updating the data indicator, there is no need for developers to manually write the corresponding code, which avoids the occupation of development and testing resources and can reduce time cost and manpower cost.
[0046] In order to solve the problem in the related art that the data reporting interface and the counter parsing module are severely coupled, and some requests may be lost when a large number of requests arrive at the server, the data processing method proposed in the present disclosure can store the request data to be parsed in the first message queue (MQ1), and pull and parse the request data from the first message queue; and the parsed data can also be sent to the second message queue (MQ2), and the parsed data can be stored in the server's memory through the second message queue. That is, the message queue (MQ) is used to link each service, and the upstream and downstream services are independently deployed, so as to achieve the decoupling of the entire link, which can play a role in shaving the peak and filling the valley of traffic, that is, it can realize the on-demand consumption of requests sent by the client, and avoid the influx of large traffic requests to crash the downstream service.
[0047] Figure 1 is a schematic diagram showing a data processing architecture according to an exemplary embodiment of the present disclosure.
[0048] Reference Figure 1 ,The data processing architecture of the present disclosure requires a database, two message queues and three service supports.
[0049] Among them, database: quality data storage database, which can be a mysql database.
[0050] Two message queues:
[0051] The first message queue: used to link the interface service and the parsing service, and can transmit the request data after preliminary formatting.
[0052] The second message queue is used to link the parsing service and the database storage service, and can transmit the formatted data after parsing.
[0053] Three services:
[0054] a. Interface service: used to receive client request data and implement authentication, idempotence and preliminary formatting functions.
[0055] b. Parsing service: The core service of the entire data processing architecture generates a data indicator parser (Counter Parser) through configuration information, and parses and processes the request data passed from the first message queue, and then sends the results obtained after the parsing process to the second message queue.
[0056] c. Database storage service: the data in the second message queue can be stored in the corresponding data table.
[0057] In this way, using message queues to link various services and independently deploying upstream and downstream services achieves decoupling of the entire link, which can play a role in shaving traffic peaks and filling valleys, that is, it can realize on-demand consumption of requests from clients and avoid large traffic requests from overwhelming downstream services. Furthermore, independent deployment of upstream and downstream services can also achieve reasonable allocation of server resources. For example, interface services and database storage services are input-output (IO) intensive services; parsing services are computing intensive services. When the system is under pressure, the service with bottlenecks can be expanded to alleviate service pressure.
[0058] Figure 2 is a flow chart showing a data processing method according to an exemplary embodiment of the present disclosure, which is applied to a parsing server.
[0059] Reference Figure 2 In step 201, the request data to be parsed may be obtained. The request data to be parsed is the client request data received and processed by the interface server. As mentioned above, the interface server may receive the request data from the client, and then perform authentication, idempotence, and preliminary formatting on the client request data to obtain the request data to be parsed to be processed by the parsing server.
[0060] In step 202, a data indicator parser may be used to parse the request data to be parsed to obtain the parsed data. The data indicator parser is a data indicator parser corresponding to each data indicator generated according to the configuration information of each data indicator after the parsing server obtains the configuration information of at least one data indicator. The data indicator may be a video downlink bit rate, a video capture frame rate, etc. As mentioned above, Figure 1 Three services are shown, namely interface service, parsing service and database warehousing service. The interface service can be implemented on the interface server, the parsing service can be implemented on the parsing server, and the database warehousing service can be implemented on the storage server. The parsing server can be used to obtain configuration information of at least one data indicator.
[0061] According to an exemplary embodiment of the present disclosure, the parsing server can receive a data indicator setting instruction, and then the parsing server can display a data indicator configuration interface in response to the data indicator setting instruction. Then, in response to the user's setting input, the configuration information of at least one data indicator can be set in the data indicator configuration interface, that is, the demander can set the configuration information of at least one data indicator in the data indicator configuration interface according to its own needs. In this way, a visual data indicator configuration interface can be provided, and the demander can arbitrarily add or modify the configuration information of the data indicator in the data indicator configuration interface according to its own needs, and then the data indicator parser corresponding to the data indicator can be generated through the configuration information of the data indicator, and the operation steps are simple, easy to understand, convenient and fast. When updating the data indicator, there is no need for developers to manually write the corresponding code, which avoids the occupation of development and testing resources, and can reduce time and labor costs.
[0062] It should be noted that the configuration information of each data indicator may include the data indicator name (counterName); data indicator code (counterId); data indicator alias (alias), for example, Chinese name; attributed text type (contentType); pre-filter (matcher), evaluator (evaluator), parser (parser), post-filter (filter), data saver (sinker) and read-time parser.
[0063] The text type is used to indicate which text type of request data can be parsed. In addition, the text type can be divided into four types: client basic information (ClientBaseInfo), runtime information (RuntimeInfo), audio transmission metadata (AtxMetaData), and video reception media information (VrxMediaInfo).
[0064] Prefilter: also known as a matcher, Predicate interface. After the data is passed in, a Boolean return value can be obtained, which is used to determine whether the current access data needs further filtering before parameter extraction.
[0065] Value extractor: According to the configuration value, select the specified implementation class, pass in the data and parameter values, and return a generic object Value, that is, extract the required data according to the input parameters and the appropriate value extraction method. For example, the data to be extracted can be single-value data, double-value data (within double 32 bits), single-character data (special for string data), percentage display (two number data), double-string data, Boolean data, etc.
[0066] Parser: After obtaining the value, you can call the configured serialization method to serialize the value into a data structure that can be put into the database, that is, you can process the extracted parameters once using the parsing method. For example, the parsing method can be data processing methods such as dividing by 1000 and multiplying by 10.
[0067] Post-filter: used to perform some filtering operations before data is saved. For example, if the data is found to be unchanged after the value is obtained, the data is found to be invalid, or a custom filtering condition is triggered, further operations on the current data can be prevented.
[0068] Data saver: call the sink method to pass in the data to be saved.
[0069] Read-time parser: It has nothing to do with the process of consuming the request data sent by the client, and is mainly used to read the configuration information in the database. It should be noted that the data in the database may be in the form of serialized data, which needs to be deserialized to be readable, so the parser is generally composed of a set of serialization and deserialization functions with the read-time parser.
[0070] According to an exemplary embodiment of the present disclosure, as described above, the configuration information of each data indicator may include text type information. First, a parsing manager may be created, that is, the parsing manager may be initialized. The parsing manager is responsible for managing the life cycle of the module, assigning different access data to the corresponding data indicator parser according to contentType, and rejecting client request data without pre-entered contentType.
[0071] Then, at least one content type manager (Content TypeManager) can be created in the parsing manager, that is, at least one content type manager can be initialized. The content type manager is mainly used to store instantiated data indicator parsers, and the content type manager can also extract the common data in the request data sent by the client, such as the application identifier (appId), account identifier (userId) and event timestamp (timestamp), in an agreed manner, and use it as a data template for subsequent processes. The content type manager can shield the heterogeneity at the contentType level and manage the main process of data parsing.
[0072] Next, according to the configuration information of each data indicator, a data indicator parser corresponding to each data indicator can be generated in the text type manager indicated by the text type information contained in the configuration information of each data indicator, that is, the instantiated data indicator parser will be stored in the corresponding text type manager according to the corresponding contentType. Figure 3 is a schematic diagram showing a text type manager according to an exemplary embodiment of the present disclosure. Figure 3 , you can create n data indicator parsers in one text type manager.
[0073] According to an exemplary embodiment of the present disclosure, the configuration information of each data indicator may include type information and code information of at least one function processor, wherein the at least one function processor is the pre-filter, valuer, parser, post-filter, etc. mentioned above.
[0074] According to the type information of each function processor in at least one function processor included in the configuration information of each data indicator, the function processor indicated by the code information of each function processor can be obtained from the function processor library indicated by the type information of each function processor. The function processor library can include multiple function processors.
[0075] For example, the types of function processors can be divided into pre-filters, evaluators, parsers and post-filters. Each type of function processor can correspond to a function processor library, that is, a function processor factory, such as a pre-filter factory (matcher factory), an evaluator factory (evaluator factory), a parser factory (parserfactory) and a post-filter factory (filter factory).
[0076] Each function processor factory can contain multiple processing function code segments of the corresponding type, and the multiple processing function code segments contained in the same function processor factory can correspond to different code information (int). For example, the type information and code information of the function processor contained in the configuration information of the three data indicators can be: Matcher (2), Matcher (5), filter (7) respectively. Based on the type information "Matcher" of the function processor and the code information "2" of the function processor, the getMatcher(2) method of the pre-filter factory (matcher factory) can be called to obtain the processing function code segment indicated by the digital code "2"; based on the type information "Matcher" of the function processor and the code information "5" of the function processor, the getMatcher(5) method of the pre-filter factory (matcher factory) can be called to obtain the processing function code segment indicated by the digital code "5"; based on the type information "filter" of the function processor and the code information "7" of the function processor, the getfilter(7) method of the post-filter factory (filterfactory) can be called to obtain the processing function code segment indicated by the digital code "7". Figure 4 is a schematic diagram illustrating a method of obtaining a function processor in a function processor library according to an exemplary embodiment of the present disclosure.
[0077] Next, at least one function processor corresponding to each acquired data indicator can be used to generate a data indicator parser corresponding to each data indicator in the text type manager indicated by the text type information included in the configuration information of each data indicator. Each data indicator parser can correspond to a counter configuration information. Figure 5 is a schematic diagram showing a data indicator parser according to an exemplary embodiment of the present disclosure. Figure 5 , each data indicator parser can contain componentized objects such as pre-filter, valuer, parser, post-filter, data saver, etc.
[0078] Figure 6 is a schematic diagram showing the hierarchical relationship among a parsing manager, a text type manager, and a data indicator parser according to an exemplary embodiment of the present disclosure. Figure 6, the parsing manager can be initialized first, and the parsing manager does not contain any content in the initialized state. Then, multiple text type managers can be created in the initialized parsing manager, and at this time, the initialized text type manager may not contain any content. Next, multiple data indicator parsers can be generated in each initialized text type manager based on the configuration information of the data indicator uploaded by the demander.
[0079] According to an exemplary embodiment of the present disclosure, as described above, the request data to be parsed can be pulled from the first message queue. The request data to be parsed is uploaded to the first message queue by the interface server, and the first message queue can be a message queue on the kafka platform. Then, the parsing server can pull the request data from the first message queue, and use the data indicator parser corresponding to each data indicator to parse and process the request data to obtain the parsed data. In this way, the message queue on the kafka platform can be used as a buffer pool for request data between the interface service and the parsing service, so as to realize the decoupling of the interface service and the parsing service, which can play a role in shaving the peak and filling the valley of traffic, and avoid the influx of large traffic requests to crash the downstream service.
[0080] According to an exemplary embodiment of the present disclosure, the parsed data can be sent to a second message queue so that the parsed data can be stored in a storage server through the second message queue, and the second message queue can be a message queue on a kafka platform. Then, the parsed data can be stored in a memory on the server side through the second message queue, that is, the parsed data can be stored in a storage server through the second message queue. In this way, the message queue on the kafka platform can be used as a buffer pool for parsed data between the parsing service and the database warehousing service, thereby realizing the decoupling of the parsing service and the database warehousing service, which can play a role in shaving the peaks and filling the valleys of traffic and avoiding the influx of large traffic requests to crash the downstream services.
[0081] It should be noted that when the upstream service sends 1 million data items per minute, and the downstream service can only consume 500,000 data items per minute, the extra 500,000 data items may cause the downstream server to crash if they are sent directly to the downstream server. At this time, if the extra 500,000 data items are stored in the message queue and consumed when the downstream service is not busy, on-demand consumption can be achieved to avoid large traffic data crashing the downstream service.
[0082] According to an exemplary embodiment of the present disclosure, the target text type contained in the request data to be parsed can be determined, and then the target text type manager whose corresponding text type is the target text type can be searched in at least one text type manager. Next, at least one data indicator parser generated in the target text type manager can be used to traverse and parse the request data to be parsed to obtain the parsed data, that is, each data indicator parser in at least one data indicator parser generated in the target text type manager can be used to parse the request data to be parsed respectively to obtain the data parsed by each data indicator parser of the request data to be parsed.
[0083] For example, assume that the target text type contained in the request data to be parsed is ClientBaseInfo. Figure 6 , assuming Figure 6 The text type corresponding to the text type manager (Content Type Manager 1) shown is ClientBaseInfo; the text type corresponding to the text type manager (Content Type Manager 2) is RuntimeInfo; the text type corresponding to the text type manager (Content Type Manager 3) is AtxMetaData; and the text type corresponding to the text type manager (Content Type Manager 4) is VrxMediaInfo. At this time, since only the text type corresponding to the text type manager (Content Type Manager 1) is the same as the target text type contained in the request data to be parsed, only the request data to be parsed can be sent to the text type manager (Content Type Manager 1), and the data indicator parser (Counter Parser 1), data indicator parser (CounterParser 2) and data indicator parser (Counter Parser 3) generated in the text type manager (Content TypeManager 1) are used to traverse and parse the request data to be parsed. In this way, compared with the method of directly sending the request data to be parsed to all text type managers for parsing and processing, the present invention selectively inputs the request data to be parsed into the text type manager whose corresponding text type matches the target text type contained in the request data to be parsed for parsing and processing, thereby reducing the number of traversals, avoiding unnecessary traversal processes and additional overhead, and can greatly save computing resources.
[0084] According to an exemplary embodiment of the present disclosure, the data type contained in the request data to be parsed can be determined, and the data type that can be parsed by each data indicator parser in at least one data indicator parser generated in the target text type manager can also be determined. Then, the data indicator parser generated in the target text type manager and capable of parsing the data type that matches the data type contained in the request data to be parsed can be used to perform traversal parsing processing on the request data to be parsed to obtain the parsed data.
[0085] For example, suppose the request data to be parsed is video data, and the data type contained in the video data is a large stream type. Figure 6 , assuming that the data type that can be parsed by the data indicator parser (Counter Parser 1) generated in the text type manager (Content Type Manager 1) is a large flow type, the data type that can be parsed by the data indicator parser (Counter Parser 2) is a small flow type, and the data type that can be parsed by the data indicator parser (Counter Parser 3) is a small flow type. At this time, since only the data type that can be parsed by the data indicator parser (Counter Parser 1) is the same as the data type contained in the video data, the video data can be sent only to the data indicator parser (Counter Parser 1) of the text type manager (Content Type Manager 1) for parsing. In this way, compared with the method of directly sending the request data to be parsed to all the data indicator parsers contained in the same text type manager for traversal parsing, the present disclosure selectively inputs the request data to be parsed into the data indicator parser whose parsable data type matches the data type contained in the request data to be parsed for parsing, which reduces the number of traversals, avoids unnecessary traversal processes and extra overhead, and can greatly save computing resources.
[0086] According to an exemplary embodiment of the present disclosure, a parameter template object (row) may be created in each text type manager in at least one text type manager. The parameter template object may include a value object and a data saver, the value object is used to temporarily store the retrieved and serialized value, and the parameter template object may send serialized data to the second message queue through a sink method. Figure 7 is a schematic diagram illustrating a parameter template object according to an exemplary embodiment of the present disclosure.
[0087] The public data contained in the request data to be parsed can be extracted, for example, the identification and account identification of the application. Then, the data indicator parser corresponding to each data indicator can be used to parse the data in the request data to be parsed except the identification and account identification of the application to obtain the parsed data. Next, the identification of the application, the account identification and the parsed data can be stored in the parameter template object in the text type manager where the data indicator parser is located. For example, the identification of the application, the account identification and the parsed data can be stored in the value object in the parameter template object. In this way, the parameter template object in the text type manager can extract the public data contained in the request data to be parsed, and the multiple parsed data obtained by traversing and parsing the request data to be parsed by multiple data indicator parsers can be stored in the value object in the parameter template object in correspondence with the public data, that is, the parameter template object in the text type manager can be reused for the traversal of each data indicator parser, which can avoid the repeated creation of the parameter template object and save computing resources.
[0088] It should be noted that after the demander updates the configuration information of the data indicator on the parsing server, the parsing server will upload the configuration information of the updated data indicator to the storage server, that is, the MySQL database. At the same time, the parsing server will send a hot update instruction to the topic of the Kafka platform. Then, the Kafka platform can broadcast the hot update instruction to multiple parsing servers that have subscribed to the topic of the Kafka platform, wherein each of the multiple parsing servers that have subscribed to the topic of the Kafka platform is provided with a parsing manager.
[0089] Next, the parsing manager in each parsing server that receives the hot update instruction will lock the consumption of all threads and pull the configuration information of the updated data indicators from the storage server, that is, the MySQL database, and then reinitialize the data indicator parser in the parsing manager based on the pulled updated data indicator configuration information.
[0090] After completing the initialization process of the data indicator parser, the parsing manager releases the lock and uses the regenerated data indicator parser to resume the consumption process of the request data sent by the client.
[0091] It should be noted that the parsing server of the configuration information of the uploaded data indicators can directly initialize the data indicator parser based on the configuration information of the data indicators uploaded by the demander this time, or it can pull the configuration information of the data indicators updated by the demander this time from the storage server, that is, the MySQL database, and initialize the data indicator parser based on the pulled updated data indicator configuration information.
[0092] Figure 8 It is a specific implementation flow chart showing a data processing method according to an exemplary embodiment of the present disclosure, which is applied to a parsing server.
[0093]
[0094]
[0095] Among them, "msgBody" contains data that needs to be parsed by the data indicator parser. The same level as "msgBody" is some public basic data, namely: "appId":"demo_app_id", "userId":"demo_user_id", "clientTime":1640608163611 / / Client sending time - millisecond timestamp.
[0096]
[0097]
[0098]
[0099] Reference Figure 8 In step 801, the Parse method provided by the parsing manager is called to pass in the request data sent by the client. The request data sent by the client may be in JSON format.
[0100] In step 802, a parsing method extracts text type information contained in the request data, and searches at least one text type manager for a text type manager whose corresponding text type matches the text type information contained in the request data.
[0101] From the previous code, we can see that the text type information contained in the request data is ClientBaseInfo; the text type of the configuration information of counterId=1, counterId=3, counterId=4 and counterId=5 is ClientBaseInfo, and the text type of the configuration information of counterId=2 is RuntimeInfo. Therefore, the data indicator parser corresponding to counterId=2 will not participate in the subsequent parsing process for the request data.
[0102] In step 803, the method for extracting common data in the text type manager is called to create a parameter template object. As mentioned above, the parameter template object may include a value object and a data saver.
[0103] At this time, the public data extracted from the request data sent by the client can be: "appId": "demo_app_id", "userId": "demo_user_id" and "clientTime": 1640608163611 (client sending time - millisecond timestamp). These extracted public data can be put into the value object contained in the parameter template object. In addition, some other public data can also be generated in the parameter template object, such as server time (server timestamp) and so on.
[0104] In step 804, multiple data indicator parsers included in the text type manager are traversed, wherein the data indicator parsers are generated according to the configuration information in the initialization phase and are placed in the corresponding text type manager.
[0105] In step 805, a data indicator parser is taken out from the text type manager, and the request data sent by the client is passed to the data indicator parser for parsing. The case where the pointer is null is ignored here. At this time, the data indicator parsers of counterId=1, counterId=3, counterId=4 and counterId=5 will be called.
[0106] In step 806, the request data sent by the client is filtered using a pre-filter.
[0107] The field that the counterId=5 data indicator parser wants to extract is "params":["ipAddress"], and the valuer is "evaluator":"extract string data", and the ipAddress field in the request data is "ipAddress":"110.132.221.9", so the extracted data is "110.132.221.9". In addition, the pre-filter of the counterId=5 data indicator parser is "matcher":"must start with 10 / 192.168 / 172", and the extracted data "110.132.221.9" does not start with 10 / 192.168 / 172, so the request data has not been filtered by the pre-filter of the counterId=5 data indicator parser and cannot continue to participate in the subsequent parsing process.
[0108] In step 807, the value extractor extracts data of a specified field from the request data sent by the client.
[0109] The field that the data indicator parser with counterId=1 wants to extract is "params":["country"], and the valuer is "evaluator":"Extract string type data", and the country field in the request data is "country":"CHN", so the extracted data is ["CHN"]. In addition, the pre-filter of the data indicator parser with counterId=1 is "matcher":"Asian countries only", and "CHN" happens to be an Asian country, so the request data has withstood the filtering of the pre-filter of the data indicator parser with counterId=1 and can continue to participate in the subsequent parsing process.
[0110] The valuer in the data indicator parser for counterId=3 is "evaluator":"extract string type data and combine with hyphens (-)", and the field to be extracted is "params":["province","city"], while the province field in the request data is "province":"Beijing", and the city field is "city":"Beijing", so the extracted data is ["Beijing"-"Beijing"]. In addition, the pre-filter of the data indicator parser for counterId=3 is "matcher":"none", so the request data has withstood the filtering of the pre-filter of the data indicator parser for counterId=3 and can continue to participate in the subsequent parsing process.
[0111] The data indicator parser for counterId=4 wants to extract the field "params":["screenHeight","screenWidth"], and the evaluator is "evaluator":"Extract 2 int32 data", while the screenHeight field in the request data is "screenHeight":720, and the screenWidth field is "screenWidth":360, so the extracted data is [720,360]. In addition, the pre-filter of the data indicator parser for counterId=4 is "matcher":"width and height are greater than 0", and the extracted screenHeight:720 and screenWidth:360 are both greater than zero, so the request data has withstood the filtering of the pre-filter of the data indicator parser for counterId=4 and can continue to participate in the subsequent parsing process.
[0112] In step 808, the parser parses the data extracted from the request data sent by the client, and the extracted data is formatted into a desired form.
[0113] The data extracted by the valuer of the data indicator parser with counterId=1 is ["CHN"], and the parser of the data indicator parser with counterId=1 is "parser":"convert the English abbreviation of the country to Chinese", so the extracted data is parsed by the parser of the data indicator parser with counterId=1 and the result is: ["China"].
[0114] The data extracted by the valuer of the data indicator parser with counterId=3 is ["Beijing"-"Beijing"], and the parser of the data indicator parser with counterId=3 is "parser":"none", so the result of parsing the extracted data through the parser of the data indicator parser with counterId=3 is: ["Beijing"-"Beijing"].
[0115] The data extracted by the valuer of the data indicator parser of counterId=4 is [720,360], and the parser of the data indicator parser of counterId=4 is "parser":"Combine two int32 data into int64 data", so the extracted data is parsed by the parser of the data indicator parser of counterId=4 and the result is: [3092376453480] (the first int32 is stored in the upper 32 bits, and the second int32 is stored in the lower 32 bits).
[0116] In step 809, the data obtained through parsing is filtered using a post-filter.
[0117] The post-filter of the data indicator parser with counterId=3 is "filter": "cannot be empty", and the result obtained by parsing with the parser of the data indicator parser with counterId=3 is: ["Beijing"-"Beijing"]. Since the content of ["Beijing"-"Beijing"] is not empty, the result obtained after parsing has withstood the filtering of the post-filter of the data indicator parser with counterId=3 and can continue to participate in the subsequent process.
[0118] The post filter of the data indicator parser with counterId=4 is "filter": "none", and the result obtained by parsing the parser of the data indicator parser with counterId=4 is: [3092376453480]. Therefore, the result obtained after parsing has withstood the filtering of the post filter of the data indicator parser with counterId=4 and can continue to participate in the subsequent process.
[0119] In step 8010, the data obtained after filtering by the post-filter is stored in the value object contained in the parameter template object created in step 803, and the data saver method provided by the data indicator parser is called to store the data in the database through the message queue.
[0120] For example, the result obtained by parsing the request data by the data indicator parser with counterId=3: ["Beijing"-"Beijing"] and the result obtained by parsing the request data by the data indicator parser with counterId=4: [3092376453480] can be stored in the value object included in the parameter template object created in step 803. At this time, the value object included in the parameter template object includes the common data in the request data sent by the client: "appId": "demo_app_id", "userId": "demo_user_id" and "clientTime": 1640608163611, and also includes the parsing result of the data indicator parser with counterId=3: ["Beijing"-"Beijing"] and the parsing result of the data indicator parser with counterId=4: [3092376453480].
[0121] Furthermore, the data saver method contained in the data indicator parser with counterId=3 can be called: "sinker": "client basic data dedicated database", and the parsing result of the data indicator parser with counterId=3: ["Beijing"-"Beijing"] can be stored in the location specified by the data saver through the message queue; the data saver method contained in the data indicator parser with counterId=4 can be called: "sinker": "client basic data dedicated database", and the parsing result of the data indicator parser with counterId=4: [3092376453480] can be stored in the location specified by the data saver through the message queue.
[0122] In step 8011, it is determined whether there is an untraversed data indicator parser in the text type manager. If there is an untraversed data indicator parser, the process returns to step 805; if there is no untraversed data indicator parser, the process proceeds to step 8012.
[0123] In step 8012, the parsing process for the request data sent by the client is completed, and the next request data access is awaited.
[0124] Fig. 9 is a block diagram showing a data processing device according to an exemplary embodiment of the present disclosure, which is applied to a parsing server.
[0125] Reference Fig. 9 , the device 900 may include an acquisition module 901 and a parsing module 902.
[0126] The acquisition module 901 can acquire the request data to be parsed. The request data to be parsed is the client request data received and processed by the interface server. As mentioned above, the interface server can receive the request data of the client, and then perform authentication, idempotence and preliminary formatting on the client request data to obtain the request data to be parsed to be processed by the parsing server.
[0127] The parsing module 902 can use a data indicator parser to parse the request data to be parsed and obtain the parsed data. The data indicator parser is a data indicator parser corresponding to each data indicator generated according to the configuration information of each data indicator after the parsing server obtains the configuration information of at least one data indicator. The data indicator can be a video downlink bit rate, a video acquisition frame rate, etc. As mentioned above, Figure 1 Three services are shown, namely interface service, parsing service and database warehousing service. The interface service can be implemented on the interface server, the parsing service can be implemented on the parsing server, and the database warehousing service can be implemented on the storage server. The parsing server can be used to obtain configuration information of at least one data indicator.
[0128] According to an exemplary embodiment of the present disclosure, the data processing device 900 may further include a receiving module, a display module and a setting module. The receiving module may receive a data indicator setting instruction, and then the display module may display a data indicator configuration interface in response to the data indicator setting instruction. Then, the setting module may set the configuration information of at least one data indicator in the data indicator configuration interface in response to the user's setting input, that is, the demander may set the configuration information of at least one data indicator in the data indicator configuration interface according to its own needs. In this way, a visual data indicator configuration interface may be provided, and the demander may add or modify the configuration information of the data indicator in the data indicator configuration interface as needed, and then the data indicator parser corresponding to the data indicator may be generated through the configuration information of the data indicator, and the operation steps are simple, easy to understand, convenient and fast. When updating the data indicator, there is no need for the developer to manually write the corresponding code, which avoids the occupation of development and testing resources, and can reduce time cost and manpower cost. Among them, the configuration information of each data indicator may include the data indicator name; data indicator code; alias of the data indicator, for example, Chinese name; attributed text type; pre-filter, valuer, parser, post-filter, data saver and read-time parser.
[0129] According to an exemplary embodiment of the present disclosure, the data processing device 900 may also include a parsing manager creation module, a text type manager creation module and a generation module. The configuration information of each data indicator may include text type information. First, the parsing manager creation module may create a parsing manager, that is, the parsing manager may be initialized. The parsing manager is responsible for managing the life cycle of the module, allocating different access data to the corresponding data indicator parser according to contentType, and rejecting client request data without pre-entered contentType.
[0130] Then, the text type manager creation module can create at least one text type manager in the parsing manager, that is, it can initialize at least one text type manager. The text type manager is mainly used to store the instantiated data indicator parser, and the text type manager can also extract the public data in the request data sent by the client, such as the application ID, account ID and event timestamp, in an agreed manner, and use it as a data template for subsequent processes. The text type manager can shield the heterogeneity at the contentType level and manage the main process of data parsing.
[0131] Next, the generation module can generate a data indicator parser corresponding to each data indicator in the text type manager indicated by the text type information contained in the configuration information of each data indicator according to the configuration information of each data indicator, that is, the instantiated data indicator parser will be stored in the corresponding text type manager according to the corresponding contentType.
[0132] According to an exemplary embodiment of the present disclosure, the configuration information of each data indicator may include type information and code information of at least one function processor, wherein the at least one function processor is the pre-filter, valuer, parser, post-filter, etc. mentioned above.
[0133] The generation module can obtain the function processor indicated by the code information of each function processor in the function processor library indicated by the type information of each function processor according to the type information of each function processor in at least one function processor included in the configuration information of each data indicator. The function processor library can include multiple function processors.
[0134] Next, the generation module may utilize at least one function processor corresponding to each acquired data indicator to generate a data indicator parser corresponding to each data indicator in the text type manager indicated by the text type information included in the configuration information of each data indicator.
[0135] According to an exemplary embodiment of the present disclosure, the acquisition module 901 can pull the request data to be parsed from the first message queue. The request data to be parsed is uploaded to the first message queue by the interface server, and the first message queue can be a message queue on the kafka platform. Then, the parsing server can pull the request data from the first message queue, and use the data indicator parser corresponding to each data indicator to parse and process the request data to obtain the parsed data. In this way, the message queue on the kafka platform can be used as a buffer pool for request data between the interface service and the parsing service to achieve decoupling of the interface service and the parsing service, which can play a role in shaving the peak and filling the valley of traffic, and avoid the influx of large traffic requests to crash the downstream service.
[0136] According to an exemplary embodiment of the present disclosure, the data processing device 900 may further include a sending module. The sending module may send the parsed data to a second message queue, so as to store the parsed data in a storage server through the second message queue, and the second message queue may be a message queue on a kafka platform. Then, the parsed data may be stored in a memory on the server side through the second message queue, that is, the parsed data may be stored in a storage server through the second message queue. In this way, the message queue on the kafka platform may be used as a buffer pool for the parsed data between the parsing service and the database warehousing service, thereby realizing the decoupling of the parsing service and the database warehousing service, and playing a role in shaving the peaks and filling the valleys of the traffic, and avoiding the influx of large traffic requests to crash the downstream services.
[0137] According to an exemplary embodiment of the present disclosure, the parsing module 902 can determine the target text type contained in the request data to be parsed, and then can search for a target text type manager whose corresponding text type is the target text type in at least one text type manager. Next, the request data to be parsed can be traversed and parsed using at least one data indicator parser generated in the target text type manager to obtain the parsed data, that is, each data indicator parser in at least one data indicator parser generated in the target text type manager can be used to parse the request data to be parsed respectively, and obtain the data of the request data to be parsed after each data indicator parser. In this way, compared with the method of directly sending the request data to be parsed to all text type managers for parsing, the present disclosure selectively inputs the request data to be parsed into the text type manager whose corresponding text type matches the target text type contained in the request data to be parsed for parsing, which reduces the number of traversals, avoids unnecessary traversal processes and extra overhead, and can greatly save computing resources.
[0138] According to an exemplary embodiment of the present disclosure, the parsing module 902 can determine the data type contained in the request data to be parsed, and can also determine the data type that can be parsed by each data indicator parser in at least one data indicator parser generated in the target text type manager. Then, the data indicator parser generated in the target text type manager and whose parsable data type matches the data type contained in the request data to be parsed can be used to traverse and parse the request data to be parsed to obtain the parsed data. In this way, compared to the method of directly sending the request data to be parsed to all data indicator parsers contained in the same text type manager for traversal and parsing, the present disclosure selectively inputs the request data to be parsed into the data indicator parser whose parsable data type matches the data type contained in the request data to be parsed for parsing, thereby reducing the number of traversals, avoiding unnecessary traversal processes and extra overhead, and greatly saving computing resources.
[0139] According to an exemplary embodiment of the present disclosure, the data processing device 900 may further include a parameter template object creation module, an extraction module, and a storage module. The parameter template object creation module may create a parameter template object in each text type manager in at least one text type manager. The parameter template object may include a value object and a data saver, the value object is used to temporarily store the retrieved and serialized value, and the parameter template object may send serialized data to the second message queue through a sink method.
[0140] The extraction module can extract the public data contained in the request data to be parsed, for example, the identification and account identification of the application. Then, the parsing module 902 can use the data indicator parser corresponding to each data indicator to parse the data in the request data to be parsed except the identification and account identification of the application, and obtain the parsed data. Next, the storage module can store the identification, account identification and parsed data of the application in the parameter template object in the text type manager where the data indicator parser is located. For example, the storage module can store the identification, account identification and parsed data of the application in the value object in the parameter template object. In this way, the parameter template object in the text type manager can extract the public data contained in the request data to be parsed, and the multiple parsed data obtained by traversing and parsing the request data to be parsed by multiple data indicator parsers can be stored in the value object in the parameter template object in correspondence with the public data, that is, the parameter template object in the text type manager can be reused for the traversal of each data indicator parser, which can avoid the repeated creation of the parameter template object and save computing resources.
[0141] Fig.10is a block diagram illustrating an electronic device 1000 according to an exemplary embodiment of the present disclosure.
[0142] Reference Fig.10 The electronic device 1000 includes at least one memory 1001 and at least one processor 1002. The at least one memory 1001 stores instructions. When the instructions are executed by the at least one processor 1002, the data processing method according to the exemplary embodiment of the present disclosure is executed.
[0143] As an example, the electronic device 1000 may be a PC, a tablet device, a personal digital assistant, a smart phone, or other device capable of executing the above instructions. Here, the electronic device 1000 is not necessarily a single electronic device, but may also be any device or circuit capable of executing the above instructions (or instruction sets) individually or in combination. The electronic device 1000 may also be part of an integrated control system or a system manager, or may be configured as a portable electronic device interconnected with a local or remote (e.g., via wireless transmission) interface.
[0144] In the electronic device 1000, the processor 1002 may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller or a microprocessor. As an example and not limitation, the processor may also include an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, etc.
[0145] The processor 1002 may execute instructions or codes stored in the memory 1001, wherein the memory 1001 may also store data. Instructions and data may also be sent and received over a network via a network interface device, wherein the network interface device may employ any known transmission protocol.
[0146] The memory 1001 may be integrated with the processor 1002, for example, by placing RAM or flash memory within an integrated circuit microprocessor or the like. In addition, the memory 1001 may include a separate device, such as an external disk drive, a storage array, or any other storage device that can be used by a database system. The memory 1001 and the processor 1002 may be operatively coupled, or may communicate with each other, such as through an I / O port, a network connection, etc., so that the processor 1002 can read files stored in the memory.
[0147] In addition, the electronic device 1000 may further include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, a mouse, a touch input device, etc.) All components of the electronic device 1000 may be connected to each other via a bus and / or a network.
[0148] According to an exemplary embodiment of the present disclosure, a computer-readable storage medium may also be provided, and when the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the above-mentioned data processing method. Examples of computer-readable storage media here include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), card storage (such as, multimedia card, secure digital (SD) card or extreme digital (XD) card), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk and any other device, any other device is configured to store computer programs and any associated data, data files and data structures in a non-transitory manner and provide the computer programs and any associated data, data files and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the above-mentioned computer-readable storage medium can be run in an environment deployed in a computer device such as a client, a host, an agent device, a server, etc. In addition, in one example, the computer program and any associated data, data files and data structures are distributed on a networked computer system, so that the computer program and any associated data, data files and data structures are stored, accessed and executed in a distributed manner by one or more processors or computers.
[0149] According to an exemplary embodiment of the present disclosure, a computer program product may also be provided, including a computer program, and when the computer program is executed by a processor, the data processing method according to the present disclosure is implemented.
[0150] According to the data processing method, device, electronic device and storage medium disclosed in the present invention, the demander can configure the configuration information of the data indicator according to its own needs, and then can generate a data indicator parser corresponding to the data indicator through the configuration information of the data indicator. When the data indicator is updated, there is no need for the developer to manually write the corresponding code, thus avoiding the occupation of development and testing resources, and reducing time and labor costs.
[0151] Furthermore, a visual data indicator configuration interface can be provided, and the demander can arbitrarily add or modify the configuration information of the data indicator in the data indicator configuration interface according to its own needs, and then generate the data indicator parser corresponding to the data indicator through the data indicator configuration information. The operation steps are simple, easy to understand, convenient and fast.
[0152] Furthermore, the message queue on the Kafka platform can be used as a buffer pool for request data between the interface service and the resolution service to achieve decoupling of the interface service and the resolution service, which can smooth out traffic peaks and fill valleys, and avoid large traffic requests from overwhelming downstream services.
[0153] Furthermore, the message queue on the Kafka platform can be used as a buffer pool for parsed data between the parsing service and the database warehousing service to decouple the parsing service and smooth out the traffic peaks, thus preventing large traffic requests from overwhelming downstream services.
[0154] Furthermore, compared to the method of directly sending the request data to be parsed to all text type managers for parsing and processing, the present invention selectively inputs the request data to be parsed into the text type manager whose corresponding text type matches the target text type contained in the request data to be parsed for parsing and processing, thereby reducing the number of traversals, avoiding unnecessary traversal processes and additional overhead, and can greatly save computing resources.
[0155] Furthermore, compared to the method of directly sending the request data to be parsed to all data indicator parsers contained in the same text type manager for traversal and parsing processing, the present invention selectively inputs the request data to be parsed into a data indicator parser whose parseable data type matches the data type contained in the request data to be parsed for parsing processing, thereby reducing the number of traversals, avoiding unnecessary traversal processes and additional overhead, and can greatly save computing resources.
[0156] Furthermore, the parameter template object in the text type manager can be reused in the traversal of each data indicator parser, which can avoid the repeated creation of the parameter template object and save computing resources.
[0157] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.
[0158] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.
Claims
1. A data processing method, applied to a parsing server, characterized in that: The data processing method comprises: Acquire request data to be parsed, wherein the request data to be parsed is client request data received and processed by the interface server; Utilizing a data indicator parser to parse the request data to be parsed to obtain parsed data, wherein the data indicator parser is a data indicator parser corresponding to each data indicator generated according to the configuration information of each data indicator after the parsing server obtains the configuration information of at least one data indicator; Before obtaining the request data to be parsed, the following steps are also included: Create a parsing manager; creating at least one text type manager in the parsing manager; The data processing method further includes: creating a parameter template object in each of the at least one text type manager; Extracting the application identifier and the account identifier contained in the request data to be parsed; The using a data indicator parser to parse the request data to be parsed includes: Utilizing the data indicator parser to parse the data in the request data to be parsed except for the identification of the application and the identification of the account, to obtain the parsed data; The data processing method further includes: The application identifier, the account identifier and the parsed data are correspondingly stored in a parameter template object in a text type manager where the data indicator parser is located.
2. The data processing method according to claim 1, characterized in that: The configuration information of each data indicator contains text type information. Before the step of obtaining the request data to be parsed, it also includes: According to the configuration information of each data indicator, a data indicator parser corresponding to each data indicator is generated in the text type manager indicated by the text type information included in the configuration information of each data indicator.
3. The data processing method according to claim 2, characterized in that: The configuration information of each data indicator contains the type information and code information of at least one function processor. The step of generating a data indicator parser corresponding to each data indicator in the text type manager indicated by the text type information included in the configuration information of each data indicator according to the configuration information of each data indicator comprises: According to the type information of each function processor in at least one function processor included in the configuration information of each data indicator, the function processor indicated by the code information of each function processor is obtained from the function processor library indicated by the type information of each function processor, wherein the function processor library includes a plurality of function processors; By utilizing the at least one function processor corresponding to each of the acquired data indicators, a data indicator parser corresponding to each of the data indicators is generated in the text type manager indicated by the text type information included in the configuration information of each of the data indicators.
4. The data processing method according to claim 2, characterized in that: The utilizing a data indicator parser to parse the request data to be parsed to obtain parsed data includes: Determining the target text type contained in the request data to be parsed; Searching for a target text type manager whose corresponding text type is the target text type in the at least one text type manager; The data indicator parser generated in the target text type manager is used to traverse and parse the request data to be parsed to obtain the parsed data.
5. The data processing method according to claim 4, characterized in that: The step of using the data indicator parser generated in the target text type manager to perform traversal parsing on the request data to be parsed includes: Determine the data type contained in the request data to be parsed; Determining the data type that can be parsed by each data indicator parser in at least one data indicator parser generated in the target text type manager; Utilizing a data indicator parser generated in the target text type manager and capable of parsing a data type that matches the data type contained in the request data to be parsed, the request data to be parsed is traversed and parsed to obtain the parsed data.
6. The data processing method according to claim 1, characterized in that: Before the step of obtaining the request data to be parsed, it also includes: Receive data indicator setting instructions; In response to the data indicator setting instruction, displaying a data indicator configuration interface; In response to a setting input by a user, configuration information of the at least one data indicator is set in the data indicator configuration interface.
7. The data processing method according to claim 1, characterized in that: The obtaining of the request data to be parsed includes: The request data to be parsed is pulled from the first message queue, wherein the request data to be parsed is uploaded to the first message queue by the interface server.
8. The data processing method according to claim 1, characterized in that: The data processing method further includes: The parsed data is sent to a second message queue, so that the parsed data is stored in a storage server through the second message queue.
9. A data processing device, applied to a parsing server, characterized in that: include: An acquisition module is configured to acquire request data to be parsed, wherein the request data to be parsed is client request data received and processed by the interface server; A parsing module, configured to parse the request data to be parsed using a data indicator parser to obtain parsed data, wherein the data indicator parser is a data indicator parser corresponding to each data indicator generated according to the configuration information of each data indicator after the parsing server obtains the configuration information of at least one data indicator; Wherein, the data processing device further includes: A parsing manager creation module, configured to create a parsing manager; a text type manager creation module, configured to create at least one text type manager in the parsing manager; The data processing device further comprises: a parameter template object creation module configured to create a parameter template object in each text type manager of the at least one text type manager; An extraction module, configured to extract an application identifier and an account identifier contained in the request data to be parsed; The parsing module is configured to parse the data except the application identifier and the account identifier in the request data to be parsed by using the data indicator parser to obtain the parsed data; The data processing device further comprises: The storage module is configured to store the application identifier, the account identifier and the parsed data in a parameter template object in the text type manager where the data indicator parser is located.
10. The data processing device according to claim 9, characterized in that The configuration information of each data indicator contains text type information. The data processing device further comprises: The generating module is configured to generate, according to the configuration information of each data indicator, a data indicator parser corresponding to each data indicator in the text type manager indicated by the text type information contained in the configuration information of each data indicator.
11. The data processing device according to claim 10, characterized in that The configuration information of each data indicator contains the type information and code information of at least one function processor. The generation module is configured to: According to the type information of each function processor in at least one function processor included in the configuration information of each data indicator, the function processor indicated by the code information of each function processor is obtained from the function processor library indicated by the type information of each function processor, wherein the function processor library includes a plurality of function processors; By utilizing the at least one function processor corresponding to each of the acquired data indicators, a data indicator parser corresponding to each of the data indicators is generated in the text type manager indicated by the text type information included in the configuration information of each of the data indicators.
12. The data processing device according to claim 10, characterized in that: The parsing module is configured to: Determining the target text type contained in the request data to be parsed; Searching for a target text type manager whose corresponding text type is the target text type in the at least one text type manager; The data indicator parser generated in the target text type manager is used to traverse and parse the request data to be parsed to obtain the parsed data.
13. The data processing device according to claim 12, characterized in that: The parsing module is configured to: Determine the data type contained in the request data to be parsed; Determining the data type that can be parsed by each data indicator parser in at least one data indicator parser generated in the target text type manager; Utilizing a data indicator parser generated in the target text type manager and capable of parsing a data type that matches the data type contained in the request data to be parsed, the request data to be parsed is traversed and parsed to obtain the parsed data.
14. The data processing device according to claim 9, characterized in that The data processing device further comprises: A receiving module, configured to receive a data indicator setting instruction; A display module, configured to display a data indicator configuration interface in response to the data indicator setting instruction; The setting module is configured to set the configuration information of the at least one data indicator in the data indicator configuration interface in response to the setting input of the user.
15. The data processing device according to claim 9, characterized in that: The acquisition module is configured to pull the request data to be parsed from the first message queue, wherein the request data to be parsed is uploaded to the first message queue by the interface server.
16. The data processing device according to claim 9, characterized in that The data processing device further comprises: The sending module is configured to send the parsed data to the second message queue, so as to store the parsed data in the storage server through the second message queue.
17. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the data processing method according to any one of claims 1 to 8.
18. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the data processing method as claimed in any one of claims 1 to 8.
