Data processing method and device, equipment and storage medium

Abstract

The invention discloses a data processing method and device, equipment and a storage medium, and relates to the technical field of data processing. The method comprises the following steps: acquiring a multi-protocol data message, and uniformly converting the multi-protocol data message into a JSON (JavaScript Object Notation) message; analyzing the JSON message through JSONPath on the basis of a preset stream processing algorithm, so as to determine target data in the JSON message; and performing data processing on the target data by using a predefined business processing logic to obtain processed data, and mapping the processed data and a business index to generate a business index result. Through the technical scheme of the invention, access verification of different protocol data and JSON conversion can be realized, and the data in the JSON format can be uniformly analyzed without hard coding analysis of each level.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data processing method, device, equipment and storage medium. Background Art

[0002] In modern data processing architecture, data sources may transmit data through multiple protocols, such as HTTP (HyperText Transfer Protocol), Dubbo, etc. Each protocol has its specific application scenarios and advantages, but these data need to be uniformly parsed and processed to form usable indicator information. Currently, JSON (JavaScript Object Notation, a format for storing and transmitting data) objects or XML (Extensible Markup Language, Extensible Markup Language) nodes are usually traversed layer by layer through hard-coded code. The code is lengthy, especially when the data level is deep, and the readability is poor and difficult to maintain. When dealing with conditional screening and filtering, the existing parsing technology is not flexible enough, and the code needs to be changed frequently. Usually, manual traversal and judgment are required to filter out the target data. When the external interface data structure changes, it needs to be adjusted step by step through the code, which is easy to cause parsing errors or exceptions. It can be seen that there is currently a lack of an efficient technical solution to integrate the data access and efficient parsing process of these protocols. Summary of the invention

[0003] In view of this, the purpose of the present invention is to provide a data processing method, device, equipment and storage medium, which can realize access verification of data of different protocols and uniformly parse data in JSON format without hard-coding parsing at each level. The specific scheme is as follows:

[0004] In a first aspect, the present application discloses a data processing method, comprising:

[0005] Obtain multi-protocol data messages, and uniformly convert the multi-protocol data messages into JSON messages;

[0006] Parsing the JSON message based on a preset stream processing algorithm through JSONPath to determine target data in the JSON message;

[0007] The target data is processed using a predefined business processing logic to obtain processed data, and the processed data is mapped to business indicators to generate business indicator results.

[0008] Optionally, the acquiring of multi-protocol data messages and converting the multi-protocol data messages uniformly into JSON messages includes:

[0009] Acquire a multi-protocol data message, and perform functional formatting verification on the multi-protocol data message;

[0010] When the multi-protocol data message is JSON data, the JSON data is directly stored in a JSON string to generate a JSON message;

[0011] When the multi-protocol data message is extensible markup language data, the extensible markup language data is converted into the JSON data, and then the JSON data is stored in a JSON string to generate a JSON message.

[0012] Optionally, the parsing the JSON message based on a preset stream processing algorithm by JSONPath to determine the target data in the JSON message includes:

[0013] Determine the data path of the JSON message;

[0014] Write a JSONPath expression based on the data path;

[0015] The JSONPath expression is query compiled based on a preset stream processing algorithm to generate a deterministic finite automaton, and query processing is performed based on the deterministic finite automaton to determine target data in the JSON message.

[0016] Optionally, query compiling the JSONPath expression based on a preset stream processing algorithm to generate a deterministic finite automaton includes:

[0017] The JSONPath expression is converted into a regular expression, and the regular expression is converted into a deterministic finite automaton.

[0018] Optionally, converting the JSONPath expression into a regular expression includes:

[0019] Define a preset conversion method, and perform query matching on the JSONPath expression based on the preset conversion method, so as to convert the JSONPath expression into a regular expression according to the query matching result; wherein the preset conversion method includes conversion methods of alphabet definition, placeholder processing, label selector conversion, array slice selector conversion, selector conversion, child axis conversion, and descendant axis conversion.

[0020] Optionally, converting the regular expression into a deterministic finite automaton comprises:

[0021] The regular expression is compiled into a non-deterministic finite automaton using the Thompson construction algorithm, and then the non-deterministic finite automaton is converted into a deterministic finite automaton using a subset construction method;

[0022] Accordingly, query processing is performed based on the deterministic finite automaton to determine the target data in the JSON message, including:

[0023] Determining an initial state of the deterministic finite automaton, and sequentially reading each regular expression through the deterministic finite automaton to determine a next state of the deterministic finite automaton according to a state transition rule;

[0024] When the next state includes a target state for indicating a successful query, a corresponding matching result is returned, and after all the regular expressions are read, the target data in the JSON message is determined according to the matching result.

[0025] Optionally, the processing the target data using a predefined business processing logic to obtain processed data, and mapping the processed data with a business indicator to generate a business indicator result includes:

[0026] Assigning a corresponding variable name to each data point or data item in the target data to determine the corresponding preliminary variable value;

[0027] Using a preset script engine, the preliminary variable values ​​are input into a preset processing algorithm to obtain processed data;

[0028] The processed data is mapped to the business indicator to generate a business indicator result.

[0029] In a second aspect, the present application discloses a data processing device, comprising:

[0030] A data conversion module, used to obtain multi-protocol data messages and uniformly convert the multi-protocol data messages into JSON messages;

[0031] A data parsing module, used to parse the JSON message based on a preset stream processing algorithm through JSONPath to determine target data in the JSON message;

[0032] The data processing module is used to process the target data using a predefined business processing logic to obtain processed data, and map the processed data with business indicators to generate business indicator results.

[0033] In a third aspect, the present application discloses an electronic device, comprising a processor and a memory; wherein the memory is used to store a computer program, and the computer program is loaded and executed by the processor to implement the data processing method as described above.

[0034] In a fourth aspect, the present application discloses a computer-readable storage medium for storing a computer program; wherein the computer program implements the data processing method as described above when executed by a processor.

[0035] The present application provides a data processing method, including: obtaining a multi-protocol data message, and uniformly converting the multi-protocol data message into a JSON message; parsing the JSON message based on a preset stream processing algorithm through JSONPath to determine target data in the JSON message; processing the target data using a predefined business processing logic to obtain processed data, and mapping the processed data with business indicators to generate business indicator results.

[0036] The beneficial technical effects of the present application are as follows: traditional JSON data traversal and conditional statement methods are cumbersome, inefficient and difficult to modify. The present invention accesses and converts JSON to data of different protocols, uses JSONPath as the core to process JSON data, and uniformly parses data in JSON format based on a preset stream processing algorithm without the need for hard-coded parsing at each level. JSONPath provides a more concise, flexible and efficient way to query, locate and extract content from JSON data, improves development efficiency for data processing, reduces learning costs, and improves code readability. In addition, the parsed target data is mapped to business indicators to associate data with specific businesses. Business indicator mapping can be adjusted according to different requirements, standards and algorithms to provide accurate and reliable data support for decision-making.

[0037] In addition, the present application provides a data processing device, equipment and storage medium, which correspond to the above-mentioned data processing method and have the same effect as above. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0039] Figure 1 A flow chart of a data processing method disclosed in this application;

[0040] Figure 2 An overall block diagram of a data processing flow disclosed in this application;

[0041] Figure 3 A schematic diagram of a data parsing engine disclosed in this application;

[0042] Figure 4 A schematic diagram of an indicator formation process disclosed in this application;

[0043] Figure 5 A schematic diagram of the structure of a data processing device disclosed in this application;

[0044] Figure 6 This is a structural diagram of an electronic device disclosed in this application. DETAILED DESCRIPTION

[0045] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0046] Currently, data transmitted through multiple protocols needs to be uniformly parsed and processed. The existing technology needs to traverse JSON objects or XML nodes layer by layer through hard-coded code. The code is lengthy, especially when the data level is deep, and the readability is poor and difficult to maintain. When dealing with conditional screening and filtering, the existing parsing technology is not flexible enough, and the code needs to be changed frequently. Usually, manual traversal and judgment are required to filter out the target data. When the external interface data structure changes, it needs to be adjusted step by step through the code, which is easy to cause parsing errors or exceptions.

[0047] To this end, the present application provides a data processing solution that can implement access verification of data of different protocols and uniformly parse data in JSON format without the need for hard-coded parsing at each level.

[0048] The embodiment of the present invention discloses a data processing method, see Figure 1 As shown, the method includes:

[0049] Step S11: Acquire multi-protocol data messages, and convert the multi-protocol data messages into JSON messages.

[0050] In an embodiment of the present application, a data front-end module is provided to receive multi-protocol data messages based on an external interface in an upstream system, such as HTTP protocol messages, Dubbo protocol messages, etc., and then convert them uniformly into JSON data format.

[0051] Specifically, when implementing access verification of different protocol data and converting JSON, the multi-protocol data message is functionally formatted and verified; if the data message is JSON data, it is directly stored in the JSON string; if the data message is XML data, the XML is converted into JSON data and stored in the JSON string to obtain a JSON message.

[0052] Step S12: Parse the JSON message through JSONPath based on a preset stream processing algorithm to determine the target data in the JSON message.

[0053] In the embodiment of the present application, JSONPath is used as the core to build a data parsing engine to process JSON data. Traditional JSON data traversal and conditional statement methods are cumbersome and inefficient, and not easy to modify. As a powerful data query language, JSONPath is suitable for parsing JSON data format. It provides a more concise, flexible and efficient way to query, locate and extract content in JSON data and obtain results. Therefore, the low-latency streaming processing method based on JSONPath uniformly parses JSON messages, intelligently selects parsing paths, flexibly locates and extracts data, and determines the target data in JSON messages. In this way, the structure of external data can be made clearer and easier to understand without the need for hard-coded parsing at each level, thereby improving the development efficiency of data processing, reducing learning costs, and improving code readability.

[0054] Step S13: Process the target data using a predefined business processing logic to obtain processed data, and map the processed data to business indicators to generate business indicator results.

[0055] In the embodiment of the present application, business logic processing is the process of secondary processing of the target data obtained after data analysis, which aims to further convert the raw data obtained by preliminary analysis into valuable business indicators through a series of predefined processing steps.

[0056] When using business processing logic to process target data, data processing is first performed to ensure that the processed data meets business requirements and goals before final output. Then, the processed data is mapped to business indicators. By associating data with specific businesses, the final business indicator results are generated. It should be noted that business indicator mapping can be adjusted according to different requirements, standards and algorithms, thereby providing accurate and reliable data support for decision-making.

[0057] like Figure 2The figure shows a schematic diagram of the overall process of integrating multi-protocol data messages, connecting them and parsing them efficiently. First, a unified external data interface is set up to obtain multi-protocol data; then the data is uniformly converted into JSON data format using the data front-end module. Furthermore, using JSONPath as the core, a data parsing engine is built to process JSON data; finally, business indicator values ​​are extracted from the parsed data to generate the final business indicator results for indicator logic processing expansion.

[0058] The present application provides a data processing method, including: obtaining a multi-protocol data message, and uniformly converting the multi-protocol data message into a JSON message; parsing the JSON message based on a preset stream processing algorithm through JSONPath to determine target data in the JSON message; processing the target data using a predefined business processing logic to obtain processed data, and mapping the processed data with business indicators to generate business indicator results.

[0059] The beneficial technical effects of the present application are as follows: traditional JSON data traversal and conditional statement methods are cumbersome, inefficient and difficult to modify. The present invention accesses and converts JSON to data of different protocols, uses JSONPath as the core to process JSON data, and uniformly parses data in JSON format based on a preset stream processing algorithm without the need for hard-coded parsing at each level. JSONPath provides a more concise, flexible and efficient way to query, locate and extract content from JSON data, improves development efficiency for data processing, reduces learning costs, and improves code readability. In addition, the parsed target data is mapped to business indicators to associate data with specific businesses. Business indicator mapping can be adjusted according to different requirements, standards and algorithms to provide accurate and reliable data support for decision-making.

[0060] Based on the above embodiment, this embodiment will specifically explain S12 in the above embodiment. In order to uniformly parse the data in JSON format without hard-coding parsing at each level, the process of parsing the JSON message based on a preset stream processing algorithm by JSONPath to determine the target data in the JSON message may include the following steps:

[0061] Determine the data path of the JSON message;

[0062] Write a JSONPath expression based on the data path;

[0063] The JSONPath expression is query compiled based on a preset stream processing algorithm to generate a deterministic finite automaton, and query processing is performed based on the deterministic finite automaton to determine target data in the JSON message.

[0064] In the embodiment of the present application, the data path of the specific JSON message of the data to be parsed is confirmed, and then a JSONPath expression is written according to the corresponding data path.

[0065] JSONPath is specifically used to extract data from JSON data. It specifies the query location through path syntax, such as selecting root nodes, attributes, array elements, etc. The commonly used JSONPath syntax is expressed as follows:

[0066] (1) Query: $segment*, the query starts with the root element $, followed by one or more segments;

[0067] (2) Segment: axis [selector (, selector)*], each segment consists of an axis and one or more selectors. The selector is the specific method used to select data, and the axis determines the direction of the query.

[0068] (3) Axis: .(child)|..(descendant); . represents the direct child node (child) of the current node; .. represents the descendant node (descendant), that is, recursively searching all nested nodes from the current node downward.

[0069] (4) selector: name selector (name), index selector (index), asterisk selector (*), array slice selector (start:end:step), filter selector (?(predicate)).

[0070] Furthermore, a low-latency stream processing algorithm based on JSONPath parses JSON data to obtain results. In an embodiment of the present application, the preset stream processing algorithm uses SST (Streaming Tree Transducer) as the underlying data model to perform streaming processing of JSON. The state machine is used to implement the step-by-step conversion of input data and generate corresponding output. The stream processing algorithm includes two parts: query compilation and query processing.

[0071] In a specific implementation, query compilation generates custom machine code by just-in-time compilation to achieve efficient query processing. First, the JSONPath expression is converted into a regular expression; then, the regular expression is converted into a deterministic finite automaton.

[0072] Specifically, when converting the regular expression, the following steps are specifically included:

[0073] Define a preset conversion method, and perform query matching on the JSONPath expression based on the preset conversion method, so as to convert the JSONPath expression into a regular expression according to the query matching result; wherein the preset conversion method includes conversion methods of alphabet definition, placeholder processing, label selector conversion, array slice selector conversion, selector conversion, child axis conversion, and descendant axis conversion.

[0074] For example, JSONPath queries are converted into regular expressions in the following way:

[0075] (1) Alphabet definition: In order to convert the simplified JSONPath query into a regular expression, we first need to define an "alphabet", that is, all the labels (such as the property names of the object) and array indexes (such as the slice range of the array) involved in the query.

[0076] (2) Placeholder processing: Since it is impossible to predict all object names and array indexes in JSON, a special placeholder is used ( ) to represent these unknown parts.

[0077] (3) Conversion of label selectors: When compiling a query, label selectors (such as specified attribute names) are directly converted to labels in regular expressions. The asterisk selector (*) means "match all labels", so it is converted to a regular expression that matches any label.

[0078] (4) Conversion of array slice selectors: For array slice selectors (e.g., [start:end]), it allows selecting array indexes within a specified range. The upper bound of the array is fixed, which means that the corresponding regular expression can be constructed based on this range.

[0079] (5) Conversion of selectors: Selectors (such as ?()) are used to filter elements that meet a certain condition. They will be converted into a "selection" of a regular subexpression, that is, selecting the part that meets a specific rule.

[0080] (6) Child axis conversion: The child axis (such as .) means to get the child attributes directly from the current object downward. This is usually converted into a connection operation, which means matching the next level of elements from the current position in the regular expression.

[0081] (7) Descendant axis conversion: The descendant axis (such as ..) means matching not only direct child elements, but also all descendant elements. It is converted into a join operation, followed by a "star operation" that matches all tags, and then another join operation to indicate that it can match deeper nested elements.

[0082] Furthermore, the JSONPath expression is rewritten according to a preset conversion method. The characters in the JSONPath expression are traversed and queried, and converted into a corresponding regular expression form according to a preset conversion method. For example: (1) Let θ=$ be a query without any fragments, then regexp(θ)=$; (2) Set is a query containing at least one fragment, then ;in: For descendant axes, for descendant axes; and is a selector The regular expression is defined as follows: (3) Tag selector: If or ,but ; (4) Star selector: If ,but ; in, ; (5) Array slice selector: If ,and ; but ; in ,and .

[0083] Specifically, converting the regular expression into a deterministic finite automaton (DFA) comprises the following steps: using the Thompson construction algorithm to compile the regular expression into a non-deterministic finite automaton, and then converting the non-deterministic finite automaton into a deterministic finite automaton by using a subset construction method.

[0084] When constructing DFA, use Identify by regular expression Describe a deterministic finite automaton (DFA) that describes all sequences of labels on the path from the root of the tree to the root of the matching subtree. The alphabet processed by the automaton is given by and placeholders Using the Thompson construction method, the regular expression is first converted into a nondeterministic finite automata (NFA), and then the subset construction method is used to convert the NFA into a DFA.

[0085] It should be pointed out that For query, DFA defines ;in: : a non-empty finite set of states; : Input alphabet; : Transfer function: ; : a starting state, ; : A collection of receiving states .

[0086] In another specific implementation, the query processing can select the best query strategy to achieve intelligent selection of the parsing path. When performing query processing, the DFA state machine performs state transitions according to each input tag when receiving the data stream until the query is completed or there is no match. For each input JSON element, a corresponding event is generated. and ,According to the input event, the DFA state machine will perform state transition and record the matching path.

[0087] In the embodiment of the present application, SST is used as the underlying data model for JSON streaming processing. Therefore, for the SST structure, query The specific implementation of SST on the input JSON stream is as follows:

[0088] ;in, : state set; Indicates the evaluation status. Indicates the generation status. Indicates an error state; : represents the input and output alphabets; , including the state set of the original automaton and the generated state set, where ; R is a set of rules that describes how to process input symbols to transition to states, including evaluation and generation of state transitions, as well as rules for the end of processing.

[0089] Furthermore, the process of performing query processing based on the deterministic finite automaton to determine the target data in the JSON message specifically includes the following steps:

[0090] Determining an initial state of the deterministic finite automaton, and sequentially reading each regular expression through the deterministic finite automaton to determine a next state of the deterministic finite automaton according to a state transition rule;

[0091] When the next state includes a target state for indicating a successful query, a corresponding matching result is returned, and after all the regular expressions are read, the target data in the JSON message is determined according to the matching result.

[0092] In the embodiment of the present application, the initialization phase is first performed: The state is used to evaluate the query. At the beginning, the stack contains the initial state of the DFA , encountering a JSON stream Event, the first conversion is based on In the initial state Label conversion, assuming Transfer to state Then it enters the evaluation phase. Whenever a new label or data node is encountered, the DFA state machine determines the next state according to the evaluation rule to perform state transition. The following is an exemplary evaluation rule: (1) ,in ,if ,but ;if ,but ;in: Indicates the state transition triggered when encountering a specific input; p: the top symbol of the stack, which represents the current state of the stack and is the object on which the operation is performed; q: the target state after the transition. When the rule is triggered, the automaton will Transfer to state q; : The result of the stack operation, p is the top element of the stack, is another symbol added to the stack; (2) .

[0093] During the evaluation phase, if no The final state is that no match is found and further evaluation is required. At this time, the state will remain status, and The next state is pushed onto the stack. Finally, the generation phase is entered to determine the final target data according to the generation rule. Once the DFA state machine reaches an accepting state, it means that the query is successful and the matching result is returned. The following is an exemplary generation rule: (1) ,in ; if ,but ;if ,but ; Here, the subscript g is a copy of the DFA state; (2) ,in ; (3) ,in .

[0094] In the generation phase, if yes The final state of , it means that a node matching the given query has been found, and the subtree of the current node needs to be output. The state will transition to status, and The next state is pushed onto the stack. Figure 3 The working principle of the overall parsing engine is shown. For more specific working processes of the above-mentioned modules, please refer to the corresponding contents disclosed above in this embodiment, which will not be repeated here.

[0095] In addition, it is understandable that the preset stream processing algorithm using SST is only an example, and other stream processing methods, such as Jsonsurfer, can also be used. The following example shows a JSON data set with a large amount of nested data under the same server resources, with a complex JSON structure containing nested objects and arrays. The query execution time based on the JMH test report is shown in Table 1:

[0096] Table 1

[0097]

[0098] It can be seen that the streaming processing method based on SST has lower latency.

[0099] Based on the above embodiment, this embodiment will specifically explain S13 in the above embodiment. In order to extract the business indicator value from the parsed data, the process of processing the target data using the predefined business processing logic to obtain processed data, and mapping the processed data with the business indicator to generate the business indicator result may include the following steps:

[0100] Assigning a corresponding variable name to each data point or data item in the target data to determine the corresponding preliminary variable value;

[0101] Using a preset script engine, the preliminary variable values ​​are input into a preset processing algorithm to obtain processed data;

[0102] The processed data is mapped to the business indicator to generate a business indicator result.

[0103] In the embodiment of the present application, the target data is further processed using business logic processing. The specific processing steps can be summarized into the following stages:

[0104] (1) Obtain and assign data variables: First, after the data parsing engine processes the target data, obtain the processed initial data, which may correspond to the original information extracted from the external system or database. Each data point or data item will be assigned a corresponding variable name (such as v1, v2, v3, etc.), which will serve as the basis for subsequent data processing and calculation.

[0105] (2) Use MVEL scripts for data processing: After obtaining preliminary variable values, use the MVEL (an expression language) script engine to input these variable values ​​into the pre-set processing algorithm. The role of the MVEL script is to further process and transform the data according to specific business rules and calculation logic. This step can include data filtering, merging, conversion, mapping and other operations to ensure that the data meets business needs and goals before final output.

[0106] (3) Business indicator mapping and result generation: Map the processed data with business indicators. By associating the data with specific businesses, the system generates the final business indicator results. Business indicator mapping can be adjusted according to different needs, standards and algorithms, thereby providing accurate and reliable data support for decision-making.

[0107] like Figure 4 The figure shows an exemplary business indicator mapping diagram. The data variables obtained after parsing are processed through a series of algorithms and converted into business indicators. These algorithms include maximum value, minimum value, sum, number of times, merged set, time difference, distance difference, etc., and provide custom methods to process complex fields for indicator logic processing expansion.

[0108] Correspondingly, the present application also discloses a data processing device, see Figure 5 As shown, the device comprises:

[0109] The data conversion module 11 is used to obtain multi-protocol data messages and convert the multi-protocol data messages into JSON messages;

[0110] A data parsing module 12, configured to parse the JSON message based on a preset stream processing algorithm through JSONPath to determine target data in the JSON message;

[0111] The data processing module 13 is used to process the target data using a predefined business processing logic to obtain processed data, and map the processed data with business indicators to generate business indicator results.

[0112] Among them, for more specific working processes of the above-mentioned modules, please refer to the corresponding contents disclosed in the aforementioned embodiments, which will not be repeated here.

[0113] It can be seen that the above scheme of this embodiment includes: obtaining multi-protocol data messages, and uniformly converting the multi-protocol data messages into JSON messages; parsing the JSON messages based on a preset stream processing algorithm through JSONPath to determine the target data in the JSON message; using a predefined business processing logic to process the target data to obtain processed data, and mapping the processed data with business indicators to generate business indicator results.

[0114] The beneficial technical effects of the present application are as follows: traditional JSON data traversal and conditional statement methods are cumbersome, inefficient and difficult to modify. The present invention accesses and converts JSON to data of different protocols, uses JSONPath as the core to process JSON data, and uniformly parses data in JSON format based on a preset stream processing algorithm without the need for hard-coded parsing at each level. JSONPath provides a more concise, flexible and efficient way to query, locate and extract content from JSON data, improves development efficiency for data processing, reduces learning costs, and improves code readability. In addition, the parsed target data is mapped to business indicators to associate data with specific businesses. Business indicator mapping can be adjusted according to different requirements, standards and algorithms to provide accurate and reliable data support for decision-making.

[0115] Furthermore, the present application also discloses an electronic device. Figure 6 This is a structural diagram of an electronic device 20 according to an exemplary embodiment, and the content in the diagram cannot be considered as any limitation on the scope of use of the present application.

[0116] Figure 6A schematic diagram of the structure of an electronic device 20 provided in an embodiment of the present application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 is used to store a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the data processing method disclosed in any of the aforementioned embodiments. In addition, the electronic device 20 in this embodiment may specifically be a computer.

[0117] In this embodiment, the power supply 23 is used to provide working voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and the external device, and the communication protocol it follows is any communication protocol that can be applied to the technical solution of the present application, and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0118] In addition, the memory 22 as a carrier for resource storage may be a read-only memory, a random access memory, a disk or an optical disk, etc. The resources stored thereon may include an operating system 221, a computer program 222 and data 223, etc. The data 223 may include various data. The storage method may be temporary storage or permanent storage.

[0119] The operating system 221 is used to manage and control the hardware devices and computer programs 222 on the electronic device 20, and may be Windows Server, Netware, Unix, Linux, etc. In addition to computer programs that can be used to complete the data processing method performed by the electronic device 20 disclosed in any of the aforementioned embodiments, the computer program 222 may further include computer programs that can be used to complete other specific tasks.

[0120] Furthermore, the embodiment of the present application also discloses a computer-readable storage medium, wherein the computer-readable storage medium mentioned here includes a random access memory (Random Access Memory, RAM), a memory, a read-only memory (Read-Only Memory, ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a magnetic disk or an optical disk, or any other form of storage medium known in the technical field. Wherein, the computer program implements the aforementioned data processing method when executed by the processor. For the specific steps of the method, reference can be made to the corresponding contents disclosed in the aforementioned embodiments, which will not be repeated here.

[0121] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0122] The steps of the data processing method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0123] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0124] The data processing method, device, equipment and storage medium provided by the present invention are introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A data processing method, characterized in that: include: Obtain multi-protocol data messages, and uniformly convert the multi-protocol data messages into JSON messages; Parsing the JSON message based on a preset stream processing algorithm through JSONPath to determine target data in the JSON message; The target data is processed using a predefined business processing logic to obtain processed data, and the processed data is mapped to business indicators to generate business indicator results.

2. The data processing method according to claim 1, characterized in that: The obtaining of multi-protocol data messages and converting the multi-protocol data messages into JSON messages includes: Acquire a multi-protocol data message, and perform functional formatting verification on the multi-protocol data message; When the multi-protocol data message is JSON data, the JSON data is directly stored in a JSON string to generate a JSON message; When the multi-protocol data message is extensible markup language data, the extensible markup language data is converted into the JSON data, and then the JSON data is stored in a JSON string to generate a JSON message.

3. The data processing method according to claim 1, characterized in that: The parsing of the JSON message based on a preset stream processing algorithm by JSONPath to determine target data in the JSON message includes: Determine the data path of the JSON message; Write a JSONPath expression based on the data path; The JSONPath expression is query compiled based on a preset stream processing algorithm to generate a deterministic finite automaton, and query processing is performed based on the deterministic finite automaton to determine target data in the JSON message.

4. The data processing method according to claim 3, characterized in that: The query compilation of the JSONPath expression based on a preset stream processing algorithm to generate a deterministic finite automaton includes: The JSONPath expression is converted into a regular expression, and the regular expression is converted into a deterministic finite automaton.

5. The data processing method according to claim 4, characterized in that: The converting the JSONPath expression into a regular expression comprises: Define a preset conversion method, and perform query matching on the JSONPath expression based on the preset conversion method, so as to convert the JSONPath expression into a regular expression according to the query matching result; wherein the preset conversion method includes conversion methods of alphabet definition, placeholder processing, label selector conversion, array slice selector conversion, selector conversion, child axis conversion, and descendant axis conversion.

6. The data processing method according to claim 4, characterized in that: The converting the regular expression into a deterministic finite automaton comprises: The regular expression is compiled into a non-deterministic finite automaton using the Thompson construction algorithm, and then the non-deterministic finite automaton is converted into a deterministic finite automaton using a subset construction method; Accordingly, query processing is performed based on the deterministic finite automaton to determine the target data in the JSON message, including: Determining an initial state of the deterministic finite automaton, and sequentially reading each regular expression through the deterministic finite automaton to determine a next state of the deterministic finite automaton according to a state transition rule; When the next state includes a target state for indicating a successful query, a corresponding matching result is returned, and after all the regular expressions are read, the target data in the JSON message is determined according to the matching result.

7. The data processing method according to any one of claims 1 to 6, characterized in that: The processing of the target data by using the predefined business processing logic to obtain processed data, and mapping the processed data with the business indicator to generate a business indicator result, includes: Assigning a corresponding variable name to each data point or data item in the target data to determine the corresponding preliminary variable value; Using a preset script engine, the preliminary variable values ​​are input into a preset processing algorithm to obtain processed data; The processed data is mapped to the business indicator to generate a business indicator result.

8. A data processing device, characterized in that: include: A data conversion module, used to obtain multi-protocol data messages and uniformly convert the multi-protocol data messages into JSON messages; A data parsing module, used to parse the JSON message based on a preset stream processing algorithm through JSONPath to determine target data in the JSON message; The data processing module is used to process the target data using a predefined business processing logic to obtain processed data, and map the processed data with business indicators to generate business indicator results.

9. An electronic device, characterized in that: The electronic device comprises a processor and a memory; wherein the memory is used to store a computer program, and the computer program is loaded and executed by the processor to implement the data processing method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: Used to store a computer program; wherein when the computer program is executed by a processor, the data processing method according to any one of claims 1 to 7 is implemented.

Images