A data processing method and device, electronic equipment and storage medium

Abstract

The embodiment of the application is suitable for the technical field of computers, and provides a data processing method and device, electronic equipment and a storage medium, wherein the data processing method comprises the following steps: determining at least one frequent item set based on hierarchical data of each of at least two Json data; each of the at least one frequent item set corresponds to one of at least two levels corresponding to the at least two Json data; the frequent item set represents a value set of a field that appears simultaneously in the corresponding level; the hierarchical data represents a field and a value of each level of the corresponding Json data; determining a dependency relationship between each field in each of the at least one frequent item set; generating a Json template based on the determined dependency relationship, and verifying to-be-verified Json data based on the Json template.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a data processing method, apparatus, electronic device, and storage medium. Background Technology

[0002] JavaScript Object Notation (JSON) is a lightweight data interchange format. JSON templates can be used to validate JSON data. However, the JSON templates generated by related technologies only contain information such as data types. Before using a JSON template, users need to manually determine the dependencies between fields in the JSON data and fill these dependencies into the JSON template. If the user is not proficient, the determined dependencies may be inaccurate, hindering the validation of JSON data using the template. Summary of the Invention

[0003] To address the aforementioned problems, embodiments of the present invention provide a data processing method, apparatus, electronic device, and storage medium to at least resolve the issue that JSON templates generated by related technologies are not conducive to users validating JSON data.

[0004] The technical solution of this invention is implemented as follows:

[0005] In a first aspect, embodiments of the present invention provide a data processing method, the method comprising:

[0006] Based on the hierarchical data of each of the at least two JSON data sets, at least one frequent itemset is determined; each frequent itemset in the at least one frequent itemset corresponds to one of the at least two hierarchies corresponding to the at least two JSON data sets; the frequent itemset represents the set of values ​​of a field that appears simultaneously in the corresponding hierarchy; the hierarchical data represents the fields and values ​​of each level of the corresponding JSON data.

[0007] Determine the dependencies between fields in each frequent itemset of the at least one frequent itemset;

[0008] Generate JSON templates based on the determined dependencies;

[0009] The JSON data to be validated is validated based on the JSON template.

[0010] In the above scheme, generating a JSON template based on the determined dependencies includes:

[0011] Write the dependency relationship into the setting field of the JSON template.

[0012] In the above scheme, determining at least one frequent itemset based on the hierarchical data of each of at least two JSON data sets includes:

[0013] Determine at least two candidate sets for each of the at least two levels;

[0014] Calculate the support of each candidate set in the at least two candidate sets; the support represents the frequency of the corresponding candidate set appearing in the at least two JSON datasets.

[0015] Based on the support, determine the frequent itemsets of each of the at least two levels.

[0016] In the above scheme, determining the dependencies between fields in each frequent itemset of the at least one frequent itemset includes:

[0017] Calculate the information entropy gain for each field in each frequent itemset of the at least one frequent itemset;

[0018] The dependency relationship is determined based on the information entropy gain.

[0019] In the above scheme, calculating the information entropy gain corresponding to each field in each frequent itemset of the at least one frequent itemset includes:

[0020] Calculate the information entropy of each frequent itemset of the at least one frequent itemset;

[0021] Calculate the conditional entropy for each field in each frequent itemset of the at least one frequent itemset;

[0022] The information entropy gain corresponding to each field in the corresponding frequent itemset is determined based on the information entropy and the conditional entropy.

[0023] In the above scheme, determining the dependency relationship based on the information entropy gain includes:

[0024] The field with the largest information entropy gain in the frequent itemset is determined as the dependent party in the dependency relationship.

[0025] In the above scheme, before determining at least one frequent itemset, the method further includes:

[0026] By flattening each of the at least two JSON data sets, hierarchical data for each of the at least two JSON data sets is obtained.

[0027] In a second aspect, embodiments of the present invention provide a data processing apparatus, the apparatus comprising:

[0028] The first determining module is used to determine at least one frequent itemset based on the hierarchical data of each of the at least two JSON data sets; each frequent itemset in the at least one frequent itemset corresponds to one of the at least two hierarchies corresponding to the at least two JSON data sets; the frequent itemset represents the set of values ​​of a field that appears simultaneously in the corresponding hierarchy; the hierarchical data represents the fields and values ​​of each level of the corresponding JSON data.

[0029] The second determining module is used to determine the dependencies between fields in each frequent itemset of the at least one frequent itemset;

[0030] The generation module is used to generate JSON templates based on the determined dependencies;

[0031] The verification module is used to verify the JSON data to be verified based on the JSON template.

[0032] Thirdly, embodiments of the present invention provide an electronic device including a processor and a memory interconnected thereto. The memory is used to store a computer program, the computer program including program instructions, and the processor is configured to invoke the program instructions to execute the steps of the data processing method provided in the first aspect of the present invention.

[0033] Fourthly, embodiments of the present invention provide a computer-readable storage medium, comprising: the computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the steps of the data processing method provided in the first aspect of the present invention.

[0034] This invention, through embodiments of at least two JSON datasets, determines at least one frequent itemset, identifies the dependencies between fields within each frequent itemset, generates a JSON template based on the determined dependencies, and then validates the JSON data to be validated using the JSON template. Each frequent itemset in the at least one frequent itemset corresponds to one of at least two levels corresponding to at least two JSON datasets. The frequent itemset represents the set of values ​​for fields that appear simultaneously in the corresponding level, and the level data represents the fields and values ​​of each level of the corresponding JSON dataset. This invention determines the dependencies between fields using frequent itemsets and generates a JSON template based on these dependencies. Compared to related technologies that require users to manually determine and write dependencies, this invention eliminates the need for manual determination and writing of dependencies, and the determined dependencies are more accurate. Therefore, when using the JSON template to validate the JSON data to be validated, the accuracy of the validation results can be improved. Attached Figure Description

[0035] Figure 1 This is a schematic diagram illustrating the implementation flow of a data processing method provided in an embodiment of the present invention;

[0036] Figure 2 This is a schematic diagram illustrating the implementation flow of another data processing method provided in an embodiment of the present invention;

[0037] Figure 3 This is a schematic diagram illustrating the implementation flow of another data processing method provided in an embodiment of the present invention;

[0038] Figure 4 This is a schematic diagram illustrating the implementation flow of another data processing method provided in an embodiment of the present invention;

[0039] Figure 5 This is a flowchart of generating a JSON schema provided in an application embodiment of the present invention;

[0040] Figure 6 This is a schematic diagram of a data processing device provided in an embodiment of the present invention;

[0041] Figure 7 This is a schematic diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0042] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0043] Data validity verification is a crucial protection mechanism for internet and IT businesses. It protects data security and maintains server security. The lack of data validity verification can lead to numerous security problems such as command injection and business downtime. Therefore, data validity verification is particularly important in secure production.

[0044] JSON templates can be used to validate JSON data. For example, JsonSchema is a type of JSON template. This embodiment of the invention will use JsonSchema as an example for illustration.

[0045] The industry typically uses JSON templates for input data validation; therefore, generating high-quality JSON templates is a crucial step in data validation. Related technologies utilize JSON template generation tools to parse the read JSON data, obtain different data types, map these data types to the data types in the JSON template, and finally obtain the JSON template.

[0046] The JSON templates generated by related technologies are very crude. Before using a JSON template to validate JSON data, users need to fill in a lot of additional information related to the JSON data to be validated, such as the dependencies between fields, naming rules, and data size. For users unfamiliar with this data, determining the dependencies and writing them into the JSON template is a challenge. Moreover, manually determined dependencies may be inaccurate, leading to inaccurate JSON templates and inaccurate validation results when users use these templates to validate JSON data.

[0047] To address the shortcomings of the aforementioned related technologies, embodiments of the present invention provide a data processing method that can at least improve the accuracy of validation results for JSON data using JSON templates. To illustrate the technical solution described in this invention, specific embodiments are provided below.

[0048] Figure 1 This is a schematic diagram illustrating the implementation flow of a data processing method provided in an embodiment of the present invention. The execution subject of the data processing method is an electronic device, including desktop computers, laptops, and servers, etc. (Reference) Figure 1 Data processing methods include:

[0049] S101, based on the hierarchical data of each of the at least two JSON data sets, determine at least one frequent itemset; each frequent itemset in the at least one frequent itemset corresponds to one of the at least two hierarchies corresponding to the at least two JSON data sets; the frequent itemset represents the set of values ​​of fields that appear simultaneously in the corresponding hierarchies; the hierarchical data represents the fields and values ​​of each level of the corresponding JSON data set.

[0050] Here, at least two JSON data are the basis for generating the JSON template in this embodiment of the invention. Each JSON data has a hierarchical structure. For example, one of the JSON data is JSON: {"opr":"list","module":" / root / ","filter":{"start":"ssss","cf":"test","success":"unknown"}}. This JSON data has two levels, namely ["opr", "module", "filter"] and ["start", "cf", "success"].

[0051] At least two JSON data points correspond to at least two levels. The number of levels in the at least two JSON data points is the number of levels in the JSON data point with the most levels.

[0052] The frequent itemsets corresponding to each level are determined. A frequent itemset refers to a set of items that appear frequently in the dataset, and an itemset is a collection of several items. In this embodiment of the invention, for at least two JSON datasets, at least one frequent itemset can be mined at each level. In this embodiment of the invention, a frequent itemset is a field that appears simultaneously in a level and the set of values ​​for that field. For example, if the first level of one JSON dataset is ["opr", "module", "filter"], and the first level of another JSON dataset is ["opr", "module"], then the frequent itemset of the first level is ["opr", "module"].

[0053] In one embodiment, before determining at least one frequent itemset, the method further includes:

[0054] By flattening each of the at least two JSON data sets, hierarchical data for each of the at least two JSON data sets is obtained.

[0055] Because JSON data has a nested, layered structure, some JSON data can have very deep hierarchies. To obtain detailed information about each level of the JSON data, it is flattened to expose the deeper layers. Flattening allows for the extraction of detailed information about each level of the JSON data, including the fields within each level and their corresponding values.

[0056] refer to Figure 2 In one embodiment, determining at least one frequent itemset based on the hierarchical data of each of at least two JSON data sets includes:

[0057] S201, determine at least two candidate sets for each of the at least two levels.

[0058] Candidate itemsets are used to obtain frequent itemsets. Itemets in the candidate itemset that meet the support condition are retained, while those that do not meet the support condition are discarded.

[0059] Currently, there are many mature algorithms for frequent itemset mining. This embodiment of the invention uses the Apriori algorithm to mine frequent itemsets. The Apriori algorithm is a frequent itemset mining algorithm used to extract frequent itemsets from association rules, that is, to mine combinations that are highly likely to appear together in all items. Association rules represent the probability of inferring another frequent itemset given a certain frequent itemset. Based on prior knowledge of the properties of frequent itemsets, the Apriori algorithm uses an iterative method of bottom-up search, generating candidate 1-itemsets to candidate K-itemsets sequentially from bottom to top.

[0060] The Apriori algorithm uses two prior principles: 1. If an itemset is not a frequent itemset, then its superset must also be a frequent itemset; 2. If an itemset is a frequent itemset, then its subsets are also frequent itemsets. Using these two prior principles can greatly reduce the number of candidate itemsets.

[0061] S202, calculate the support of each candidate set in the at least two candidate sets; the support represents the frequency of the corresponding candidate set appearing in the at least two JSON data.

[0062] Support refers to the frequency with which a set appears in all transactions. In this embodiment of the invention, support refers to the frequency or number of times a candidate set appears in at least two JSON data sets.

[0063] S203, determine the frequent itemsets of each of the at least two levels based on the support.

[0064] Here, candidate itemsets with support less than the minimum support (minsup) can be deleted, and the remaining candidate itemsets can be identified as frequent itemsets. In practice, usually only one frequent itemset is retained, which can be the candidate itemset with the highest support.

[0065] S102, determine the dependencies between fields in each frequent itemset of the at least one frequent itemset.

[0066] Here, dependency means that the values ​​of each field in a frequent itemset are dependent on each other. For example, a frequent itemset includes field A and field B, where the value of field A depends on the value of field B. Then there is a dependency relationship between field A and field B, with field A being the dependent party and field B being the dependent party. The value of field A depends on the value of field B.

[0067] The dependency relationship here can refer only to the magnitude of the values, such as the value of field A depending on the value of field B, meaning the value of field B is greater than the value of field A. The dependency relationship can also refer to the values ​​of different fields following a functional relationship, such as the value of field B being a multiple of the value of field A.

[0068] Assuming the value of B depends on the value of A, the principle behind calculating dependency relationship using information entropy gain is that reclassifying the dataset according to the value of A will result in a lower information entropy, meaning a higher information entropy gain. Therefore, by dividing the dataset according to A and B respectively and calculating the information entropy gain, the partitioning scheme with the largest gain represents the dependency relationship.

[0069] refer to Figure 3 In one embodiment, determining the dependencies between fields in each frequent itemset of the at least one frequent itemset includes:

[0070] S301, calculate the information entropy gain corresponding to each field in each frequent itemset of the at least one frequent itemset.

[0071] Information entropy refers to the probability of the occurrence of a specific piece of information (the probability of the occurrence of a discrete random event). Information entropy is essentially uncertainty; the greater the entropy, the greater the uncertainty.

[0072] Conditional entropy refers to the uncertainty of a random variable under a given condition.

[0073] Information entropy gain equals information entropy minus conditional entropy. Information entropy gain represents the degree to which information uncertainty is reduced under a given condition.

[0074] refer to Figure 4 In one embodiment, calculating the information entropy gain corresponding to each field in each frequent itemset of the at least one frequent itemset includes:

[0075] S401, calculate the information entropy of each frequent itemset of the at least one frequent itemset.

[0076] Calculate the information entropy of frequent itemsets using the following formula:

[0077]

[0078] Where D is the complete dataset, i.e., all fields and their corresponding values ​​in the frequent itemset, and P...k This represents the proportion of each combination value in D.

[0079] For example, a frequent itemset includes fields A, B, and C, where A corresponds to the value 'a', B corresponds to the value 'b', and C corresponds to the value 'c'. Suppose the value combination (a1, b1, c1) appears 3 times in the frequent itemset, and the frequent itemset contains a total of 17 values. Then, the P of this value combination... k It is 3 / 17.

[0080] S402, calculate the conditional entropy corresponding to each field in each frequent itemset of the at least one frequent itemset.

[0081] For each frequent itemset, select a field and use the values ​​of this field to partition D. V For each of the divided subsets, the information entropy of each subset is calculated and weighted to obtain the conditional entropy.

[0082] For example, suppose a frequent itemset includes fields A, B, and C. The combinations of values ​​for this frequent itemset are: (a1, b1, c1) has 3 sets, (a1, b2, c1) has 10 sets, and (a2, b1, c2) has 4 sets.

[0083] If D is one of the 17 possible combinations of values ​​mentioned above, then the information entropy is:

[0084]

[0085] Calculate the conditional entropy when A, B, and C are the first nodes respectively, and select the node with the largest gain as the root node.

[0086] For example, the conditional entropy corresponding to field B is:

[0087]

[0088] Similarly, the conditional entropy corresponding to fields A and C can be calculated.

[0089] S403, determine the information entropy gain corresponding to each field in the corresponding frequent itemset based on the information entropy and the conditional entropy.

[0090] Using the above example, we continue to calculate the information entropy gain corresponding to field B. The information entropy gain corresponding to field B is: Ent(D) - Ent(Y|B); the information entropy gain corresponding to field A is: Ent(D) - Ent(Y|A).

[0091] By analogy, the information entropy gain corresponding to each field can be obtained.

[0092] S302, determine the dependency relationship based on the information entropy gain.

[0093] Based on the above example, the node with the largest information entropy gain is selected as the splitting node. Assuming field B is selected, then A and C are considered to depend on B. The dependency relationship between A and C can also be calculated using the same method. In practical applications, it is usually unnecessary to mine overly complex dependencies, as overly complex dependencies are not necessarily accurate.

[0094] In one embodiment, determining the dependency based on the information entropy gain includes:

[0095] The field with the largest information entropy gain in the frequent itemset is determined as the dependent party in the dependency relationship.

[0096] For example, in the above example, the information entropy gain corresponding to field B is the largest, so field B is determined as the dependent, and A and C depend on B.

[0097] S103, Generate JSON templates based on the determined dependencies.

[0098] For example, the JSON template is JsonSchema, which is used to describe JSON data and can be used to validate JSON data.

[0099] In one embodiment, generating a JSON template based on the determined dependencies includes:

[0100] Write the dependency relationship into the setting field of the JSON template.

[0101] Here, the field can be set to a "description" field. The "description" field is used to describe the field information as a prompt for the user, making it easier for the user to verify the JSON data.

[0102] The process of generating a JSON template can refer to relevant technologies. For example, different types of data can be mapped to the data types of the JSON template to obtain the JSON template.

[0103] For each level, multiple dependencies can be obtained. These dependencies can be written into a JSON template. When users are validating JSON data, they can use the JSON template to validate the JSON data.

[0104] In practical applications, multiple JSON schemas can be generated based on at least two JSON data sets. Each JSON schema can be compared, and identical JSON schemas can be merged.

[0105] S104, Validate the JSON data to be validated based on the JSON template.

[0106] When validating JSON data using a JSON template, the following steps are included:

[0107] Step 1: Obtain the fields and corresponding values ​​of the JSON data to be validated.

[0108] Step 2: Obtain the dependencies in the JSON template.

[0109] Step one and step two can be performed simultaneously, or either step one can be performed first.

[0110] Step 3: Determine whether each field and its corresponding value in the JSON data to be validated conforms to the above dependency relationships. If they conform, the JSON data to be validated passes validation. If they do not conform, the JSON data to be validated fails validation.

[0111] For example, suppose we have a batch of HTTP request data, placed according to different URLs, with multiple request data under each URL, and each request data is in JSON format. Generate a unique JSONSchema that can match each URL.

[0112] First, obtain the field information and values ​​for each level. Then, use a frequent itemset mining algorithm to mine frequent itemsets at each level, calculate the conditional entropy of the field values ​​corresponding to each frequent itemset, calculate the maximum information entropy gain, and classify these as field dependencies. Generate the JSONSchema body and write the dependencies into the JSONSchema.

[0113] Assume the code for generating the JSON schema is as follows:

[0114] "title":"This is a schema that matches body of url:..cgibinaccess_saas.cgi.please pay much more attention to the following key properties:{'opr':0.9036402569593148,'module':0.9271948608137045,'cf':0.9400428265524625}, values ​​mean support degree.the possible dependency relation is'opr'(3.2)<-['module'(1.8),'cf'(0.6)],values ​​mean information gain."

[0115] The frequent itemset is {'opr':0.9036402569593148, 'module':0.9271948608137045, 'cf':0.9400428265524625}. The dependency is 'opr'(3.2)<-['module'(1.8), 'cf'(0.6)], which indicates that the values ​​of the "cf" and "module" fields depend on the value of the "opr" field. 3.2, 1.8, and 0.6 represent the order of the field values: "opr" > "module" > "cf". There is no specific limit to the exact values, as long as the order of the values ​​is satisfied.

[0116] Suppose there is a JSON data to be validated that is an HTTP request, and the body of the HTTP request is as follows:

[0117] “start”: 0,

[0118] "limit": 50,

[0119] "cf": "c64da58266dbd3a1ea960596e94515ac",

[0120] "module":" / mod-monitor / abnormal-traffic / index"

[0121] The JSON data was validated using the JSONSchema described above. Because the opr field is missing, the JSON data does not match the dependency relationship, so the validation failed.

[0122] After modifying the HTTP request based on the verification results, the modified HTTP request body data segment is as follows:

[0123] “opr”: 2,

[0124] “start”: 0,

[0125] "limit": 50,

[0126] "cf": 0,

[0127] “module”:“1”

[0128] The JSON data was validated using the JSONSchema described above. The fields “cf”, “module”, and “opr” were not missing, and the values ​​of the “cf”, “module”, and “opr” fields conformed to the above dependency relationship. Therefore, the validation passed.

[0129] By using JSON templates to validate the validity of request data, only valid request data is allowed to access the system, while invalid request data is blocked, thus enhancing data security.

[0130] This invention is applicable to various data verification scenarios, such as HTTP request verification, Web parameter verification, and backend API verification. The technical solutions of this invention can be used in these scenarios.

[0131] This invention identifies at least one frequent itemset by analyzing the hierarchical data of each of at least two JSON datasets. Dependencies between fields within each frequent itemset are determined, and a JSON template is generated based on these dependencies. The JSON template is then used to validate the JSON data to be validated. The JSON template describes the JSON data, and each frequent itemset corresponds to one of at least two hierarchies in the at least two JSON datasets. A frequent itemset represents the set of values ​​for fields that appear simultaneously in the corresponding hierarchical level, and the hierarchical data represents the fields and values ​​of each level in the corresponding JSON data. This invention identifies dependencies between fields using frequent itemsets, and the generated JSON template eliminates the need for manual dependency identification and writing compared to related technologies, thus improving the user experience. Furthermore, the identified dependencies are more precise, improving the accuracy of validation results when using the JSON template to validate the JSON data.

[0132] refer to Figure 5 , Figure 5 This is a flowchart of a JsonSchema generation process provided in an application embodiment of the present invention. The JsonSchema generation process includes:

[0133] First, the data is loaded in batches of JSON data. The batch JSON data is then preprocessed to extract the fields and values ​​for each level.

[0134] Next, key field mining is performed. For each level, frequent itemsets are calculated using a frequent itemset mining algorithm, and these frequent itemsets are used as key fields. The Apriori algorithm can be used here to mine frequent itemsets.

[0135] Then, dependency mining is performed. For each frequent itemset, the information entropy gain corresponding to each field in each frequent itemset is calculated, and the dependency is determined based on the information entropy gain.

[0136] Finally, generate a JsonSchema and write the dependencies into the "description" field of the JsonSchema.

[0137] The JSONSchema generated by the application examples of this invention can be directly used to validate JSON data, avoiding the need for users to manually input information into the JSONSchema and improving the user experience. Furthermore, the determined dependencies are more precise, which can improve the accuracy of validation results for JSON data validated using JSON templates.

[0138] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0139] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0140] It should be noted that the technical solutions described in the embodiments of the present invention can be combined arbitrarily without conflict.

[0141] In addition, in the embodiments of the present invention, "first," "second," etc. are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0142] refer to Figure 6 , Figure 6 This is a schematic diagram of a data processing device provided in an embodiment of the present invention, such as... Figure 6 As shown, the device includes: a first determining module, a second determining module, and a generating module.

[0143] The first determining module is used to determine at least one frequent itemset based on the hierarchical data of each of the at least two JSON data sets; each frequent itemset in the at least one frequent itemset corresponds to one of the at least two hierarchies corresponding to the at least two JSON data sets; the frequent itemset represents the set of values ​​of a field that appears simultaneously in the corresponding hierarchy; the hierarchical data represents the fields and values ​​of each level of the corresponding JSON data.

[0144] The second determining module is used to determine the dependencies between fields in each frequent itemset of the at least one frequent itemset;

[0145] The generation module is used to generate JSON templates based on the determined dependencies;

[0146] The verification module is used to verify the JSON data to be verified based on the JSON template.

[0147] In one embodiment, the generation module generates a JSON template based on the determined dependencies, for:

[0148] Write the dependency relationship into the setting field of the JSON template.

[0149] In one embodiment, when determining at least one frequent itemset based on the hierarchical data of each of at least two JSON data sets, the first determining module is configured to:

[0150] Determine at least two candidate sets for each of the at least two levels;

[0151] Calculate the support of each candidate set in the at least two candidate sets; the support represents the frequency of the corresponding candidate set appearing in the at least two JSON datasets.

[0152] Based on the support, determine the frequent itemsets of each of the at least two levels.

[0153] In one embodiment, when determining the dependencies between fields in each frequent itemset of the at least one frequent itemset, the second determining module is configured to:

[0154] Calculate the information entropy gain for each field in each frequent itemset of the at least one frequent itemset;

[0155] The dependency relationship is determined based on the information entropy gain.

[0156] In one embodiment, when calculating the information entropy gain corresponding to each field in each frequent itemset of the at least one frequent itemset, the second determining module is used to:

[0157] Calculate the information entropy of each frequent itemset of the at least one frequent itemset;

[0158] Calculate the conditional entropy for each field in each frequent itemset of the at least one frequent itemset;

[0159] The information entropy gain corresponding to each field in the corresponding frequent itemset is determined based on the information entropy and the conditional entropy.

[0160] In one embodiment, when determining the dependency based on the information entropy gain, the second determining module is used to:

[0161] The field with the largest information entropy gain in the frequent itemset is determined as the dependent party in the dependency relationship.

[0162] In one embodiment, the device further includes:

[0163] The processing module is used to obtain hierarchical data for each of the at least two JSON data by flattening each JSON data.

[0164] In practical applications, the first determining module, the second determining module, and the generating module can be implemented by processors in electronic devices, such as central processing units (CPUs), digital signal processors (DSPs), microcontroller units (MCUs), or field-programmable gate arrays (FPGAs).

[0165] It should be noted that the data processing apparatus provided in the above embodiments is only illustrated by the division of the above modules. In practical applications, the above processing can be assigned to different modules as needed, that is, the internal structure of the apparatus can be divided into different modules to complete all or part of the processing described above. In addition, the data processing apparatus and data processing method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0166] Based on the hardware implementation of the above program modules, and in order to implement the method of the embodiments of this application, the embodiments of this application also provide an electronic device. Figure 7 This is a schematic diagram of the hardware structure of the electronic device according to an embodiment of this application, as shown below. Figure 7 As shown, the electronic device includes:

[0167] A communication interface enables information exchange with other devices, such as network devices.

[0168] The processor, connected to the communication interface, enables information interaction with other devices and, when running a computer program, executes the methods provided by one or more technical solutions on the electronic device side. The computer program is stored in memory.

[0169] Of course, in practical applications, the various components in an electronic device are coupled together through a bus system. It can be understood that the bus system is used to achieve communication and connection between these components. In addition to the data bus, the bus system also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, in... Figure 7 The general will label all buses as bus systems.

[0170] The memory in this application embodiment is used to store various types of data to support the operation of the electronic device. Examples of such data include any computer program used to operate on the electronic device.

[0171] It is understood that memory can be volatile or non-volatile, or both. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), ferromagnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM); magnetic surface memory can be disk storage or magnetic tape storage. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Synchronous Static Random Access Memory (SSRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDRSDRAM), Enhanced Synchronous Dynamic Random Access Memory (ESDRAM), SyncLink Dynamic Random Access Memory (SLDRAM), and Direct Rambus Random Access Memory (DRRAM).The memories described in the embodiments of this application are intended to include, but are not limited to, these and any other suitable types of memories.

[0172] The methods disclosed in the embodiments of this application can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by integrated logic circuits in the processor's hardware or by instructions in software form. The processor may be a general-purpose processor, a DSP, or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. A general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the methods disclosed in the embodiments of this application can be directly manifested as execution by a hardware decoding processor, or execution by a combination of hardware and software modules in the decoding processor. The software modules may be located in a storage medium, which is located in memory. The processor reads the program from the memory and, in conjunction with its hardware, completes the steps of the aforementioned method.

[0173] Optionally, when the processor executes the program, it implements the corresponding processes implemented by the electronic device in the various methods of the embodiments of this application. For the sake of brevity, these will not be described in detail here.

[0174] In an exemplary embodiment, this application also provides a storage medium, namely a computer storage medium, specifically a computer-readable storage medium, such as a first memory storing a computer program, which can be executed by a processor of an electronic device to complete the steps described in the aforementioned method. The computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, Flash Memory, magnetic surface memory, optical disc, or CD-ROM.

[0175] In the several embodiments provided in this application, it should be understood that the disclosed apparatus, electronic devices, and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components may be combined, or integrated into another system, or some features may be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed may be through some interfaces, and the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0176] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0177] In addition, each functional unit in the various embodiments of this application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0178] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.

[0179] Alternatively, if the integrated units described above are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.

[0180] It should be noted that the technical solutions described in the embodiments of this application can be combined arbitrarily without conflict.

[0181] In addition, in this application example, "first," "second," etc. are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0182] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A data processing method, characterized in that, The method includes: Based on the hierarchical data of each of the at least two JSON data sets, at least one frequent itemset is determined; each frequent itemset in the at least one frequent itemset corresponds to one of the at least two hierarchies corresponding to the at least two JSON data sets; the frequent itemset represents the set of values ​​of a field that appears simultaneously in the corresponding hierarchy; the hierarchical data represents the fields and values ​​of each level of the corresponding JSON data. Calculate the information entropy of each frequent itemset of the at least one frequent itemset; Calculate the conditional entropy for each field in each frequent itemset of the at least one frequent itemset; The information entropy gain corresponding to each field in the corresponding frequent itemset is determined based on the information entropy and the conditional entropy. The field with the largest information entropy gain in the frequent itemset is determined as the dependent party in the dependency relationship; Generate a JSON template based on the aforementioned dependencies; The JSON data to be validated is validated based on the JSON template.

2. The method according to claim 1, characterized in that, The process of generating a JSON template based on the determined dependencies includes: Write the dependency relationship into the setting field of the JSON template.

3. The method according to claim 1, characterized in that, The determination of at least one frequent itemset based on the hierarchical data of each of at least two JSON data sets includes: Determine at least two candidate sets for each of the at least two levels; Calculate the support of each candidate set in the at least two candidate sets; the support represents the frequency of the corresponding candidate set appearing in the at least two JSON datasets. Based on the support, determine the frequent itemsets of each of the at least two levels.

4. The method according to claim 1, characterized in that, Before determining at least one frequent itemset, the method further includes: By flattening each of the at least two JSON data sets, hierarchical data for each of the at least two JSON data sets is obtained.

5. A data processing apparatus, characterized in that, include: The first determining module is used to determine at least one frequent itemset based on the hierarchical data of each of at least two JSON data sets. Each frequent itemset in the at least one frequent itemset corresponds to one of the at least two levels corresponding to the at least two JSON data; The frequent itemset represents the set of values ​​of a field that appears simultaneously in the corresponding level; The hierarchical data represents the fields and values ​​of each level of the corresponding JSON data; The second determining module is used to calculate the information entropy of each frequent itemset of the at least one frequent itemset; Calculate the conditional entropy for each field in each frequent itemset of the at least one frequent itemset; Based on the information entropy and the conditional entropy, determine the information entropy gain corresponding to each field in the frequent itemset; determine the field with the largest information entropy gain in the frequent itemset as the dependent party in the dependency relationship; The generation module is used to generate a JSON template based on the dependencies. The verification module is used to verify the JSON data to be verified based on the JSON template.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the data processing method as described in any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, the computer program including program instructions that, when executed by a processor, cause the processor to perform the data processing method as described in any one of claims 1 to 4.

Images