Key-value pair path generation method, device, electronic device and computer-readable medium

Abstract

The embodiments of the present disclosure disclose a key-value pair path generation method, device, electronic device and computer-readable medium. A specific implementation of the method includes: performing data structure verification processing on a semi-structured credit information set, and screening the data structure verification result set to obtain a target data structure verification result set; performing structural anomaly detection and semantic replacement processing on the semi-structured credit information set to obtain a semi-structured semantic credit information set; parsing the semi-structured semantic credit information set and the semi-structured credit information set to obtain a semi-structured credit information set after parsing; generating a key-value pair path information set; extracting a key-value pair data set; storing the key-value pair path information set, the key-value pair data set and the semi-structured credit information set in a non-relational database. This implementation can improve the quality of key-value pair data and the key-value pair path generation rate, reduce the waste of storage resources, and improve the user experience.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of computer technology, and in particular to a method, device, electronic device, and computer-readable medium for generating a key-value pair path. Background Art

[0002] The data of JSON (JavaScript Object Notation) document is a lightweight data exchange format, which can be used in the data exchange and parsing fields of various businesses. For the generation of key-value pair paths, the method usually adopted is: recursively parse the semi-structured credit information set to obtain the key-value pair data location information set. Then, the path of the key-value pair data location information set is constructed to obtain the key-value pair path. Finally, the key-value pair data in the semi-structured credit information set is extracted and stored through the key-value pair path.

[0003] However, it is found in practice that when the above method is used to generate key-value pair paths, the following technical problems often occur: Since semi-structured data may contain complex nested structures and abnormal data, recursive parsing is to parse the semi-structured credit information set in memory, which requires a large amount of memory resources and generates a large number of erroneous and redundant key-value pair paths. The key-value pair data obtained contains a large amount of redundant and erroneous data, resulting in a large system load, a waste of storage resources, and a prolonged time for key-value pair path generation, resulting in a low user experience.

[0004] In the process of adopting technical solutions to solve the above-mentioned technical problem 1, the following technical problem 2 is often accompanied: there are a large number of credit information with the same semantics but different expressions in the semi-structured credit information set, which causes the repeated storage of a large amount of redundant information, resulting in a waste of storage resources. For the above-mentioned technical problem 2, the conventional solution is generally: determine the cosine similarity of the key-value pair keywords in the semi-structured credit information as a keyword similarity value set. Then, the key-value pair keyword set corresponding to at least one keyword similarity greater than or equal to the preset similarity threshold is filtered out from the keyword similarity value set for deduplication and storage. However, the above-mentioned conventional solution still has the following problem: determining the semantic redundancy between key-value pair keywords only by the cosine similarity of the keywords is prone to missed detection and false detection of keyword semantic redundancy, and there is a large amount of redundant data in the semi-structured credit information set, resulting in a waste of storage resources.

[0005] In the process of adopting technical solutions to solve the above-mentioned technical problem one, the following technical problem three is often accompanied: how to accurately determine the extraction path of the key-value pair data included in the semi-structured credit information to improve the accuracy and efficiency of extracting the key-value pair data set and shorten the extraction time. For the above-mentioned technical problem three, the conventional solution is generally: extract the semantic features in the semi-structured credit information through a natural language model to generate a key-value pair extraction path. However, the above-mentioned conventional solution still has the following problems: the key-value pair extraction path is generated through a natural language model. Since the natural language model only extracts the semantic information of the semi-structured credit information, there is a lack of a nested hierarchical relationship between key-value pairs, and there are a large number of key-value pairs with different extraction positions but the same semantics in the semi-structured credit information data set, resulting in low accuracy of the generated key-value pair path information, long extraction time, low user experience and waste of storage resources.

[0006] The above information disclosed in this Background section is only for enhancement of understanding of the background of the present disclosure concept and therefore it may contain information that does not form the prior art that is already known in this country to a person of ordinary skill in the art. Summary of the invention

[0007] The content of this disclosure is used to introduce concepts in a brief form, which will be described in detail in the detailed implementation section below. The content of this disclosure is not intended to identify the key features or essential features of the technical solution claimed for protection, nor is it intended to limit the scope of the technical solution claimed for protection.

[0008] Some embodiments of the present disclosure propose a key-value pair path generation method, device, electronic device and computer-readable medium to solve one or more of the technical problems mentioned in the above background technology section.

[0009] In a first aspect, some embodiments of the present disclosure provide a key-value pair path generation method, comprising: obtaining a semi-structured credit information set; obtaining a semi-structured credit information set, wherein the semi-structured credit information in the semi-structured credit information set is user credit status information and enterprise credit status information stored in JSON format; performing data structure verification processing on each semi-structured credit information in the semi-structured credit information set to generate a data structure verification result and obtain a data structure verification result set; screening out at least one data structure verification result representing a verification failure from the data structure verification result set as a target data structure verification result set; performing structural anomaly detection on the semi-structured credit information set corresponding to the target data structure verification result set to obtain a structural anomaly credit information set; Each structural abnormal credit information in the abnormal credit information set is semantically replaced to generate semi-structured semantic credit information and obtain a semi-structured semantic credit information set; the above-mentioned semi-structured semantic credit information set and the semi-structured credit information set corresponding to at least one data structure verification result that has passed the representation verification are parsed to obtain a parsed semi-structured semantic credit information set and a parsed semi-structured credit information set as the parsed semi-structured credit information set; based on the above-mentioned parsed semi-structured credit information set, a key-value pair path information set is generated; based on the above-mentioned key-value pair path information set, a key-value pair data set of the key-value pair group set included in the above-mentioned parsed semi-structured credit information set is extracted; the above-mentioned key-value pair path information set, the above-mentioned key-value pair data set and the above-mentioned semi-structured credit information set are stored in a non-relational database.

[0010] In a second aspect, some embodiments of the present disclosure provide a key-value pair path generation device, including: an acquisition unit, configured to acquire a semi-structured credit information set, wherein the semi-structured credit information in the above-mentioned semi-structured credit information set is user credit status information and enterprise credit status information stored in JSON format; a data structure verification unit, configured to perform data structure verification processing on each semi-structured credit information in the above-mentioned semi-structured credit information set to generate a data structure verification result and obtain a data structure verification result set; a screening unit, configured to screen out at least one data structure verification result representing a verification failure from the above-mentioned data structure verification result set as a target data structure verification result set; a structural anomaly detection unit, configured to perform structural anomaly detection on the semi-structured credit information set corresponding to the above-mentioned target data structure verification result set to obtain a structural anomaly credit information set; a semantic replacement unit, configured to perform structural anomaly detection on the above-mentioned semi-structured credit information set corresponding to the above-mentioned target data structure verification result set to obtain a structural anomaly credit information set; and a semantic replacement unit, configured to perform structural anomaly detection on the above-mentioned semi-structured credit information set corresponding to the above-mentioned target data structure verification result set. Each structural abnormality credit information in the structural abnormality credit information set is semantically replaced to generate semi-structured semantic credit information and obtain a semi-structured semantic credit information set; a parsing unit is configured to parse the above-mentioned semi-structured semantic credit information set and the semi-structured credit information set corresponding to at least one data structure verification result that has passed the representation verification to obtain a parsed semi-structured semantic credit information set and a parsed semi-structured credit information set as a parsed semi-structured credit information set; a generating unit is configured to generate a key-value pair path information set based on the above-mentioned parsed semi-structured credit information set; an extracting unit is configured to extract a key-value pair data set of the key-value pair group set included in the above-mentioned parsed semi-structured credit information set based on the above-mentioned key-value pair path information set; a storage unit is configured to store the above-mentioned key-value pair path information set, the above-mentioned key-value pair data set and the above-mentioned semi-structured credit information set in a non-relational database.

[0011] In a third aspect, some embodiments of the present disclosure provide an electronic device comprising: one or more processors; a storage device on which one or more programs are stored, and when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation manner in the first aspect.

[0012] In a fourth aspect, some embodiments of the present disclosure provide a computer-readable medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the method described in any implementation manner in the first aspect is implemented.

[0013] The above-mentioned embodiments of the present disclosure have the following beneficial effects: the key-value pair path generation method of some embodiments of the present disclosure can improve the quality of key-value pair data and the key-value pair path generation rate, reduce the waste of storage resources, and improve the user experience. Specifically, the reason why the load of the relevant system is large, the storage resources are wasted, the time of key-value pair path generation is extended, and the user experience is low is that: due to the complex nested structure, diverse structure types and abnormal data contained in the semi-structured data, and the recursive parsing is to parse the semi-structured credit information set in the memory, a large amount of memory resources are required, a large number of erroneous and redundant key-value pair paths are generated, and the key-value pair data obtained has a large amount of redundant and erroneous data, resulting in a large load on the system, a waste of storage resources, and an extension of the time of key-value pair path generation, and the user experience is low. Based on this, the image segmentation method of some embodiments of the present disclosure can first obtain a semi-structured credit information set, wherein the semi-structured credit information in the semi-structured credit information set is user credit status information and enterprise credit status information stored in JSON format. Here, the semi-structured credit information set is used for subsequent data structure verification. Secondly, a data structure verification process is performed on each semi-structured credit information in the above-mentioned semi-structured credit information set to generate a data structure verification result and obtain a data structure verification result set. Here, a large amount of abnormal data that does not conform to the data structure can be removed through data structure verification processing, reducing the amount of computational data and reducing the system load. Again, at least one data structure verification result that does not pass the characterization verification is screened out from the above-mentioned data structure verification result set as the target data structure verification result set. Here, the target data structure verification result set is used for subsequent structural and semantic anomaly detection. Next, a structural anomaly detection is performed on the semi-structured credit information set corresponding to the above-mentioned target data structure verification result set to obtain a structural anomaly credit information set. Here, the semi-structured credit information set with structural anomalies can be removed, improving the data quality of the semi-structured credit information set and reducing the data volume. Subsequently, a semantic replacement process is performed on each structural anomaly credit information in the above-mentioned structural anomaly credit information set to generate semi-structured semantic credit information and obtain a semi-structured semantic credit information set. Here, the semi-structured credit information set is subjected to anomaly replacement detection from coarse to fine granularity, which can improve the data quality of the semi-structured credit information and reduce the amount of computational data. Afterwards, the semi-structured semantic credit information set and the semi-structured credit information set corresponding to at least one data structure verification result that has passed the characterization verification are parsed to obtain the parsed semi-structured semantic credit information set and the parsed semi-structured credit information set as the parsed semi-structured credit information set. Here, the system parsing load can be reduced, and it is easy to understand the data structure of the semi-structured credit information, which is convenient for the subsequent generation of key-value pair path information. Then, based on the above-mentioned parsed semi-structured credit information set, a key-value pair path information set is generated.Here, the key-value pair path information can be quickly and automatically generated to improve the accuracy of the generated key-value pair path information. Then, according to the above key-value pair path information set, the key-value pair data set of the key-value pair group set included in the above-mentioned semi-structured credit information set after analysis is extracted. Here, the rate of extracting key-value pair data and the quality of key-value pair data can be improved. Finally, the above key-value pair path information set, the above key-value pair data set and the above semi-structured credit information set are stored in a non-relational database. Here, due to the high accuracy of the key-value pair path information and key-value pair data, the waste of storage resources can be reduced, and it is convenient to subsequently increase the rate of extracting data and reduce the extraction time, thereby improving the user experience. It can be obtained that the key-value pair path generation method can reduce the workload of manually extracting the key-value pair path, improve the key-value pair path generation rate and the quality of the key-value pair data, reduce the waste of storage resources, and improve the user experience by performing anomaly detection on the semi-structured credit information set, removing a large amount of redundant error data, and automatically and quickly generating an accurate key-value pair path through data analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that components and elements are not necessarily drawn to scale.

[0015] Figure 1 is a flow chart of some embodiments of a key-value pair path generation method according to the present disclosure;

[0016] Figure 2 is a schematic diagram of the structure of some embodiments of the key-value pair path generation device according to the present disclosure;

[0017] Figure 3 It is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0018] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0019] It should also be noted that, for ease of description, only the parts related to the invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of the present disclosure can be combined with each other.

[0020] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0021] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0022] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0023] The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0024] Figure 1 The process 100 of some embodiments of the key-value pair path generation method according to the present disclosure is shown. The key-value pair path generation method comprises the following steps:

[0025] Step 101, obtaining a semi-structured credit information set.

[0026] In some embodiments, the execution subject (e.g., electronic device) of the key-value pair path generation method can obtain a semi-structured credit information set from multiple data sources through an API (Application Programming Interface) interface through a wired connection or a wireless connection. Among them, the semi-structured credit information in the semi-structured credit information set can be enterprise credit status information and user credit status information stored in the form of a JSON data structure. The enterprise credit status information can be information related to the state and state changes of the enterprise's credit information. The enterprise credit status information can include but is not limited to at least one of the following: basic enterprise information, enterprise financial information, and enterprise project operation information. The user credit status information can be information related to the credit status and credit status changes of the user's credit information. The user credit status information can include but is not limited to at least one of the following: basic user information, user credit query information, user credit information, and user credit evaluation information. The user credit evaluation information can represent whether the user has returned the loan normally and abided by laws and regulations on the agreed date. The data source can be a storage server for obtaining the semi-structured credit information set. For example, the multiple data sources can include but are not limited to at least one of the following: government departments and national credit reporting agencies.

[0027] Step 102: Perform data structure verification processing on each semi-structured credit information in the semi-structured credit information set to generate a data structure verification result and obtain a data structure verification result set.

[0028] In some embodiments, the above-mentioned execution entity may perform data structure verification processing on each semi-structured credit information in the above-mentioned semi-structured credit information set to generate a data structure verification result and obtain a data structure verification result set. Among them, the above-mentioned data structure verification result may include: the data structure of the semi-structured credit information meets the requirements of the constraint information of the semi-structured template, and the data structure of the semi-structured credit information does not meet the requirements of the constraint information of the semi-structured template. The above-mentioned semi-structured template may be a specification document that describes JSON data and defines the constraints such as the data structure, type, and value range of JSON data. In practice, the above-mentioned execution entity may use a verification tool to perform data structure verification processing on each semi-structured credit information in the above-mentioned semi-structured credit information set to generate a data structure verification result and obtain a data structure verification result set. Among them, the above-mentioned verification tool may be a JSON-Schema tool.

[0029] In some optional implementations of some embodiments, the above-mentioned performing data structure verification processing on each semi-structured credit information in the above-mentioned semi-structured credit information set to generate a data structure verification result and obtain a data structure verification result set may include the following steps:

[0030] The first step is to classify the above-mentioned semi-structured credit information set to obtain a category credit information set. The category credit information in the above-mentioned category credit information set may be information describing the credit data of an enterprise or an individual from different aspects. The above-mentioned different aspects may include but are not limited to at least one of the following: basic information, guarantee information, and loan information.

[0031] The second step is to obtain a semi-structured credit information template corresponding to each category of credit information in the above category credit information set to obtain a semi-structured credit information template set. The above semi-structured credit information template may be a binding document defining the semi-structured credit information.

[0032] The third step is to generate a semi-structured description information set based on the above-mentioned semi-structured credit information template set. The semi-structured description information in the above-mentioned semi-structured description information set may be information constraining the data type of the semi-structured credit information, defining the attributes of the semi-structured credit information including the values ​​corresponding to the keys, and the key keywords that are indispensable in the semi-structured credit information. The above-mentioned key keywords may be the words corresponding to the keys in the above-mentioned semi-structured credit information template.

[0033] As an example, the above-mentioned execution entity can use the json-schema generator to generate a semi-structured description information set according to the above-mentioned semi-structured credit information template set.

[0034] The fourth step is to perform data structure verification processing on each semi-structured credit information in the semi-structured credit information set according to the semi-structured description information set to generate a data structure verification result and obtain a data structure verification result set.

[0035] As an example, the execution entity may perform key-value pair field matching processing on each semi-structured credit information in the semi-structured credit information set and the semi-structured description information in the semi-structured description information set to obtain a matching result set as a data structure verification result set.

[0036] Step 103: Filter out at least one data structure verification result representing a verification failure from the data structure verification result set as a target data structure verification result set.

[0037] In some embodiments, the execution subject may filter out at least one data structure verification result representing a verification failure from the data structure verification result set as the target data structure verification result set.

[0038] Step 104, performing structural anomaly detection on the semi-structured credit information set corresponding to the target data structure verification result set to obtain a structural anomaly credit information set.

[0039] In some embodiments, the execution entity may perform structural anomaly detection on the semi-structured credit information set corresponding to the target data structure verification result set to obtain a structural anomaly credit information set. The structural anomaly credit information in the structural anomaly credit information set may be credit information that does not conform to the data types of the key and value in the key-value pair defined by the semi-structured template. The data type of the key may be a string type. The data type of the value may be one of the following: string type, numerical type, array type, Boolean type, object type, null value. The object type may be an unordered set of key-value pairs, wherein the key-value pairs in the key-value pair set may also be strings, numerical values, objects, arrays, Boolean values, and null values.

[0040] In some optional implementations of some embodiments, the above-mentioned performing structural anomaly detection on the semi-structured credit information set corresponding to the above-mentioned target data structure verification result set to obtain the structural anomaly credit information set may include the following steps:

[0041] In the first step, for each semi-structured credit information in the semi-structured credit information set corresponding to the target data structure verification structure set, the following generation steps are performed:

[0042] Sub-step 1, determining the data nesting type information of the semi-structured credit information to obtain the data nesting type information. The data nesting type information may be information for describing whether the key-value pair data in the semi-structured credit information includes multiple levels of key-value pair data.

[0043] Sub-step 2, in response to determining that the data nesting type information is the first data nesting type information, the semi-structured credit information is flattened for the first time to obtain dimensional credit information. The first data nesting type information may be type information in which there is no key-value pair nested key-value pair. The dimensional credit information may be information in the form of a two-dimensional list. The two-dimensional list may be a list with the key of the semi-structured credit information as the horizontal coordinate and the value of the semi-structured credit information as the vertical coordinate.

[0044] Sub-step 3, in response to determining that the data nesting type information is the second data nesting type information, the semi-structured credit information is subjected to a second flattening process to obtain hierarchical credit information. The second data nesting type information may be type information including at least one layer of nesting. The hierarchical credit information may be credit information of a nested hierarchical relationship of key-value pairs of semi-structured credit information stored in the form of a hash table (HashMap). The second flattening process may be a flattening process performed using a hash table storage method.

[0045] Sub-step 4: perform key-value keyword traversal extraction processing on the above-mentioned dimensional credit information or hierarchical credit information to obtain a key-value keyword group. The key-value keyword in the above-mentioned key-value keyword group may be a keyword corresponding to a key in the above-mentioned semi-structured credit information.

[0046] Sub-step 5: determining the semi-structured credit information template corresponding to the semi-structured credit information. The determination may be to select an information template with the same data type as the semi-structured credit information from the semi-structured credit information template set.

[0047] Sub-step 6, performing key-value keyword traversal extraction processing on the semi-structured credit information template to obtain a template keyword group, wherein the template keyword in the template keyword group may be a keyword corresponding to a key in the semi-structured credit information template.

[0048] Sub-step 7, generating structural abnormality credit information based on the key-value keyword group and the template keyword group, wherein the structural abnormality credit information may be abnormal credit information composed of keywords that cannot be successfully matched between the key-value keyword group and the template keyword group.

[0049] As an example, the execution subject may match the key-value keyword group and the template keyword group to obtain a matching result set. Then, at least one key-value keyword in the matching result set that represents an unsuccessful match is determined as structurally abnormal credit information.

[0050] In some optional implementations of some embodiments, generating structure abnormality credit information according to the key-value keyword group and the template keyword group may include the following steps:

[0051] The first step is to sort the key value keyword group and the template keyword group to obtain a key value keyword sequence and a template keyword sequence. The keyword sorting can be sorting by the first letter of the pinyin if the key value keyword is a Chinese word, or by the first letter of the English word if the key value keyword is an English word.

[0052] In the second step, each template keyword in the template keyword sequence is matched with at least one key-value keyword in the key-value keyword sequence corresponding to the template keyword to generate a keyword matching result group, thereby obtaining a keyword matching result group set. The at least one key-value keyword corresponding to the template keyword may be at least one key-value keyword with the same first letter and the same nesting level.

[0053] In the third step, at least one keyword matching result indicating a successful match is selected from the above keyword matching result set as a target keyword matching result set.

[0054] The fourth step is to determine the key-value attribute field of the key-value keyword and the template attribute field of the template keyword corresponding to each target keyword matching result in the target keyword matching result group according to the dimensional credit information or hierarchical credit information and the semi-structured credit information template, and obtain a key-value attribute field group and a template attribute field group. Among them, the key-value attribute field in the key-value attribute field group can be the field corresponding to the value in the semi-structured credit information. The template attribute field in the template attribute field group can be the field corresponding to the value in the semi-structured credit information template.

[0055] As an example, the execution subject may first determine the key value position information of the key value keyword corresponding to each target keyword matching result in the target keyword matching result group in the dimensional credit information or hierarchical credit information, and the template position information of the template keyword in the semi-structured credit information template, and obtain the key value position information group and the template position information group. Then, using the key value position information group and the template position information group, extract the key value attribute field group and the template attribute field group from the dimensional credit information or hierarchical credit information and the semi-structured credit information template.

[0056] Step 5: Perform attribute type verification processing on each template attribute field in the template attribute field group and the key value attribute field corresponding to the template attribute field in the key value attribute field group to obtain an attribute type verification result group. The attribute type verification processing may be a verification to determine whether the attribute type of the key value attribute field is consistent with the attribute type specified by the template attribute field.

[0057] Step 6: Determine the keyword abnormal type information of each key-value verification keyword in the key-value verification keyword group and each key-value matching keyword in the key-value matching keyword group, and obtain the keyword abnormal type information group, wherein the above-mentioned key-value verification keyword group may be the key-value keyword group corresponding to at least one attribute type verification result group in the above-mentioned attribute type verification result group that represents an unsuccessful verification. The above-mentioned key-value matching keyword group may be the key-value keyword group corresponding to at least one keyword matching result in the above-mentioned keyword matching result group that represents an unsuccessful matching. The above-mentioned keyword abnormal type information may include: key-value keyword redundancy abnormal type, key-value keyword missing abnormal type and attribute type abnormal type. The above-mentioned key-value keyword redundancy abnormal type may be a type in which the key-value keyword in the above-mentioned keyword matching result group exists in the key-value keyword group but cannot be matched in the template keyword group. The above-mentioned key-value keyword missing abnormal type may be a type in which the template keyword in the above-mentioned keyword matching group exists in the template keyword but cannot be matched in the key-value keyword group. The above-mentioned attribute type abnormal type may be a type in which the above-mentioned keyword matching result represents a successful match, but the attribute type of the value attribute field corresponding to the key-value keyword representing the successful match is different from the attribute type of the value attribute field corresponding to the template keyword.

[0058] In the seventh step, based on the key-value verification keyword group and the key-value matching keyword group, each keyword anomaly type information in the keyword anomaly type information group is embedded into the semi-structured credit information to obtain structured anomaly credit information.

[0059] As an example, the execution subject may first determine the keyword position information of the key-value verification keyword group and the key-value matching keyword group in the semi-structured credit information to obtain a keyword position information group. Then, determine the keyword anomaly type information of the keyword anomaly type information group corresponding to each keyword position information in the keyword position information group to obtain a position anomaly type information group. Finally, embed the position anomaly type information group into the position group corresponding to the keyword position information group in the semi-structured credit information to obtain the structural anomaly credit information.

[0060] Step 105, performing semantic replacement processing on each structurally abnormal credit information in the structurally abnormal credit information set to generate semi-structured semantic credit information, thereby obtaining a semi-structured semantic credit information set.

[0061] In some embodiments, the execution subject may perform semantic replacement processing on each structural abnormality credit information in the structural abnormality credit information set to generate semi-structured semantic credit information and obtain a semi-structured semantic credit information set. The semi-structured semantic credit information may be credit information obtained by replacing redundant abnormality types of key-value keywords in the structural abnormality credit information. The semantic replacement may be a replacement of abnormal keywords with key-value keywords in a semi-structured template.

[0062] Considering the problems of the above conventional solutions, facing the above technical problem 2: determining the semantic redundancy between key-value pairs of keywords only through the cosine similarity of keywords is prone to missed detection and false detection of keyword semantic redundancy, and there is a large amount of redundant data in the semi-structured credit information set, resulting in a waste of storage resources. Combining the advantages / technical status, we can decide to adopt the following solution.

[0063] In some optional implementations of some embodiments, the semantic replacement processing of each structurally abnormal credit information in the structurally abnormal credit information set to generate semi-structured semantic credit information to obtain the semi-structured semantic credit information set may include the following steps:

[0064] In the first step, according to the keyword abnormality type information, the key value keyword is extracted from the above structure abnormality credit information set to obtain a key value redundant keyword group set. Among them, the key value redundant keyword in the above key value redundant keyword group set can be the key value keyword corresponding to the keyword abnormality type information of the keyword redundant abnormal type.

[0065] As an example, the execution subject may first extract at least one keyword abnormality type information whose keyword abnormality type information is a keyword redundant abnormality type from the structure abnormality credit information set. Then, a key value keyword group is extracted from each keyword abnormality type information in the at least one keyword abnormality type information as a key value redundant keyword group to obtain a key value redundant keyword group set.

[0066] The second step is to embed the above-mentioned key-value redundant keyword group set and the corresponding key-value redundant attribute field group set, and the obtained template keyword group set and the corresponding template attribute field group set to obtain a key-value redundant word embedding feature vector group set and a template word embedding feature vector group set. Among them, the key-value redundant word embedding feature vector in the above-mentioned key-value redundant word embedding feature vector group set can represent the semantic information of the key-value redundant keyword. The template word embedding feature vector in the above-mentioned template word embedding feature vector group set can represent the semantic information of the template keyword.

[0067] As an example, the execution subject may first classify the key-value redundant keyword group set and the obtained template keyword group set to obtain a first category key-value keyword set, a second category key-value keyword set, a first category template keyword set and a second category template keyword set. Among them, the first category key-value keyword set and the first category template keyword may be Chinese keywords. The second category key-value keyword set and the second category template keyword may be English keywords. Secondly, the first category key-value keyword set and the first category template keyword set are input into the first keyword embedding model to obtain a first key-value feature vector set and a first template feature vector set. Among them, the first keyword embedding model may be a model that represents the input Chinese keywords by word embedding. For example, the first keyword embedding model may be FastText. Then, the second category key-value keyword set and the second category template keyword set are input into the second keyword embedding model to obtain a second key-value feature vector set and a second template feature vector set. Among them, the second keyword embedding model may be a model that represents the input English keywords by word embedding. For example, the second keyword embedding model may be Word2Vec. Finally, the first key-value feature vector set and the second key-value feature vector set are determined as a key-value redundant word embedding feature vector set, and the first template feature vector set and the second template feature vector set are determined as a template word embedding feature vector set.

[0068] The third step is to perform feature mapping processing on the above key-value redundant word embedding feature vector set and the above template word embedding feature vector set to obtain a key-value redundant mapping feature vector set and a template mapping feature vector set. The feature mapping processing can use a linear regression algorithm to map the key-value redundant word embedding feature vector set and the above template word embedding feature vector set of different dimensions to the same space.

[0069] In the fourth step, for each template mapping feature vector in the template mapping feature vector set and each key value redundant mapping feature vector in the key value redundant mapping feature vector set, the following keyword similarity determination step is performed:

[0070] Sub-step 1: Determine the semantic distance value between the template mapping feature vector and the key-value redundant mapping feature vector, wherein the semantic distance value may represent the Euclidean distance between the template mapping feature vector and the key-value redundant mapping feature vector.

[0071] As an example, the execution entity may first determine the dimensional feature mean of each dimensional feature in the key-value redundant mapping feature vector. Secondly, determine the arithmetic square root of the cumulative sum of the squares of the difference between each dimensional feature in the key-value redundant mapping feature vector and the dimensional feature mean as the dimensional feature standard deviation. Then, determine the arithmetic square root of the cumulative sum of the squares of the ratio of the difference between each dimensional feature in the template mapping feature vector and each dimensional feature of the key-value redundant mapping feature vector and the dimensional feature standard deviation as the semantic distance value.

[0072] Sub-step 2: determine the arithmetic square root of the feature dimension value of the above template mapping feature vector as the dimension value.

[0073] Sub-step 3, determining the difference between the above dimension value and the above semantic distance value as a normalized index value.

[0074] Sub-step 4: input the normalization index value into a preset normalization function to obtain a normalization function value. The preset normalization function may be a Sigmoid normalization function.

[0075] Sub-step 5, determining the difference between the preset threshold and the normalized function value as the first keyword similarity value between the template mapping feature vector and the key value redundancy mapping feature vector. The preset threshold may be a pre-set value. The preset threshold may be 1.

[0076] Sub-step 6: determining the cosine distance between the template mapping feature vector and the key value redundancy mapping feature vector as the second keyword similarity value.

[0077] Sub-step 7: determining the Jaccard coefficient between the template mapping feature vector and the key-value redundancy mapping feature vector as the third keyword similarity value.

[0078] Sub-step 8: determining an average of the first keyword similarity value, the second keyword similarity value, and the third keyword similarity value as the keyword similarity value.

[0079] Sub-step 9, determining the template semantic path of the above-mentioned template mapping feature vector and the key-value semantic path of the above-mentioned key-value redundant mapping feature vector. Among them, the above-mentioned template semantic path can be the path of the template keyword corresponding to the template mapping feature vector in the semi-structured credit information template from the beginning of the semi-structured credit information template to the location of the template keyword. The above-mentioned key-value semantic path can be the path of the key-value keyword corresponding to the key-value redundant mapping feature vector from the beginning of the structural abnormal credit information to the location of the key-value keyword. In practice, the above-mentioned execution entity can firstly perform graph structure components on the structural abnormal credit information corresponding to the above-mentioned key-value redundant mapping feature vector and the semi-structured credit information template corresponding to the above-mentioned template mapping feature vector to obtain credit information graph data and template credit information graph data. Then, keyword traversal is performed on the above-mentioned credit information graph data and the above-mentioned template credit information graph data to obtain the template semantic path and the key-value semantic path.

[0080] Sub-step 10, determining the semantic path similarity between the template semantic path and the key-value semantic path, and obtaining a semantic path similarity value. The semantic path similarity value can represent the similarity between the template semantic path and the key-value semantic path in terms of path length, keyword nodes passed by the path, and weight coefficients of edges passed by the path.

[0081] Sub-step 11, performs weighted summation processing on the above-mentioned keyword similarity value and the above-mentioned semantic path similarity value to obtain a semantic similarity value. Among them, the above-mentioned semantic similarity value can be a similarity value obtained by combining different weights of keyword similarity and semantic path similarity. In practice, the above-mentioned execution subject can first use a heuristic algorithm to determine the key weight value and the semantic weight value of the above-mentioned keyword similarity value and the above-mentioned semantic path similarity value. Among them, the above-mentioned heuristic hybrid algorithm can be but is not limited to one of the following: particle swarm algorithm, ant colony algorithm, simulated annealing algorithm. Then, the sum of the product of the above-mentioned keyword similarity value and the key weight value and the product of the above-mentioned semantic path similarity value and the above-mentioned semantic weight value is determined as the semantic similarity value.

[0082] The fifth step is to screen each semantic similarity value group in the obtained semantic similarity value group set as the target semantic similarity value, wherein the semantic similarity value group is a similarity value group of the key-value redundant mapping feature vector and each template mapping feature vector in the template mapping feature vector set. The screening may be to select the keyword similarity value with the largest value. The target keyword similarity value may be the similarity value with the largest value in the keyword similarity value group.

[0083] The sixth step is to replace the key-value redundant keyword set corresponding to the target semantic similarity value set with the template keyword set to obtain a semi-structured semantic credit information set, and to deduplicate the above semi-structured semantic credit information set to obtain a deduplicated semi-structured semantic credit information set, and to store the deduplicated semi-structured semantic credit information set.

[0084] The above-mentioned first to sixth steps and their related contents serve as an inventive point of an embodiment of the present disclosure, which solves the second technical problem mentioned in the background technology: "Determining the semantic redundancy between key-value pairs of keywords only through the cosine similarity of keywords is prone to missed detection and false detection of keyword semantic redundancy. There is a large amount of redundant data in the semi-structured credit information set, resulting in a waste of storage resources." The factors that lead to the waste of storage resources are often as follows: Determining the semantic redundancy between key-value pairs of keywords only through the cosine similarity of keywords is prone to missed detection and false detection of keyword semantic redundancy. There is a large amount of redundant data in the semi-structured credit information set, resulting in a waste of storage resources. If the above-mentioned factors are solved, the effect of reducing the waste of storage resources can be achieved. In order to achieve this effect, the present disclosure first extracts keywords from the structural abnormal credit information set according to the keyword abnormality type information to obtain a key-value redundant keyword group set. The keyword abnormality type can further reduce the replacement range of semantic replacement, reduce unnecessary semantic replacement, and improve the efficiency of semantic replacement. Secondly, embedding and vectorizing the key-value redundant attribute field group set and the template attribute field group set can improve the semantic richness of keywords and reduce the sparsity of keyword data, and can improve the accuracy of the subsequent feature cosine similarity calculation. Then, the keyword similarity is determined by the average of the first keyword similarity, the second keyword similarity, and the third keyword similarity, which can improve the accuracy of keyword similarity. And by combining path similarity and keyword similarity to determine the similarity between semi-structured credit information and semi-structured credit information templates, the accuracy of keyword similarity can be further improved, and the omissions and false detections determined by keyword semantic similarity can be reduced. Finally, by semantic similarity and preset similarity thresholds, structurally abnormal credit information is semantically replaced, deduplicated, and stored, which can improve the accuracy of semantic replacement and reduce the storage resources for storing semi-structured semantic credit information sets.

[0085] Step 106, parse the semi-structured semantic credit information set and the semi-structured credit information set corresponding to at least one data structure verification result that has passed the representation verification, and obtain a parsed semi-structured semantic credit information set and a parsed semi-structured credit information set as the parsed semi-structured credit information set.

[0086] In some embodiments, the execution subject may parse the semi-structured semantic credit information set and the semi-structured credit information set corresponding to at least one data structure verification result that has passed the characterization verification, and obtain a parsed semi-structured semantic credit information set and a parsed semi-structured credit information set as a parsed semi-structured credit information set. The parsed semi-structured credit information in the parsed semi-structured credit information set may be credit information that displays the relationship between key-value pairs in JSON data in the form of a dictionary. The keys in the dictionary correspond to the keys in the semi-structured semantic credit information. The values ​​in the dictionary correspond to the values ​​in the semi-structured semantic credit information.

[0087] As an example, the execution subject can use the JSON module that comes with Python to parse the semi-structured semantic credit information set and the semi-structured credit information set corresponding to at least one data structure verification result that has passed the representation verification, and obtain the parsed semi-structured semantic credit information set and the parsed semi-structured credit information set as the parsed semi-structured credit information set. The JSON module that comes with Python can be a function that serializes and deserializes JSON. For example, the function can be json.dumps().

[0088] Step 107: Generate a key-value pair path information set based on the parsed semi-structured credit information set.

[0089] In some embodiments, the execution subject may generate a key-value pair path information set according to the semi-structured credit information set after parsing, wherein the key-value pair path information in the key-value pair path information set may be the location information of the description value attribute field in the semi-structured credit information after parsing.

[0090] As an example, the execution subject may perform a traversal loop on the semi-structured credit information set after the analysis to obtain a key-value pair path information set.

[0091] In some optional implementations of some embodiments, generating a key-value pair path information set according to the semi-structured credit information set after the above analysis may include the following steps:

[0092] In the first step, each parsed semi-structured credit information in the above-mentioned parsed semi-structured credit information set is converted into a graph structure to generate semi-structured graph data and obtain a semi-structured graph data set. The above-mentioned semi-structured graph data may be a graph representing the semantic information and topological information of the parsed semi-structured credit information. The nodes in the above-mentioned semi-structured graph data may be key-value keywords. The edges in the above-mentioned semi-structured graph data may be edges that describe the hierarchical relationship between key-value keywords.

[0093] As an example, the execution entity may use a dictionary to perform graph structured conversion on each parsed semi-structured credit information in the parsed semi-structured credit information set to generate semi-structured graph data and obtain a semi-structured graph data set.

[0094] The second step is to perform semantic coding on the semi-structured graph data set to obtain a graph semantic coding vector set. The graph semantic coding vector in the graph semantic coding vector set may be a vector representing the semantic information of the node. The semantic coding process may be one-hot coding.

[0095] The third step is to perform structural coding on the semi-structured graph data set to obtain a graph structure coding vector set. The graph structure coding vectors in the graph structure coding vector set can represent the topological information of the semi-structured graph data. The structural coding process can be a structural coding process performed using a graph neural network.

[0096] The fourth step is to determine the adjacent node group of each semi-structured graph node in the semi-structured graph node set included in each semi-structured graph data in the semi-structured graph data set to obtain each adjacent node group set. The adjacent nodes in each adjacent node group set may be semi-structured graph nodes that can be obtained by the semi-structured graph node through a node path. The node path may be a path consisting of 1, 2 or 3 nodes.

[0097] The fifth step is to embed each adjacent node group in the above adjacent node group sets to obtain an adjacent node embedding feature vector group set. The adjacent node embedding feature vector in the above adjacent node embedding feature vector set can be a feature vector obtained by embedding the semantic information and topological information corresponding to the above adjacent node group into the semi-structured graph node. In practice, the above execution subject can use a graph embedding algorithm to embed each adjacent node group in the above adjacent node group sets to obtain an adjacent node embedding feature vector group set.

[0098] The sixth step is to perform feature fusion processing on the graph semantic encoding vector set, the graph structure encoding vector set and the adjacent node latent feature vector set to obtain a node feature vector set.

[0099] In the seventh step, the above node feature vector set is input into the attention mechanism layer to obtain the node weight feature vector set.

[0100] The eighth step is to generate a key-value pair path information set based on the above node weight feature vector set, and to extract the key-value pair data set in the above semi-structured credit information set according to the above key-value pair path information set, and store it.

[0101] As an example, the execution entity may input the node weight feature vector set into a neural network model based on a self-attention mechanism to obtain a key-value pair path information set.

[0102] The above-mentioned first to eighth steps and their related contents, as an inventive point of an embodiment of the present disclosure, solve the technical problem three mentioned in the background technology, "generating a key-value pair extraction path through a natural language model, because the natural language model only extracts the semantic information of semi-structured credit information, lacks a nested hierarchical relationship between key-value pairs, and there are a large number of key-value pair data with different extraction positions but the same semantics in the semi-structured credit information data set, resulting in low accuracy of the generated key-value pair path information, long extraction time, low user experience and waste of storage resources". The factors that lead to low accuracy of the generated key-value pair path information, long extraction time, low user experience and waste of storage resources are often as follows: generating a key-value pair extraction path through a natural language model, because the natural language model only extracts the semantic information of semi-structured credit information, lacks a nested hierarchical relationship between key-value pairs, and there are a large number of key-value pair data with different extraction positions but the same semantics in the semi-structured credit information data set. If the above factors are solved, the accuracy of the generated key-value pair path information can be improved, the extraction time can be shortened, the user experience can be improved, and the waste of storage resources can be reduced. In order to achieve this effect, the present disclosure firstly performs a graph structured conversion on the semi-structured credit information after parsing, taking into account the nested relationship of the key-value pair data in the semi-structured credit information. Secondly, the semi-structured graph data set is semantically encoded and structurally encoded, and the semantic information and structural information of the key-value pair data of the semi-structured data after parsing are considered at the same time, which can improve the accuracy of the key-value pair path generated subsequently. Then, the adjacent nodes of each node included in the semi-structured graph data are determined and embedded, and the nested hierarchical relationship of the key-value pair data can be further extracted by considering the association relationship between the node and other nodes. Finally, according to the above-mentioned node weight feature vector set, a key-value pair path information set is generated, and according to the above-mentioned key-value pair path information set, the key-value pair data set in the above-mentioned semi-structured credit information set is extracted and stored, which can improve the accuracy of the generated key-value pair path information, shorten the extraction time, improve the user experience and reduce the waste of storage resources.

[0103] Step 8: Generate a key-value pair path information set based on the above node weight feature vector set.

[0104] In some optional implementations of some embodiments, generating a key-value pair path information set according to the semi-structured credit information set after the above analysis may include the following steps:

[0105] The first step is to input the above-mentioned parsed semi-structured credit information set into a semi-structured graph model to obtain a set of semi-structured node feature vectors. The above-mentioned semi-structured graph model can be a model that converts the input parsed semi-structured credit information set into a graph data structure and extracts the feature vectors of the nodes in the graph data structure. For example, the above-mentioned semi-structured graph model can be a model of a structure conversion convolutional neural network layer and a heterogeneous information network. The semi-structured node feature vectors in the above-mentioned semi-structured node feature vector set can represent the semantic information of the key data in the semi-structured credit information.

[0106] The second step is to input the semi-structured node feature vector set into the self-attention mechanism layer to obtain a node weight feature vector set. The node weight feature vector in the node weight feature vector set may be a vector that assigns different weight values ​​to each semi-structured node feature vector in the semi-structured node feature vector set.

[0107] The first step is to determine the key-value graph path group corresponding to each node weight feature vector in the above node weight feature vector set to obtain a key-value graph path group set. The key-value graph path in the above key-value graph path group set can be a node path formed by the semi-structured graph node corresponding to the node weight feature vector passing through the semi-structured graph node other than the semi-structured graph node.

[0108] In the second step, for each key-value graph path group in the key-value graph path group set, perform the following determination steps:

[0109] Sub-step 1, determining the path length of each key-value graph path in the key-value graph path group to obtain a path length group, wherein the path length in the path length group may be the number of semi-structured graph nodes included in the key-value graph path.

[0110] Sub-step 2, determining the data type group of each node included in each key-value graph path in the key-value graph path group. The data type in the data type group can be the data type of the value corresponding to the key-value keyword. The data type can be one of the following: string type, numeric type, array type, Boolean type, object type, null value.

[0111] Sub-step 3, determining the number of array indexes of each key-value graph path in the key-value graph path group as the number of array indexes, and obtaining an array index number group. The array indexes can represent the front-to-back order between the semi-structured graph nodes in the key-value graph path.

[0112] Sub-step 4, determining the data nesting depth of each key-value pair graph path in the key-value pair graph path group, and obtaining a data nesting depth value group. The data nesting depth value in the data nesting depth value group may be the nesting level of the semi-structured credit information where the semi-structured graph node corresponding to the key-value pair graph path is located.

[0113] Sub-step 5, performing path complexity evaluation based on the path length group, the data type group, the data index quantity group and the data nesting depth value group to obtain a path complexity value group. The path complexity evaluation class can characterize the complexity of the key-value pair graph path.

[0114] As an example, the execution subject may input the path length group, the data type group, the data index quantity group and the data nesting depth value group into a path evaluation model to obtain a path complexity value group. The path evaluation model may be a convolutional neural network model that assigns different weights to the input path length group, data type group, data index quantity group and data nesting depth value.

[0115] Sub-step 6, filtering the above path complexity value group to obtain a filtered path complexity value group. The filtered path complexity values ​​in the above filtered path complexity value group may be path complexity values ​​greater than or equal to a preset complexity threshold. The above preset complexity threshold may be a preset maximum threshold for determining path complexity. For example, the above preset complexity threshold may be 0.7.

[0116] Sub-step 7, determining the key-value pair graph path group corresponding to the above-screened path complexity value group as the key-value pair path group.

[0117] The third step is to determine the obtained key-value pair path group set as the key-value pair path information set.

[0118] Step 108: extracting a key-value pair data set of a key-value pair group set included in the semi-structured credit information set after parsing according to the key-value pair path information set.

[0119] In some embodiments, the execution subject may extract a key-value pair data set of a key-value pair group set included in the semi-structured credit information set after the analysis according to the key-value pair path information set. The key-value pair data in the key-value pair data set may be data for extracting data to be extracted from the semi-structured credit information in the form of a dictionary. The key in the dictionary form may be a value attribute field in the semi-structured credit information. The value in the dictionary form may be the key-value pair path information of the value attribute field.

[0120] As an example, the execution entity may utilize a recursive loop function to extract the key-value pair data set of the key-value pair group set included in the semi-structured credit information set according to the key-value pair path information set.

[0121] Step 109: store the key-value pair path information set, the key-value pair data set and the semi-structured credit information set in a non-relational database.

[0122] In some embodiments, the execution entity may store the key-value pair path information set, the key-value pair data set, and the semi-structured credit information set in a non-relational database, wherein the non-relational database may be a NoSQL database.

[0123] The above-mentioned embodiments of the present disclosure have the following beneficial effects: the key-value pair path generation method of some embodiments of the present disclosure can improve the quality of key-value pair data and the key-value pair path generation rate, reduce the waste of storage resources, and improve the user experience. Specifically, the reason why the load of the relevant system is large, the storage resources are wasted, the time of key-value pair path generation is extended, and the user experience is low is that: due to the complex nested structure, diverse structure types and abnormal data contained in the semi-structured data, and the recursive parsing is to parse the semi-structured credit information set in the memory, a large amount of memory resources are required, a large number of erroneous and redundant key-value pair paths are generated, and the key-value pair data obtained has a large amount of redundant and erroneous data, resulting in a large load on the system, a waste of storage resources, and an extension of the time of key-value pair path generation, and thus the user experience is low. Based on this, the image segmentation method of some embodiments of the present disclosure can first obtain a semi-structured credit information set. Here, the semi-structured credit information set is used for subsequent data structure verification. Secondly, each semi-structured credit information in the above-mentioned semi-structured credit information set is subjected to data structure verification processing to generate a data structure verification result, and a data structure verification result set is obtained. Here, a large amount of abnormal data that does not conform to the data structure can be removed through data structure verification processing, reducing the amount of computing data and reducing the system load. Again, at least one data structure verification result that characterizes the failure of the verification is screened out from the above-mentioned data structure verification result set as the target data structure verification result set. Here, the target data structure verification result set is used for subsequent structural and semantic anomaly detection. Next, a structural anomaly detection is performed on the semi-structured credit information set corresponding to the above-mentioned target data structure verification result set to obtain a structural anomaly credit information set. Here, the semi-structured credit information set with structural anomalies can be removed to improve the data quality of the semi-structured credit information set and reduce the amount of data. Subsequently, each structural anomaly credit information in the above-mentioned structural anomaly credit information set is semantically replaced to generate semi-structured semantic credit information to obtain a semi-structured semantic credit information set. Here, the semi-structured credit information set is subjected to anomaly replacement detection from coarse to fine granularity, which can improve the data quality of the semi-structured credit information and reduce the amount of computing data. Afterwards, the above-mentioned semi-structured semantic credit information set and the semi-structured credit information set corresponding to at least one data structure verification result that has passed the representation verification are parsed to obtain the parsed semi-structured semantic credit information set and the parsed semi-structured credit information set as the parsed semi-structured credit information set. Here, the system parsing load can be reduced, and it is easy to understand the data structure of the semi-structured credit information, which is convenient for the subsequent generation of key-value pair path information. Then, based on the above-mentioned parsed semi-structured credit information set, a key-value pair path information set is generated. Here, the key-value pair path information can be quickly and automatically generated, thereby improving the accuracy of the generated key-value pair path information.Then, according to the key-value pair path information set, extract the key-value pair data set of the key-value pair group set included in the semi-structured credit information set after the above analysis. Here, the rate of extracting key-value pair data and the quality of key-value pair data can be improved. Finally, the key-value pair path information set, the key-value pair data set and the semi-structured credit information set are stored in a non-relational database. Here, due to the high accuracy of the key-value pair path information and the key-value pair data, the waste of storage resources can be reduced, and it is convenient to subsequently increase the rate of extracting data and reduce the extraction time, thereby improving the user experience. It can be obtained that the key-value pair path generation method can reduce the workload of manually extracting the key-value pair path, improve the key-value pair path generation rate and the quality of the key-value pair data, reduce the waste of storage resources, and improve the user experience by performing anomaly detection on the semi-structured credit information set, removing a large amount of redundant error data, and automatically and quickly generating an accurate key-value pair path through data analysis.

[0124] Further references Figure 2 As an implementation of the methods shown in the above figures, the present disclosure provides some embodiments of a key-value pair path generation device, and these device embodiments are similar to Figure 1 Corresponding to the method embodiments shown, the key-value pair path generation device can be specifically applied to various electronic devices.

[0125] like Figure 2As shown, a key-value pair path generation device 200 includes: an acquisition unit 201, a data structure verification unit 202, a screening unit 203, a structural anomaly detection unit 204, a semantic replacement unit 205, a parsing unit 206, a generation unit 207, an extraction unit 208 and a storage unit 209. The acquisition unit 201 is configured to: acquire a semi-structured credit information set, wherein the semi-structured credit information in the semi-structured credit information set is user credit status information and enterprise credit status information stored in JSON format. The data structure verification unit 202 is configured to: perform data structure verification processing on each semi-structured credit information in the semi-structured credit information set to generate a data structure verification result and obtain a data structure verification result set. The screening unit 203 is configured to: filter out at least one data structure verification result representing a failure of verification from the data structure verification result set as a target data structure verification result set. The structural anomaly detection unit 204 is configured to: perform structural anomaly detection on the semi-structured credit information set corresponding to the target data structure verification result set to obtain a structural anomaly credit information set. The semantic replacement unit 205 is configured to: perform semantic replacement processing on each structural abnormality credit information in the above structural abnormality credit information set to generate semi-structured semantic credit information and obtain a semi-structured semantic credit information set. The parsing unit 206 is configured to: perform parsing processing on the above semi-structured semantic credit information set and the semi-structured credit information set corresponding to at least one data structure verification result that has passed the representation verification, and obtain a parsed semi-structured semantic credit information set and a parsed semi-structured credit information set as a parsed semi-structured credit information set. The generating unit 207 is configured to: generate a key-value pair path information set according to the above parsed semi-structured credit information set. The extracting unit 208 is configured to: extract a key-value pair data set of the key-value pair group set included in the above parsed semi-structured credit information set according to the above key-value pair path information set. The storage unit 209 is configured to: store the above key-value pair path information set, the above key-value pair data set and the above semi-structured credit information set in a non-relational database.

[0126] It can be understood that the various units recorded in the key-value pair path generation device 200 and the reference Figure 1 Therefore, the operations, features and beneficial effects described above for the method are also applicable to the key-value pair path generation device 200 and the units included therein, and will not be described in detail here.

[0127] Reference below Figure 3 , which shows a structural schematic diagram of an electronic device (eg, an electronic device) 300 suitable for implementing some embodiments of the present disclosure. Figure 3The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0128] like Figure 3 As shown, the electronic device 300 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 302 or a program loaded from a storage device 308 into a random access memory (RAM) 303. In the RAM 303, various programs and data required for the operation of the electronic device 300 are also stored. The processing device 301, the ROM 302, and the RAM 303 are connected to each other via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0129] Typically, the following devices may be connected to the I / O interface 305: input devices 306 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 307 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 308 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 309. The communication devices 309 may allow the electronic device 300 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 3 The electronic device 300 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead. Figure 3 Each block shown in the figure may represent one device, or may represent multiple devices as required.

[0130] In particular, according to some embodiments of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, some embodiments of the present disclosure include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In some such embodiments, the computer program can be downloaded and installed from the network through the communication device 309, or installed from the storage device 308, or installed from the ROM 302. When the computer program is executed by the processing device 301, the above-mentioned functions defined in the method of some embodiments of the present disclosure are executed.

[0131] It should be noted that the computer-readable medium in some embodiments of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In some embodiments of the present disclosure, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In some embodiments of the present disclosure, the computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0132] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (Hyper Text Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0133] The above-mentioned computer-readable medium may be included in the above-mentioned electronic device; or it may exist independently without being installed in the electronic device. The above-mentioned computer-readable medium carries one or more programs. When the above-mentioned one or more programs are executed by the electronic device, the electronic device: obtains a semi-structured credit information set, wherein the semi-structured credit information in the above-mentioned semi-structured credit information set is user credit status information and enterprise credit status information stored in JSON format; performs data structure verification processing on each semi-structured credit information in the above-mentioned semi-structured credit information set to generate a data structure verification result and obtain a data structure verification result set; filters out at least one data structure verification result representing a verification failure from the above-mentioned data structure verification result set as the target data structure verification result set; performs structural anomaly detection on the semi-structured credit information set corresponding to the above-mentioned target data structure verification result set to obtain a structural anomaly credit information set; Each structural abnormality credit information in the structural abnormality credit information set is semantically replaced to generate semi-structured semantic credit information and obtain a semi-structured semantic credit information set; the above-mentioned semi-structured semantic credit information set and the semi-structured credit information set corresponding to at least one data structure verification result that has passed the representation verification are parsed to obtain a parsed semi-structured semantic credit information set and a parsed semi-structured credit information set as the parsed semi-structured credit information set; based on the above-mentioned parsed semi-structured credit information set, a key-value pair path information set is generated; based on the above-mentioned key-value pair path information set, a key-value pair data set of the key-value pair group set included in the above-mentioned parsed semi-structured credit information set is extracted; the above-mentioned key-value pair path information set, the above-mentioned key-value pair data set and the above-mentioned semi-structured credit information set are stored in a non-relational database.

[0134] Computer program code for performing the operations of some embodiments of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0135] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0136] The units described in some embodiments of the present disclosure may be implemented by software or by hardware. The described units may also be arranged in a processor, for example, may be described as: a processor including an acquisition unit, a data structure verification unit, a screening unit, a structural anomaly detection unit, a semantic replacement unit, a parsing unit, a generation unit, an extraction unit and a storage unit. The names of these units do not constitute a limitation on the units themselves in some cases, for example, the acquisition unit may also be described as a "unit for acquiring a semi-structured credit information set".

[0137] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0138] The above descriptions are only some preferred embodiments of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but should also cover other technical solutions formed by any combination of the above-mentioned technical features or their equivalent features without departing from the above-mentioned inventive concept. For example, the above-mentioned features are replaced with the technical features with similar functions disclosed in the embodiments of the present disclosure (but not limited to) and the technical solutions formed.

Claims

1. A key-value pair path generation method, comprising: Acquire a semi-structured credit information set, wherein the semi-structured credit information in the semi-structured credit information set is user credit status information and enterprise credit status information stored in JSON format; Performing data structure verification processing on each semi-structured credit information in the semi-structured credit information set to generate a data structure verification result, and obtaining a data structure verification result set, wherein performing data structure verification processing on each semi-structured credit information in the semi-structured credit information set to generate a data structure verification result, and obtaining a data structure verification result set, includes: Classifying the semi-structured credit information set to obtain a category credit information set; Acquire a semi-structured credit information template corresponding to each category of credit information in the category credit information set to obtain a semi-structured credit information template set; Generating a semi-structured description information set according to the semi-structured credit information template set; According to the semi-structured description information set, performing data structure verification processing on each semi-structured credit information in the semi-structured credit information set to generate a data structure verification result, thereby obtaining a data structure verification result set; Filtering out at least one data structure verification result representing a verification failure from the data structure verification result set as a target data structure verification result set; Performing structural anomaly detection on the semi-structured credit information set corresponding to the target data structure verification result set to obtain the structural anomaly credit information set, wherein performing structural anomaly detection on the semi-structured credit information set corresponding to the target data structure verification result set to obtain the structural anomaly credit information set includes: For each semi-structured credit information in the semi-structured credit information set corresponding to the target data structure verification result set, the following generation steps are performed: Determining data nesting type information of the semi-structured credit information to obtain data nesting type information; In response to determining that the data nesting type information is the first data nesting type information, performing a first flattening process on the semi-structured credit information to obtain dimensional credit information; In response to determining that the data nesting type information is the second data nesting type information, performing a second flattening process on the semi-structured credit information to obtain hierarchical credit information; Performing key-value keyword traversal extraction processing on the dimension credit information or the hierarchical credit information to obtain a key-value keyword group; Determining a semi-structured credit information template corresponding to the semi-structured credit information; Performing key-value keyword traversal extraction processing on the semi-structured credit information template to obtain a template keyword group; Generating structural abnormality credit information according to the key value keyword group and the template keyword group; Performing semantic replacement processing on each structurally abnormal credit information in the structurally abnormal credit information set to generate semi-structured semantic credit information, thereby obtaining a semi-structured semantic credit information set; Parsing the semi-structured semantic credit information set and the semi-structured credit information set corresponding to at least one data structure verification result that has passed the representation verification to obtain a parsed semi-structured semantic credit information set and a parsed semi-structured credit information set as the parsed semi-structured credit information set; Generate a key-value pair path information set according to the parsed semi-structured credit information set; extract a key-value pair data set of the key-value pair group set included in the parsed semi-structured credit information set according to the key-value pair path information set; The key-value pair path information set, the key-value pair data set and the semi-structured credit information set are stored in a non-relational database.

2. The method according to claim 1, wherein: The generating structure abnormality credit information according to the key value keyword group and the template keyword group includes: Sorting the key-value keyword group and the template keyword group by keywords to obtain a key-value keyword sequence and a template keyword sequence; Performing matching processing on each template keyword in the template keyword sequence and at least one key-value keyword in the key-value keyword sequence corresponding to the template keyword to generate a keyword matching result group, thereby obtaining a keyword matching result group set; Filtering at least one keyword matching result indicating a successful match from the keyword matching result set as a target keyword matching result set; According to the dimensional credit information or the hierarchical credit information and the semi-structured credit information template, determine the key attribute field of the key keyword and the template attribute field of the template keyword corresponding to each target keyword matching result in the target keyword matching result group, and obtain a key attribute field group and a template attribute field group; Performing attribute type verification processing on each template attribute field in the template attribute field group and the key value attribute field corresponding to the template attribute field in the key value attribute field group to obtain an attribute type verification result group; Determine keyword abnormality type information of each key-value verification keyword in the key-value verification keyword group and each key-value matching keyword in the key-value matching keyword group, and obtain a keyword abnormality type information group, wherein the key-value verification keyword group is a key-value keyword group corresponding to at least one attribute type verification result group representing an unsuccessful verification in the attribute type verification result group, and the key-value matching keyword group is a key-value keyword group corresponding to at least one keyword matching result representing an unsuccessful matching in the keyword matching result group; According to the key-value verification keyword group and the key-value matching keyword group, each keyword anomaly type information in the keyword anomaly type information group is embedded into the semi-structured credit information to obtain structured anomaly credit information.

3. The method according to claim 1, wherein: The generating a key-value pair path information set according to the parsed semi-structured credit information set comprises: Inputting the parsed semi-structured credit information set into a semi-structured graph model to obtain a semi-structured node feature vector set; Inputting the semi-structured node feature vector set into the self-attention mechanism layer to obtain a node weight feature vector set; Determine a key-value pair graph path group corresponding to each node weight feature vector in the node weight feature vector set to obtain a key-value pair graph path group set; For each key-value graph path group in the key-value graph path group set, the following determination steps are performed: Determine the path length of each key-value pair graph path in the key-value pair graph path group to obtain a path length group; Determine a data type group of each node included in each key-value graph path in the key-value graph path group; Determine the number of array indexes of each key-value pair graph path in the key-value pair graph path group as the number of array indexes, and obtain an array index number group; Determine the data nesting depth of each key-value pair graph path in the key-value pair graph path group to obtain a data nesting depth value group; Performing a path complexity evaluation according to the path length group, the data type group, the data index quantity group, and the data nesting depth value group to obtain a path complexity value group; Screening the path complexity value group to obtain a screened path complexity value group; Determine the key-value pair graph path group corresponding to the filtered path complexity value group as the key-value pair path group; The obtained key-value pair path group set is determined as a key-value pair path set.

4. A key-value pair path generation device, comprising: An acquisition unit is configured to acquire a semi-structured credit information set, wherein the semi-structured credit information in the semi-structured credit information set is user credit status information and enterprise credit status information stored in JSON format; The data structure verification unit is configured to perform data structure verification processing on each semi-structured credit information in the semi-structured credit information set to generate a data structure verification result and obtain a data structure verification result set, wherein the data structure verification processing on each semi-structured credit information in the semi-structured credit information set to generate a data structure verification result and obtain a data structure verification result set includes: classifying the semi-structured credit information set to obtain a category credit information set; acquiring a semi-structured credit information template corresponding to each category credit information in the category credit information set to obtain a semi-structured credit information template set; generating a semi-structured description information set according to the semi-structured credit information template set; performing data structure verification processing on each semi-structured credit information in the semi-structured credit information set according to the semi-structured description information set to generate a data structure verification result and obtain a data structure verification result set; A screening unit is configured to screen out at least one data structure verification result representing a verification failure from the data structure verification result set as a target data structure verification result set; The structural anomaly detection unit is configured to perform structural anomaly detection on the semi-structured credit information set corresponding to the target data structure verification result set to obtain the structural anomaly credit information set, wherein the structural anomaly detection on the semi-structured credit information set corresponding to the target data structure verification result set to obtain the structural anomaly credit information set includes: for each semi-structured credit information in the semi-structured credit information set corresponding to the target data structure verification result set, performing the following generation steps: determining data nesting type information of the semi-structured credit information to obtain data nesting type information; in response to determining that the data nesting type information is the first data nesting type information information, performing a first flattening process on the semi-structured credit information to obtain dimensional credit information; in response to determining that the data nesting type information is the second data nesting type information, performing a second flattening process on the semi-structured credit information to obtain hierarchical credit information; performing a key-value keyword traversal extraction process on the dimensional credit information or the hierarchical credit information to obtain a key-value keyword group; determining a semi-structured credit information template corresponding to the semi-structured credit information; performing a key-value keyword traversal extraction process on the semi-structured credit information template to obtain a template keyword group; generating structurally abnormal credit information according to the key-value keyword group and the template keyword group; A semantic replacement unit is configured to perform semantic replacement processing on each structural abnormality credit information in the structural abnormality credit information set to generate semi-structured semantic credit information and obtain a semi-structured semantic credit information set; A parsing unit is configured to parse the semi-structured semantic credit information set and the semi-structured credit information set corresponding to at least one data structure verification result that has passed the representation verification, to obtain a parsed semi-structured semantic credit information set and a parsed semi-structured credit information set as the parsed semi-structured credit information set; A generating unit, configured to generate a key-value pair path information set according to the parsed semi-structured credit information set; an extracting unit configured to extract a key-value pair data set of a key-value pair group set included in the parsed semi-structured credit information set according to the key-value pair path information set; The storage unit is configured to store the key-value pair path information set, the key-value pair data set and the semi-structured credit information set in a non-relational database.

5. An electronic device comprising: one or more processors; a storage device having one or more programs stored thereon, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 3.

6. A computer readable medium having a computer program stored thereon, wherein: When the computer program is executed by a processor, the method according to any one of claims 1 to 3 is implemented.

Images