Semi-structured data analysis method and device and medium

Abstract

The invention discloses a semi-structured data analysis method and device and a medium, and the method comprises the steps: collecting semi-structured data from a preset data source, recognizing the data type of the semi-structured data, and determining an analysis mode corresponding to the semi-structured data according to the data type; wherein the analysis mode is divided into a direct analysis mode and an indirect analysis mode according to whether the data content can be directly obtained or not; on the basis of a direct analysis mode, determining an organization relationship among content elements in the semi-structured data, and reading data contents corresponding to the semi-structured data according to the organization relationship; based on the indirect analysis mode, performing region division on the semi-structured data, and obtaining data content corresponding to the semi-structured data through each data component obtained after division; and generating an embedded vector corresponding to the data content, and storing the embedded vector into a vector database to realize analysis of the semi-structured data.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a semi-structured data analysis method, device and medium. Background Art

[0002] With the rapid development of information technology, data storage formats are becoming more and more diverse. Semi-structured data plays an important role in practical applications because of its flexible structure and easy expansion. Semi-structured data contains both elements with a clear structure and certain free-format content, such as XML and HTML. The effective parsing and vectorization of these data are crucial for data mining, information retrieval, natural language processing and other fields. Existing semi-structured data parsing methods often rely on template matching or rule definition, which is difficult to adapt to the dynamic changes of data structure. Summary of the invention

[0003] In order to solve the above problems, this application proposes a semi-structured data parsing method, including:

[0004] Collecting semi-structured data from a preset data source, identifying the data type of the semi-structured data, and determining a parsing mode corresponding to the semi-structured data according to the data type; wherein the parsing mode is divided into a direct parsing mode and an indirect parsing mode according to whether the data content can be directly obtained;

[0005] Based on the direct parsing mode, determining the organizational relationship between the content elements in the semi-structured data, and reading the data content corresponding to the semi-structured data according to the organizational relationship;

[0006] Based on the indirect parsing mode, the semi-structured data is divided into regions, and data content corresponding to the semi-structured data is obtained through each data component obtained after the division;

[0007] Generate an embedding vector corresponding to the data content, and store the embedding vector in a vector database to implement parsing of the semi-structured data.

[0008] In an implementation of the present application, determining the organizational relationship between the content elements in the semi-structured data based on the direct parsing mode specifically includes:

[0009] Based on the direct parsing mode, according to the file organization mode of the semi-structured data, determining whether the unstructured elements in the semi-structured data and the corresponding structured elements are in the same format file; wherein the file organization mode includes the organization mode between the style file and the content file;

[0010] If so, it is determined that the organizational relationship between the content elements in the semi-structured data is a nested relationship; if not, it is a parallel relationship.

[0011] In one implementation of the present application, reading data content corresponding to the semi-structured data according to the organizational relationship specifically includes:

[0012] In the case where the organizational relationship is the nested relationship, extracting the data file where the data content is located from the semi-structured data, and reading the data content in the data file according to the grammatical structure of the data file;

[0013] In the case where the organizational relationship is the parallel relationship, data content corresponding to the semi-structured data is read according to the grammatical structure of the semi-structured data.

[0014] In one implementation of the present application, based on the indirect parsing mode, the semi-structured data is divided into regions, and the data content corresponding to the semi-structured data is obtained through each data component obtained after the division, specifically including:

[0015] Based on the indirect parsing mode, the position coordinates corresponding to each data component in the semi-structured data are determined by a preset target detection model, and the semi-structured data is divided into regions according to the position coordinates to obtain a plurality of data components;

[0016] Determine the data attributes corresponding to the data components, and determine whether the data components need to be processed in blocks according to the data attributes;

[0017] If so, the data components are divided into blocks to obtain a number of sub-data contents after division, and a characteristic routing id corresponding to the sub-data contents is generated; wherein the characteristic routing id corresponds to the data attribute.

[0018] In one implementation of the present application, after generating the embedding vector corresponding to the data content, the method further includes:

[0019] For different semi-structured data, the corresponding data contents are subjected to association analysis to determine the association types between the data contents; wherein the association types include strong association relationships and weak association relationships;

[0020] If the association type between the data contents is a strong association relationship, the corresponding embedding vectors are fused to obtain a fused fused embedding vector, and the fused embedding vector is stored in a vector database;

[0021] If the association type between the data contents is a weak association relationship, a corresponding association index is generated for the embedded vector, so as to realize association retrieval of the embedded vector through the association index.

[0022] In one implementation of the present application, the corresponding data contents are subjected to association analysis to determine the association type between the data contents, specifically including:

[0023] For different semi-structured data, performing structural feature analysis on the semi-structured data to determine whether the structural features of the semi-structured data are the same;

[0024] If yes, obtain a structure tree corresponding to the semi-structured data, and determine the structural similarity between the extracted data contents according to the tree edit distance between the structure trees;

[0025] If not, calculating the content similarity between the embedding vectors;

[0026] The association type between the data contents is determined according to the structural similarity and / or the content similarity.

[0027] In one implementation of the present application, determining the association type between the data contents according to the structural similarity and the content similarity specifically includes:

[0028] assigning corresponding weights to the structural similarity and the content similarity respectively, and performing weighted summation of the structural similarity and the content similarity according to the weights to obtain a corresponding comprehensive similarity;

[0029] The type of association between the data contents is determined according to the magnitude relationship between the comprehensive similarity and a preset similarity threshold.

[0030] In one implementation of the present application, generating an embedding vector corresponding to the data content, the method further includes:

[0031] Generate sub-embedding vectors corresponding to the sub-data contents respectively;

[0032] According to the characteristic routing ID carried by the sub-data content, the routing relationship corresponding to the sub-embedding vector is determined, and according to the routing relationship, the corresponding sub-embedding vectors are spliced ​​in sequence to obtain the embedding vector corresponding to the data content.

[0033] The present application provides a semi-structured data parsing device, the device comprising:

[0034] at least one processor;

[0035] and, a memory communicatively coupled to the at least one processor;

[0036] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to:

[0037] Collecting semi-structured data from a preset data source, identifying the data type of the semi-structured data, and determining a parsing mode corresponding to the semi-structured data according to the data type; wherein the parsing mode is divided into a direct parsing mode and an indirect parsing mode according to whether the data content can be directly obtained;

[0038] Based on the direct parsing mode, determining the organizational relationship between the content elements in the semi-structured data, and reading the data content corresponding to the semi-structured data according to the organizational relationship;

[0039] Based on the indirect parsing mode, the semi-structured data is divided into regions, and data content corresponding to the semi-structured data is obtained through each data component obtained after the division;

[0040] Generate an embedding vector corresponding to the data content, and store the embedding vector in a vector database to implement parsing of the semi-structured data.

[0041] The embodiment of the present application provides a non-volatile computer storage medium storing computer executable instructions, wherein the computer executable instructions are configured as follows:

[0042] Collecting semi-structured data from a preset data source, identifying the data type of the semi-structured data, and determining a parsing mode corresponding to the semi-structured data according to the data type; wherein the parsing mode is divided into a direct parsing mode and an indirect parsing mode according to whether the data content can be directly obtained;

[0043] Based on the direct parsing mode, determining the organizational relationship between the content elements in the semi-structured data, and reading the data content corresponding to the semi-structured data according to the organizational relationship;

[0044] Based on the indirect parsing mode, the semi-structured data is divided into regions, and data content corresponding to the semi-structured data is obtained through each data component obtained after the division;

[0045] Generate an embedding vector corresponding to the data content, and store the embedding vector in a vector database to implement parsing of the semi-structured data.

[0046] The semi-structured data parsing method proposed in this application can bring the following beneficial effects:

[0047] Identify the data type of semi-structured data and determine the corresponding parsing mode according to the data type. Different parsing modes can adapt to the dynamic changes of data structure and improve the flexibility and adaptability of data parsing. Through the direct parsing mode, the organizational relationship between the content elements in the semi-structured data can be accurately captured to ensure the accurate reading of the data content, while the indirect parsing mode can achieve effective acquisition of data content by dividing the data area, further enhancing the reliability of parsing. The parsed data content is converted into an embedded vector and stored in a vector database, providing data support for applications such as data mining and information retrieval. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0049] Figure 1 A schematic diagram of a process for analyzing semi-structured data provided in an embodiment of the present application;

[0050] Figure 2 A schematic diagram of the structure of a semi-structured data parsing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0052] The technical solutions provided by various embodiments of the present application are described in detail below in conjunction with the accompanying drawings.

[0053] like Figure 1 As shown, a semi-structured data parsing method provided in an embodiment of the present application includes:

[0054] S101: Collect semi-structured data from a preset data source, identify the data type of the semi-structured data, and determine a parsing mode corresponding to the semi-structured data according to the data type; wherein the parsing mode is divided into a direct parsing mode and an indirect parsing mode according to whether the data content can be directly obtained.

[0055] Semi-structured data is a data type between structured data and unstructured data, and is widely used in scenarios such as web services, configuration files, and data storage. In order to reduce the difficulty of parsing semi-structured data, it is necessary to convert it into structured data to meet the data mining, data retrieval and other functions under different application requirements. Structured data is parsed into embedded vectors. The vector database composed of the embedded vectors corresponding to each semi-structured data can not only effectively improve the data processing efficiency during the relevant data analysis process, but also support more complex data analysis and mining tasks, and is applicable to more diverse scenarios.

[0056] If you want to finally build a vector database, you need to collect semi-structured data from the preset data source according to actual business needs. The data source here can be a database, data file or API interface, etc. Semi-structured data includes multiple data types, such as HTML, XML, PDF, etc. For semi-structured data of different data types, the parsing mode used is also different. Therefore, after collecting semi-structured data, you must first clarify its corresponding parsing mode. The parsing mode is divided into two types according to whether the data content can be directly obtained, namely direct parsing mode and indirect parsing mode. The direct parsing mode is suitable for semi-structured data that can directly extract the required information, such as XML and TEX files, and directly identify the data content in the file. The indirect parsing mode is suitable for semi-structured data that requires some intermediate operation to obtain the required information, such as PDF files. First, it is necessary to perform image partitioning before the data content can be finally identified.

[0057] S102: Based on the direct parsing mode, determine the organizational relationship between the content elements in the semi-structured data, and read the data content corresponding to the semi-structured data according to the organizational relationship.

[0058] For semi-structured data suitable for direct parsing mode, such as XML, TEX, HTML, DOCX and other files, in direct parsing mode, there is a certain organizational relationship between the various content elements contained therein, and the organizational relationship is used to represent the positional relationship between the data structures of the various content elements, including nested relationships and parallel relationships. After clarifying the organizational relationship, the required data content can be read from the semi-structured data according to the organizational relationship.

[0059] Specifically, content elements refer to the various components that constitute semi-structured data, including structured elements and unstructured elements. Structured elements refer to parts with clear formats and rules, such as style content such as tags, attributes, nodes, etc. Unstructured elements are the opposite. They do not have clear formats and rules, and usually include data content such as text, images, and videos. Most of the style content of semi-structured data coexists with the data content in the same file. However, the style content of some semi-structured data is not only stored through independent style files, but also coexists with the data content. Therefore, according to the file organization method of semi-structured data, that is, the organization method of style files and content files, it can be clearly determined whether unstructured elements and structured elements are located in the same format file. If they are located in the same format file, it means that the data content still has another separate style file to store the style content in addition to the style content parallel to it, so the data content and the style content are in a nested relationship. On the contrary, if the data content and the style content exist together in one data file, then the two are in a parallel relationship.

[0060] Based on different organizational relationships, when reading the corresponding type of semi-structured data, the reading method adopted is also different. When the organizational relationship is a nested relationship, it is first necessary to extract the data file where the data content is located from the semi-structured data, and then read the data content in the data file according to the grammatical structure of the data file. For example, for DOCX data with nested relationships between content elements, it is first necessary to decompress it to obtain a data file and multiple style files, and extract the data file document.xml containing the data content from it. Since the data file is still an XML format file, it is necessary to further read the data content to be obtained according to the grammatical structure of the XML file. For semi-structured data with parallel organizational relationships, the data content can be read directly according to the grammatical structure of the semi-structured data. For example, for XML, TEX, and HTML data with parallel relationships between content elements, taking XML data as an example, the semi-structured data is converted into a tree structure through an XML parser, and the tree structure is traversed to obtain the text content under each element node in turn, and finally the corresponding data content is output.

[0061] S103: Based on the indirect parsing mode, the semi-structured data is divided into regions, and data content corresponding to the semi-structured data is obtained through each data component obtained after the division.

[0062] For semi-structured data suitable for indirect parsing mode, such as PDF data, since it is impossible to read the data directly, the specific grammatical structure of the data is not directly parsed when reading the data content. Instead, the data area is divided according to the logical layout of the data, and the required data content is extracted from the data area.

[0063] Specifically, the position coordinates corresponding to each data component in the semi-structured data are determined by a preset target detection model, and the semi-structured data is regionally divided according to the position coordinates to obtain several data components. Each data component contains its position in the data and the corresponding content. After the semi-structured data is split into several data components, the content of each data component is analyzed to determine its corresponding data attributes, such as data type, format, length, etc., and then according to the data attributes, it is determined whether the data component needs to be processed in blocks. Block processing is usually used to process large fields or complex structures. If the structure of the data component is relatively complex or the length is relatively long, in order to improve the processing speed of the data, it is preferred to block such data components to obtain several sub-data contents after division. Each sub-data content has its corresponding feature routing id, which corresponds to the data attribute. In the subsequent processing, the feature routing id can be used to more conveniently realize the identification and processing of each sub-data content, reflecting the organizational logic of the data content. Among them, the feature routing id should be named according to the organization method of the data attributes in which it is located. For example, the text part is divided into 3 blocks of sub-data content, then these sub-data contents can be named A1, A2, A3 or 1-1, 1-2, 1-3 or text1, text2, text3, etc. The specific id form is not limited in this embodiment of the present application and can be set according to actual needs.

[0064] S104: Generate an embedding vector corresponding to the data content, and store the embedding vector in a vector database to implement parsing of the semi-structured data.

[0065] After completing the parsing of the semi-structured data through the two different parsing modes S102 and S103, it is necessary to use the selected embedding model, such as the text-embedding-ada-002 model, to convert the data content into an embedding vector. The generated embedding vector and the data identifier associated with it (such as the feature routing ID) are stored in the vector database, and the corresponding index is constructed to accelerate retrieval. By converting the content of the semi-structured data into an embedding vector and using the vector database for efficient storage and retrieval, not only the processing efficiency of the semi-structured data is improved, but also strong support is provided for subsequent data analysis and utilization.

[0066] It should be noted that semi-structured data is eventually split into several sub-parts in the indirect parsing mode. Therefore, when vectorizing it, it is also necessary to generate corresponding sub-embedding vectors for each sub-part. Each sub-embedding vector carries a corresponding feature routing id. According to the feature routing id, the routing relationship corresponding to different sub-data contents can be clarified. The routing relationship is used to represent the organizational structure between sub-data contents. In this way, according to the routing relationship, the sub-embedding vectors can be gradually spliced, and finally the complete embedding vector corresponding to the data content is obtained.

[0067] In one embodiment, since the above-mentioned embedding vectors come from semi-structured data in different data sources, there may be certain correlations between the semi-structured data in terms of data content. Therefore, after generating the embedding vectors of the data content, the embedding vectors can be further processed based on the correlation types between different data contents, thereby realizing deep mining and utilization of the data.

[0068] Specifically, for different semi-structured data, the corresponding data content is analyzed to determine the type of association between the data contents. Among them, the association type includes strong association and weak association. Strong association usually refers to the direct and close connection between data contents, such as similar semantics, the same entity reference, etc., while weak association refers to the existence of a certain connection between data contents, but this connection is relatively loose or indirect, such as similar topics, related concepts, etc.

[0069] The type of association between data contents can be evaluated by the content similarity or structural similarity between the data contents. First, for different semi-structured data, the structural feature analysis of the semi-structured data is performed to determine whether the structural features of each semi-structured data are the same. Structural features are used to indicate the file structure type where the data content is located. If the structural features corresponding to the data content are the same, it indicates that the file structure type is the same. For example, the data content in XML files and DOCX files are both in xml format, then it can be determined that their structural features are the same. At this time, obtain the structure tree corresponding to the semi-structured data, and determine the structural similarity between the extracted data contents based on the tree edit distance between the structure trees. The specific formula is as follows: S=1-e, where S represents structural similarity and e represents edit distance.

[0070] e max

[0071] Distance, e maxis the maximum edit distance, which is expressed as the maximum total number of nodes on the structure tree. If the structural features are different, the information of the structural features is ignored at this time, and the content similarity between the embedded vectors is directly calculated. In this way, the type of association between the data contents can be determined according to the structural similarity and / or content similarity. Specifically, the structural similarity and content similarity have different weights, which can be allocated according to the application scenario of the semi-structured data. For semi-structured data with stronger content bias, more attention should be paid to the similarity between the contents, and the weight allocated to the content similarity is relatively higher. After clarifying the weights corresponding to the structural similarity and content similarity respectively, the structural similarity and content similarity are weighted and summed according to the weights to obtain the corresponding comprehensive similarity. Then, according to the size relationship between the comprehensive similarity and the preset similarity threshold, the type of association between the data contents can be determined. When the comprehensive similarity is greater than the preset similarity threshold, the associated content is a strong association type, otherwise it is a weak association type. For data content that does not involve structural similarity, it can be directly evaluated for similarity based on content similarity.

[0072] Based on different types of association, the processing measures taken for embedded vectors are also different. If there is a strong correlation between the data contents, then both the data subject and the content itself have a strong correlation. In order to reduce the redundant information between the data contents, the embedded vectors can be fused to fuse the embedded vectors corresponding to the data contents with strong correlation into a fused embedded vector as a whole, thereby realizing the integration between the data and forming a more complete and comprehensive data representation. It should be noted that if there is a strong correlation between data content A and data content B, and there is a strong correlation between data content B and data content C, this does not mean that data content A can be directly fused with data content C. It is also necessary to conduct specific analysis on it. If there is no strong correlation between A and C, then data content A and data content B, and data content B and data content C can be fused separately.

[0073] If the data contents are weakly correlated, it means that although there is a certain connection between the data contents, the connection is relatively loose or indirect. By generating an associated index for the corresponding embedded vector, a complete retrieval view can be formed. In this way, the associated retrieval of the embedded vector can be realized through the associated index during retrieval, which can enable subsequent analysis and retrieval to cover a wider range of information.

[0074] The above are embodiments of the method proposed in this application. Based on the same idea, some embodiments of this application also provide devices and non-volatile computer storage media corresponding to the above methods.

[0075] Figure 2A schematic diagram of the structure of a semi-structured data parsing device provided in an embodiment of the present application. Figure 2 As shown, including:

[0076] at least one processor; and,

[0077] at least one processor is communicatively connected to a memory; wherein,

[0078] The memory stores instructions that can be executed by at least one processor, and the instructions are executed by at least one processor so that the at least one processor can execute a semi-structured data parsing method as described in any of the above items.

[0079] The embodiment of the present application provides a non-volatile computer storage medium storing computer executable instructions, wherein the computer executable instructions are configured as follows:

[0080] A semi-structured data parsing method as described in any of the above items.

[0081] Each embodiment in this application is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device and medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0082] The devices and media provided in the embodiments of the present application correspond one-to-one to the methods. Therefore, the devices and media also have similar beneficial technical effects as the corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the devices and media will not be repeated here.

[0083] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0084] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0085] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0086] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0087] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0088] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0089] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0090] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0091] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A semi-structured data analysis method, characterized in that: The method comprises: Collecting semi-structured data from a preset data source, identifying the data type of the semi-structured data, and determining a parsing mode corresponding to the semi-structured data according to the data type; wherein the parsing mode is divided into a direct parsing mode and an indirect parsing mode according to whether the data content can be directly obtained; Based on the direct parsing mode, determining the organizational relationship between the content elements in the semi-structured data, and reading the data content corresponding to the semi-structured data according to the organizational relationship; Based on the indirect parsing mode, the semi-structured data is divided into regions, and data content corresponding to the semi-structured data is obtained through each data component obtained after the division; Generate an embedding vector corresponding to the data content, and store the embedding vector in a vector database to implement parsing of the semi-structured data.

2. A semi-structured data analysis method according to claim 1, characterized in that: Determining the organizational relationship between the content elements in the semi-structured data based on the direct parsing mode specifically includes: Based on the direct parsing mode, according to the file organization mode of the semi-structured data, determining whether the unstructured elements in the semi-structured data and the corresponding structured elements are in the same format file; wherein the file organization mode includes the organization mode between the style file and the content file; If so, it is determined that the organizational relationship between the content elements in the semi-structured data is a nested relationship; if not, it is a parallel relationship.

3. A semi-structured data analysis method according to claim 2, characterized in that: Reading data content corresponding to the semi-structured data according to the organizational relationship specifically includes: In the case where the organizational relationship is the nested relationship, extracting the data file where the data content is located from the semi-structured data, and reading the data content in the data file according to the grammatical structure of the data file; In the case where the organizational relationship is the parallel relationship, data content corresponding to the semi-structured data is read according to the grammatical structure of the semi-structured data.

4. A semi-structured data analysis method according to claim 1, characterized in that: Based on the indirect parsing mode, the semi-structured data is divided into regions, and data content corresponding to the semi-structured data is obtained through each data component obtained after the division, specifically including: Based on the indirect parsing mode, the position coordinates corresponding to each data component in the semi-structured data are determined by a preset target detection model, and the semi-structured data is divided into regions according to the position coordinates to obtain a plurality of data components; Determine the data attributes corresponding to the data components, and determine whether the data components need to be processed in blocks according to the data attributes; If so, the data components are divided into blocks to obtain a number of sub-data contents after division, and a characteristic routing id corresponding to the sub-data contents is generated; wherein the characteristic routing id corresponds to the data attribute.

5. A semi-structured data analysis method according to claim 1, characterized in that: After generating the embedding vector corresponding to the data content, the method further includes: For different semi-structured data, the corresponding data contents are subjected to association analysis to determine the association types between the data contents; wherein the association types include strong association relationships and weak association relationships; If the association type between the data contents is a strong association relationship, the corresponding embedding vectors are fused to obtain a fused fused embedding vector, and the fused embedding vector is stored in a vector database; If the association type between the data contents is a weak association relationship, a corresponding association index is generated for the embedded vector, so as to realize association retrieval of the embedded vector through the association index.

6. A semi-structured data analysis method according to claim 5, characterized in that: Performing association analysis on the corresponding data content to determine the association type between the data contents, specifically including: For different semi-structured data, performing structural feature analysis on the semi-structured data to determine whether the structural features of the semi-structured data are the same; If yes, obtain a structure tree corresponding to the semi-structured data, and determine the structural similarity between the extracted data contents according to the tree edit distance between the structure trees; If not, calculating the content similarity between the embedding vectors; The association type between the data contents is determined according to the structural similarity and / or the content similarity.

7. A semi-structured data analysis method according to claim 6, characterized in that: Determining the association type between the data contents according to the structural similarity and the content similarity, specifically including: assigning corresponding weights to the structural similarity and the content similarity respectively, and performing weighted summation of the structural similarity and the content similarity according to the weights to obtain a corresponding comprehensive similarity; The type of association between the data contents is determined according to the magnitude relationship between the comprehensive similarity and a preset similarity threshold.

8. A semi-structured data analysis method according to claim 4, characterized in that: Generating an embedding vector corresponding to the data content, the method further includes: Generate sub-embedding vectors corresponding to the sub-data contents respectively; According to the characteristic routing ID carried by the sub-data content, the routing relationship corresponding to the sub-embedding vector is determined, and according to the routing relationship, the corresponding sub-embedding vectors are spliced ​​in sequence to obtain the embedding vector corresponding to the data content.

9. A semi-structured data analysis device, characterized in that: The device comprises: at least one processor; and, a memory communicatively coupled to the at least one processor; The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute a semi-structured data parsing method as described in any one of claims 1-8.

10. A non-volatile computer storage medium storing computer executable instructions, characterized in that: The computer executable instructions are configured to: A semi-structured data parsing method as described in any one of claims 1 to 8.

Images