Information processing method and related device

Abstract

The invention provides an information processing method and a related device. The embodiment of the invention can be applied to various scenes such as artificial intelligence. The method comprises the steps of obtaining M text blocks corresponding to text information to be processed and M pieces of time identification information corresponding to the M text blocks; inputting the M text blocks into a word vector generation network to generate M text coding features; the M text coding features are input into a clustering network, N clustering clusters are generated, each clustering cluster comprises K text blocks, and the similarity of every two text coding features corresponding to every two text blocks belonging to the same clustering cluster is larger than or equal to a similarity threshold value; and generating N data blood relationship maps according to the N clustering clusters and the M pieces of time identification information. According to the method provided by the embodiment of the invention, the data consanguinity analysis can be carried out on the unstructured text information, the use scene of most data of the user can be covered, the risk of data security is reduced, and the data security protection capability is improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to an information processing method and related devices. Background Art

[0002] Data lineage describes the paths and relationships along which data flows and transforms within a system. It documents the data's origin, processing, transformation steps, and dependencies with other data. Analyzing data within information systems using data lineage analysis can help users understand the data's origins, assess data changes and impacts, identify data quality issues, and trace and repair data errors or anomalies.

[0003] However, existing data lineage analysis methods are mainly aimed at structured data scenarios, relying on database audit logs and SQL execution statements. They do not cover the majority of unstructured text data and are unable to perform data lineage analysis on large blocks of text data. As a result, a large amount of text data is out of the coverage of data lineage analysis, thus creating certain data security risks. Summary of the Invention

[0004] The embodiments of the present application provide an information processing method and related devices, which enhance the ability to perform data lineage analysis on structured text information, thereby improving the data security protection capability for unstructured text information.

[0005] One aspect of the present application provides an information processing method, comprising:

[0006] Obtain M text blocks corresponding to the text information to be processed and M time identification information corresponding to the M text blocks, wherein the M text blocks are obtained by segmenting the text information, and M>1;

[0007] Encode M text blocks through the word vector generation network to generate M text encoding features;

[0008] The similarity of M text encoding features is calculated through a clustering network to generate N clusters, where each cluster includes K text blocks, and the similarity of the text encoding features corresponding to any two text blocks in the same cluster is greater than or equal to the similarity threshold, M ≥ N ≥ 1, K > 1;

[0009] According to the N clusters and the M time identification information, N data lineage graphs are generated, wherein the data lineage graphs are used to represent the association relationship between the K time identification information corresponding to the K text blocks belonging to the same cluster.

[0010] Another aspect of the present application provides an information processing device, comprising:

[0011] A text block acquisition module is used to acquire M text blocks corresponding to the text information to be processed and M time identification information corresponding to the M text blocks, wherein the M text blocks are obtained by segmenting the text information, and M>1;

[0012] The text block encoding module is used to encode M text blocks through the word vector generation network to generate M text encoding features;

[0013] The text block clustering module is used to calculate the similarity of M text encoding features through a clustering network to generate N clusters, where each cluster includes K text blocks, and the similarity of the text encoding features corresponding to any two text blocks in the same cluster is greater than or equal to the similarity threshold, M ≥ N ≥ 1, K > 1;

[0014] The data lineage graph generation module is used to generate N data lineage graphs based on N clusters and M time identification information, wherein the data lineage graph is used to represent the association relationship between K time identification information corresponding to K text blocks belonging to the same cluster.

[0015] In another implementation of the embodiment of the present application, the word vector generation network includes a semantic feature extraction layer and a feature embedding layer; a text block encoding module, and is further used to:

[0016] Input M text blocks into the semantic feature extraction layer in the word vector generation network, and generate M text semantic features through the semantic feature extraction layer;

[0017] The text semantic features are input into the feature embedding layer in the word vector generation network, and M text encoding features are generated through the feature embedding layer.

[0018] In another implementation of the embodiment of the present application, the text block clustering module is further configured to:

[0019] Input M text encoding features into the clustering network, calculate the cosine similarity of each pair of the M text encoding features through the clustering network, and obtain L cosine similarity values, where L = M × (M-1) / 2;

[0020] According to the L cosine similarity values, the M text encoding features are grouped to obtain N encoding feature groups, wherein the cosine similarity value of each two text encoding features belonging to the same encoding feature group is greater than or equal to the similarity threshold;

[0021] According to N encoding feature groups, M text blocks are grouped to obtain N clusters.

[0022] In another implementation of the embodiment of the present application, the data lineage graph generation module is further configured to:

[0023] For each of the N clusters, execute:

[0024] Generate K text block nodes based on the K text blocks in each cluster;

[0025] According to the K time identification information corresponding to the K text blocks in each cluster, establish the edge relationship between the K text block nodes;

[0026] According to the K text block nodes and the edge relationships between the K text block nodes, a data lineage graph corresponding to the cluster is generated.

[0027] In another implementation of the embodiment of the present application, the data lineage graph generation module is further configured to:

[0028] Arrange the K time identification information corresponding to the K text blocks in each cluster in ascending order;

[0029] Sort the K text block nodes according to the K time identification information in ascending order;

[0030] According to the sorted K text block nodes, edge relationships between the K text block nodes are established.

[0031] In another implementation of the embodiment of the present application, the text block acquisition module is further configured to:

[0032] Get the text information to be processed;

[0033] Perform target identifier recognition on the text information to be processed to obtain P target identifiers, where P ≥ M;

[0034] According to P target identifiers, the text information to be processed is segmented to obtain P candidate text blocks;

[0035] Count the number of words in the P candidate text blocks to obtain P word count information corresponding to the P candidate text blocks;

[0036] According to the P word count information, M text blocks are determined from the P candidate text blocks, wherein the word count information corresponding to each text block in the M text blocks is greater than or equal to a preset word count.

[0037] In another implementation of the embodiment of the present application, the text block acquisition module is further configured to:

[0038] Receive file path information and target file format;

[0039] Traverse the file path information to obtain at least one candidate file;

[0040] Determining, according to the target file format, at least one target file whose file format satisfies the target file format from the at least one candidate file;

[0041] According to the target file format, at least one target file is parsed to read the content of the at least one target file to obtain text information to be processed.

[0042] In another implementation of the embodiment of the present application, the information processing device further includes a storage module. Specifically, the storage module is configured to:

[0043] Determine at least one file name information corresponding to the M text blocks according to the file path information;

[0044] Generate M pieces of attribute information according to the M word count information corresponding to the M text blocks, at least one file name information corresponding to the M text blocks, and the M time identification information corresponding to the M text blocks;

[0045] Store M attribute information.

[0046] In another implementation of the embodiment of the present application, the information processing device further includes a query module. Specifically, the query module is configured to:

[0047] Receive query text;

[0048] Determine the data lineage graph containing the query text according to the query text, and return the data lineage graph containing the query text.

[0049] In another implementation of the embodiment of the present application, the information processing device further includes a word vector generation network training module. Specifically, the word vector generation network training module is used to:

[0050] Obtaining a first sample data set, wherein the first sample data set includes at least one set of first sample data, each set of first sample data includes a first training sample text, a first positive sample text, and a first negative sample text, semantic information of the first training sample text is the same as semantic information of the first positive sample text, and semantic information of the first training sample text is different from semantic information of the first negative sample text;

[0051] Get the word vector to generate the initial network;

[0052] The initial network for generating word vectors is trained according to the first sample data set to adjust network parameters of the initial network for generating word vectors, thereby obtaining a trained network for generating word vectors.

[0053] In another implementation of the embodiment of the present application, the information processing device further includes a clustering network training module. Specifically, the clustering network training module is configured to:

[0054] Obtaining a second sample data set and a second sample data label set, wherein the second sample data set includes at least one second training sample text encoding;

[0055] Get the initial clustering network;

[0056] Inputting the second sample data set into the clustering initial network, and generating a predicted clustering result through the clustering initial network;

[0057] The clustering initial network is trained according to the second sample data label set and the predicted clustering result to adjust the network parameters of the clustering initial network to obtain the clustering network.

[0058] Another aspect of the present application provides a computer device, comprising:

[0059] memories, transceivers, processors, and bus systems;

[0060] Wherein, the memory is used to store programs;

[0061] The processor is used to execute the program in the memory, including executing the above-mentioned methods;

[0062] The bus system is used to connect the memory and the processor so that the memory and the processor can communicate with each other.

[0063] Another aspect of the present application provides a computer-readable storage medium, wherein instructions are stored in the computer-readable storage medium. When the computer-readable storage medium is run on a computer, the computer is enabled to execute the above-mentioned methods.

[0064] Another aspect of the present application provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the methods provided by the above aspects.

[0065] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:

[0066] The present application provides an information processing method and related devices, the method comprising obtaining M text blocks corresponding to the text information to be processed, encoding the M text blocks through a word vector generation network, obtaining M text encoding features for representing the semantic information of the M text blocks, performing similarity calculation on the M text encoding features through a clustering network, clustering the M text blocks according to the calculated similarity values, clustering the text blocks with higher semantic similarity into the same cluster cluster, establishing a data kinship relationship between the text blocks belonging to the same cluster cluster, and displaying the data kinship relationship in the form of a data kinship graph. The present application implements The method provided in the example can realize data lineage analysis of unstructured text information to be processed by segmenting and encoding the text information to be processed, which can cover the usage scenarios of most user data, reduce data security risks, and improve data security protection capabilities; using word vector generation networks and clustering networks, this application can more accurately calculate the similarity of text encoding features, generate more meaningful clusters, and thus provide a more accurate data lineage graph; generating a data lineage graph based on clusters and text attributes can clearly show the relationship between text blocks belonging to the same cluster, helping users better understand the ins and outs of text data. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 A schematic diagram of the architecture of an information processing system provided in one embodiment of the present application;

[0068] Figure 2 A flowchart of an information processing method provided in one embodiment of the present application;

[0069] Figure 3 A schematic diagram of a data lineage graph provided in one embodiment of the present application;

[0070] Figure 4 A flowchart of an information processing method provided in another embodiment of the present application;

[0071] Figure 5 A schematic diagram of a network processing process for generating word vectors provided in one embodiment of the present application;

[0072] Figure 6 A flowchart of an information processing method provided in another embodiment of the present application;

[0073] Figure 7 A schematic diagram of a clustering process provided in one embodiment of the present application;

[0074] Figure 8 A flowchart of an information processing method provided in another embodiment of the present application;

[0075] Figure 9A schematic diagram of a data lineage graph provided in one embodiment of the present application;

[0076] Figure 10 A flowchart of an information processing method provided in another embodiment of the present application;

[0077] Figure 11 A flowchart of an information processing method provided in another embodiment of the present application;

[0078] Figure 12 A schematic diagram of text information segmentation provided in an embodiment of the present application;

[0079] Figure 13 A flowchart of an information processing method provided in another embodiment of the present application;

[0080] Figure 14 A flowchart of an information processing method provided in another embodiment of the present application;

[0081] Figure 15 A flowchart of text information preprocessing provided in one embodiment of the present application;

[0082] Figure 16 A flowchart of an information processing method provided in another embodiment of the present application;

[0083] Figure 17 A flowchart of an information processing method provided in another embodiment of the present application;

[0084] Figure 18 A schematic diagram of a network training process for generating word vectors provided in an embodiment of the present application;

[0085] Figure 19 A flowchart of an information processing method provided in another embodiment of the present application;

[0086] Figure 20 A schematic diagram of a process for training a clustering network according to an embodiment of the present application;

[0087] Figure 21 A flowchart of an information processing method provided in one embodiment of the present application;

[0088] Figure 22 A schematic diagram of the architecture of a system for executing an information processing method provided in one embodiment of the present application;

[0089] Figure 23 A schematic diagram of the structure of an information processing device provided in one embodiment of the present application;

[0090] Figure 24 A schematic structural diagram of an information processing device provided in another embodiment of the present application;

[0091] Figure 25 A schematic structural diagram of an information processing device provided in another embodiment of the present application;

[0092] Figure 26 A schematic structural diagram of an information processing device provided in another embodiment of the present application;

[0093] Figure 27 A schematic structural diagram of an information processing device provided in another embodiment of the present application;

[0094] Figure 28 A schematic diagram of the server structure provided for an embodiment of the present application. DETAILED DESCRIPTION

[0095] An embodiment of the present application provides an information processing method, which can perform data lineage analysis on unstructured text information by segmenting and encoding the text information to be processed, covering a wider range of data types and reducing data security risks.

[0096] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can, for example, be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "corresponding to" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0097] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0098] The embodiments of the present application relate to artificial intelligence (AI) and natural language processing (NLP), and are designed based on the natural language processing model.

[0099] Artificial Intelligence (AI) refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI also involves studying the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making.

[0100] Artificial intelligence (AI) technology is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0101] Natural language processing (NLP) is a key area of ​​research in computer science and artificial intelligence. It studies the theories and methods that enable effective communication between humans and computers using natural language. Natural language processing (NLP) integrates linguistics, computer science, and mathematics. Therefore, research in this field involves natural language—the language we use in everyday life—and is closely linked to the study of linguistics. Natural language processing technologies typically include text processing, semantic understanding, machine translation, robotic question answering, and knowledge graphs.

[0102] To facilitate understanding of the technical solutions provided in the embodiments of the present application, some key terms used in the embodiments of the present application are explained here:

[0103] Word vectors: Word vectors are a method used in natural language processing to represent words. They convert natural language words into vectorized representations in a high-dimensional space, making them easier for computers to process and analyze. Simply put, a word vector represents a word as a numerical vector. These vectors are typically obtained through unsupervised learning on large amounts of text. The goal is to ensure that words with similar meanings are placed close together in the vector space.

[0104] Data lineage: Data lineage is the relationship between data generated during the data processing and circulation process. It provides a means to explore data relationships and is used to obtain the path through which data flows. Data lineage is a description of the path and related relationships along which data flows and transforms within a system. It records the data's source, processing, conversion steps, and dependencies with other data. Data lineage can help users understand the source and flow of data, allowing for better data management and maintenance. By understanding the data's evolution, users can identify data quality issues and promptly address them. Furthermore, data lineage analysis is crucial for data governance, helping users establish trust and transparency in their data. Furthermore, in data analysis and data science projects, understanding data lineage can help more accurately understand the data and avoid erroneous analytical results.

[0105] Data Security Governance Center (DSGC): A Data Security Governance Center is an organization or platform that centrally manages and controls data security. Its primary goal is to ensure the security of an organization's data, including protecting the confidentiality, integrity, and availability of data.

[0106] Sentence Transformer model: A class of Transformer models that can generate vector representations for sentences and text chunks. Sentence Transformer models can compute text embeddings for over 100 languages ​​and use them for common tasks such as semantic text similarity, semantic search, and synonym mining.

[0107] DBSCAN (Density-Based Spatial Clustering of Applications with Noise) is an unsupervised density-based spatial clustering algorithm used to identify dense regions in a dataset and group them into clusters. The core idea of ​​DBSCAN is to determine whether data points belong to the same cluster based on their density.

[0108] In scenarios such as service operation and management, fault analysis, and data security, it is necessary to analyze text data within the information system through data lineage analysis methods so that users can understand the origins and development of text data, judge the changes and impacts of text data, identify text data quality issues, and trace and repair text data when errors or anomalies occur.

[0109] Related technologies require data lineage analysis to analyze information data, which requires support from database audit logs or SQL statements. However, in real-world scenarios, the proportion of text data recorded by audit logs or SQL statements is relatively small, while most scenarios lack such records. Therefore, there is no way to perform data lineage analysis on text data without audit logs or SQL statements. This makes data outside the scope of data lineage analysis, posing a potential data security risk.

[0110] Therefore, an embodiment of the present application provides an information processing method that can perform data lineage analysis on text data without audit logs or SQL statement records and on text data of unstructured text, and display the association relationship between multiple text data with data lineage relationships in the form of a data lineage graph, helping users to better understand the origins and development of text data.

[0111] For easier understanding, see Figure 1 , Figure 1 This is an application environment diagram of the information processing method in the embodiment of the present application, such as Figure 1 As shown, the information processing method in the embodiment of the present application is applied to an information processing system. The information processing system includes: a server and a terminal device; wherein the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), and big data and artificial intelligence platforms. The terminal can be a smart phone, tablet computer, laptop computer, desktop computer, smart speaker, smart watch, etc., but is not limited to this. The terminal and the server can be directly or indirectly connected by wired or wireless communication, and the embodiment of the present application is not limited here.

[0112] The server first obtains M text blocks corresponding to the text information to be processed, and M time identification information corresponding to the M text blocks, where the M text blocks are obtained by segmenting the text information; then, the M text blocks are encoded through a word vector generation network to generate M text encoding features; then, the M text encoding features are calculated for similarity through a clustering network to generate N cluster clusters, where each cluster cluster includes K text blocks, and the similarity of the pairwise text encoding features corresponding to the two text blocks belonging to the same cluster cluster is greater than or equal to the similarity threshold; finally, the server generates N data lineage graphs based on the N cluster clusters and the M time identification information, where the data lineage graphs are used to characterize the association relationship between the K time identification information corresponding to the K text blocks belonging to the same cluster cluster.

[0113] The following describes the information processing method in this application from the perspective of the server. Figure 2 The information processing method provided in the embodiment of the present application includes: Step S110 to Step S140. Specifically:

[0114] S110 : Obtain M text blocks corresponding to the text information to be processed, and M time identification information corresponding to the M text blocks.

[0115] The M text blocks are obtained by segmenting the text information, and M>1.

[0116] It is understood that first, the text information is acquired, and then the text information is segmented to obtain M text blocks corresponding to the text information. The text information can be composed of at least one text data, which can be an article or a text paragraph. Each text data corresponds to its text attributes, and the text attributes include the creation time of the text. The text blocks are obtained by segmenting the text information. The segmentation method can be based on the target identifier in the text information and the segmentation is performed based on a preset number of words. The target identifier can be a line break, a preset punctuation mark, or a preset character.

[0117] Exemplarily, L articles are obtained, data cleaning is performed on the L articles, and text information is formed from the L articles after data cleaning. Target identifiers in the text information are identified, and the target identifiers include line breaks and preset punctuation marks. The target identifiers include: "\n" (line break), "." (period), "?" (question mark), "!" (exclamation mark), and "······" (ellipsis). According to the identified target identifiers, the text information is segmented to obtain P candidate text blocks, and the word count of the P candidate text blocks is counted. M text blocks with a word count greater than the preset word count are determined from the P candidate text blocks. Since the M text blocks are derived from L articles, based on the text creation time of the L articles, the M time identification information corresponding to the M text blocks is determined, and the time identification information for the text blocks derived from the same article is the same.

[0118] S120. Encode the M text blocks through a word vector generation network to generate M text encoding features.

[0119] It is understandable that M text blocks are input into the word vector generation network in batches or sequentially, and the word vector generation network outputs M text encoding features for encoding the M text blocks. The word vector generation network is a trained network that can achieve encoding according to the semantic information of the text blocks. The word vector generation network extracts semantic features of the M text blocks to obtain M text semantic features, and then the M text semantic features are subjected to feature embedding processing to obtain M text encoding features. Text blocks with similar semantic information have similar text encoding features.

[0120] Preferably, after obtaining the text encoding features, the text encoding features need to be persistently stored to facilitate subsequent analysis and avoid repeated calculations. Methods for storing text encoding features include but are not limited to: local vector index library, cloud vector database, and Elasticsearch.

[0121] S130 , performing similarity calculation on the M text encoding features through a clustering network to generate N clusters.

[0122] Each cluster includes K text blocks, and the similarity of the text encoding features corresponding to any two text blocks in the same cluster is greater than or equal to a similarity threshold, M≥N≥1, K>1.

[0123] It can be understood that M text encoding features are input into the clustering network, and the M text blocks are classified by the clustering network to generate N clusters. The clustering network is a trained network that can calculate the similarity of the M text encoding features and classify the M text blocks according to the calculated similarity results. M text encoding features are input into the clustering network, and the clustering network calculates the similarity of the M text encoding features to obtain L similarity results (L = M × (M-1) / 2); based on the L similarity results, the M text encoding features are grouped to obtain N encoding feature groups; based on the N encoding feature groups, the M text blocks are grouped to obtain N clusters.

[0124] S140 , generating N data lineage graphs based on the N clusters and the M time identification information.

[0125] The data lineage graph is used to represent the association relationship between K time identification information corresponding to K text blocks belonging to the same cluster.

[0126] It can be understood that a corresponding data lineage graph is generated for each cluster. In each cluster, the text blocks belonging to the cluster are represented by nodes, and the text block nodes belonging to the same cluster are sorted in the order of time identification information. The sorted text block nodes are connected by edges, and the text block nodes with smaller time identification information point to the text block nodes with larger time identification information.

[0127] For example, assume there are 6 text blocks (M=6). Among them, text block 1 is: "The Key Management Service (KMS) is a security management service that uses a third-party certified hardware security module (HSM) to generate and protect keys." Text block 2 is: "The Key Management System helps users easily create and manage keys, meeting the key management needs of users across multiple applications and businesses, and helping users implement compliance requirements." Text block 3 is: "The Key Management System is a security management service that uses a third-party certified hardware security module (HSM) to create and protect keys." Text block 4 is: "The Key Management Service (KMS) is a security management service that generates and protects keys using a third-party certified hardware security module (HSM). Text block 5 is: "As a security management service, the Key Management Service (KMS) uses a third-party certified hardware security module (HSM) to generate and protect keys." Text block 6 is: "The Key Management System helps users easily build and manage keys, meeting the key management needs of users across multiple applications and businesses, and promoting users to achieve compliance standards."

[0128] Input these 6 text blocks into the word vector generation network, encode these 6 through the word vector generation network, and obtain 6 text encoding features: the text encoding feature 1 corresponding to text block 1 is: [0.04531337,-0.01646684,0.01226383,···,-0.02124389,-0.02557556,-0.02724724], the text encoding feature 2 corresponding to text block 2 is: [0.04531337,-0.01646684,0.01226383,···,-0.02124389,-0.02557556,-0.02724724], the text encoding feature 3 corresponding to text block 2 is: [0.04531337,-0.01646684,0.01226383,···,-0.02124389,-0.02557556,-0.02724724], the text encoding feature 4 corresponding to text block 2 is: [0.04531337,-0.01646684,0.01226383,···,-0.02124389,-0.02557556,-0.02724724], the text encoding feature 5 corresponding to text block 2 is: [0.04531337,-0.01646684,0.01226383,···, Code feature 2 is: [0.05465307,-0.01805847,0.02885371,···,-0.01930028,-0.00418899,-0.03872178], and the text encoding feature 3 corresponding to text block 3 is: [0.04242388,-0.01511716,0.01020334,···,-0.0306508,- 0.02594302,-0.02818255], the text encoding feature 4 corresponding to text block 4 is: [0.04069821,-0.01605092,0.01149077,···,-0.03230501,-0.02621504,-0.02876401], the text encoding feature 5 corresponding to text block 5 is: [0.04069821,-0.01 605092,0.01149077,···,-0.03230501,-0.02621504,-0.02876401], the text encoding feature 6 corresponding to text block 6 is: [0.05468520,-0.01809763,0.02887462,···,-0.01931156-0.00419746,-0.03874539].

[0129] The similarity values ​​of the six text encoding features are calculated pairwise to obtain 15 similarity values. According to the 15 similarity values, the six text blocks are divided into two clusters, where the first cluster includes text block 1, text block 3, text block 4 and text block 5, and the second cluster includes text block 2 and text block 6.

[0130] For the first cluster including text block 1, text block 3, text block 4 and text block 5, generate text block 1 node, text block 3 node, text block 4 node and text block 5 node. According to the time identification information corresponding to the four text blocks, arrange these four text blocks in chronological order to obtain: text block 3 node, text block 5 node, text block 1 node, text block 4 node, and generate the edge relationship of these four text blocks to generate a data lineage graph, such as Figure 3 shown.

[0131] The method provided in the embodiment of the present application can realize data lineage analysis of unstructured text information by segmenting and encoding the text information to be processed, which can cover the usage scenarios of most user data, reduce data security risks, and improve data security protection capabilities; using word vector generation networks and clustering networks, the present application can more accurately calculate the similarity of text encoding features, generate more meaningful clusters, and thus provide a more accurate data lineage graph; generating a data lineage graph based on clusters and text attributes can clearly display the association relationship between text blocks belonging to the same cluster, helping users better understand the ins and outs of text data.

[0132] In this application Figure 2 In an optional embodiment of the information processing method provided by the corresponding embodiment, the word vector generation network includes a semantic feature extraction layer and a feature embedding layer; see Figure 4 , step S120 further includes sub-steps S121 to S122. Specifically:

[0133] S121. Input M text blocks into the semantic feature extraction layer in the word vector generation network, and generate M text semantic features through the semantic feature extraction layer.

[0134] It's understandable that in order to subsequently identify similarities among the M text blocks, these blocks need to be semantically identified and encoded. The word embedding network can employ a Sentence Transformer model, which aims to vectorize each input text block. This vectorized representation is a digitized vector that characterizes the semantic features of the text block.

[0135] Furthermore, the word vector generation network includes a semantic feature extraction layer and a feature embedding layer. First, M text blocks are input into the semantic feature extraction layer, and the semantic feature extraction layer extracts M semantic information corresponding to the M text blocks to obtain M text semantic features. The function of the semantic feature extraction layer is to extract meaningful semantic features from the text blocks. Specifically, the semantic feature extraction layer may use natural language processing technology to analyze the text. It may perform lexical analysis, syntactic analysis, or semantic analysis on the text to extract key information, such as features of words, parts of speech, syntactic structure, and themes. By processing the M text blocks, the semantic feature extraction layer generates M text semantic features. These semantic features are abstract representations of the meaning of the text block. They capture the key information in the text for subsequent processing and analysis. For example, the semantic feature extraction layer may represent each text block as a vector, where each dimension of the vector may represent a different semantic feature.

[0136] S122. Input the M text semantic features into the feature embedding layer in the word vector generation network, and generate M text encoding features through the feature embedding layer.

[0137] It can be understood that the M text semantic features output by the semantic feature extraction layer are input into the feature embedding layer, where they are embedded and represented digitally as vectors, generating M text encoding features. The function of the feature embedding layer is to map the input text semantic features into continuous low-dimensional vectors, which are called text encoding features. Specifically, the feature embedding layer may use a neural network or other machine learning algorithm to process the input text semantic features. During training, the feature embedding layer learns how to map text semantic features into low-dimensional vectors that can effectively represent the meaning of the text. Through the processing of the feature embedding layer, each text semantic feature is mapped into a low-dimensional vector, which constitutes the M text encoding features. These text encoding features are compact representations of text semantic features, capturing key information in the text and having a low dimensionality, which facilitates subsequent processing and analysis. For example, the feature embedding layer may represent each text semantic feature as a 50-dimensional vector, where each dimension represents a certain aspect of the text semantic feature. These vectors can be used to compare and measure the similarity or difference between different texts.

[0138] For easier understanding, see Figure 5 , Figure 5 : This is a schematic diagram of the word vector generation network processing process provided by the embodiment of the present application. Taking text block 1 as an example, text block 1 is: "Key Management Service (KMS) is a security management service that uses a third-party certified hardware security module HSM (Hardware Security Module) to generate and protect keys." Input text block 1 into the word vector generation network generated by the Sentence Transformers model. First, the semantic information of text block 1 is identified by the semantic feature extraction layer in the word vector generation network to obtain text semantic features; then, the text semantic features are input into the feature embedding layer, and the text semantic features are digitally vectorized by the feature embedding layer to obtain the text encoding features corresponding to text block 1: [0.04531337, -0.01646684, 0.01226383, ···, -0.02124389, -0.02557556, -0.02724724].

[0139] The method provided in the embodiment of the present application converts text blocks into text semantic features and text encoding features through a semantic feature extraction layer and a feature embedding layer, thereby realizing semantic representation of the text. This semantic representation method can better capture the meaning of the text and provide more meaningful information for subsequent processing and analysis; mapping text semantic features to low-dimensional text encoding features achieves information compression and simplification. This can reduce the dimension and complexity of the data, improve computing efficiency and processing speed; by training the word vector generation network, the learned feature embedding layer can perform universal semantic representation for different texts. This helps to improve the generalization ability of the model, enabling it to process a variety of different texts.

[0140] In this application Figure 2 In an optional embodiment of the information processing method provided in the corresponding embodiment, please refer to Figure 6 , step S130 further includes sub-steps S131 to S133. Specifically:

[0141] S131. Input the M text encoding features into a clustering network, and calculate the cosine similarity between any two of the M text encoding features through the clustering network to obtain L cosine similarity values.

[0142] Wherein, L=M×(M-1) / 2.

[0143] As can be understood, first, M text encoding features are input into the clustering network. A clustering network is a model or algorithm used to group or cluster data. Next, the clustering network is used to calculate the cosine similarity between each of the M text encoding features. Cosine similarity is a commonly used similarity metric that measures the cosine of the angle between two vectors. Specifically, it can be calculated by taking the dot product of two vectors and then dividing it by the product of their norms. Since the cosine similarity between two pairs is to be calculated, M comparisons are required. For the i-th text encoding feature and the j-th text encoding feature, the cosine similarity value is calculated between them. Finally, L cosine similarity values ​​are obtained. The calculation formula for L is L = M × (M-1) / 2. This is because for M text encoding features, the number of possible combinations between each pair is M × (M-1). However, since each comparison produces two similarity values ​​(one for comparing i and j, and the other for comparing j and i), it is necessary to divide by 2 to obtain the actual number of cosine similarity values, L. The L cosine similarity values ​​calculated reflect the degree of similarity between each text encoding feature and other text encoding features. These similarity values ​​can be used in subsequent grouping or clustering operations to group similar text encoding features together. This calculation process aims to discover relationships and patterns between texts by comparing the similarities between text encoding features.

[0144] S132. Group the M text encoding features according to the L cosine similarity values ​​to obtain N encoding feature groups.

[0145] Among them, the cosine similarity value of each pair of text encoding features belonging to the same encoding feature group is greater than or equal to the similarity threshold.

[0146] It can be understood that, according to L cosine similarity values, M text encoding features are grouped to obtain N encoding feature groups. This means that the M text encoding features are divided into different groups according to the cosine similarity value. Specifically, the cosine similarity value of each two text encoding features belonging to the same encoding feature group is greater than or equal to the similarity threshold. This similarity threshold is a key parameter that determines which text encoding features will be classified into the same group. If the cosine similarity value of two text encoding features is greater than or equal to the similarity threshold, then they are considered to be similar and are divided into the same encoding feature group. If the cosine similarity value is less than the similarity threshold, then they are divided into different encoding feature groups. In this way, the text encoding features can be grouped according to similarity so that the text encoding features belonging to the same group have a higher similarity to a certain extent. For example, if the similarity threshold is set to 0.8, then for a text encoding feature A, if its cosine similarity value with other text encoding features is greater than or equal to 0.8, then it will be divided into one encoding feature group; and if its cosine similarity value with some text encoding features is less than 0.8, then it will be divided into different encoding feature groups.

[0147] S133. Group the M text blocks according to the N coding feature groups to obtain N clusters.

[0148] It can be understood that this step is the process of associating text blocks with coding feature groups. Each text block will be assigned to a corresponding cluster according to its associated coding feature group. Specifically, each text block will be assigned to a cluster corresponding to the same coding feature group to which its coding feature belongs. In this way, text blocks belonging to the same cluster have similarities in coding features. In this way, M text blocks are grouped into N clusters, and the text blocks in each cluster have similar features or meanings to some extent. This method of grouping text blocks according to coding feature groups can help to effectively organize and analyze large amounts of text data. Clusters can be used to subsequently generate data lineage graphs.

[0149] For example, see Figure 7 , Figure 7Schematic diagram of the clustering process provided by the embodiment of the present application. For the six text encoding features corresponding to the six text blocks: the text encoding feature 1 corresponding to text block 1 is: [0.04531337, -0.01646684, 0.01226383, ···, -0.02124389, -0.02557556, -0.02724724], the text encoding feature 2 corresponding to text block 2 is: [0.05465307, -0. 01805847,0.02885371,···,-0.01930028,-0.00418899,-0.03872178], the text encoding feature 3 corresponding to text block 3 is: [0.04242388,-0.01511716,0.01020334,···,-0.0306508,-0.02594302,-0. 02818255], the text encoding feature 4 corresponding to text block 4 is: [0.04069821,-0.01605092,0.01149077,···,-0.03230501,-0.02621504,-0.02876401], the text encoding feature 5 corresponding to text block 5 is: [0.04069821,-0.01605092, 0.01149077,···,-0.03230501,-0.02621504,-0.02876401], and the text encoding feature 6 corresponding to text block 6 is: [0.05468520,-0.01809763,0.02887462,···,-0.01931156-0.00419746,-0.03874539].

[0150] The similarity values ​​of the six text encoding features are calculated for each pair, resulting in 15 similarity values. The six text encoding features are then grouped according to the 15 similarity values ​​to obtain two encoding feature groups. The six text blocks are then clustered according to the two encoding feature groups to obtain two clusters. The first cluster includes text blocks 1, 3, 4, and 5, and the second cluster includes text blocks 2 and 6. The text blocks in each cluster have a data lineage relationship.

[0151] The method provided in the embodiment of the present application can discover the degree of similarity between texts by calculating the cosine similarity between text encoding features, which helps to identify texts with similar themes or content and provides a basis for the subsequent generation of data lineage graphs; grouping text encoding features according to a similarity threshold so that similar text encoding features are grouped together. This grouping method can better organize and manage text data and facilitate further analysis and application; and grouping text blocks according to the encoding feature groups to obtain clusters. The formation of clusters can help discover patterns and structures in text data.

[0152] In this application Figure 2 In an optional embodiment of the information processing method provided in the corresponding embodiment, please refer to Figure 8 , step S140 further includes sub-steps S141 to S143. Specifically:

[0153] For each of the N clusters, execute:

[0154] S141 . Generate K text block nodes according to the K text blocks in each cluster.

[0155] It can be understood that for each cluster, the text blocks within each cluster are abstracted into text block nodes. Nodes can be considered abstract entities representing text blocks. Each node is given a unique name or identifier so that it can be distinguished and referenced in subsequent steps. By converting each text block into a node, the text content within the cluster can be discretized.

[0156] S142 : Establish edge relationships between the K text block nodes according to the K time identification information corresponding to the K text blocks in each cluster.

[0157] It can be understood that the step of establishing edge relationships between the K text block nodes is based on the K time stamp information corresponding to the K text blocks in each cluster. This means that the association between the text blocks is determined based on their time stamp information. Specifically, the time stamp information can be a timestamp, sequence number, or other information indicating a temporal order associated with each text block. These time stamps can be used to determine the relative order or temporal precedence between the text blocks. The following are possible ways to establish edge relationships: 1) Sequential relationship: If the time stamp information indicates the order in which the text blocks were generated, then edges can be established based on this order. For example, if the time stamp of text block 1 is earlier than that of text block 2, then a directed edge can be established between text block 1 and text block 2, from text block 1 to text block 2, indicating that text block 1 precedes text block 2 in time. 2) Dependency relationship: If the time stamp information reflects a dependency relationship between text blocks, such as if one text block is generated based on another, then a corresponding edge relationship can be established. This edge relationship can represent a causal or dependency relationship between text blocks. 3) Concurrent Relationships: If the time stamp information indicates that certain text blocks were generated simultaneously or are related, undirected edges can be established between these blocks to represent their connection. The purpose of establishing edge relationships is to accurately reflect the chronological order or other relationships between text blocks in the data lineage graph. Such edge relationships help understand the logical relationships between text blocks and their evolution over time.

[0158] S143. Generate a data lineage graph corresponding to the clusters according to the K text block nodes and the edge relationships between the K text block nodes.

[0159] It is understood that the step of generating a corresponding data lineage graph based on the K text block nodes in the clusters and the edge relationships between them aims to transform the text block nodes and edge relationships in the clusters into a visual data lineage graph, thereby better understanding and analyzing the relationships between text blocks. Specifically, a data lineage graph is a visual graphical representation used to display the relationships and flows between data. In a data lineage graph, nodes represent data entities, and edges represent the relationships between data entities. By mapping the text block nodes and edge relationships in the clusters into the data lineage graph, an intuitive graphical representation can be created to demonstrate the relationships and flows between text blocks. The following are the general steps for generating a data lineage graph: Node drawing: Based on the K text block nodes in the clusters, corresponding nodes are drawn in the data lineage graph. Each node can represent a text block, typically represented by a graphical symbol or label. Edge drawing: Based on the edge relationships in the clusters, corresponding edges are drawn in the data lineage graph. Edges can represent relationships between text blocks, typically represented by arrows or lines. The direction and style of the edge relationship can be determined by the type of edge relationship (e.g., sequential, dependency, or simultaneous). Layout optimization: In order to make the data lineage graph clearer and easier to understand, layout optimization can be performed. The purpose of layout optimization is to arrange the nodes in the graph reasonably according to the relationship between nodes and edges, and avoid the intersection and overlap of edges. Common layout algorithms include force-directed layout, hierarchical layout, etc. Visual display: The generated data lineage graph is visualized. Graphical interface tools or visualization libraries can be used to create and display data lineage graphs. By observing the data lineage graph, users can understand the relationship and flow between text blocks, so as to better understand and analyze the text data in the clusters. By generating a data lineage graph, users can intuitively understand the relationship and flow between text blocks in the clusters. The data lineage graph can help users discover the dependencies, sequence and other associations between text blocks, so as to better understand the generation and evolution process of text data.

[0160] For example, based on the cluster analysis results, text block nodes are created for text blocks in the same cluster. At the same time, edge relationships are established between nodes based on the modification time sequence of the files to which the text blocks belong. The attributes of the text block nodes include user, file path, file name, number of characters, creation time, and modification time. The attributes of the edges include update time. The schematic diagram of the data lineage graph is as follows: Figure 9 shown.

[0161] The method provided in the embodiment of the present application abstracts text blocks into nodes, establishes connections between text blocks through edge relationships, and generates a data lineage graph. This visual display method makes the relationship between text blocks more intuitive and clear, which is convenient for users to understand and analyze; by establishing edge relationships, the time sequence, dependency or simultaneous relationship between text blocks can be clarified, which helps users better understand the logical relationship between text blocks and discover potential patterns and associations; the data lineage graph provides a powerful tool for the analysis of text data. Users can observe the nodes and edge relationships in the graph to deeply study the flow and evolution process of text blocks, thereby obtaining valuable insights and information.

[0162] In this application Figure 8 In an optional embodiment of the information processing method provided in the corresponding embodiment, please refer to Figure 10 , step S142 further includes sub-steps S1421 to S1423. Specifically:

[0163] S1421 , arranging the K time identification information corresponding to the K text blocks in each cluster in ascending order.

[0164] It is understandable that the time identification information can be a timestamp, date, time sequence, etc. associated with each text block. Arranging in ascending order means arranging these time identification information in order from small to large. The purpose of doing this is to organize the text blocks in chronological order for subsequent processing or analysis. By arranging in ascending order, it can be ensured that the text blocks in each cluster are ordered in time. For example, if the time identification information is a timestamp, then the ascending order will sort the text blocks in chronological order. This helps to better understand the temporal relationship between text blocks, such as which text block was generated first and which text block was generated later. By arranging the time identification information in ascending order, the temporal sequence of the text blocks can be more intuitively observed and analyzed.

[0165] S1422. Sort the K text block nodes according to the K time identification information arranged in ascending order.

[0166] It is understandable that the time identification information has been arranged in ascending order in step S1421. Now, it is necessary to sort the corresponding text block nodes according to these arranged time identification information. The sorting process can be understood as associating the text block nodes with their corresponding time identification information and rearranging the text block nodes in the ascending order of the time identification information. The purpose of doing so is to organize the text block nodes in chronological order for subsequent processing or analysis. By sorting the text block nodes according to the time identification information, it is possible to ensure that the text blocks in each cluster are ordered in time. By sorting the text block nodes, the chronological order of the text blocks can be observed and analyzed more intuitively.

[0167] S1423. Establish edge relationships between the K text block nodes based on the sorted K text block nodes.

[0168] It can be understood that, based on the sorted K text block nodes, edge relationships are established between the K text block nodes. The sorted text block node list is traversed. For each text block node i, its sorted position i is found. For text block node i, among the nodes preceding position i, the node j closest to i is found. An edge relationship is established between text block nodes i and j. This process is repeated until all sorted text block nodes have been traversed. Through this process, edge relationships can be established between the sorted text block nodes.

[0169] The method provided in the embodiment of the present application arranges the time identification information in ascending order and sorts the text block nodes according to the sorted time identification information, so that the text block nodes are organized in chronological order, which helps to better understand the temporal relationship between text blocks, such as the order in which events occur; establishing edge relationships between text block nodes can form a data lineage graph, which clearly shows the relationship between text blocks.

[0170] In this application Figure 2 In an optional embodiment of the information processing method provided in the corresponding embodiment, please refer to Figure 11 , step S110 further includes sub-steps S111 to S115. Specifically:

[0171] S111. Obtain text information to be processed.

[0172] It is understandable that the text information may be composed of at least one text data, and the text data may be an article or a text paragraph. Each text data corresponds to its text attribute, and the text attribute includes the creation time of the text.

[0173] S112: Perform target identifier recognition on the text information to be processed to obtain P target identifiers.

[0174] Among them, P≥M.

[0175] It is understood that the target identifier can be a line break, a preset punctuation mark or a preset character, which plays a role in separating or identifying different parts in the text. These target identifiers can be found by analyzing and identifying the text information.

[0176] S113. Segment the text information to be processed according to the P target identifiers to obtain P candidate text blocks.

[0177] It can be understood that the text information is divided into multiple parts according to the position of the target identifier. The position of each target identifier is a segmentation point, which divides the text information into different paragraphs or blocks. In this way, by segmenting according to the target identifier, P candidate text blocks are obtained. Each candidate text block is a relatively independent part of the text information, separated by the target identifier. For example, if the target identifier is a line break, then the text information can be divided into individual lines according to the position of the line break; if the target identifier is a specific punctuation mark, then the text information may be divided into blocks separated by these punctuation marks. The P candidate text blocks obtained may represent different paragraphs, sentences, topic paragraphs, etc. in the text information. These candidate text blocks will serve as the basic units for subsequent processing.

[0178] S114 , performing word count on the P candidate text blocks to obtain P word count information corresponding to the P candidate text blocks.

[0179] It is understandable that the number of characters contained in each candidate text block is calculated to obtain the corresponding word count information. The word count information is used to describe the length or size of each candidate text block. By counting the number of words, the specific size of each candidate text block can be understood. For example, a candidate text block may contain 100 characters, another candidate text block may contain 200 characters, and so on. In this way, P word count information corresponding to P candidate text blocks is obtained. Each word count information corresponds to a specific candidate text block. This word count information can be used in subsequent steps. For example, according to preset conditions or requirements, candidate text blocks that meet a certain word count range can be screened out, or candidate text blocks can be classified or sorted according to the word count information. The purpose of word count statistics is to obtain the specific size information of each candidate text block for further processing or analysis. It can help understand the characteristics of the text block, such as length, complexity, etc.

[0180] S115 . Determine M text blocks from the P candidate text blocks based on the P word count information.

[0181] The word count information corresponding to each text block in the M text blocks is greater than or equal to a preset word count.

[0182] It will be appreciated that, based on the previously obtained P word count information, M text blocks are determined from the P candidate text blocks. The key condition here is that the word count information corresponding to each determined text block must be greater than or equal to a preset word count. This means that only text blocks that meet a certain word count requirement are selected. Specifically, based on a preset word count standard, text blocks with word count information greater than or equal to the standard are filtered out from the P candidate text blocks. These text blocks that meet the requirements are then determined as the M text blocks. The M text blocks are selected based on the word count information. This may be based on specific needs or application scenarios. For example, in some cases, only longer or more complex text blocks may be of interest. Therefore, by setting a preset word count, these text blocks of a certain size can be selected. Alternatively, based on the word count information, text blocks that are too short or do not meet specific requirements can be excluded, allowing for greater focus on analyzing and processing the important text content. The process of determining the M text blocks may involve comparison and judgment. The word count information of each candidate text block is checked one by one, and those that meet the requirements are marked or selected.

[0183] For example, see Figure 12 , Figure 12 It is a schematic diagram of text information segmentation provided in an embodiment of the present application. First, obtain the text information "Key Management Service (KMS) is a security management service that uses a third-party certified hardware security module HSM (Hardware Security Module) to generate and protect keys. The key management system helps users easily create and manage keys, meet the key management needs of users for multiple applications and multiple businesses, and help users implement compliance requirements." Then, identify the target identifier of the text information, and the target identifier includes: "\n" (line break), "." (period), "?" (question mark), "!" (exclamation mark), "······" (ellipsis). The text information is segmented according to the identified target identifier to obtain two candidate text blocks: Candidate text block 1 is "Key Management Service (KMS) is a security management service that uses a third-party certified hardware security module HSM (Hardware Security Module) Module) to generate and protect keys." Candidate text block 2 is "The key management system helps users easily create and manage keys, meeting the key management needs of users across multiple applications and businesses, and helping users implement compliance requirements." Then, the number of characters in these two candidate text blocks is counted, and candidate text blocks that do not meet the preset character count are removed. Since both candidate text blocks meet the preset character count, they are retained and become the text blocks.

[0184] The method provided in the embodiments of the present application, by acquiring text information and performing target identifier identification and segmentation, can decompose original text data into smaller, easier-to-process candidate text blocks, which helps to better understand and analyze the structure and content of the text; count the words in the candidate text blocks, and determine M text blocks based on a preset word count, so that text blocks of a specific length or importance can be screened out, which helps focus on key information and reduce redundant and unnecessary text processing; determine the text blocks based on word count information, improving the accuracy and specificity of the selection. This makes subsequent processing and analysis more efficient and meaningful.

[0185] In this application Figure 11 In an optional embodiment of the information processing method provided in the corresponding embodiment, please refer to Figure 13 , sub-step S111 further includes sub-steps S1111 to S1114. Specifically:

[0186] S1111. Receive file path information and target file format.

[0187] It can be understood that the file path information and target file format sent by the user through the terminal device can be received, and the text information that needs to be processed can be obtained based on the file path information and the target file format. The file path information is used to indicate the location of the file in the system, and the target file format is used to indicate the file type or format expected by the user.

[0188] S1112: Traverse the file path information to obtain at least one candidate file.

[0189] It is understandable that, according to the file path information provided, traverse the possible files. In this way, one or more candidate files relevant to the path can be found.

[0190] S1113. Determine, according to the target file format, at least one target file from at least one candidate file, whose file format satisfies the target file format.

[0191] It is understandable that based on the target file format requirement, candidate files whose file formats match the target format are screened out, and only files that meet the format requirement are determined as target files.

[0192] S1114. Parse at least one target file according to the target file format to read the content of the at least one target file to obtain text information to be processed.

[0193] It is understood that, depending on the format of the target file, the selected target file is parsed, which may involve opening the file, reading the content therein, and converting it into text information.

[0194] Exemplarily, the user specifies the file path to be analyzed or the name of the data bucket where COS is stored on the cloud. The file path traversal step will perform a traversal operation based on the path and data bucket name provided by the user, obtain all files in the target format, and form a file list. Next, the file format parsing step reads the files in sequence according to the file list, and parses them according to the corresponding file format to read the text content in the file. The supported file formats include common file formats such as docx, pptx, json, txt, markdown, etc. The supported file formats are dynamically extensible and dynamically loaded in the form of plug-ins. For example, the file format parsing step reads a docx file, calls the parsing plug-in, obtains the text content in the file, and obtains the text information.

[0195] The method provided in the embodiment of the present application, by receiving the file path and the target file format, the system can quickly locate and filter out files that meet the requirements, thereby improving the efficiency of file processing; filtering is performed according to the target file format to ensure that the acquired files meet the user's expectations, avoiding unnecessary errors or misunderstandings; supports multiple common file formats and has the ability to dynamically expand, so that the system can adapt to different file types and requirements and has better flexibility; parses the target file and reads the text content, so that the user can easily obtain the required information, providing a basis for further processing or analysis.

[0196] In this application Figure 13 In an optional embodiment of the information processing method provided in the corresponding embodiment, please refer to Figure 14 Sub-step S115 is followed by sub-steps S116 to S118. Specifically:

[0197] S116. Determine at least one file name information corresponding to the M text blocks according to the file path information.

[0198] As will be appreciated, in order to associate text blocks with the files they belong to, the file path information can be used to locate the specific file corresponding to each text block and obtain its file name. File path information generally refers to a description of the path or location leading to a specific file. File path information can be used to determine the file's storage location within a computer system. By parsing the file path information, files associated with that path can be found. Searching and identifying the specific file corresponding to each text block is performed based on the file path information. This involves file system operations such as traversing the folder structure, searching for specific files, or matching files using known file naming conventions. Determining at least one file name means finding one or more files associated with the text block. This depends on the specific content of the file path information and the system's search capabilities. If the file path information is accurate and unique, a single corresponding file name can usually be accurately determined. However, if the file path is ambiguous or has multiple possible matches, multiple file names may be returned. Determining the file name information allows a text block to be associated with a specific file. This is very useful for further processing, analysis, or manipulation. For example, different documents, file types, or sources can be identified based on the file name, allowing for more targeted processing.

[0199] S117 , generating M pieces of attribute information according to the M word count information corresponding to the M text blocks, the at least one file name information corresponding to the M text blocks, and the M time identification information corresponding to the M text blocks.

[0200] It will be appreciated that the word count information, file name information, and timestamp information of a text block are combined to form attribute information for each text block. This attribute information can be used for further analysis, processing, or storage. The word count information indicates the length or size of each text block, which can reflect the content volume or complexity of the text block. The file name information provides the name of the specific file associated with the text block, which can be used to identify the file or source to which the text block belongs. The timestamp information may indicate the creation time, modification time, or other time-related information of each text block, which can be used to understand the chronological order or timestamp of the text blocks. By combining these different information, attribute information corresponding to each text block is generated. This attribute information can be a comprehensive description or representation of the characteristics of the text block. For example, attribute information can include the word count, file name, timestamp, etc. The purpose of generating attribute information is to give each text block more characteristics and descriptions to facilitate further processing, analysis, or classification.

[0201] S118. Store M pieces of attribute information.

[0202] It is understood that the generated M attribute information is stored for subsequent use or further processing, and can be stored in a database, file, or other suitable storage medium for retrieval and use when needed.

[0203] For example, see Figure 15 , Figure 15 This is a flowchart for text information preprocessing provided by an embodiment of the present application. First, the user specifies the file path to be analyzed or the name of the data bucket where COS is stored on the cloud. The file path traversal step performs a traversal operation based on the user-provided path and data bucket name, obtaining all files in the target format and forming a file list. Next, the file format parsing step sequentially reads the files from the file list and parses them according to the corresponding file format to read the text content within them. Supported file formats include common file formats such as docx, pptx, json, txt, and markdown. Supported file formats are dynamically extensible and can be dynamically loaded as plug-ins. For example, the file format parsing step reads a docx file and calls a parsing plug-in to obtain the text content within the file. Then, the text block segmentation step segments the text content into text blocks of a certain character size. Segmentation is performed based on a certain character count, with text blocks segmented into pre-set sizes. For example, text blocks with 200 characters are segmented. The step for removing text blocks with substandard Chinese characters counts the number of Chinese characters in a text block. If the number does not reach a pre-set threshold, the text block is discarded. Finally, the step of recording text block attributes records information such as the file name, number of characters, etc. to which a text block belongs as attribute information of the text block.

[0204] The method provided in the embodiment of the present application provides a clearer source and contextual information for the text block by associating it with the file to which it belongs. This is very helpful for subsequent analysis and processing, such as classification based on file type or specific processing based on file source. The attribute information generated by combining word count information, file name information, and time stamp information provides a more comprehensive description of the text block. This attribute information can be used for various purposes, such as data analysis, text classification, and search, increasing the understanding and utilization value of the text block. The attribute information is stored to facilitate subsequent management and retrieval of the text block. The required text block can be quickly found based on specific attribute conditions, improving data availability and efficiency.

[0205] In this application Figure 2 In an optional embodiment of the information processing method provided in the corresponding embodiment, please refer to Figure 16 , after step S140, further steps S210 to S220 are included. Specifically:

[0206] S210: Receive query text.

[0207] It is understandable that the query text input by the user is received. The query text can be specific content that the user wants to search for or understand, such as a keyword, a sentence or a text.

[0208] S220: Determine a data lineage graph containing the query text according to the query text, and return the data lineage graph containing the query text.

[0209] It is understandable that, based on the received query text, the lineage graph containing the query text is determined among the N data lineage graphs generated previously. The specific determination method may involve comparing or matching the query text with the text blocks in the data lineage graph. By finding the data lineage graph containing the query text, the system can return the lineage graph to the user. The data lineage graph graphically displays the time identification information association relationship between text blocks belonging to the same cluster. In this way, the user can understand the text blocks related to the query text and their time sequence relationship by observing the data lineage graph.

[0210] The method provided in the embodiment of the present application is to help users quickly locate and obtain information related to the query text, and present it in an intuitive manner, so that users can further analyze and understand the relationship between text data.

[0211] In this application Figure 2 In an optional embodiment of the information processing method provided in the corresponding embodiment, please refer to Figure 17 The information processing method further includes a process of training the word vector generation network, including steps S310 to S330. Specifically:

[0212] S310: Obtain a first sample data set.

[0213] Among them, the first sample data set includes at least one group of first sample data, each group of first sample data includes a first training sample text, a first positive sample text and a first negative sample text, the semantic information of the first training sample text is the same as the semantic information of the first positive sample text, and the semantic information of the first training sample text is different from the semantic information of the first negative sample text.

[0214] It is understood that the first sample dataset includes at least one set of first sample data, each of which consists of three parts: a first training sample text, a first positive sample text, and a first negative sample text. The first training sample text and the first positive sample text share the same semantic information, while the first training sample text and the first negative sample text have different semantic information. This design allows the word embedding network to learn how to distinguish between texts with similar and different semantics. Through this design, the first sample dataset provides rich information to help the word embedding network learn and understand the semantics of text. During training, the network continuously adjusts its parameters based on the similarities between the first training sample text and the first positive sample text, as well as the differences between the first training sample text and the first negative sample text, to improve its ability to represent text semantics. For example, if the first training sample text is "cats like to eat fish," then the first positive sample text might be "cats love to eat fish," while the first negative sample text might be "dogs like to eat fish." During training, the network learns the semantic similarities between the words "cat" and "fish," as well as the semantic differences between "cat" and "dog." This sample data design helps the network better capture the semantic relationships between texts, thereby improving the quality and accuracy of word vector generation. By training on multiple sets of such sample data, the network can gradually learn a wider range of semantic knowledge, providing more valuable information for subsequent text processing and analysis tasks. The ratio of the number of first training sample texts to the number of first positive sample texts is set to 1:1, and the ratio of the number of first training sample texts to the number of first negative sample texts is set to 1:3.

[0215] S320: Obtain word vectors to generate an initial network.

[0216] It's understood that the initial network for generating word vectors can be an existing model or a new network trained from scratch. The initial network has some default parameters that will be adjusted during training. The initial network for generating word vectors can use a Sentence Transformer-like model architecture. This model is typically trained on large-scale corpora and can be used for a variety of natural language processing tasks, such as text classification, sentiment analysis, and machine translation.

[0217] S330: Train the initial network for generating word vectors according to the first sample data set to adjust network parameters of the initial network for generating word vectors, and obtain a trained network for generating word vectors.

[0218] It is understandable that the first sample data set is used to train the initial network for generating word vectors. Through training, the network will learn how to generate corresponding word vectors based on the input text.

[0219] Specifically, the network will adjust its own parameters based on the comparison between the first training sample text and the first positive sample text, so that the generated word vector can better reflect the semantic information of the text. At the same time, the network will also learn how to distinguish between the first training sample text and the first negative sample text to avoid mapping texts with different semantics into similar word vector spaces. Through continuous iterative training, the performance of the word vector generation network will gradually improve, and eventually a trained word vector generation network will be obtained. The purpose of this training process is to enable the word vector generation network to accurately capture the semantic information of the text and to distinguish texts with different semantics. In this way, in practical applications, the word vector generation network can generate accurate word vectors based on the input text, providing strong support for subsequent processing and analysis.

[0220] During the training process, each set of first sample data in the first sample dataset (including the first training sample text, the first positive sample text, and the first negative sample text) is input into the initial network for word vector generation. The network performs calculations and processing based on the input text data, attempting to learn and understand the semantic relationships between the texts. By interacting with the first sample dataset, the network adjusts its parameters based on the feedback from the training data. These parameters may include connection weights in the neural network, activation function parameters, the number of layers, and so on. The purpose of adjusting parameters is to enable the network to better capture the semantic patterns and features in the first sample dataset. Through continuous training and parameter adjustment, the initial network for word vector generation gradually learns how to generate more accurate and meaningful word vectors based on the input text. These word vectors can be used in subsequent natural language processing tasks such as text classification, semantic similarity calculation, and information retrieval. During the training process, optimization algorithms such as stochastic gradient descent (SGD) or its variants are typically used to update the network parameters. These algorithms calculate the gradients of the parameters based on the loss function of the training data (such as cross-entropy loss or mean squared error) and update the parameters in a direction that reduces the loss. Ultimately, after sufficient training iterations, a trained word vector generation network is obtained. This network has learned some useful knowledge and patterns through interaction with the first sample dataset and can be used to generate word vectors related to the first sample dataset. It should be noted that the training process may require a certain amount of time and computing resources, depending on the complexity of the network, the size of the sample dataset, and the choice of training algorithm.

[0221] For example, see Figure 18 , Figure 18This is a schematic diagram of the word vector generation network training process provided by an embodiment of the present application. First, prepare a training data set, which consists of positive and negative sentence pairs. For a first training sample text, construct sentences with similar semantic information as the first positive sample text, and sentences with dissimilar semantic information as negative samples. An example of a set of first sample data is as follows: if the first training sample text is "cats like to eat fish", then the first positive sample text may be "cats love to eat fish", and the first negative sample text may be "dogs like to eat fish", "big fish like to eat small fish", or "cats love to catch mice". Then, set the various parameters for word vector generation network training. The epoch parameter indicates the number of times the training is performed on the entire data set, and the batch size parameter indicates the size of the batch data input during training. The Temperature parameter is used to control the distribution of the vector space. After the parameters are set, start training the word vector generation network. Use the test data set to test the performance of the model. If it does not meet the standard, repeat the above training process; if it meets the standard, output the trained word vector generation network. The parameter setting uses a grid search strategy to test different parameter combinations until the parameters with optimal performance are obtained.

[0222] The method provided in the embodiment of the present application, by training the initial network for generating word vectors, the network can learn word vectors that more accurately represent the semantics of the text, and such word vectors can provide more valuable information in subsequent natural language processing tasks, such as better performance in text classification, semantic similarity calculation, etc.; using the first sample data set for training can enable the model to learn the differences between different semantic texts, thereby improving the generalization ability of the model, which means that the model can better capture semantic information when processing new texts and reduce the risk of overfitting; by comparing the first training sample text, the first positive sample text and the first negative sample text, the model can better understand the semantic relationship between texts, including similarities and differences, which helps the model make more accurate judgments in tasks such as semantic analysis and information retrieval; training the initial network for generating word vectors according to the first sample data set can adjust the parameters of the network to make it more suitable for processing text data for specific tasks. This can improve the performance and efficiency of the model; by setting different proportions of the first training sample text, the first positive sample text and the first negative sample text, more training data is provided, which helps to enrich the learning experience of the model and further improve the performance of the model.

[0223] In this application Figure 2 In an optional embodiment of the information processing method provided in the corresponding embodiment, please refer to Figure 19 The information processing method further includes a process of training the clustering network, including steps S410 to S440. Specifically:

[0224] S410: Obtain a second sample data set and a second sample data label set.

[0225] The second sample data set includes at least one second training sample text encoding.

[0226] It will be understood that the second sample data set is a set of data used to train the clustering network, including at least one second training sample text encoding. These text encodings can be a representation of text content, such as vectors generated using a word embedding network. The second sample data label set is the label or category information corresponding to the second sample data set. These labels are used to indicate the category or cluster to which each second training sample text belongs. For example, labels can be integers, strings, or other identifiers representing different categories. By obtaining the second sample data set and the second sample data label set, sample texts can be associated with their categories. In this way, when training the clustering network, the network can learn how to predict the corresponding category or cluster based on the text encodings. In practical applications, the quality and representativeness of the second sample data set are crucial to the training effectiveness of the clustering network. If the data set contains sufficiently diverse and representative sample texts, and the label assignments are accurate, the clustering network can better learn the relationship between text and categories during training, thereby improving clustering accuracy. Furthermore, the size and number of the second sample data set can also affect the training effect. Generally speaking, larger data sets provide more information and generalization capabilities, but they also increase training time and computational cost. Therefore, when selecting the second sample dataset, it is necessary to weigh the data quantity and quality based on the specific situation. In addition, the selection of the second training sample text encoding also needs to take into account the characteristics and requirements of the clustering task. Different text encoding methods may have different effects on the clustering results, so it is necessary to select an encoding method that is suitable for the task. In general, the purpose of obtaining the second sample dataset and the second sample data label set is to provide sufficient training data for the clustering network and guide the network to learn the correct clustering pattern through the label information.

[0227] S420: Obtain an initial clustering network.

[0228] It's understandable that the initial clustering network is an untrained network. This network may have some default parameter settings and has not yet learned how to correctly cluster samples. This initial clustering network is typically a predefined model structure with adjustable parameters that are optimized during subsequent training. There are various ways to obtain the initial clustering network. One common approach is to use existing clustering algorithms and model structures, such as density-based clustering, hierarchical clustering, or neural network-based clustering methods. These algorithms and model structures each have specific characteristics and application scenarios, and can be selected based on specific needs and data characteristics. The obtained initial clustering network typically has some default parameter settings, which may affect the network's learning ability and clustering results. For example, the initial clustering network may include parameters such as the number of neurons, connection structure, and activation function. The choice of these parameters affects the network's ability to process input data and extract features. In practical applications, selecting an appropriate initial clustering network is crucial. Different clustering algorithms and model structures may exhibit different performance when processing different types of data. Some clustering algorithms may be more suitable for high-dimensional data, while others may be more effective when handling large datasets. Therefore, when selecting the initial clustering network, factors such as the characteristics of the data, the clustering goals, and requirements need to be considered. Furthermore, after obtaining the initial clustering network, it is often necessary to further adjust and optimize it to suit the specific task and data. This may include fine-tuning network parameters, introducing additional features or constraints, and so on. By continuously adjusting and improving the initial clustering network, the performance of the network and the accuracy of the clustering results can be improved.

[0229] S430: Input the second sample data set into the initial clustering network, and generate a predicted clustering result through the initial clustering network.

[0230] S440: Training the initial clustering network according to the second sample data label set and the predicted clustering result to adjust the network parameters of the initial clustering network to obtain a clustering network.

[0231] It is understood that the predicted clustering results are compared with the second sample data label set. Based on the comparison results, an appropriate training algorithm (such as the backpropagation algorithm) is used to adjust the parameters of the initial clustering network to make the predicted results closer to the true labels. Through continuous iteration and adjustment, the initial clustering network gradually learns how to correctly cluster the samples, ultimately obtaining a trained clustering network.

[0232] For example, see Figure 20 , Figure 20It is a flow chart of the training clustering network provided in an embodiment of the present application. First, a second sample data set is obtained, the initial clustering network is trained using the second sample data set, the eps and min_samples parameters of the initial clustering network are set, and the DBScan model is trained as a clustering model. The eps parameter represents the maximum distance between two data points that will be considered to be in a neighborhood, and the min_samples parameter represents the minimum number of data points that can be clustered into one class. Then, the trained clustering network is tested using a test data set. If the standard is not met, the above training process is repeated; if the standard is met, the trained clustering network is output. The parameter setting uses a grid search strategy to test different parameter combinations until the parameters with optimal performance are obtained.

[0233] The method provided in the embodiment of the present application trains a clustering network to effectively cluster the input text data. By comparing with the second sample data label set and adjusting parameters, the clustering network can learn the inherent structure and pattern of the sample data, thereby improving the accuracy of clustering.

[0234] For ease of understanding, the following Figure 21 and Figure 22 Introducing an information processing method, Figure 21 is a flowchart of the information processing method provided in the embodiment of the present application, Figure 22 This is a schematic diagram of the architecture of the system for executing the information processing method provided in the embodiment of the present application. Figure 21 The information processing method is divided into two stages: data processing stage and data lineage mining. Among them, the data processing stage is jointly implemented by three modules: data preprocessing module, word vector generation module, and word vector storage module; the data lineage mining stage is jointly implemented by three modules: cluster analysis module, data lineage graph construction module, and lineage graph display module. The information processing method provided in the embodiment of the present application inputs the original text information and outputs the data lineage graph generated by the analysis. Specifically: the data preprocessing module is responsible for cleaning and segmenting the original text information to form text blocks. The word vector generation module is responsible for using the word vector generation network to perform vectorization calculation on the text blocks and output the vectorized representation of each text block: text encoding features. The word vector storage module is responsible for storing the generated text encoding features to facilitate retrieval and cluster analysis. The cluster analysis module is responsible for performing cluster analysis on all text encoding features to find text blocks whose similarity exceeds a threshold. Similar text blocks have data lineage relationships. The data lineage graph construction module is responsible for constructing text block nodes, establishing edges between similar text blocks, adding various attributes to the nodes, and forming a data lineage graph of user assets. Finally, the bloodline graph display module is responsible for generating various bloodline graphs based on the query text entered by the user and displaying them on the front end.

[0235] See also Figure 22 First, the file path information and target file format are received. The data preprocessing module reads the file path information and extracts text information from various documents. The filtering strategy is used to clean the text information, remove errors and non-standard text information, retain the text information required by the user, and segment the text information according to the preset number of words to form text blocks. Next, the text block is input into the word vector generation module. The word vector generation network in the word vector generation module vectorizes the text block and generates text encoding features. Then, the text encoding features are stored in the word vector storage module. The word vector storage module performs persistent storage of the text encoding features to facilitate subsequent analysis and reduce the cost of repeated calculations. Then, the text block and text encoding features are input into the clustering module. The clustering network calculates the similarity of the text encoding features. The text blocks are clustered according to the similarity calculation results and cluster clusters are output. Texts belonging to the same cluster have data lineage relationships. Then, the data lineage graph construction module establishes text block nodes and connecting edges between nodes with lineage relationships, and adds attribute data to the nodes and edges. Finally, the user enters the query text, retrieves the relevant nodes and edges, and displays the data lineage graph containing the query text on the front end for the user to view and analyze.

[0236] The method provided in the embodiment of the present application enhances the user's data lineage analysis capability for unstructured text data, thereby improving the data security protection capability for unstructured text data; it can automatically generate a data lineage graph for unstructured data, greatly reducing the pressure of manual analysis and improving analysis efficiency; the word vector generation network can learn the user's professional field text in a targeted manner according to user needs, thereby better performing data lineage analysis on the text in the user's professional field and improving the accuracy of data lineage analysis.

[0237] The information processing device in this application is described in detail below. Figure 23 . Figure 23 This is a schematic diagram of an embodiment of an information processing device 10 in an embodiment of the present application. The information processing device 10 includes: a text block acquisition module 110, a text block encoding module 120, a text block clustering module 130 and a data lineage graph generation module 140;

[0238] The text block acquisition module 110 is configured to acquire M text blocks corresponding to the text information to be processed and M time identification information corresponding to the M text blocks, wherein the M text blocks are obtained by segmenting the text information, and M>1;

[0239] A text block encoding module 120 is configured to encode M text blocks using a word vector generation network to generate M text encoding features;

[0240] The text block clustering module 130 is configured to calculate similarity of M text encoding features using a clustering network to generate N clusters, where each cluster includes K text blocks, and the similarity of the text encoding features corresponding to any two text blocks in the same cluster is greater than or equal to a similarity threshold, where M≥N≥1 and K>1;

[0241] The data lineage graph generation module 140 is used to generate N data lineage graphs based on N clusters and M time identification information, wherein the data lineage graph is used to represent the association relationship between K time identification information corresponding to K text blocks belonging to the same cluster.

[0242] The device provided in the embodiment of the present application can realize data lineage analysis of unstructured text information by segmenting and encoding the text information to be processed, which can cover the usage scenarios of most user data, reduce data security risks, and improve data security protection capabilities; using word vector generation networks and clustering networks, the present application can more accurately calculate the similarity of text encoding features, generate more meaningful clusters, and thus provide a more accurate data lineage graph; generating a data lineage graph based on clusters and text attributes can clearly display the association relationship between text blocks belonging to the same cluster, helping users better understand the ins and outs of text data.

[0243] In this application Figure 23 In an optional embodiment of the information processing device provided by the corresponding embodiment, the word vector generation network includes a semantic feature extraction layer and a feature embedding layer; the text block encoding module 120 is further used to:

[0244] Input M text blocks into the semantic feature extraction layer in the word vector generation network, and generate M text semantic features through the semantic feature extraction layer;

[0245] The text semantic features are input into the feature embedding layer in the word vector generation network, and M text encoding features are generated through the feature embedding layer.

[0246] The device provided in the embodiment of the present application converts text blocks into text semantic features and text encoding features through a semantic feature extraction layer and a feature embedding layer, thereby realizing semantic representation of the text. This semantic representation method can better capture the meaning of the text and provide more meaningful information for subsequent processing and analysis; mapping text semantic features to low-dimensional text encoding features achieves information compression and simplification. This can reduce the dimension and complexity of the data, improve computing efficiency and processing speed; by training the word vector generation network, the learned feature embedding layer can perform universal semantic representation of different texts. This helps to improve the generalization ability of the model, enabling it to process a variety of different texts.

[0247] In this application Figure 23 In an optional embodiment of the information processing device provided by the corresponding embodiment, the text block clustering module 130 is further configured to:

[0248] Input M text encoding features into the clustering network, calculate the cosine similarity of each pair of the M text encoding features through the clustering network, and obtain L cosine similarity values, where L = M × (M-1) / 2;

[0249] According to the L cosine similarity values, the M text encoding features are grouped to obtain N encoding feature groups, wherein the cosine similarity value of each two text encoding features belonging to the same encoding feature group is greater than or equal to the similarity threshold;

[0250] According to N encoding feature groups, M text blocks are grouped to obtain N clusters.

[0251] The device provided in the embodiment of the present application can find the similarity between texts by calculating the cosine similarity between text encoding features, which helps to identify texts with similar themes or content and provides a basis for the subsequent generation of data lineage graphs; grouping text encoding features according to a similarity threshold so that similar text encoding features are grouped together. This grouping method can better organize and manage text data and facilitate further analysis and application; and grouping text blocks according to encoding feature groups to obtain clusters. The formation of clusters can help discover patterns and structures in text data.

[0252] In this application Figure 23 In an optional embodiment of the information processing device provided by the corresponding embodiment, the data lineage graph generation module 140 is further configured to:

[0253] For each of the N clusters, execute:

[0254] Generate K text block nodes based on the K text blocks in each cluster;

[0255] According to the K time identification information corresponding to the K text blocks in each cluster, establish the edge relationship between the K text block nodes;

[0256] According to the K text block nodes and the edge relationships between the K text block nodes, a data lineage graph corresponding to the cluster is generated.

[0257] The device provided in the embodiment of the present application abstracts text blocks into nodes, and establishes connections between text blocks through edge relationships to generate a data lineage graph. This visual display method makes the relationship between text blocks more intuitive and clear, which is convenient for users to understand and analyze; by establishing edge relationships, the time sequence, dependency or simultaneous relationship between text blocks can be clarified, which helps users better understand the logical relationship between text blocks and discover potential patterns and associations; the data lineage graph provides a powerful tool for the analysis of text data. Users can observe the nodes and edge relationships in the graph to deeply study the flow and evolution process of text blocks, thereby obtaining valuable insights and information.

[0258] In this application Figure 23 In an optional embodiment of the information processing device provided by the corresponding embodiment, the data lineage graph generation module 140 is further configured to:

[0259] Arrange the K time identification information corresponding to the K text blocks in each cluster in ascending order;

[0260] Sort the K text block nodes according to the K time identification information in ascending order;

[0261] According to the sorted K text block nodes, edge relationships between the K text block nodes are established.

[0262] The device provided in the embodiment of the present application arranges the time identification information in ascending order and sorts the text block nodes according to the sorted time identification information, so that the text block nodes are organized in chronological order, which helps to better understand the temporal relationship between text blocks, such as the order in which events occur; establishing edge relationships between text block nodes can form a data lineage graph, which clearly shows the relationship between text blocks.

[0263] In this application Figure 23 In an optional embodiment of the information processing device provided by the corresponding embodiment, the text block acquisition module 110 is further configured to:

[0264] Get the text information to be processed;

[0265] Perform target identifier recognition on the text information to be processed to obtain P target identifiers, where P ≥ M;

[0266] According to P target identifiers, the text information to be processed is segmented to obtain P candidate text blocks;

[0267] Count the number of words in the P candidate text blocks to obtain P word count information corresponding to the P candidate text blocks;

[0268] According to the P word count information, M text blocks are determined from the P candidate text blocks, wherein the word count information corresponding to each text block in the M text blocks is greater than or equal to a preset word count.

[0269] The device provided in the embodiments of the present application can decompose original text data into smaller, easier-to-process candidate text blocks by acquiring text information and performing target identifier recognition and segmentation, which helps to better understand and analyze the structure and content of the text. It also counts the words in the candidate text blocks and determines M text blocks based on a preset word count, making it possible to screen out text blocks of a specific length or importance, which helps focus on key information and reduces redundant and unnecessary text processing. The determination of text blocks based on word count information improves the accuracy and pertinence of the selection. This makes subsequent processing and analysis more efficient and meaningful.

[0270] In this application Figure 23 In an optional embodiment of the information processing device provided by the corresponding embodiment, the text block acquisition module 110 is further configured to:

[0271] Receive file path information and target file format;

[0272] Traverse the file path information to obtain at least one candidate file;

[0273] According to the target file format, determining at least one target file whose file format satisfies the target file format from the at least one candidate file;

[0274] According to the target file format, at least one target file is parsed to read the content of the at least one target file to obtain text information to be processed.

[0275] The device provided in the embodiment of the present application, by receiving the file path and the target file format, the system can quickly locate and filter out files that meet the requirements, thereby improving the efficiency of file processing; filtering is performed according to the target file format to ensure that the acquired files meet the user's expectations, avoiding unnecessary errors or misunderstandings; supports multiple common file formats and has the ability to dynamically expand, so that the system can adapt to different file types and requirements and has better flexibility; parses the target file and reads the text content, so that the user can easily obtain the required information, providing a basis for further processing or analysis.

[0276] In this application Figure 23 In an optional embodiment of the information processing device provided in the corresponding embodiment, please refer to Figure 24 The information processing device further includes a storage module 150. Specifically, the storage module 150 is configured to:

[0277] Determine at least one file name information corresponding to the M text blocks according to the file path information;

[0278] Generate M pieces of attribute information according to the M word count information corresponding to the M text blocks, at least one file name information corresponding to the M text blocks, and the M time identification information corresponding to the M text blocks;

[0279] Store M attribute information.

[0280] The device provided in the embodiment of the present application provides a clearer source and contextual information for the text block by associating it with the file to which it belongs. This is very helpful for subsequent analysis and processing, such as classification based on file type or specific processing based on the file source. The attribute information generated by combining word count information, file name information, and time stamp information provides a more comprehensive description of the text block. This attribute information can be used for various purposes, such as data analysis, text classification, and search, increasing the understanding and utilization value of the text block. The attribute information is stored to facilitate the subsequent management and retrieval of the text block. The required text block can be quickly found based on specific attribute conditions, improving data availability and efficiency.

[0281] In this application Figure 23 In an optional embodiment of the information processing device provided in the corresponding embodiment, please refer to Figure 25 The information processing device further includes a query module, specifically, a query module 200, which is used to:

[0282] Receive query text;

[0283] Determine the data lineage graph containing the query text according to the query text, and return the data lineage graph containing the query text.

[0284] The device provided in the embodiment of the present application is designed to help users quickly locate and obtain information related to the query text, and present it in an intuitive manner, so that users can further analyze and understand the relationship between text data.

[0285] In this application Figure 23 In an optional embodiment of the information processing device provided in the corresponding embodiment, please refer to Figure 26 The information processing device 10 further includes a word vector generation network training module 300. Specifically, the word vector generation network training module 300 is used to:

[0286] Obtaining a first sample data set, wherein the first sample data set includes at least one set of first sample data, each set of first sample data includes a first training sample text, a first positive sample text, and a first negative sample text, semantic information of the first training sample text is the same as semantic information of the first positive sample text, and semantic information of the first training sample text is different from semantic information of the first negative sample text;

[0287] Get the word vector to generate the initial network;

[0288] The initial network for generating word vectors is trained according to the first sample data set to adjust network parameters of the initial network for generating word vectors, thereby obtaining a trained network for generating word vectors.

[0289] The device provided in the embodiment of the present application, by training the initial network for generating word vectors, the network can learn word vectors that more accurately represent the semantics of the text, such that word vectors can provide more valuable information in subsequent natural language processing tasks, such as better performance in text classification, semantic similarity calculation, etc.; using the first sample data set for training can enable the model to learn the differences between different semantic texts, thereby improving the generalization ability of the model, which means that the model can better capture semantic information when processing new texts and reduce the risk of overfitting; by comparing the first training sample text, the first positive sample text and the first negative sample text, the model can better understand the semantic relationship between the texts, including similarities and differences, which helps the model make more accurate judgments in tasks such as semantic analysis and information retrieval; training the initial network for generating word vectors according to the first sample data set can adjust the parameters of the network to make it more suitable for processing text data for specific tasks. This can improve the performance and efficiency of the model; by setting different proportions of the first training sample text, the first positive sample text and the first negative sample text, more training data is provided, which helps to enrich the learning experience of the model and further improve the performance of the model.

[0290] In this application Figure 23 In an optional embodiment of the information processing device provided in the corresponding embodiment, please refer to Figure 27 The information processing device further includes a clustering network training module 400. Specifically, the clustering network training module 400 is used to:

[0291] Obtaining a second sample data set and a second sample data label set, wherein the second sample data set includes at least one second training sample text encoding;

[0292] Get the initial clustering network;

[0293] Inputting the second sample data set into the clustering initial network, and generating a predicted clustering result through the clustering initial network;

[0294] The clustering initial network is trained according to the second sample data label set and the predicted clustering result to adjust the network parameters of the clustering initial network to obtain the clustering network.

[0295] The apparatus provided in the embodiment of the present application trains a clustering network to effectively cluster input text data. By comparing with a second sample data label set and adjusting parameters, the clustering network can learn the inherent structure and pattern of the sample data, thereby improving the accuracy of clustering.

[0296] Figure 28 : This is a schematic diagram of a server structure provided in an embodiment of the present application. The server 300 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPUs) 322 (for example, one or more processors) and memories 332, and one or more storage media 330 (for example, one or more massive storage devices) for storing application programs 342 or data 344. Among them, the memories 332 and the storage media 330 may be temporary storage or permanent storage. The program stored in the storage medium 330 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations on the server. Furthermore, the central processing unit 322 may be configured to communicate with the storage medium 330 to execute a series of instruction operations in the storage medium 330 on the server 300.

[0297] The server 300 may also include one or more power supplies 326, one or more wired or wireless network interfaces 350, one or more input and output interfaces 358, and / or one or more operating systems 341, such as Windows Server 2003 or Windows Server 2003R. TM , Mac OS X TM , Unix TM ,Linux TM , FreeBSD TM etc.

[0298] The steps performed by the server in the above embodiment can be based on the Figure 28 The server structure shown.

[0299] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0300] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.

[0301] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0302] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0303] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0304] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. An information processing method, characterized in that: include: Obtaining M text blocks corresponding to the text information to be processed and M time identification information corresponding to the M text blocks, wherein the M text blocks are obtained by segmenting the text information to be processed, and M>1; Encode the M text blocks through a word vector generation network to generate M text encoding features; Performing similarity calculation on the M text encoding features through a clustering network to generate N clusters, wherein each cluster includes K text blocks, and the similarity of the text encoding features corresponding to any two text blocks in the same cluster is greater than or equal to a similarity threshold, M≥N≥1, K>1; N data lineage graphs are generated based on the N clusters and the M time identification information, wherein the data lineage graphs are used to represent the association relationship between the K time identification information corresponding to the K text blocks belonging to the same cluster.

2. The information processing method according to claim 1, wherein: The word vector generation network includes a semantic feature extraction layer and a feature embedding layer; The M text blocks are encoded by a word vector generation network to generate M text encoding features, including: Inputting the M text blocks into the semantic feature extraction layer in the word vector generation network, and generating M text semantic features through the semantic feature extraction layer; The M text semantic features are input into the feature embedding layer in the word vector generation network, and M text encoding features are generated through the feature embedding layer.

3. The information processing method according to claim 1, wherein: The similarity calculation is performed on the M text encoding features through the clustering network, and N clusters are output, including: Inputting the M text encoding features into a clustering network, calculating the cosine similarity of each of the M text encoding features through the clustering network to obtain L cosine similarity values, where L = M × (M-1) / 2; Grouping the M text encoding features according to the L cosine similarity values ​​to obtain N encoding feature groups, wherein the cosine similarity values ​​of any two text encoding features in the same encoding feature group are greater than or equal to a similarity threshold; The M text blocks are grouped according to the N coding feature groups to obtain N clusters.

4. The information processing method according to claim 1, wherein: Generating N data lineage graphs according to the N clusters and the M time identification information includes: For each of the N clusters, execute: Generate K text block nodes according to the K text blocks in each cluster; Establishing edge relationships between the K text block nodes according to the K time identification information corresponding to the K text blocks in each cluster; A data lineage graph corresponding to the cluster is generated according to the K text block nodes and the edge relationships between the K text block nodes.

5. The information processing method according to claim 4, wherein: The step of establishing edge relationships between the K text block nodes according to the K time identification information corresponding to the K text blocks in each cluster includes: Arrange the K time identification information corresponding to the K text blocks in each cluster in ascending order; Sort the K text block nodes according to the K time identification information arranged in ascending order; According to the sorted K text block nodes, edge relationships between the K text block nodes are established.

6. The information processing method according to claim 1, wherein: The step of obtaining M text blocks corresponding to the text information to be processed includes: Obtaining the text information to be processed; Performing target identifier identification on the text information to be processed to obtain P target identifiers, where P ≥ M; Segment the text information to be processed according to the P target identifiers to obtain P candidate text blocks; Perform word count on the P candidate text blocks to obtain P word count information corresponding to the P candidate text blocks; The M text blocks are determined from the P candidate text blocks according to the P word count information, wherein the word count information corresponding to each text block in the M text blocks is greater than or equal to a preset word count.

7. The information processing method according to claim 6, wherein: The obtaining of the text information to be processed includes: Receive file path information and target file format; Traversing the file path information to obtain at least one candidate file; According to the target file format, determining at least one target file whose file format satisfies the target file format from the at least one candidate file; According to the target file format, the at least one target file is parsed to read the content of the at least one target file to obtain the text information to be processed.

8. The information processing method according to claim 7, wherein: After determining the M text blocks from the P candidate text blocks, the method further includes: Determine at least one file name information corresponding to the M text blocks according to the file path information; Generate M pieces of attribute information according to the M word count information corresponding to the M text blocks, at least one file name information corresponding to the M text blocks, and the M time identification information corresponding to the M text blocks; The M attribute information is stored.

9. The information processing method according to claim 1, wherein: After generating N data lineage graphs, the method further includes: Receive query text; A data lineage graph containing the query text is determined according to the query text, and the data lineage graph containing the query text is returned.

10. The information processing method according to any one of claims 1 to 9, wherein: The method further comprises: Obtaining a first sample data set, wherein the first sample data set includes at least one group of first sample data, each group of first sample data includes a first training sample text, a first positive sample text, and a first negative sample text, semantic information of the first training sample text is the same as semantic information of the first positive sample text, and semantic information of the first training sample text is different from semantic information of the first negative sample text; Get the word vector to generate the initial network; The initial network for generating word vectors is trained according to the first sample data set to adjust network parameters of the initial network for generating word vectors and obtain the trained network for generating word vectors.

11. The information processing method according to any one of claims 1 to 9, wherein: The method further comprises: Acquire a second sample data set and a second sample data label set, wherein the second sample data set includes at least one second training sample text encoding; Get the initial clustering network; Inputting the second sample data set into the initial clustering network, and generating a predicted clustering result through the initial clustering network; The clustering initial network is trained according to the second sample data label set and the predicted clustering result to adjust the network parameters of the clustering initial network to obtain the clustering network.

12. An information processing device, characterized in that: include: A text block acquisition module is used to acquire M text blocks corresponding to the text information to be processed and M time identification information corresponding to the M text blocks, wherein the M text blocks are obtained by segmenting the text information to be processed, and M>1; A text block encoding module is used to encode the M text blocks through a word vector generation network to generate M text encoding features; A text block clustering module is configured to calculate similarity of the M text encoding features using a clustering network to generate N clusters, wherein each cluster includes K text blocks, and the similarity of the text encoding features corresponding to any two text blocks in the same cluster is greater than or equal to a similarity threshold, where M≥N≥1 and K>1; The data lineage graph generation module is used to generate N data lineage graphs based on the N clusters and the M time identification information, wherein the data lineage graph is used to represent the association relationship between the K time identification information corresponding to the K text blocks belonging to the same cluster.

13. A computer device, characterized in that: include: Memory, processor, and bus system; Wherein, the memory is used to store programs; The processor is configured to execute the program in the memory, including executing the information processing method according to any one of claims 1 to 11; The bus system is used to connect the memory and the processor so that the memory and the processor can communicate with each other.

14. A computer-readable storage medium comprising instructions, characterized in that: When the information processing method is executed on a computer, the computer is caused to execute the information processing method according to any one of claims 1 to 11.

15. A computer program product comprising a computer program, characterized in that The computer program is used by a processor to execute the information processing method according to any one of claims 1 to 11.

Images