Information identification method and device

Abstract

The embodiment of the invention discloses an information identification method and device. The method comprises the steps of obtaining original data in a database, slicing the original data to obtain data slices, obtaining auxiliary information corresponding to the data slices, generating embedded vectors according to the auxiliary information and the data slices, and obtaining query information of a user after the query information is obtained. Obtaining at least one candidate data slice related to the query information according to the content of the data slices and the embedded vector, obtaining a correlation score of each candidate data slice and the query information through a pre-trained score model, and obtaining a target data slice from the candidate data slices according to the correlation scores, and obtaining an identification result of the security data node according to the target data slice through the large language model, wherein the identification result comprises node information and side information. Therefore, the information identification cost and difficulty can be reduced, and the scene coverage rate can be improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to an information identification method and device. Background Art

[0002] In the field of computer technology, constructing call relationship graphs between code nodes has widespread applications in data traceability, system architecture optimization, and security auditing. Call relationship graphs between code nodes include a structured representation of nodes and edges, where nodes represent code entities (such as functions, classes, and variables), and edges represent call, inheritance, or data transfer relationships. Therefore, accurately acquiring node and edge information is crucial for constructing call relationship graphs between code nodes.

[0003] In existing technologies, obtaining node and edge information primarily relies on static code analysis engines, supplemented by dynamic analysis and collection engines. Static code analysis extracts node and edge relationships by parsing the AST (Abstract Syntax Tree) structure and matching predefined rules. However, due to the diversity of programming language syntax, differences in framework encapsulation, and dynamic characteristics, static code analysis suffers from high rule maintenance costs, low scenario coverage, and difficulty adapting to personalized coding patterns. Summary of the Invention

[0004] In view of this, an object of the embodiments of the present invention is to provide an information recognition method and device, which can reduce the cost and difficulty of information recognition and improve scenario coverage.

[0005] In a first aspect, an embodiment of the present invention provides an information identification method, the method comprising:

[0006] Obtaining original data from a database, wherein the original data includes text information in a file, and the file includes at least one of a code file, a MarkDown file, and a TXT file;

[0007] Slicing the original data to obtain data slices;

[0008] Obtaining auxiliary information corresponding to the data slice, where the auxiliary information includes one or more of a slice function, a usage scenario, and a correlation with a data link;

[0009] generating an embedding vector based on the auxiliary information and the data slice;

[0010] After obtaining the user's query information, obtaining at least one candidate data slice related to the query information according to the content of the data slice and the embedding vector;

[0011] Obtaining a relevance score between each candidate data slice and the query information through a pre-trained scoring model;

[0012] Obtaining a target data slice from the candidate data slices according to the relevance score;

[0013] The recognition result of the security data node is obtained according to the target data slice through the large language model, and the recognition result includes node information and edge information.

[0014] In some embodiments, slicing the original data to obtain data slices specifically includes:

[0015] The original data is sliced ​​according to a predetermined segmentation granularity to obtain data slices, where the predetermined segmentation granularity includes at least one of a row, a function, a class, and a paragraph.

[0016] In some embodiments, the node information includes one or more of a node identifier, a node type, a node description, and a node attribute.

[0017] In some embodiments, the side information includes one or more of data flow direction, link relationship, and transmission mode.

[0018] In a second aspect, an embodiment of the present invention provides an information identification method, the method comprising:

[0019] Get the user's query information;

[0020] Acquire a slice library, wherein the slice library includes multiple data slices;

[0021] Acquire at least one target data slice related to the query information in the slice library;

[0022] The recognition result of the security data node is obtained according to the target data slice through the large language model, and the recognition result includes node information and edge information.

[0023] In some embodiments, obtaining a slice library includes:

[0024] Obtaining original data from a database, wherein the original data includes text information in a file, and the file includes at least one of a code file, a MarkDown file, and a TXT file;

[0025] Slicing the original data to obtain data slices;

[0026] The slice library is generated according to the data slices.

[0027] In some embodiments, obtaining the slice library further comprises:

[0028] Obtaining auxiliary information corresponding to the data slice, where the auxiliary information includes one or more of a slice function, a usage scenario, and a correlation with a data link;

[0029] An embedding vector is generated according to the auxiliary information and the data slice.

[0030] In some embodiments, obtaining at least one target data slice related to the query information in the slice library includes:

[0031] Acquire at least one candidate data slice related to the query information in the slice library using a predetermined retrieval algorithm;

[0032] Obtaining a relevance score between each candidate data slice and the query information through a pre-trained scoring model;

[0033] The target data slice is obtained from the candidate data slices according to the relevance score.

[0034] In some embodiments, obtaining at least one candidate data slice related to the query information from the slice library using a predetermined retrieval algorithm includes at least one of the following:

[0035] Obtaining an embedding vector corresponding to the query information, calculating a vector similarity between the embedding vector corresponding to the query information and the embedding vector corresponding to the data slice, and obtaining a candidate data slice related to the query information based on the vector similarity;

[0036] Acquire candidate data slices related to the query information in the slice library based on keyword retrieval;

[0037] Based on rule retrieval, candidate data slices related to the query information are obtained in the slice library.

[0038] In a third aspect, an embodiment of the present invention provides an information identification device, the device comprising:

[0039] A data acquisition unit, configured to acquire original data from a database, wherein the original data includes text information in a file, and the file includes at least one of a code file, a MarkDown file, and a TXT file;

[0040] a slicing unit, configured to slice the original data to obtain data slices;

[0041] an auxiliary information acquisition unit, configured to acquire auxiliary information corresponding to the data slice, the auxiliary information including one or more of a slice function, a usage scenario, and a correlation with a data link;

[0042] an embedding vector generating unit, configured to generate an embedding vector according to the auxiliary information and the data slice;

[0043] a candidate slice acquiring unit, configured to, after acquiring the user's query information, acquire at least one candidate data slice related to the query information based on the content of the data slice and the embedding vector;

[0044] A score calculation unit, configured to obtain a relevance score between each candidate data slice and the query information using a pre-trained score model;

[0045] a screening unit, configured to obtain a target data slice from the candidate data slices according to the relevance score;

[0046] A result recognition unit is used to obtain a recognition result of a security data node according to the target data slice through a large language model, and the recognition result includes node information and edge information.

[0047] In a fourth aspect, an embodiment of the present invention provides an information identification device, the device comprising:

[0048] A query information acquisition unit, used to acquire the user's query information;

[0049] A slice library acquisition unit, configured to acquire a slice library, wherein the slice library includes a plurality of data slices;

[0050] a target data acquisition unit, configured to acquire at least one target data slice related to the query information from the slice library;

[0051] An information identification unit is used to obtain an identification result of a security data node according to the target data slice through a large language model, wherein the identification result includes node information and edge information.

[0052] In a fifth aspect, an embodiment of the present invention provides an electronic device comprising a memory and a processor, wherein the memory is used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method as described in the first and second aspects.

[0053] In a sixth aspect, an embodiment of the present invention provides a computer program product, which includes a computer program. When the computer program runs on a computer, the computer executes the methods described in the first and second aspects above.

[0054] In a seventh aspect, an embodiment of the present invention provides a computer-readable storage medium storing computer program instructions, which, when executed by a processor, implement the method according to the first and second aspects.

[0055] The technical solution of the embodiment of the present invention obtains raw data from a database, slices the raw data to obtain data slices, obtains auxiliary information corresponding to the data slices, generates embedding vectors based on the auxiliary information and the data slices, and after obtaining the user's query information, obtains at least one candidate data slice related to the query information based on the content of the data slice and the embedding vector. A pre-trained scoring model is used to obtain a relevance score between each candidate data slice and the query information. A target data slice is obtained from the candidate data slices based on the relevance score. A large language model is used to obtain a security data node identification result based on the target data slice. The identification result includes node information and edge information. This can reduce the cost and difficulty of information identification and improve scenario coverage. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] The above and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings, in which:

[0057] Figure 1 is a schematic diagram of an information identification system according to an embodiment of the present invention;

[0058] Figure 2 is a flow chart of an information identification method according to an embodiment of the present invention;

[0059] Figure 3 is a flowchart of obtaining candidate data slices according to an embodiment of the present invention;

[0060] Figure 4 is a flow chart of an information identification method according to another embodiment of the present invention;

[0061] Figure 5 is a schematic diagram of an information identification device according to an embodiment of the present invention;

[0062] Figure 6 is a schematic diagram of an information identification device according to another embodiment of the present invention;

[0063] Figure 7 is a schematic diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0064] The present application is described below based on the following embodiments, but the present application is not limited to these embodiments. In the detailed description of the present application below, certain specific details are described in detail. Those skilled in the art can fully understand the present application without the description of these details. To avoid obscuring the essence of the present application, well-known methods, processes, procedures, components, and circuits are not described in detail.

[0065] Furthermore, persons of ordinary skill in the art will appreciate that the figures provided herein are for illustration purposes only and are not necessarily drawn to scale.

[0066] Unless the context clearly requires otherwise, words like “include”, “comprising” and the like throughout this application should be interpreted as including rather than exclusive or exhaustive; that is, as meaning “including but not limited to”.

[0067] In the description of this application, it should be understood that the terms "first", "second", etc. are used for descriptive purposes only and should not be understood to indicate or imply relative importance. In addition, in the description of this application, unless otherwise specified, "plurality" means two or more.

[0068] Where the solutions described in this specification and in the examples involve the processing of personal information, such processing will be conducted with a legitimate basis (e.g., with the consent of the personal information subject or as necessary for the performance of a contract) and only within the prescribed or agreed scope. A user's refusal to process personal information other than that required for basic functions will not affect the user's use of these basic functions.

[0069] In the data processing field, the Data Flow Lineage Graph (DFLG) serves as a core tool for representing the call relationships and data flows between code nodes. Its construction relies on the integration of static code analysis and dynamic tracing technologies. This graph consists of two key elements: nodes and edges. Nodes represent code or file entities, such as functions, classes, and variables, while edges represent call relationships or data transfer paths.

[0070] The construction of node information and edge call relationships is based on the AST static code analysis engine, supplemented by the dynamic analysis and collection engine.

[0071] In many scenarios (such as Node, JavaScript, and multiple TypeScript environments), dynamic analysis engines are difficult to build cost-effectively and runtime data is difficult to obtain. Therefore, the recognition performance of the foundational static code analysis engine is crucial. Static code engines are typically built using a combination of AST and rules. This leads to high costs associated with adapting to various customized coding support requirements that general rules cannot cover. Specifically, the difficulties and challenges they face include the following:

[0072] First, there's the sheer diversity of programming languages. Besides commonly used application languages ​​like Java and JavaScript, there's also a smaller but larger number of long-tail languages, such as Go, Python, C / C++, and Rust. Each language has its own unique grammatical structure, and AST generation requires individual adaptation based on the language, which is both costly and difficult to implement.

[0073] Second, there's a wide variety of frameworks. Every language supports multiple frameworks. For example, the mainstream frameworks supported by the Node language include egg, midway, begg, express, koa, and more. Different frameworks have varying implementations, encapsulation, and invocation methods, resulting in significant variations in the standard way application code is written. Furthermore, uncommon, newly emerging, and evolving frameworks all require individual adaptation and timely maintenance. Consequently, framework issues can significantly complicate engine adaptation.

[0074] Third, the individuality of code writing. Different programmers have different programming styles, naming methods, packaging methods, and syntax usage, which leads to different code styles and makes adaptation difficult.

[0075] Fourth, complex scenarios such as proxies, multi-layer encapsulation, implicit calls, cross-file parsing, and dynamic builds. When the code contains proxies, multi-layer encapsulation, implicit calls, runtime dynamic loading, dynamic builds (such as generating an HSF instance based on files in a specified directory), and incomplete code (missing dependencies or npm packages), static code parsing often becomes significantly more difficult, or even impossible.

[0076] In response to the above problems, the present invention proposes an information identification method that can reduce the cost and difficulty of information identification, reduce the risk of production environment deployment, and improve scenario coverage and data integrity.

[0077] Figure 1 Schematic diagram of the information identification system according to an embodiment of the present invention. Figure 1 As shown, the information identification system of the embodiment of the present invention can be deployed in various electronic devices, such as servers, computers, etc. Specifically, the information identification system includes a data system 11, a data preprocessing module 12, an embedding model 13, a keyword-based retrieval module 14, a rule-based retrieval module 15, a vector-based retrieval module 16, a relevance score calculation module 17, and a large language model 18.

[0078] The data system 11 is used to store raw data, which is the data to be processed, that is, the data that needs to be identified. The data to be processed includes multiple nodes. A node refers to a specific processing unit or component in a system or application. It can be a service, function, module, or any code block that can perform a specific task. The purpose of the embodiment of the present invention is to identify data security nodes in the raw data and their call relationships. The raw system can be implemented through various data systems such as data warehouses, message queues or event stream platforms, log management systems, file systems, and cloud storage.

[0079] The data preprocessing module 12 is used to preprocess the raw data to obtain a slice library. Specifically, the data preprocessing module 12 reads the raw data from the data system 11 and slices the raw data according to a predetermined granularity to obtain data slices. The predetermined granularity includes at least one of line-level, function-level, class-level, and paragraph-level. Auxiliary information is extracted from the data slices, including one or more of the following: slice function, usage scenario, and relevance to data links.

[0080] The embedding model 13 is used to convert the input data into an embedding vector. Specifically, the embedding model 13 generates an embedding vector corresponding to the data slice based on the data slice and the corresponding auxiliary information, and generates a corresponding embedding vector based on the query information input by the user.

[0081] The keyword-based retrieval module 14 is used to obtain candidate data slices related to the query information in the slice library based on keyword retrieval. Specifically, the keywords of the query information can be obtained by performing word segmentation processing on the query information. The keywords corresponding to each data slice in the slice library can be obtained by performing word segmentation processing on the data slice. The keywords of the query information are matched with the keywords of the data slice, and the matching method can be implemented by fuzzy matching, regular matching or full-text search. The matching degree is calculated based on factors such as the frequency and position of the matched keywords. According to the matching degree, the candidate data slices are obtained according to the predetermined screening conditions. For example, the data slice with a matching degree greater than or equal to the predetermined threshold is selected as the candidate data slice.

[0082] The rule-based retrieval module 15 is used to obtain candidate data slices related to the query information in the slice library based on rule retrieval. Specifically, based on expert experience, specific retrieval rules are formulated. For example, matching a certain field, etc. A matching tool is determined, and the matching tool includes a regular expression, SQL (Structured Query Language) function or other rule matching tool. The query information is converted into a query statement, for example, into a regular expression or SQL syntax, etc. The data slices in the slice library are loaded, and the query statement obtained above is used to match the target field in the data slice, and the data slices that meet the conditions are screened as the candidate data slices.

[0083] The vector-based retrieval module 16 is configured to calculate vector similarity between the embedding vector corresponding to the query information and the embedding vector corresponding to the data slice. The vector similarity can be calculated using various existing methods. For example, the vector similarity can be the Euclidean distance, Manhattan distance, or cosine similarity between the embedding vector of the query information and the embedding vector of the data slice. Data slices with vector similarity greater than or equal to a predetermined threshold are selected as candidate data slices.

[0084] The correlation score calculation module 17 is used to obtain the correlation score between each candidate data slice and the query information through a pre-trained scoring model. Among them, the scoring model can be various re-ranking models. The scoring model first pre-processes the input data (such as query information and candidate slice data), including word segmentation, conversion into a format acceptable to the model, etc. The input data is converted into a vector representation. The correlation score between the query information and the vector representation of the candidate slice data is calculated by a vector similarity algorithm. Among them, the scoring model can be implemented by models such as BgeReranker, ColbertRerank, CohereRerank, JinaRerank, etc. The candidate data slice with a correlation score greater than or equal to a predetermined threshold is selected as the target data slice. Alternatively, the candidate data slices are sorted in descending order according to the correlation score, and the candidate data slice ranked in the front predetermined position is selected as the target data slice.

[0085] The large language model 18 is used to obtain raw data related to the target data slice from the raw data, and perform recognition processing on the raw data related to the target data slice to obtain a recognition result. The recognition result includes node information and edge information corresponding to the target data slice. The node information includes one or more of a node identifier, a node type, a node description, and node attributes. The node identifier is used to uniquely identify the node, such as a node ID or name. The node type is used to describe the role or function of the node, such as a data source, an ETL (Extract, Transform, Load) process, a database, an application programming interface (API), etc. The node description is used to provide a detailed description of the operation performed by the node, such as "extracting user information from a database." Node attributes are any additional information related to the node. The edge information includes one or more of data flow direction, link relationship, and transmission mode. The data flow direction indicates which node the data flows from and to which node it reaches. Link relationships are links composed of call relationships between nodes within an application, such as which nodes call it or which nodes it will call. The transmission mode describes how the data is transmitted, such as through API calls, message queues, file transfers, etc.

[0086] The embodiment of the present invention obtains raw data from a database, slices the raw data to obtain data slices, obtains auxiliary information corresponding to the data slices, generates embedding vectors based on the auxiliary information and the data slices, and after obtaining the user's query information, obtains at least one candidate data slice related to the query information based on the content of the data slice and the embedding vector. A pre-trained scoring model is used to obtain a relevance score between each candidate data slice and the query information. A target data slice is obtained from the candidate data slices based on the relevance score. A large language model is used to obtain a security data node recognition result based on the target data slice. The recognition result includes node information and edge information. This reduces the cost and difficulty of information recognition and improves scenario coverage.

[0087] Figure 2 FIG. 1 is a flow chart of an information identification method according to an embodiment of the present invention. Figure 2 As shown, the information identification method of the embodiment of the present invention is executed by the server and specifically includes the following steps:

[0088] Step S110: Obtain original data from the database.

[0089] In this embodiment, the original data is the data to be processed, that is, the data that needs to be identified. The data to be processed includes multiple nodes. A node refers to a specific processing unit or component in a system or application. It can be a service, function, module or any code block that can perform a specific task. The purpose of this embodiment of the present invention is to identify the data security nodes in the original data and their calling relationships.

[0090] The original data may be data in various data warehouses, message queues or event stream platforms, log management systems, file systems, cloud storage and other data systems.

[0091] Data security nodes, also known as data asset nodes, are key locations or links in raw data for processing sensitive information. Identifying data security nodes can help organizations better understand and manage their data flows, such as privacy protection, compliance checks, and risk assessments.

[0092] The raw data includes text information in a file, and the file includes at least one of a code file, a MarkDown file, and a TXT file. Among them, a code file is a file containing the source code of a computer program, which is usually saved in the form of plain text and uses a specific file extension to identify its programming language or purpose. A Markdown file (MD file for short) is a lightweight and easy-to-read markup language file format with an extension of ".md" or ".markdown". A TXT file is a plain text file format, usually using ".txt" as the file extension. This file format is mainly used to store text data that has not undergone any special encoding or formatting.

[0093] Step S120: Slice the original data to obtain data slices.

[0094] In this embodiment, after obtaining the original data, the original data is sliced ​​to obtain data slices. Specifically, the original data is sliced ​​according to a predetermined granularity to obtain data slices, and the predetermined granularity includes at least one of line-level, function-level, class-level, and paragraph-level.

[0095] For code files, Markdown files, and TXT files, you can split the original data into multiple data slices based on the granularity of paragraphs, lines, etc. For code files, you can also split the original data into multiple data slices based on the granularity of functions, classes, etc.

[0096] Step S130: Acquire auxiliary information corresponding to the data slice.

[0097] In this embodiment, the auxiliary information includes one or more of slicing functions, usage scenarios, and relevance to data links. Among them, the slicing function is used to describe the functional implementation of the data slice, such as the functions, parameters and other information of the data slice. The usage scenario is the purpose of the data slice in actual application. The relevance to the data link includes relevance and irrelevance, among which relevance to the data link means that when data operations (such as database access, API calls) are involved, the data slice is the flow path of the data, or indirectly acts on the data link by affecting the data processing process (such as configuration management, logging). Irrelevant to the data link means irrelevant to the data link.

[0098] The acquisition of auxiliary information can be completed by relevant technical personnel and provided to the server; it can also be completed automatically by a computer, for example, by analyzing data slices through a large language model to obtain auxiliary information.

[0099] Step S140: Generate an embedding vector according to the auxiliary information and the data slice.

[0100] In this embodiment, an embedding vector is generated based on the auxiliary information and the data slice using a pre-acquired embedding model, wherein the embedding model can be implemented by various existing models, such as BGE-M3, Gte-Qwen2, Text-Embedding-3-Large, etc.

[0101] Taking Text-Embedding-3-Large as an example, Text-Embedding-3-Large is a text embedding model. After obtaining the data slice and the corresponding auxiliary information, the data slice and the corresponding auxiliary information are input into Text-Embedding-3-Large, and the embedding vector is generated by the Text-Embedding-3-Large model.

[0102] Specifically, the Text-Embedding-3-Large model generates text embedding vectors based on natural language processing techniques in deep learning, particularly models based on the Transformer architecture. These models can convert input text into fixed-length vector representations that capture the text's semantic information and contextual relationships.

[0103] First, the model is pre-trained to obtain the Text-Embedding-3-Large model. Then, the concatenated text is generated by concatenating the content of the data slices and the corresponding auxiliary information. This concatenated text is decomposed into a series of tokens (tokens), typically including words, subwords, and some special tags (such as [CLS] and [SEP]). Each token is mapped to a vector in a high-dimensional space, namely the token embedding. Furthermore, a positional embedding is added to encode the token's position in the sequence. These position-informed token embeddings are passed through a multi-layer Transformer encoder. Each encoder layer includes a multi-head self-attention mechanism, which allows the model to focus on different parts of the input sequence at different positions within the same layer. A feed-forward neural network is then used to further transform the feature representation. After all encoder layers, the model generates a corresponding hidden state for each input token. Finally, the embedding of the [CLS] token in the last layer is used as the embedding vector for the entire concatenated text.

[0104] In some embodiments, to facilitate subsequent calculations, the obtained embedding vector may be normalized to obtain an embedding vector of fixed length.

[0105] Finally, a slice library is generated according to the data slices and the corresponding embedding vectors, where the slice library includes multiple data slices and the embedding vector corresponding to each data slice.

[0106] Step S150: Obtain the user's query information.

[0107] In this embodiment, the server receives query information input by the user, wherein the query information is related to the content of the data security node that the user wants to identify. The query information can be in various forms, such as keywords or phrases, data snippets, natural language descriptions, etc.

[0108] For keywords or phrases: Users can provide keywords or phrases to describe the information of the required node. For example, if the user wants to manage sensitive information involving user information, the query information can be "get user information" and so on.

[0109] For data fragments: users can directly provide a small data fragment (such as a code snippet) as query information, hoping to find nodes with similar or exact matching functions.

[0110] For natural language description: Users can use natural language to describe the node they want to query, for example, "I want to find the node that gets user information."

[0111] Step S160: Obtain at least one candidate data slice related to the query information according to the content of the data slice and the embedding vector.

[0112] In this embodiment, at least one candidate data slice related to the query information is obtained from the slice library according to the content of the data slice and the embedding vector.

[0113] in, Figure 3 FIG. 1 is a flowchart of obtaining candidate data slices according to an embodiment of the present invention. Figure 3 As shown, obtaining the candidate data slices includes one or more of the following steps S161-S163:

[0114] Step S161: Obtain an embedding vector corresponding to the query information, calculate a vector similarity between the embedding vector corresponding to the query information and the embedding vector corresponding to the data slice, and obtain candidate data slices related to the query information based on the vector similarity.

[0115] In this embodiment, an embedding vector is generated based on the query information using a pre-acquired embedding model. The embedding model can be implemented using various existing models, such as BGE-M3, Gte-Qwen2, and Text-Embedding-3-Large. The specific implementation method can refer to step S140 above and will not be further described in detail in this embodiment of the present invention. The length of the embedding vector for the query information is the same as the length of the embedding vector for the data slice.

[0116] After obtaining the embedding vector of the query information, a vector similarity is calculated between the embedding vector corresponding to the query information and the embedding vector corresponding to the data slice. The vector similarity can be calculated using various existing methods. For example, the vector similarity can be the Euclidean distance, Manhattan distance, or cosine similarity between the embedding vector of the query information and the embedding vector of the data slice. Data slices whose vector similarity is greater than or equal to a predetermined threshold are selected as candidate data slices.

[0117] Step S162: Acquire candidate data slices related to the query information in the slice library based on keyword retrieval.

[0118] In this embodiment, keywords of the query information can be obtained by performing word segmentation processing on the query information. Keywords corresponding to each data slice in the slice library can be obtained by performing word segmentation processing on the data slice. The keywords of the query information are matched with the keywords of the data slice. The matching method can be implemented by fuzzy matching, regular matching or full-text search. The matching degree is calculated based on factors such as the frequency and position of the matched keywords. Candidate data slices are obtained according to the matching degree and predetermined screening conditions. For example, a data slice with a matching degree greater than or equal to a predetermined threshold is selected as the candidate data slice.

[0119] Step S163: retrieve candidate data slices related to the query information in the slice library based on the rules.

[0120] In this embodiment, specific retrieval rules are formulated based on expert experience. For example, matching a certain field, etc. A matching tool is determined, and the matching tool includes a regular expression, SQL (Structured Query Language) function or other rule matching tool. The query information is converted into a query statement, for example, into a regular expression or SQL syntax, etc. The data slices in the slice library are loaded, and the query statement obtained above is used to match the target field in the data slice, and the data slices that meet the conditions are screened as the candidate data slices.

[0121] In the embodiment of the present invention, candidate data slices can be obtained based on any one or more of the three retrieval methods in steps S1611-S1613. If multiple methods are used to obtain candidate data slices at the same time, it is necessary to deduplicate the candidate data slices obtained in the multiple methods to avoid repeated calculations.

[0122] Step S170: Obtain a relevance score between each candidate data slice and the query information through a pre-trained scoring model.

[0123] In this embodiment, after the candidate data slices are obtained, the relevance score between each candidate data slice and the query information is obtained through a pre-trained scoring model.

[0124] In some embodiments, the scoring model can be various re-ranking models. The scoring model first pre-processes the input data (e.g., query information and candidate slice data), including tokenization and conversion to a format acceptable to the model. The input data is converted into a vector representation. A vector similarity algorithm is used to calculate the correlation score between the vector representations of the query information and the candidate slice data.

[0125] Among them, the scoring model can be implemented by models such as BgeReranker, ColbertRerank, CohereRerank, and JinaRerank.

[0126] Among them, BgeReranker is a reranking model based on a variant of BERT (Bidirectional Encoder Representations from Transformers), which is used to rerank preliminary retrieval results to improve the relevance of the final retrieval results.

[0127] ColbertRerank uses the ColBERT (Collection-Contextualized Longformer with BERT) model for reranking. ColBERT is an improved Transformer model specifically designed to improve the matching accuracy between documents and queries in information retrieval tasks.

[0128] CohereRerank is a reranking model that uses deep learning technology to optimize the quality of text matching.

[0129] JinaRerank is a reranking model used within the Jina framework, an open source neural search engine framework that can be used to improve the relevance scores of search results.

[0130] Step S180: Obtain a target data slice from the candidate data slices according to the correlation score.

[0131] In this embodiment, candidate data slices with a correlation score greater than or equal to a predetermined threshold are selected as target data slices. Alternatively, candidate data slices are sorted in descending order of correlation scores, and the candidate data slice ranked first in a predetermined order is selected as the target data slice.

[0132] Thus, through the above steps, the target data slice related to the query information can be obtained.

[0133] Step S190: Obtain recognition results of security data nodes according to the target data slice using a large language model.

[0134] In this embodiment, a large language model (LLM) refers to a deep learning model with a large number of parameters and high computing power, often designed to understand and generate natural language or text. The trained large language model is deployed on a server, and a connection is established between the large language model and a database. After obtaining a target data slice, the target data slice is provided to the large language model. The large language model obtains raw data related to the target data slice from the original data, and the large language model performs recognition processing on the raw data related to the target data slice to obtain a recognition result.

[0135] The recognition result includes node information and edge information corresponding to the target data slice.

[0136] The node information includes one or more of a node identifier, a node type, a node description, and node attributes. A node identifier uniquely identifies a node, such as a node ID or name. A node type describes the role or function of a node, such as a data source, an ETL (Extract, Transform, Load) process, a database, or an application programming interface (API). A node description provides a detailed description of the operation performed by the node, such as "extracting user information from a database." Node attributes are any additional information associated with a node.

[0137] This edge information includes one or more of data flow direction, link relationships, and transmission methods. Data flow direction indicates which node the data flows from and to. Link relationships are links composed of the call relationships between nodes within an application, such as which nodes call it or which nodes it will call. The transmission method describes how the data is transmitted, such as through API calls, message queues, file transfers, etc.

[0138] Furthermore, the identification results of the obtained security data nodes are saved to facilitate subsequent query and use.

[0139] The embodiment of the present invention obtains original data from a database, slices the original data to obtain data slices, obtains auxiliary information corresponding to the data slices, generates an embedding vector based on the auxiliary information and the data slices, and after obtaining the user's query information, obtains at least one candidate data slice related to the query information based on the content of the data slice and the embedding vector, obtains the correlation score between each candidate data slice and the query information through a pre-trained scoring model, obtains the target data slice from the candidate data slice based on the correlation score, and obtains the recognition result of the security data node based on the target data slice through a large language model. The recognition result includes node information and edge information. As a result, a series of steps in the prior art such as adaptation code language, framework, personalized writing, AST analysis, static engine construction, and rule addition can be omitted, thereby reducing the cost and difficulty of information recognition, reducing the risk of production environment deployment, and improving scenario coverage and data integrity.

[0140] Figure 4 FIG. 1 is a flow chart of an information identification method according to another embodiment of the present invention. Figure 4 As shown, the information identification method of the embodiment of the present invention is executed by the server and specifically includes the following steps:

[0141] Step S210: Obtain the user's query information.

[0142] Step S220: Acquire a slice library, where the slice library includes multiple data slices.

[0143] Step S230: Obtain at least one target data slice related to the query information in the slice library.

[0144] Step S240: Obtain recognition results of security data nodes according to the target data slice using a large language model, where the recognition results include node information and edge information.

[0145] In some embodiments, slicing the original data to obtain data slices specifically includes:

[0146] The original data is sliced ​​according to a predetermined segmentation granularity to obtain data slices, where the predetermined segmentation granularity includes at least one of a row, a function, a class, and a paragraph.

[0147] In some embodiments, the node information includes one or more of a node identifier, a node type, a node description, and a node attribute.

[0148] In some embodiments, the side information includes one or more of data flow direction, link relationship, and transmission mode.

[0149] In some embodiments, obtaining a slice library includes:

[0150] Obtaining original data from a database, wherein the original data includes text information in a file, and the file includes at least one of a code file, a MarkDown file, and a TXT file;

[0151] Slicing the original data to obtain data slices;

[0152] The slice library is generated according to the data slices.

[0153] In some embodiments, obtaining the slice library further comprises:

[0154] Obtaining auxiliary information corresponding to the data slice, where the auxiliary information includes one or more of a slice function, a usage scenario, and a correlation with a data link;

[0155] An embedding vector is generated according to the auxiliary information and the data slice.

[0156] In some embodiments, obtaining at least one target data slice related to the query information in the slice library includes:

[0157] Acquire at least one candidate data slice related to the query information in the slice library using a predetermined retrieval algorithm;

[0158] Obtaining a relevance score between each candidate data slice and the query information through a pre-trained scoring model;

[0159] The target data slice is obtained from the candidate data slices according to the relevance score.

[0160] In some embodiments, obtaining at least one candidate data slice related to the query information from the slice library using a predetermined retrieval algorithm includes at least one of the following:

[0161] Obtaining an embedding vector corresponding to the query information, calculating a vector similarity between the embedding vector corresponding to the query information and the embedding vector corresponding to the data slice, and obtaining a candidate data slice related to the query information based on the vector similarity;

[0162] Acquire candidate data slices related to the query information in the slice library based on keyword retrieval;

[0163] Based on rule retrieval, candidate data slices related to the query information are obtained in the slice library.

[0164] The embodiment of the present invention obtains original data from a database, slices the original data to obtain data slices, obtains auxiliary information corresponding to the data slices, generates an embedding vector based on the auxiliary information and the data slices, and after obtaining the user's query information, obtains at least one candidate data slice related to the query information based on the content of the data slice and the embedding vector, obtains the correlation score between each candidate data slice and the query information through a pre-trained scoring model, obtains the target data slice from the candidate data slice based on the correlation score, and obtains the recognition result of the security data node based on the target data slice through a large language model. The recognition result includes node information and edge information. As a result, a series of steps in the prior art such as adaptation code language, framework, personalized writing, AST analysis, static engine construction, and rule addition can be omitted, thereby reducing the cost and difficulty of information recognition, reducing the risk of production environment deployment, and improving scenario coverage and data integrity.

[0165] Figure 5 FIG is a schematic diagram of an information identification device according to an embodiment of the present invention. Figure 5 As shown, the information identification device of an embodiment of the present invention includes a data acquisition unit 51, a segmentation unit 52, an auxiliary information acquisition unit 53, an embedding vector generation unit 54, a candidate slice acquisition unit 55, a score calculation unit 56, a screening unit 57, and a result identification unit 58. The data acquisition unit 51 is used to acquire raw data from a database. The raw data includes text information in a file, including at least one of a code file, a MarkDown file, and a TXT file. The segmentation unit 52 is used to slice the raw data to obtain data slices. The auxiliary information acquisition unit 53 is used to acquire auxiliary information corresponding to the data slices. The auxiliary information includes one or more of the following: slice function, usage scenario, and relevance to data links. The embedding vector generation unit 54 is used to generate an embedding vector based on the auxiliary information and the data slices. The candidate slice acquisition unit 55 is used to, after acquiring the user's query information, acquire at least one candidate data slice related to the query information based on the content of the data slice and the embedding vector. The score calculation unit 56 is used to obtain a relevance score for each candidate data slice with the query information using a pre-trained scoring model. The screening unit 57 is used to obtain a target data slice from the candidate data slices according to the relevance score. The result identification unit 58 is used to obtain a recognition result of a security data node according to the target data slice using a large language model, wherein the recognition result includes node information and edge information.

[0166] The embodiment of the present invention obtains original data from a database, slices the original data to obtain data slices, obtains auxiliary information corresponding to the data slices, generates an embedding vector based on the auxiliary information and the data slices, and after obtaining the user's query information, obtains at least one candidate data slice related to the query information based on the content of the data slice and the embedding vector, obtains the correlation score between each candidate data slice and the query information through a pre-trained scoring model, obtains the target data slice from the candidate data slice based on the correlation score, and obtains the recognition result of the security data node based on the target data slice through a large language model. The recognition result includes node information and edge information. As a result, a series of steps in the prior art such as adaptation code language, framework, personalized writing, AST analysis, static engine construction, and rule addition can be omitted, thereby reducing the cost and difficulty of information recognition, reducing the risk of production environment deployment, and improving scenario coverage and data integrity.

[0167] Figure 6 FIG is a schematic diagram of an information identification device according to another embodiment of the present invention. Figure 6 As shown, the information identification device of the embodiment of the present invention includes a query information acquisition unit 61, a slice library acquisition unit 62, a target data acquisition unit 63 and an information identification unit 64. Among them, the query information acquisition unit 61 is used to obtain the user's query information. The slice library acquisition unit 62 is used to obtain a slice library, and the slice library includes multiple data slices. The target data acquisition unit 63 is used to obtain at least one target data slice related to the query information in the slice library. The information identification unit 64 is used to obtain the recognition result of the security data node based on the target data slice through a large language model, and the recognition result includes node information and edge information.

[0168] The embodiment of the present invention obtains original data from a database, slices the original data to obtain data slices, obtains auxiliary information corresponding to the data slices, generates an embedding vector based on the auxiliary information and the data slices, and after obtaining the user's query information, obtains at least one candidate data slice related to the query information based on the content of the data slice and the embedding vector, obtains the correlation score between each candidate data slice and the query information through a pre-trained scoring model, obtains the target data slice from the candidate data slice based on the correlation score, and obtains the recognition result of the security data node based on the target data slice through a large language model. The recognition result includes node information and edge information. As a result, a series of steps in the prior art such as adaptation code language, framework, personalized writing, AST analysis, static engine construction, and rule addition can be omitted, thereby reducing the cost and difficulty of information recognition, reducing the risk of production environment deployment, and improving scenario coverage and data integrity.

[0169] Figure 7Schematic diagram of an electronic device according to an embodiment of the present invention. In this embodiment, the electronic device 7 includes a server, a terminal, etc. Figure 7 As shown, the electronic device 7 includes: at least one processor 71; a memory 72 communicatively connected to the at least one processor 71; and a communication component 73 communicatively connected to the scanning device, the communication component 73 receiving and sending data under the control of the processor 71; wherein the memory 72 stores instructions that can be executed by the at least one processor 71, and the instructions are executed by the at least one processor 71 to implement the above-mentioned information recognition method.

[0170] Specifically, the electronic device includes: one or more processors 71 and a memory 72, Figure 7 A processor 71 is taken as an example. The processor 71 and the memory 72 may be connected via a bus or other means. Figure 7 In the example above, a bus connection is used. Memory 72, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer executable programs, and modules. Processor 71 executes the non-volatile software programs, instructions, and modules stored in memory 72 to perform various functional applications and data processing of the device, thereby implementing the above-mentioned information recognition method.

[0171] The memory 72 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and application programs required for at least one function; the data storage area may store a list of options, etc. In addition, the memory 72 may include a high-speed random access memory and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 72 may optionally include a memory remotely located relative to the processor 71, and these remote memories may be connected to an external device via a network. Examples of the aforementioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0172] One or more modules are stored in the memory 72 and, when executed by one or more processors 71 , perform the information identification method in any of the above method embodiments.

[0173] The above-mentioned product can execute the method provided in the embodiment of this application, and has the functional modules and beneficial effects corresponding to the execution method. For technical details not fully described in this embodiment, please refer to the method provided in the embodiment of this application.

[0174] The embodiment of the present invention obtains raw data from a database, slices the raw data to obtain data slices, obtains auxiliary information corresponding to the data slices, generates embedding vectors based on the auxiliary information and the data slices, and after obtaining the user's query information, obtains at least one candidate data slice related to the query information based on the content of the data slice and the embedding vector. A pre-trained scoring model is used to obtain a relevance score between each candidate data slice and the query information. A target data slice is obtained from the candidate data slices based on the relevance score. A large language model is used to obtain a security data node recognition result based on the target data slice. The recognition result includes node information and edge information. This reduces the cost and difficulty of information recognition and improves scenario coverage.

[0175] Another embodiment of the present invention relates to a non-volatile storage medium for storing a computer-readable program, wherein the computer-readable program is used to enable a computer to execute part or all of the above method embodiments.

[0176] That is, those skilled in the art will understand that all or part of the steps in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a program, which is stored in a storage medium and includes a number of instructions for causing a device (which may be a single-chip microcomputer, chip, etc.) or a processor to execute all or part of the steps of the methods described in the embodiments of the present application. The aforementioned storage medium includes: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc., various media that can store program code.

[0177] The foregoing is merely a preferred embodiment of the present application and is not intended to limit the present application. Persons skilled in the art will readily appreciate that various modifications and variations are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present application are intended to be within the scope of protection of the present application.

Claims

1. An information identification method, characterized in that: The method comprises: Obtaining original data from a database, wherein the original data includes text information in a file, and the file includes at least one of a code file, a MarkDown file, and a TXT file; Slicing the original data to obtain data slices; Obtaining auxiliary information corresponding to the data slice, where the auxiliary information includes one or more of a slice function, a usage scenario, and a correlation with a data link; generating an embedding vector based on the auxiliary information and the data slice; After obtaining the user's query information, obtaining at least one candidate data slice related to the query information according to the content of the data slice and the embedding vector; Obtaining a relevance score between each candidate data slice and the query information through a pre-trained scoring model; Obtaining a target data slice from the candidate data slices according to the relevance score; The recognition result of the security data node is obtained according to the target data slice through the large language model, and the recognition result includes node information and edge information.

2. The method according to claim 1, characterized in that The slicing of the original data to obtain data slices is specifically as follows: The original data is sliced ​​according to a predetermined segmentation granularity to obtain data slices, where the predetermined segmentation granularity includes at least one of a row, a function, a class, and a paragraph.

3. The method according to claim 1, characterized in that The node information includes one or more of a node identifier, a node type, a node description, and a node attribute.

4. The method according to claim 1, wherein The side information includes one or more of data flow direction, link relationship and transmission mode.

5. An information identification method, characterized in that: The method comprises: Get the user's query information; Acquire a slice library, wherein the slice library includes multiple data slices; Acquire at least one target data slice related to the query information in the slice library; The recognition result of the security data node is obtained according to the target data slice through the large language model, and the recognition result includes node information and edge information.

6. The method according to claim 5, characterized in that The obtaining of the slice library comprises: Obtaining original data from a database, wherein the original data includes text information in a file, and the file includes at least one of a code file, a MarkDown file, and a TXT file; Slicing the original data to obtain data slices; The slice library is generated according to the data slices.

7. The method according to claim 6, characterized in that The obtaining of the slice library further comprises: Obtaining auxiliary information corresponding to the data slice, where the auxiliary information includes one or more of a slice function, a usage scenario, and a correlation with a data link; An embedding vector is generated according to the auxiliary information and the data slice.

8. The method according to claim 7, characterized in that Acquiring at least one target data slice related to the query information in the slice library includes: Acquire at least one candidate data slice related to the query information in the slice library using a predetermined retrieval algorithm; Obtaining a relevance score between each candidate data slice and the query information through a pre-trained scoring model; The target data slice is obtained from the candidate data slices according to the relevance score.

9. The method according to claim 8, characterized in that The step of obtaining at least one candidate data slice related to the query information from the slice library using a predetermined retrieval algorithm includes at least one of the following: Obtaining an embedding vector corresponding to the query information, calculating a vector similarity between the embedding vector corresponding to the query information and the embedding vector corresponding to the data slice, and obtaining a candidate data slice related to the query information based on the vector similarity; Acquire candidate data slices related to the query information in the slice library based on keyword retrieval; Based on rule retrieval, candidate data slices related to the query information are obtained in the slice library.

10. An information recognition device, characterized in that: The device comprises: A data acquisition unit, configured to acquire original data from a database, wherein the original data includes text information in a file, and the file includes at least one of a code file, a MarkDown file, and a TXT file; a slicing unit, configured to slice the original data to obtain data slices; an auxiliary information acquisition unit, configured to acquire auxiliary information corresponding to the data slice, the auxiliary information including one or more of a slice function, a usage scenario, and a correlation with a data link; an embedding vector generating unit, configured to generate an embedding vector according to the auxiliary information and the data slice; a candidate slice acquiring unit, configured to, after acquiring the user's query information, acquire at least one candidate data slice related to the query information based on the content of the data slice and the embedding vector; A score calculation unit, configured to obtain a relevance score between each candidate data slice and the query information using a pre-trained score model; a screening unit, configured to obtain a target data slice from the candidate data slices according to the relevance score; A result recognition unit is used to obtain a recognition result of a security data node according to the target data slice through a large language model, and the recognition result includes node information and edge information.

11. An information recognition device, characterized in that: The device comprises: A query information acquisition unit, used to acquire the user's query information; A slice library acquisition unit, configured to acquire a slice library, wherein the slice library includes a plurality of data slices; a target data acquisition unit, configured to acquire at least one target data slice related to the query information from the slice library; An information identification unit is used to obtain an identification result of a security data node according to the target data slice through a large language model, wherein the identification result includes node information and edge information.

12. An electronic device comprising a memory and a processor, characterized in that: The memory is configured to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method according to any one of claims 1 to 9.

13. A computer program product comprising a computer program, characterized in that When the computer program is run on a computer, the computer executes the method according to any one of claims 1 to 9.

14. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: The computer program instructions implement the method according to any one of claims 1 to 9 when executed by a processor.

Images