Data search method and device, electronic equipment and computer readable medium
Through the combination of word segmentation, word frequency statistics and feature extraction models, the problem of low direct string matching efficiency in code leakage investigation is solved, and efficient and accurate code snippet search is achieved.
Patent Information
- Application Number
- CN202411416659.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-11
- Publication Date
- 2024-12-31
AI Technical Summary
In the process of code leakage investigation, the existing technology uses string matching in a large number of code files directly to match code fragments, which is inefficient and difficult to efficiently find the file containing the fragment.
The target keyword is determined through word segmentation, word frequency statistics and importance scores, and the feature extraction model is called to extract the feature values. A similarity search is performed based on the feature vector library, and a feature vector with similarity higher than the preset threshold is found to determine the target search results.
Improves search efficiency, accuracy and efficiency when code leaks, and enables quick find files containing specific code snippets.
Smart Images
Figure CN119226445A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data search technology, and in particular to a data search method, device, electronic device and computer-readable medium. Background Art
[0002] In the process of troubleshooting code leaks, a large number of code files are searched according to a given code snippet to find the file containing the code snippet. Through direct matching, a string is matched in a large number of code files according to a given code snippet to find the code file where the snippet appears. It is inefficient to only match the string in a large number of code files according to the code snippet to find the code file where the snippet appears. Summary of the invention
[0003] In view of this, embodiments of the present application provide a data search method, device, electronic device and computer-readable medium, which can solve the problem of low efficiency of existing methods of only performing string matching in a large number of code files based on code snippets to find code files where the snippet appears.
[0004] To achieve the above object, according to one aspect of an embodiment of the present application, a data search method is provided, comprising:
[0005] In response to a data search request, corresponding source code data is obtained, and word segmentation is performed on the source code data to obtain individual words;
[0006] Determine the length of each word, and count the frequency of each word in the source code data based on the length;
[0007] According to the word frequency, determine the importance score of each word;
[0008] Based on the importance score, the target keywords are determined, and then the target database and target data fragments are determined based on the target keywords;
[0009] The feature extraction model is called to extract the feature value of the target data segment, obtain the feature vector library corresponding to the target database, search in the feature vector library based on the feature value, and then determine the target search result data based on the feature vector whose similarity is higher than the preset threshold.
[0010] Optionally, the source code data includes private source code data and open source source code data, and the frequency of each word in the source code data is counted according to the length, including:
[0011] The number of times words with a length greater than a preset threshold appear in the private source code data and the open source source code data is counted respectively.
[0012] Optionally, based on the word frequency, determine the importance score of each word, including:
[0013] Determine the weight corresponding to the number of times a word with a length greater than a preset threshold appears in the open source code data;
[0014] The set basic weight is obtained, and the importance score of each word is determined based on the number of times, weights, and basic weights of words with a length greater than a preset threshold that appear in private source code data and open source source code data.
[0015] Optionally, target keywords are determined based on importance scores, including:
[0016] The words corresponding to the importance scores greater than the preset score threshold are determined as target keywords.
[0017] Optionally, according to the target keyword, determining the target database and the target data segment includes:
[0018] Assembling the target keywords according to the preset sorting type and the preset sorting method to obtain assembled keywords;
[0019] Search data warehouses and code bases based on assembly keywords to obtain corresponding target databases and target data fragments.
[0020] Optionally, determining target search result data according to feature vectors whose similarity is higher than a preset threshold includes:
[0021] The data in the target database corresponding to the feature vector with a similarity higher than a preset threshold is determined as the target search result data, wherein the similarity is the similarity between the feature value and each feature vector in the feature vector library.
[0022] In addition, the present application also provides a data search device, comprising:
[0023] A word segmentation unit is configured to obtain corresponding source code data in response to a data search request, and perform word segmentation on the source code data to obtain individual words;
[0024] A word frequency counting unit is configured to determine the length of each word and count the word frequency of each word in the source code data according to the length;
[0025] A first determination unit is configured to determine the importance score of each word according to the word frequency;
[0026] A second determination unit is configured to determine a target keyword based on the importance score, and further determine a target database and a target data segment according to the target keyword;
[0027] The data search unit is configured to call a feature extraction model to extract feature values of a target data segment, obtain a feature vector library corresponding to a target database, search in the feature vector library based on the feature values, and then determine target search result data based on feature vectors whose similarity is higher than a preset threshold.
[0028] Optionally, the source code data includes private source code data and open source source code data, and the word frequency statistics unit is further configured to:
[0029] The number of times words with a length greater than a preset threshold appear in the private source code data and the open source source code data is counted respectively.
[0030] Optionally, the first determining unit is further configured to:
[0031] Determine the weight corresponding to the number of times a word with a length greater than a preset threshold appears in the open source code data;
[0032] The set basic weight is obtained, and the importance score of each word is determined based on the number of occurrences, weights, and basic weights of words with a length greater than a preset threshold in the private source code data and the open source source code data.
[0033] Optionally, the second determining unit is further configured to:
[0034] The words corresponding to the importance scores greater than the preset score threshold are determined as target keywords.
[0035] Optionally, the second determining unit is further configured to:
[0036] Assembling the target keywords according to the preset sorting type and the preset sorting method to obtain assembled keywords;
[0037] Search data warehouses and code bases based on assembly keywords to obtain corresponding target databases and target data fragments.
[0038] Optionally, the data search unit is further configured to:
[0039] The data in the target database corresponding to the feature vector with a similarity higher than a preset threshold is determined as the target search result data, wherein the similarity is the similarity between the feature value and each feature vector in the feature vector library.
[0040] In addition, the present application also provides a data search electronic device, including: one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by one or more processors, the one or more processors implement the data search method as described above.
[0041] In addition, the present application also provides a computer-readable medium on which a computer program is stored, and when the program is executed by a processor, the data search method as described above is implemented.
[0042] To achieve the above objective, according to another aspect of the embodiments of the present application, a computer program product is provided.
[0043] A computer program product according to an embodiment of the present application includes a computer program, and when the program is executed by a processor, the data search method provided by the embodiment of the present application is implemented.
[0044] One embodiment of the above invention has the following advantages or beneficial effects: the application obtains the corresponding source code data in response to a data search request, performs word segmentation on the source code data to obtain each word; determines the length of each word, and counts the word frequency of each word in the source code data according to the length; determines the importance score of each word according to the word frequency; determines the target keyword based on the importance score, and then determines the target database and target data segment according to the target keyword; calls the feature extraction model to extract the feature value of the target data segment, obtains the feature vector library corresponding to the target database, searches in the feature vector library based on the feature value, and then determines the target search result data according to the feature vector with a similarity higher than a preset threshold. Improve the search efficiency of leaked data when data leakage occurs.
[0045] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with the specific implementation manner. BRIEF DESCRIPTION OF THE DRAWINGS
[0046] The accompanying drawings are used to better understand the present application and do not constitute an improper limitation on the present application.
[0047] Figure 1 is a schematic diagram of the main process of a data search method according to an embodiment of the present application;
[0048] Figure 2 is a schematic diagram of the main process of a data search method according to an embodiment of the present application;
[0049] Figure 3 It is a main flow chart of a data search method according to an embodiment of the present application;
[0050] Figure 4 is a schematic diagram of main units of a data search device according to an embodiment of the present application;
[0051] Figure 5 is an exemplary system architecture diagram to which the embodiments of the present application can be applied;
[0052] Figure 6It is a structural diagram of a computer system of a terminal device or server suitable for implementing an embodiment of the present application. DETAILED DESCRIPTION
[0053] The following is a description of the exemplary embodiments of the present application in conjunction with the accompanying drawings, including various details of the embodiments of the present application to facilitate understanding, and they should be considered as merely exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present application. Similarly, for the sake of clarity and conciseness, the description of known functions and structures is omitted in the following description. It should be noted that the acquisition, transmission, storage, use, and processing of data in the technical solution of the present application are in compliance with the relevant provisions of national laws and regulations. It should be noted that in the embodiments of the present application, certain software, components, models, and other existing solutions in the industry may be mentioned, which should be considered as exemplary, and their purpose is only to illustrate the feasibility of the implementation of the technical solution of the present application, but it does not mean that the applicant has or must have used the solution. In the technical solution of the present application, the collection, analysis, use, transmission, storage, and other aspects of the user's personal information involved are in compliance with the provisions of relevant laws and regulations, and are used for legal and reasonable purposes, and are not shared, leaked, or sold outside of these legal uses, and are subject to supervision and management by regulatory authorities. Take necessary measures to prevent illegal access to such personal information, ensure that persons with access to personal information comply with relevant laws and regulations, and ensure the security of personal information. Once such personal information is no longer needed, risks should be minimized by limiting or even prohibiting data collection and / or deleting data.
[0054] When used, including in certain related applications, user privacy is protected by de-identifying data, such as by removing specific identifiers when used, controlling the amount or specificity of data stored, controlling how data is stored, and / or other methods of de-identification.
[0055] Figure 1 is a schematic diagram of the main process of a data search method according to an embodiment of the present application, such as Figure 1 As shown, the data search method mainly includes the following steps S101 to S105.
[0056] Step S101, in response to a data search request, obtain corresponding source code data, and perform word segmentation on the source code data to obtain various words.
[0057] In this embodiment, the execution subject of the data search method (for example, a server) can receive a data search request through a wired connection or a wireless connection. The data to be searched can be leaked code data. After receiving the data search request, the execution subject can obtain the source code data corresponding to the request. For example, the source code data can include private source code and part of the open source source code. The source code data is segmented based on the length of the vocabulary to obtain each vocabulary.
[0058] Step S102, determining the length of each word, and counting the frequency of each word in the source code data according to the length.
[0059] In the source code data, the words whose length is greater than a preset threshold are counted for word frequency to obtain the frequency of each word.
[0060] Step S103, determining the importance score of each word according to the word frequency.
[0061] Determine the source code data type corresponding to the vocabulary corresponding to each word frequency, determine the weight of the vocabulary corresponding to each word frequency according to the source code data type, and then determine the importance score of each word based on the weight and word frequency. The importance score depends on the source code data type, so that the determined importance score is more accurate.
[0062] Step S104, determining the target keyword based on the importance score, and then determining the target database and the target data segment according to the target keyword.
[0063] Specifically, determining the target keyword based on the importance score includes: determining the vocabulary corresponding to the importance score greater than a preset score threshold as the target keyword.
[0064] The target keywords are stored in a keyword library (e.g., a code keyword library). The target keywords are used to search the warehouse and search the code to accurately obtain the target database (e.g., the code warehouse corresponding to the target keywords) and the target data fragment (e.g., the code fragment information corresponding to the target keywords).
[0065] Specifically, according to the target keywords, the target database and the target data fragment are determined, including: assembling the target keywords according to a preset sorting type and a preset sorting method to obtain assembled keywords; searching the data warehouse and the code library based on the assembled keywords to obtain the corresponding target database and the target data fragment.
[0066] According to the parameters configured when searching the warehouse and searching the code, such as the preset sorting type and the preset sorting method, the target keywords are assembled with the corresponding preset sorting type and the preset sorting method to obtain the assembled keywords, so as to facilitate the accurate search of the data warehouse and the code library according to the assembled keywords to obtain the corresponding target database and target data fragment. Among them, the target database can be the data warehouse where the suspected leaked code is located, and the target data fragment can be the file or record where the suspected leaked code is located.
[0067] Step S105, calling the feature extraction model to extract the feature value of the target data segment, obtaining the feature vector library corresponding to the target database, searching in the feature vector library based on the feature value, and then determining the target search result data according to the feature vector whose similarity is higher than the preset threshold.
[0068] The code analysis large language pre-training model (such as CodeBERT) is trained through a keyword library (such as a code keyword library) so that the similarity of each code feature value from the positive sample is as close to 1 as possible, and the similarity of each code feature value from the negative sample is as close to 0 as possible. Through this step, a suitable code feature extraction model is obtained. Among them, for the construction of positive samples, it is necessary to use the syntax analysis tree AST tool to parse the keyword library (such as the code keyword library), and then randomly replace the synonymous attribute names, variable names, and method names as positive samples (similarity = 1.0); for different codes (including codes after AST analysis and synonymous replacement), construct negative samples (similarity = 0.0).
[0069] Specifically, the target search result data is determined based on the feature vector whose similarity is higher than a preset threshold, including: determining the data in the target database corresponding to the feature vector whose similarity is higher than the preset threshold as the target search result data, wherein the similarity is the similarity between the feature value and each feature vector in the feature vector library.
[0070] The target database may be a data warehouse where the suspected leaked code is located, and the target data segment may be a file or record where the suspected leaked code is located. By determining the data in the target database corresponding to the feature vector with a similarity higher than a preset threshold, the target search result data can be accurately determined.
[0071] This embodiment obtains the corresponding source code data in response to a data search request, performs word segmentation on the source code data to obtain each word; determines the length of each word, and counts the word frequency of each word in the source code data according to the length; determines the importance score of each word according to the word frequency; determines the target keyword based on the importance score, and then determines the target database and target data segment according to the target keyword; calls the feature extraction model to extract the feature value of the target data segment, obtains the feature vector library corresponding to the target database, searches in the feature vector library based on the feature value, and then determines the target search result data according to the feature vector with a similarity higher than a preset threshold. Improve the search efficiency of leaked data when data leakage occurs.
[0072] Figure 2 is a schematic diagram of the main flow of a data search method according to an embodiment of the present application. Figure 2 As shown, the data search method mainly includes the following steps S201 to S206.
[0073] Step S201, in response to a data search request, obtain corresponding source code data, and perform word segmentation on the source code data to obtain various words.
[0074] Source code data includes private source code data and open source source code data. The execution subject can remove meaningless words in the source code data to achieve word segmentation processing of the source code data, and can quickly and accurately obtain each word.
[0075] Step S202, determining the length of each word, and counting the number of times words with a length greater than a preset threshold appear in the private source code data and the open source source code data.
[0076] The length of each word refers to the number of characters in each word. The execution entity may compare the length of each word with a preset threshold to determine the number of times a word having a length greater than the preset threshold appears in the private source code data and the open source source code data.
[0077] Step S203, determining the weight corresponding to the number of times a word having a length greater than a preset threshold appears in the open source code data.
[0078] The execution subject can determine the source code data type corresponding to the vocabulary with a length greater than the preset threshold, call the weight configuration database according to the source code data type, and search the key-value pair corresponding to the source code data type from the weight configuration database, and determine the weight corresponding to the number of times the vocabulary with a length greater than the preset threshold appears in the open source source code data through the key-value pair found. Specifically, the value in the key-value pair found can be determined as the weight corresponding to the number of times the vocabulary with a length greater than the preset threshold appears in the open source source code data, so as to ensure the configurability and accuracy of the weight determination.
[0079] Step S204, obtaining the set basic weight, and determining the importance score of each word based on the number of occurrences, weights and basic weights of words with a length greater than a preset threshold in the private source code data and the open source source code data.
[0080] The execution entity can obtain the word frequency of each word in the source code data after performing word segmentation on the source code data, and determine the number of words whose word frequency is lower than the preset word frequency threshold, and determine the basic weight according to the number of words whose word frequency is lower than the preset word frequency threshold to avoid excessive weight of words with too low frequency, thereby ensuring the accuracy of data search.
[0081] For example, the importance score is calculated for each word whose length is greater than a preset threshold. The formula is as follows:
[0082] Importance score of a certain word = (number of times it appears in private source code - number of times it appears in open source code * weight a) / (number of times it appears in private source code + number of times it appears in open source code * weight a + basic weight b).
[0083] Among them, weight a is used to control the weight ratio of feature words appearing in open source source code (for example, set to 5), and basic weight b is used to avoid excessive weight of words with low frequency. If a word appears more times in private source code and less times in open source code, the importance score of the word will be higher. If the importance score exceeds a certain threshold, it will enter the code keyword library and can be used to optimize the code feature extraction model to improve the accuracy of data search.
[0084] Step S205, determining the target keyword based on the importance score, and then determining the target database and the target data segment according to the target keyword.
[0085] The words corresponding to the importance scores greater than the preset score threshold are determined as target keywords. The target keywords are used to search the warehouse and search the code respectively, so as to accurately obtain the target database (such as the code warehouse corresponding to the target keyword) and the target data fragment (such as the code fragment information corresponding to the target keyword).
[0086] Step S206, calling the feature extraction model to extract the feature value of the target data segment, obtaining the feature vector library corresponding to the target database, searching in the feature vector library based on the feature value, and then determining the target search result data according to the feature vector whose similarity is higher than the preset threshold.
[0087] For example, the obtained feature extraction model, such as a code feature extraction model, is used to calculate the code features of the target data fragment. The code feature extraction model can be used to calculate the feature values of the private source code, and obtain the code number, project name, code file name, code file content, and code feature value; download code number, download project name, download code file name, download code content, and download code feature value. The code number and code feature value are input into a feature vector library, such as a code feature vector library to construct initial data. For each downloaded code feature value, a search is performed in the code feature vector library, and the code data with a similarity higher than a threshold is determined to be suspected to be similar, and then the final output result is obtained. The final output result can be: download code number, download project name, download code file name, download code content, and suspected leaked code result data (i.e., the target search result data of the embodiment of the present application), such as the code number, project name, code file name, and code file content of the suspected similar code, which can be used for subsequent analysis by personnel.
[0088] Figure 3It is a main flow diagram of a data search method according to an embodiment of the present application. In the embodiment of the present application, elasticsearch: is a search server based on Lucene. The implementation principle is divided into the following steps. First, the user submits the data to the Elasticsearch database, and then the corresponding statement is segmented through the segmentation controller, and its weight and segmentation result are stored in the data together. When the user searches for data, the results are ranked and scored according to the weight, and then the returned results are presented to the user. Index: It is a logical storage of elasticsearch, which can be understood as a database in a relational database. es can store index data on one server, or it can be stored on multiple servers after sharding. Each index has one or more shards, and each shard can have multiple copies. In es, an index object can store multiple objects with different purposes. Different objects in a single index can be distinguished by the index type (index_type), which can be understood as a table in a relational database. Each index type can have a different structure, but different index types cannot set different types for the same attribute. The main entity stored in es is called a document, which can be understood as a row of records in a table in a relational database. Each document consists of multiple fields. es is an unstructured database. Each document can have different fields and has a unique identifier. Word segmentation: accept a string as input and split the string into independent words or vocabulary units (some punctuation characters may be discarded). Elasticsearch will build indexes based on these words to prepare for search. Exact value search: that is, term query, which is used for exact value matching. These exact values may be numbers, times, Booleans, or those not_analyzed strings. If term query is used, es will search in the specified field according to the given string. For example, if the given string is "hello", all the original texts that have been word-segmented and contain the word hello will be queried. Combined filter: three keywords can be used to combine complex query conditions. must: all statements must match, which is equivalent to AND, must_not: all statements cannot match, which is equivalent to NOT, should: at least one statement must match, which is equivalent to OR. Match query: match query, this query will process the given string through word segmentation, and the query results will be returned in order of relevance. Phrase matching: Similar to the match query, the match_phrase query first parses the query string into a list of terms, then searches for those terms, but only retains documents that contain all of the search terms in the same position as the search terms.For example, a phrase search for quick fox may not match any documents because no document contains the word quick followed by fox. Word frequency: It is a common weighting technique used in intelligence retrieval and text mining to evaluate the repetition of a word for a document or a set of documents in a field in a corpus. The importance of a word increases proportionally with the number of times it appears in a document, but it decreases inversely with the frequency of its appearance in the corpus. LLM (Large Language Mode), LLM refers to a large-scale language model built based on deep learning technology. LLM is a powerful natural language processing tool that can understand and generate text. Here are some introductions and important concepts about LLM: Pre-training: LLM is usually built by pre-training on a large text corpus. The pre-training stage refers to the model learning the statistical laws and semantic relationships of the language under unsupervised conditions. Pre-training enables the model to acquire rich language knowledge and representation capabilities. Contextual understanding: LLM can generate coherent text through contextual understanding. It can understand the semantics, context and related information of the previous text and generate subsequent text based on this information. This makes LLM have excellent performance in tasks such as question answering, dialogue generation, and automatic summarization. Multi-domain application: LLM can adapt to natural language processing tasks in multiple fields. It can be used for various tasks such as machine translation, text summarization, sentiment analysis, named entity recognition, dialogue systems, etc. Through fine-tuning or transfer learning, LLM can be adjusted to specific fields according to specific tasks. Generating text: LLM is able to generate text related to the input context. This makes it widely used in scenarios such as generating dialogues, automatic writing, language creation, and chatbots. The generated text is usually based on the model's learning and understanding of a large-scale corpus. Prompt refers to the input text or question provided to the model to guide it to generate the corresponding reply or output. Prompt is a way to interact with the model. It can be a short sentence, a question, a description, or any other form of text.
[0089] The overall structure of this program is as follows Figure 3As shown in the figure, it mainly includes four modules: ① code keyword module, ② code snippet acquisition module, ③ code feature training and calculation module, and ④ code feature search module. Specifically, ① code keyword module, this part finally generates code keywords for searching in the code repository. These keywords are composed of two parts, namely TF-IDF calculation keywords and manual maintenance keywords. Specifically, for private source code and some open source source code, keywords are calculated by TF-IDF. TF-IDF (term frequency-inverse document frequency) is a commonly used weighting technology for information retrieval and data mining. TF is term frequency, and IDF is inverse document frequency. Word segmentation is performed uniformly. For words with a length greater than 2, word frequency statistics are performed. For each word with a length greater than the preset threshold, the importance score is calculated. The formula is as follows: Importance score of a certain word = (number of occurrences in private source code - number of occurrences in open source code * weight a) / (number of occurrences in private source code + number of occurrences in open source code * weight a + basic weight b). Among them, weight a is used to control the weight ratio of feature words appearing in open source source code (for example, set to 5), and basic weight b is used to avoid overweighting words with low frequency. If a word appears more times in private source code and less times in open source code, the importance score of the word will be higher. Words with importance scores exceeding a certain threshold will be included in the code keyword library. Manual maintenance of keywords can manually supplement and maintain the company's important feature words.
[0090] ② Code snippet acquisition module, the code snippet acquirer of this module can execute code search and download interfaces, specifically, it can execute search interfaces of different code repositories, such as GITHUB interface, GITEE interface, other repository interfaces, etc. Take GITHUB interface as an example: Search repository Search repositories, GET / search / repositories.
[0091] When searching for a repository, the optional parameters are:
[0092] The keyword q can contain other attributes, such as user, topic, etc. The format is q= <keyword>+user: <username>+language: <language>.
[0093] Search code Search Code, GET / search / code. When searching, only the default branch will be involved, in most cases, the branch is master. Only files smaller than 384KB will be searched. When searching source code, you must include a keyword. For example, language:go is invalid, but amazing language:go is valid. When searching code, the optional parameters are as follows:
[0094] Similarly, you can add parameter values such as filename, language, user, etc. The final code repository + code snippet information is the target database + target data snippet.
[0095] ③ Code feature training and calculation module: First, the code needs to be preprocessed: remove the comments; remove the declarations that introduce other modules, packages, classes, etc.; all blank spaces are replaced with 1 space; secondly, the preprocessed code needs to be data enhanced, that is, for the training data preparation part, positive and negative samples need to be constructed. For the positive sample construction, it is necessary to use the syntax analysis tree AST tool to parse, and then randomly replace the synonymous attribute names, variable names, and method names as positive samples (similarity = 1.0); for different codes (including codes after AST analysis and synonymous replacement), construct negative samples (similarity = 0.0); then, use the above prepared training data to train the code feature extraction model, that is, train the code analysis large language pre-training model (such as CodeBERT), so that the similarity of each code feature value from the positive sample is as close to 1 as possible, and the similarity of each code feature value from the negative sample is as close to 0 as possible. This step obtains a suitable code feature extraction model.
[0096] Finally, the obtained code feature extraction model is used to calculate the code features. The code feature extraction model can be used to calculate the feature values of private source code to obtain the code number, project name, code file name, code file content, and code feature value; download code number, download project name, download code file name, download code content, and download code feature value.
[0097] ④ Code feature search module: Input the code feature value obtained by calculating the code number and code feature into the code feature vector library to construct the initial data. For each download code feature value, search in the code feature vector library, and determine the code data with a similarity higher than the threshold as suspected similar code data, and then obtain the final output result, which is: download code number, download project name, download code file name, download code content, and suspected leaked code result data (i.e., the target search result data of the embodiment of the present application), such as the code number, project name, code file name, and code file content of the suspected similar code, which can be used for subsequent analysis by personnel.
[0098] The embodiments of the present application can adapt to similar code searches in which the names of classes, functions, variables, etc. in the code have changed but the subject logic remains unchanged. Code features can be updated through self-learning. Use automatic calculation + manual maintenance of keywords to improve the accuracy of code snippet searches. Establish a code feature extraction and self-learning mechanism to effectively deal with situations where the code has undergone certain changes, and the matching is more accurate. Construct a code feature vector retrieval mechanism to increase the speed of code feature searches. It can repeatedly self-iterate learning and support repeated execution, and can periodically execute tasks to maintain the timeliness of search results. It can be implemented to quickly and accurately search in a large number of code files based on the given code snippet, find the file containing the code snippet, improve data search efficiency, increase the reference value of search results, and more accurately match suspected leaked code.
[0099] Figure 4 is a schematic diagram of the main units of the data search device according to an embodiment of the present application. Figure 4 As shown, the data search device 400 includes a word segmentation unit 401 , a word frequency statistics unit 402 , a first determination unit 403 , a second determination unit 404 and a data search unit 405 .
[0100] The word segmentation unit 401 is configured to obtain corresponding source code data in response to a data search request, and perform word segmentation on the source code data to obtain various words.
[0101] The word frequency counting unit 402 is configured to determine the length of each word and count the word frequency of each word in the source code data according to the length.
[0102] The first determining unit 403 is configured to determine the importance score of each word according to the word frequency.
[0103] The second determination unit 404 is configured to determine a target keyword based on the importance score, and further determine a target database and a target data segment according to the target keyword.
[0104] The data search unit 405 is configured to call the feature extraction model to extract the feature value of the target data segment, obtain the feature vector library corresponding to the target database, search in the feature vector library based on the feature value, and then determine the target search result data according to the feature vector whose similarity is higher than a preset threshold.
[0105] In some embodiments, the source code data includes private source code data and open source code data, and the word frequency counting unit 402 is further configured to: count the number of times a word with a length greater than a preset threshold appears in the private source code data and the open source code data respectively.
[0106] In some embodiments, the first determination unit 403 is further configured to: determine the weight corresponding to the number of times a word with a length greater than a preset threshold appears in the open source source code data; obtain the set basic weight, and determine the importance score of each word based on the number of times a word with a length greater than the preset threshold appears in the private source code data and the open source source code data, the weight and the basic weight.
[0107] In some embodiments, the second determining unit 404 is further configured to: determine a word corresponding to an importance score greater than a preset score threshold as a target keyword.
[0108] In some embodiments, the second determination unit 404 is further configured to: assemble the target keywords according to a preset sorting type and a preset sorting method to obtain assembled keywords; search the data warehouse and the code library based on the assembled keywords to obtain the corresponding target database and target data fragments.
[0109] In some embodiments, the data search unit 405 is further configured to: determine the data in the target database corresponding to the feature vector with a similarity higher than a preset threshold as the target search result data, wherein the similarity is the similarity between the feature value and each feature vector in the feature vector library.
[0110] It should be noted that the data search method and the data search device of the present application have a corresponding relationship in terms of specific implementation contents, so the repeated contents will not be described again.
[0111] Figure 5 An exemplary system architecture 500 is shown to which the data search method or data search device according to the embodiments of the present application can be applied.
[0112] like Figure 5 As shown, system architecture 500 may include terminal devices 501, 502, 503, a network 504 and a server 505. Network 504 is used to provide a medium for communication links between terminal devices 501, 502, 503 and server 505. Network 504 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.
[0113] Users can use terminal devices 501, 502, 503 to interact with server 505 through network 504 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 501, 502, 503, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only for example).
[0114] The terminal devices 501 , 502 , and 503 may be various electronic devices having a data search processing screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.
[0115] Server 505 can be a server that provides various services, such as a background management server that provides support for data search requests submitted by users using terminal devices 501, 502, and 503 (only as an example). The background management server can respond to the data search request, obtain the corresponding source code data, perform word segmentation on the source code data to obtain each word; determine the length of each word, and count the word frequency of each word in the source code data according to the length; determine the importance score of each word according to the word frequency; determine the target keyword based on the importance score, and then determine the target database and target data segment based on the target keyword; call the feature extraction model to extract the feature value of the target data segment, obtain the feature vector library corresponding to the target database, search in the feature vector library based on the feature value, and then determine the target search result data according to the feature vector with a similarity higher than a preset threshold. Improve the search efficiency of leaked data when data leakage occurs.
[0116] It should be noted that the data search method provided in the embodiment of the present application is generally executed by the server 505 , and accordingly, the data search device is generally set in the server 505 .
[0117] It should be understood that Figure 5 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.
[0118] Reference below Figure 6 , which shows a schematic diagram of the structure of a computer system 600 of a terminal device suitable for implementing an embodiment of the present application. Figure 6 The terminal device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.
[0119] like Figure 6 As shown, the computer system 600 includes a central processing unit (CPU) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage part 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the computer system 600 are also stored. The CPU 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.
[0120] The following components are connected to the I / O interface 605: an input section 606 including a keyboard, a mouse, etc.; an output section 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, a modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 610 as needed, so that a computer program read therefrom is installed into the storage section 608 as needed.
[0121] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 609, and / or installed from the removable medium 611. When the computer program is executed by the central processing unit (CPU) 601, the above-mentioned functions defined in the system of the present application are executed.
[0122] It should be noted that the computer-readable medium shown in the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. Computer-readable storage media may include, for example, but are not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.
[0123] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the above-mentioned module, program segment or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0124] The units involved in the embodiments described in the present application may be implemented by software or hardware. The units described may also be set in a processor, for example, it may be described as: a processor includes a word segmentation unit, a word frequency statistics unit, a first determination unit, a second determination unit, and a data search unit. The names of these units do not, in some cases, constitute limitations on the units themselves.
[0125] As another aspect, the present application also provides a computer-readable medium, which may be included in the device described in the above embodiment; or it may exist independently and not be assembled into the device. The above computer-readable medium carries one or more programs. When the above one or more programs are executed by a device, the device responds to a data search request, obtains the corresponding source code data, performs word segmentation on the source code data to obtain each word; determines the length of each word, and counts the word frequency of each word in the source code data according to the length; determines the importance score of each word according to the word frequency; determines the target keyword based on the importance score, and then determines the target database and target data segment according to the target keyword; calls the feature extraction model to extract the feature value of the target data segment, obtains the feature vector library corresponding to the target database, searches in the feature vector library based on the feature value, and then determines the target search result data according to the feature vector with a similarity higher than a preset threshold. Improve the search efficiency of leaked data when data leakage occurs.
[0126] The computer program product of the present application includes a computer program, and when the computer program is executed by a processor, the data search method in the embodiment of the present application is implemented.
[0127] According to the technical solution of the embodiment of the present application, the efficiency of searching for leaked data when data leakage occurs can be improved.
[0128] The above specific implementations do not constitute a limitation on the protection scope of this application. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of this application should be included in the protection scope of this application.< / language> < / username> < / keyword>
Claims
1. A data search method, characterized in that: include: In response to a data search request, corresponding source code data is acquired, and word segmentation is performed on the source code data to obtain individual words; Determine the length of each word, and count the frequency of each word in the source code data according to the length; Determining the importance score of each word according to the word frequency; Based on the importance score, determining a target keyword, and then determining a target database and a target data segment according to the target keyword; A feature extraction model is called to extract the feature value of the target data segment, a feature vector library corresponding to the target database is obtained, a search is performed in the feature vector library based on the feature value, and then target search result data is determined based on feature vectors whose similarity is higher than a preset threshold.
2. The method according to claim 1, characterized in that The source code data includes private source code data and open source source code data, and the counting of the frequency of each word in the source code data according to the length includes: The number of times words having a length greater than a preset threshold value appear in the private source code data and the open source source code data is counted respectively.
3. The method according to claim 2, characterized in that Determining the importance score of each word according to the word frequency includes: Determine a weight corresponding to the number of times a word having a length greater than a preset threshold appears in the open source code data; The set basic weight is obtained, and the importance score of each word is determined based on the number of times the word with a length greater than a preset threshold appears in the private source code data and the open source source code data, the weight and the basic weight.
4. The method according to claim 1, characterized in that: Determining the target keyword based on the importance score includes: The words corresponding to the importance scores greater than the preset score threshold are determined as target keywords.
5. The method according to claim 1, characterized in that The step of determining a target database and a target data segment according to the target keyword includes: Assembling the target keywords according to a preset sorting type and a preset sorting method to obtain assembled keywords; The data warehouse and the code base are searched based on the assembly keywords to obtain the corresponding target database and target data fragments.
6. The method according to any one of claims 1 to 5, characterized in that The step of determining target search result data according to feature vectors with a similarity higher than a preset threshold value includes: The data in the target database corresponding to the feature vector with a similarity higher than a preset threshold is determined as the target search result data, wherein the similarity is the similarity between the feature value and each feature vector in the feature vector library.
7. A data search device, characterized in that: include: A word segmentation unit is configured to obtain corresponding source code data in response to a data search request, and perform word segmentation on the source code data to obtain individual words; A word frequency counting unit is configured to determine the length of each word, and count the word frequency of each word in the source code data according to the length; A first determining unit is configured to determine the importance score of each word according to the word frequency; A second determination unit is configured to determine a target keyword based on the importance score, and further determine a target database and a target data segment according to the target keyword; The data search unit is configured to call a feature extraction model to extract the feature value of the target data segment, obtain a feature vector library corresponding to the target database, search in the feature vector library based on the feature value, and then determine the target search result data according to the feature vector whose similarity is higher than a preset threshold.
8. The device according to claim 7, characterized in that The source code data includes private source code data and open source source code data, and the word frequency statistics unit is further configured to: The number of times words having a length greater than a preset threshold value appear in the private source code data and the open source source code data is counted respectively.
9. The device according to claim 8, characterized in that The first determining unit is further configured to: Determine a weight corresponding to the number of times a word having a length greater than a preset threshold appears in the open source code data; The set basic weight is obtained, and the importance score of each word is determined based on the number of times the word with a length greater than a preset threshold appears in the private source code data and the open source source code data, the weight and the basic weight.
10. The device according to claim 7, characterized in that The second determining unit is further configured to: The words corresponding to the importance scores greater than the preset score threshold are determined as target keywords.
11. The device according to claim 7, characterized in that The second determining unit is further configured to: Assembling the target keywords according to a preset sorting type and a preset sorting method to obtain assembled keywords; The data warehouse and the code base are searched based on the assembly keywords to obtain the corresponding target database and target data fragments.
12. The device according to any one of claims 7 to 11, characterized in that The data search unit is further configured to: The data in the target database corresponding to the feature vector with a similarity higher than a preset threshold is determined as the target search result data, wherein the similarity is the similarity between the feature value and each feature vector in the feature vector library.
13. A data search electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6.
14. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.
15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.
Citation Information
Cited By
Construction method and system for local knowledge information database
CN120162317A