Data classification retrieval method and system based on AI

Abstract

The invention discloses an AI-based data classification retrieval method and system, and the method comprises the steps: obtaining a to-be-retrieved data segment, and extracting a keyword group; performing classification identification on the keywords, establishing a search index, and performing retrieval in a database based on the search index to obtain a search tree; extracting a first feature vector of the search index and a second feature vector of the search tree, and calculating similarity; and performing optimization based on a similarity arrangement result. According to the method, the to-be-retrieved data segment is split, the keyword groups in the to-be-retrieved data segment are extracted according to the word frequency and the weight, and the keyword groups are classified and pooled, so that the complexity of data is reduced, the accuracy of the retrieval process is improved, and meanwhile, the retrieval efficiency is improved through similarity calculation of the first feature vector and the second feature vector. The quality of the retrieval result is improved, the problem of semantic deviation in a traditional text matching mode is solved, and the retrieval efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the field of big data information technology, and in particular to an AI-based data classification and retrieval method and system. Background Art

[0002] With the rapid development of technologies such as the Internet and the Internet of Things, the amount of data has exploded. In the field of artificial intelligence, a large amount of data is collected for model training, testing, and optimization. For example, in the field of image recognition, there are massive image data from various monitoring devices, satellite remote sensing, medical imaging equipment, etc.; in the field of natural language processing, there are large amounts of text data such as social media texts, news articles, and academic papers.

[0003] Currently, users select keywords for search content and enter the keywords. The system searches the database for documents or records containing these keywords through simple text matching, and then filters out the search results and presents the results to the user.

[0004] However, the above text matching method has poor processing capabilities for synonyms and near-synonyms, which in turn causes semantic deviation from the search content and fails to present the content required by the user. Summary of the invention

[0005] Based on this, the purpose of the present invention is to provide an AI-based data classification retrieval method and system, which aims to solve the problem that the text matching method in the current retrieval has poor processing ability for synonyms and near-synonyms, which in turn leads to semantic deviation from the retrieval content and inability to present the user's required content.

[0006] To achieve the above objectives, the present invention proposes an AI-based data classification retrieval method, which includes:

[0007] Obtain the data segment to be searched and extract the keyword group;

[0008] Classify and identify the keywords, establish a search index, and search in a database based on the search index to obtain a search tree;

[0009] Extracting a first feature vector of the search index and a second feature vector of the search tree, and calculating similarity;

[0010] The results are arranged based on similarity and optimized.

[0011] In summary, according to the data classification and retrieval method based on AI proposed by the present invention, based on the acquired data segment to be retrieved, the frequency and weight of each word in the data segment to be retrieved are calculated, and the keyword group is extracted; the keyword group is classified and pooled for analysis, keywords with certain weight values ​​are screened out, and a search index is established. According to the established search index, several search trees close to the data segment to be retrieved are established in the database according to semantic analysis; the first feature vector of the search index and the second feature vector of each search tree are extracted, and the similarity is calculated using the first feature vector and the second feature vector to obtain the correlation between each second feature vector and the first feature vector. And the loss of the current search index establishment is calculated by the loss function, so as to dynamically adjust the preset weight value for optimization. The present invention reduces the complexity of the data and improves the accuracy of the retrieval process by splitting the data segment to be retrieved, extracting the keyword group in the data segment to be retrieved according to the frequency and weight, and classifying and pooling the keyword group, while improving the quality of the retrieval results by calculating the similarity between the first feature vector and the second feature vector, overcoming the problem of semantic deviation in the traditional text matching method, and improving the retrieval efficiency.

[0012] According to one aspect of the above technical solution, the step of obtaining the data segment to be searched and extracting the keyword group specifically includes:

[0013] According to the acquired data segment to be retrieved, identification marks are added to the first and last ends of the data segment to be retrieved, and keywords in the data segment to be retrieved are extracted through semantic model recognition;

[0014] Calculate the word frequency in the data segment to be retrieved:

[0015]

[0016] Among them, T i represents the frequency of word i in the data segment to be retrieved, n i is the number of times word i appears in the data segment to be retrieved, ∑ k n k is the total number of words in the data segment to be retrieved.

[0017] According to one aspect of the above technical solution, after obtaining the word frequency, the inverse document frequency F of word i is calculated i :

[0018]

[0019] Where N is the total number of documents, df i is the number of documents containing word i;

[0020] Based on the data segment to be retrieved, calculate the weight W of word i P :

[0021] W P =∑ i T i ×F i

[0022] The keyword group is extracted according to the weight and the semantic model recognition result.

[0023] According to one aspect of the above technical solution, the step of classifying and identifying the keywords, establishing a search index, and searching in a database based on the search index to obtain a search tree specifically includes: performing pooling analysis on the classified word groups, classifying and identifying the keywords of the keyword groups, so as to obtain word groups with at least noun, adjective, and verb classification identification respectively;

[0024] Filter out keywords whose weight values ​​are not less than the preset weight values, take the words with the largest weight value as the root node, and split the extended nodes in order according to the weight value arrangement rule from large to small to construct a search index; based on the search index, take the synonyms and / or near-synonyms of the word group as the root node in the database, use the semantic model and the matching mechanism to semantically encode the search index, and by searching for the code whose semantic encoding similarity is greater than the preset similarity value, establish a search tree in the database according to the code. According to one aspect of the above technical solution, the step of extracting the first feature vector of the search index and the second feature vector of the search tree and calculating the similarity specifically includes:

[0025] Constructing the first feature vector of the search index: arranging the weight values ​​in the search index from large to small in a regular manner to form a first feature vector; and extracting the second feature vector of the search tree, and calculating the similarity between the first feature vector and the second feature vector:

[0026] d(u,0)=u

[0027] d(0,v)=v

[0028] d(u,v)=min{d(u-1,v)+1,d(u,v-1)+1,d(u-1,v-1)+[a[u]≠b[v]]} wherein a is the first feature vector of the search index, b is the second feature vector of the search tree, d(u,v) is the edit distance required to convert the first u characters in the first feature vector into the first v characters in the second feature vector, a[u] is the u-th character in the first feature vector, b[v] is the v-th character in the second feature vector, and [a[u]≠b[v]] is an indicator function whose value is 1 if [a[u]≠b[v]] and 0 if [a[u]=b[v]]; based on the edit distance, the similarity between the first feature vector and the second feature vector is obtained, and the search results are presented based on the similarity.

[0029] According to one aspect of the above technical solution, the step of optimizing based on the similarity arrangement result specifically includes:

[0030] In the process of pooling and classification of keyword groups, the loss function is:

[0031]

[0032] Wherein, c is the number of words, d is the category of the word group, and y is the eg Represented as the one-hot encoding of the true category label for the e-th sample. If sample e belongs to category g, then y eg =1, if sample e does not belong to category g, then y eg =0, Represented as a probability vector for each category;

[0033] Based on the loss function, the preset weight value is dynamically adjusted to reduce the loss of the pooling process.

[0034] The present invention further proposes an AI-based data classification retrieval system, which is used to implement the above-mentioned AI-based data classification retrieval method. The AI-based data classification retrieval system includes:

[0035] An extraction module is used to obtain the data segment to be searched and extract the keyword group;

[0036] Establishing a module for classifying and identifying the keywords, establishing a search index, and searching in a database based on the search index to obtain a search tree;

[0037] A calculation module, used for extracting the first feature vector of the search index and the second feature vector of the search tree, and calculating the similarity;

[0038] The optimization module is used to arrange the results based on similarity and perform optimization.

[0039] The present invention also proposes a computer-readable storage medium having a computer program stored thereon, wherein the program implements the above-mentioned AI-based data classification and retrieval method when executed by a processor.

[0040] The present invention provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned AI-based data classification and retrieval method when executing the computer program.

[0041] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description or learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 This is a flow chart of the AI-based data classification retrieval method in Embodiment 1 of the present invention;

[0043] Figure 2 This is a schematic diagram of the structure of the AI-based data classification and retrieval system in Embodiment 2 of the present invention;

[0044] Figure 3 This is a structural block diagram of an electronic device in Embodiment 4 of the present invention. DETAILED DESCRIPTION

[0045] In order to facilitate the understanding of the present invention, the present invention will be described more fully below with reference to the relevant drawings. Several embodiments of the present invention are given in the drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the disclosure of the present invention more thorough and comprehensive.

[0046] It should be noted that when an element is referred to as being "fixed to" another element, it may be directly on the other element or there may be a central element. When an element is considered to be "connected to" another element, it may be directly connected to the other element or there may be a central element at the same time. The terms "vertical", "horizontal", "left", "right" and similar expressions used herein are for illustrative purposes only.

[0047] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used herein in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used herein includes any and all combinations of one or more related listed items.

[0048] Embodiment 1

[0049] like Figure 1 The flowchart of a data classification retrieval method based on AI in the first embodiment of the present invention is shown. The data classification retrieval method based on AI includes the following steps S01 to S04, wherein:

[0050] S01. Obtain the data segment to be searched and extract the keyword group.

[0051] The data segment to be retrieved is input, and identification marks are added to the beginning and the end of the data segment to be retrieved respectively for use in the semantic model for identification, thereby extracting keywords from the data segment to be retrieved.

[0052] The extraction of keywords is based on the frequency of the words appearing in the data segment to be retrieved, and the semantic position of the keywords in the semantic analysis.

[0053] The formula for calculating the frequency of keywords in the data segment to be searched is:

[0054]

[0055] Among them, T i represents the frequency of word i in the data segment to be retrieved, n i is the number of times word i appears in the data segment to be retrieved, ∑ k n k is the total number of words in the data segment to be retrieved.

[0056] Through the above data calculation formula, it can be obtained that the frequency of occurrence of a keyword in the data segment to be searched can reflect the importance of the keyword in the data segment to be searched.

[0057] Furthermore, the inverse document frequency F of word i i :

[0058]

[0059] Where N is the total number of documents, df i is the number of documents containing word i. Inverse document frequency can be used to represent weighted techniques for information retrieval and text mining, and is mainly used to quantify the importance of word i in the set of data segments to be retrieved. It is worth noting that if a word appears in many documents, then the word may be a common word and has low importance in distinguishing the content of the document; conversely, if a word only appears in a few documents, then it may have high specificity and is important in distinguishing the content of the document.

[0060] Based on the word frequency and inverse document frequency of word i, calculate the weight W of word i P :

[0061] W P =∑i T i ×F i

[0062] According to the weight W P And the recognition results of the semantic model are used to extract keyword clusters.

[0063] S02. Classify and identify the keywords, establish a search index, and search in a database based on the search index to obtain a search tree.

[0064] According to the keyword group obtained in step S01, the classified word group is pooled and analyzed to reduce the data dimension, retain the most important feature information, select the edge features as the most representative, and reduce the complexity of the data. At the same time, the pooling analysis can help reduce the amount of calculation and prevent overfitting, making it easier for subsequent classifiers to process these features.

[0065] The keyword group is classified to obtain word groups having at least classification identifiers such as nouns, pronouns, verbs, adjectives, numerals, quantifiers, adverbs, prepositions, etc.

[0066] The word groups after pooling analysis and classification are screened to select keywords with weight values ​​not less than the preset weight values, and the words with the largest weight values ​​are used as the root nodes, and the extended nodes are split according to the arrangement rules from large to small weight values ​​to build a search index. The search index is used to enter the database to directly retrieve the user's desired search content.

[0067] Further, according to the constructed search index, the synonyms and / or near synonyms of the word group are used as root nodes in the database, and the synonyms and / or near synonyms of the word group are searched through the matching mechanism of the semantic model. According to the semantic coding, the synonyms and / or near synonyms with semantic coding similarity greater than the preset similarity value code are searched. The above synonyms and / or near synonyms are semantically edited, and then several search trees are established in the database. The search tree is the content retrieved by the user.

[0068] S03. Extract the first feature vector of the search index and the second feature vector of the search tree, and calculate the similarity.

[0069] By calculating the similarity between the first feature vector of the search index and the second feature vector of the search tree in the database, the search results are presented to the user according to the arrangement rule from high to low similarity to improve the accuracy of the search.

[0070] The output process of the first feature vector is: the weight values ​​in the search index are arranged in a regular order from large to small to form the first feature vector; similarly, each search tree outputs its own second feature vector in the same way. The similarity between the search tree and the search index is obtained by calculating the first feature vector and the second feature vector:

[0071] d(u,0)=u

[0072] d(0,v)=v

[0073] d(u,v)=min{d(u-1,v)+1,d(u,v-1)+1,d(u-1,v-1)+[a[u]≠b[v]]}

[0074] Where a is the first eigenvector of the search index, b is the second eigenvector of the search tree, d(u,v) is the edit distance required to convert the first u characters in the first eigenvector into the first v characters in the second eigenvector, a[u] is the u-th character in the first eigenvector, b[v] is the v-th character in the second eigenvector, and [a[u]≠b[v]] is the indicator function, which is 1 if [a[u]≠b[v]] and 0 if [a[u]=b[v]].

[0075] By calculating the minimum number of edits required to convert a search index into a search tree, the smaller the edit distance is, the higher the similarity is.

[0076] S04. Optimize the results based on the similarity arrangement.

[0077] Since there is loss in the process of pooling keywords in step S02 and in the process of extracting keywords, it is necessary to optimize the loss. The loss function is:

[0078]

[0079] Wherein, c is the number of words, d is the category of the word group, and y is the eg Represented as the one-hot encoding of the true category label for the e-th sample. If sample e belongs to category g, then y eg =1, if sample e does not belong to category g, then y eg =0, Represented as a probability vector for each category.

[0080] Based on the above loss function, the preset weight value is dynamically adjusted to reduce the loss of the pooling process.

[0081] If the loss function value is higher than the preset loss value, the preset weight value is lowered to reduce the pooling loss; if the loss function value is lower than the preset loss value, the preset weight value is increased to improve the accuracy of the retrieval.

[0082] In summary, according to the data classification and retrieval method based on AI proposed by the present invention, based on the acquired data segment to be retrieved, the frequency and weight of each word in the data segment to be retrieved are calculated, and the keyword group is extracted; the keyword group is classified and pooled for analysis, keywords with certain weight values ​​are screened out, and a search index is established. According to the established search index, several search trees close to the data segment to be retrieved are established in the database according to semantic analysis; the first feature vector of the search index and the second feature vector of each search tree are extracted, and the similarity is calculated using the first feature vector and the second feature vector to obtain the correlation between each second feature vector and the first feature vector. And the loss of the current search index establishment is calculated by the loss function, so as to dynamically adjust the preset weight value for optimization. The present invention reduces the complexity of the data and improves the accuracy of the retrieval process by splitting the data segment to be retrieved, extracting the keyword group in the data segment to be retrieved according to the frequency and weight, and classifying and pooling the keyword group, while improving the quality of the retrieval results by calculating the similarity between the first feature vector and the second feature vector, overcoming the problem of semantic deviation in the traditional text matching method, and improving the retrieval efficiency.

[0083] Embodiment 2

[0084] Another aspect of the present invention further provides an AI-based data classification retrieval system, which is used to implement the AI-based data classification retrieval method in the first embodiment. Figure 2 , which is a schematic diagram of the structure of AI-based data classification retrieval in Embodiment 2 of the present invention, wherein the AI-based data classification retrieval system comprises:

[0085] The extraction module 11 is used to obtain the data segment to be searched and extract the keyword group;

[0086] Establishing module 12, used for classifying and identifying the keywords, establishing a search index, and searching in a database based on the search index to obtain a search tree;

[0087] A calculation module 13 is used to extract the first feature vector of the search index and the second feature vector of the search tree, and calculate the similarity;

[0088] The optimization module 14 is used to optimize the results of similarity arrangement.

[0089] The data segment to be retrieved is input, and identification marks are added to the beginning and the end of the data segment to be retrieved respectively for use in the semantic model for identification, thereby extracting keywords from the data segment to be retrieved.

[0090] The extraction of keywords is based on the frequency of the words appearing in the data segment to be retrieved, and the semantic position of the keywords in the semantic analysis.

[0091] The formula for calculating the frequency of keywords in the data segment to be searched is:

[0092]

[0093] Among them, T i represents the frequency of word i in the data segment to be retrieved, n i is the number of times word i appears in the data segment to be retrieved, ∑ k n k is the total number of words in the data segment to be retrieved.

[0094] Through the above data calculation formula, it can be obtained that the frequency of occurrence of a keyword in the data segment to be searched can reflect the importance of the keyword in the data segment to be searched.

[0095] Furthermore, the inverse document frequency F of word i i :

[0096]

[0097] Where N is the total number of documents, df i is the number of documents containing word i. Inverse document frequency can be used to represent weighted techniques for information retrieval and text mining, and is mainly used to quantify the importance of word i in the set of data segments to be retrieved. It is worth noting that if a word appears in many documents, then the word may be a common word and has low importance in distinguishing the content of the document; conversely, if a word only appears in a few documents, then it may have high specificity and is important in distinguishing the content of the document.

[0098] Based on the word frequency and inverse document frequency of word i, calculate the weight W of word i P :

[0099] W P =∑ i T i ×F i

[0100] According to the weight W P And the recognition results of the semantic model are used to extract keyword clusters.

[0101] Based on the obtained keyword groups, pooling analysis is performed on the classified word groups to reduce the data dimension, retain the most important feature information, select edge features as the most representative, and thus reduce the complexity of the data. At the same time, pooling analysis can help reduce the amount of calculation and prevent overfitting, making it easier for subsequent classifiers to process these features.

[0102] The keyword group is classified to obtain word groups having at least classification identifiers such as nouns, pronouns, verbs, adjectives, numerals, quantifiers, adverbs, prepositions, etc.

[0103] The word groups after pooling analysis and classification are screened to select keywords with weight values ​​not less than the preset weight values, and the words with the largest weight values ​​are used as the root nodes, and the extended nodes are split according to the arrangement rules from large to small weight values ​​to build a search index. The search index is used to enter the database to directly retrieve the user's desired search content.

[0104] Further, according to the constructed search index, the synonyms and / or near synonyms of the word group are used as root nodes in the database, and the synonyms and / or near synonyms of the word group are searched through the matching mechanism of the semantic model. According to the semantic coding, the synonyms and / or near synonyms with semantic coding similarity greater than the preset similarity value code are searched. The above synonyms and / or near synonyms are semantically edited, and then several search trees are established in the database. The search tree is the content retrieved by the user.

[0105] By calculating the similarity between the first feature vector of the search index and the second feature vector of the search tree in the database, the search results are presented to the user according to the arrangement rule from high to low similarity to improve the accuracy of the search.

[0106] The output process of the first feature vector is: the weight values ​​in the search index are arranged in a regular order from large to small to form the first feature vector; similarly, each search tree outputs its own second feature vector in the same way. The similarity between the search tree and the search index is obtained by calculating the first feature vector and the second feature vector:

[0107] d(u,0)=u

[0108] d(0,v)=v

[0109] d(u,v)=min{d(u-1,v)+1,d(u,v-1)+1,d(u-1,v-1)+[a[u]≠b[v]]}

[0110] Where a is the first eigenvector of the search index, b is the second eigenvector of the search tree, d(u,v) is the edit distance required to convert the first u characters in the first eigenvector into the first v characters in the second eigenvector, a[u] is the u-th character in the first eigenvector, v[v] is the v-th character in the second eigenvector, and [a[u]≠b[v]] is the indicator function, which is 1 if [a[u]≠b[b]] and 0 if [a[u]=b[v]].

[0111] By calculating the minimum number of edits required to convert a search index into a search tree, the smaller the edit distance is, the higher the similarity is.

[0112] Since there is loss in the process of keyword pooling and keyword extraction, it is necessary to optimize the loss. The loss function is:

[0113]

[0114] Wherein, c is the number of words, d is the category of the word group, and y is the eg Represented as the one-hot encoding of the true category label for the e-th sample. If sample e belongs to category g, then y eg =1, if sample e does not belong to category g, then y eg =0, Represented as a probability vector for each category.

[0115] Based on the above loss function, the preset weight value is dynamically adjusted to reduce the loss of the pooling process.

[0116] If the loss function value is higher than the preset loss value, the preset weight value is lowered to reduce the pooling loss; if the loss function value is lower than the preset loss value, the preset weight value is increased to improve the accuracy of the retrieval.

[0117] In summary, according to the data classification and retrieval system based on AI proposed by the present invention, based on the acquired data segment to be retrieved, the frequency and weight of each word in the data segment to be retrieved are calculated, and the keyword group is extracted; the keyword group is classified and pooled for analysis, keywords with certain weight values ​​are screened out, and a search index is established. According to the established search index, several search trees close to the data segment to be retrieved are established in the database according to semantic analysis; the first feature vector of the search index and the second feature vector of each search tree are extracted, and the similarity is calculated using the first feature vector and the second feature vector to obtain the correlation between each second feature vector and the first feature vector. And the loss of the current search index establishment is calculated by the loss function, so as to dynamically adjust the preset weight value for optimization. The present invention reduces the complexity of the data and improves the accuracy of the retrieval process by splitting the data segment to be retrieved, extracting the keyword group in the data segment to be retrieved according to the frequency and weight, and classifying and pooling the keyword group, while improving the quality of the retrieval results by calculating the similarity between the first feature vector and the second feature vector, overcoming the problem of semantic deviation in the traditional text matching method, and improving the retrieval efficiency.

[0118] Embodiment 3

[0119] On the other hand, the present invention further proposes a computer-readable storage medium having one or more computer programs stored thereon, which, when executed by a processor, implement the above-mentioned AI-based data classification and retrieval method.

[0120] Those skilled in the art will appreciate that the logic or steps represented in the flowchart or otherwise described herein, for example, may be considered as an ordered list of executable instructions for implementing logical functions, and may be embodied in any computer-readable storage medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For purposes of this specification, a "computer-readable storage medium" may be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0121] More specific examples of computer-readable storage media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable storage medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or processing in another suitable manner as necessary, and then stored in a computer memory.

[0122] Embodiment 4

[0123] Figure 3 This is a structural block diagram of an electronic device provided in Example 4. The electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and when the processor executes the program, the AI-based data classification and retrieval method in the above-mentioned embodiment is implemented. Figure 3 The electronic device 30 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0124] like Figure 3 As shown, the electronic device 30 may be in the form of a general-purpose computing device, for example, it may be a server device. The components of the electronic device 30 may include, but are not limited to: at least one processor 31, at least one memory 32, and a bus 33 connecting different system components (including the memory 32 and the processor 31).

[0125] The bus 33 includes a data bus, an address bus, and a control bus.

[0126] The memory 32 may include a volatile memory, such as a RAM 321 (Random Access Memory), and / or a cache memory 322 , and may further include a ROM 323 (Read Only Memory).

[0127] The memory 32 may also include a program tool 325 having a set (at least one) of program modules 324, such program modules 324 including but not limited to: an operating system, one or more application programs, other program modules and program data, each of which or some combination may include the implementation of a network environment.

[0128] The processor 31 executes various functional applications and data processing by running the computer programs stored in the memory 32, such as the AI-based data classification and retrieval method described above in the present invention.

[0129] The electronic device 30 may also communicate with one or more external devices 34 (e.g., keyboards, pointing devices, etc.). Such communication may be performed via an I / O interface 35 (input / output interface). Furthermore, the model generating electronic device 30 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 36. Figure 3 As shown, the network adapter 36 communicates with other modules of the model generation electronic device 30 via the bus 33. It should be understood that although Figure 3 Not shown, other hardware and / or software modules may be used in conjunction with the model-generated electronic device 30, including but not limited to: microcode, device drivers, redundant processors, disk drive arrays, RAID (disk array) systems, tape drives, and data backup storage systems.

[0130] It should be noted that, although several units / modules or subunits / modules of electronic equipment are mentioned in the above detailed description, this division is only exemplary and not mandatory. In fact, according to an embodiment of the present invention, the features and functions of two or more units / modules described above can be concretized in one unit / module. On the contrary, the features and functions of one unit / module described above can be further divided into being concretized by multiple units / modules. In the description of this specification, the description of reference terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0131] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A data classification retrieval method based on AI, characterized in that: The AI-based data classification retrieval method includes: Obtain the data segment to be searched and extract the keyword group; Classify and identify the keywords, establish a search index, and search in a database based on the search index to obtain a search tree; Extracting a first feature vector of the search index and a second feature vector of the search tree, and calculating similarity; The results are arranged based on similarity and optimized.

2. The AI-based data classification retrieval method according to claim 1, characterized in that: The step of obtaining the data segment to be searched and extracting the keyword group specifically includes: According to the acquired data segment to be retrieved, identification marks are added to the first and last ends of the data segment to be retrieved, and keywords in the data segment to be retrieved are extracted through semantic model recognition; Calculate the word frequency in the data segment to be retrieved: Among them, T i represents the frequency of word i in the data segment to be retrieved, n i is the number of times word i appears in the data segment to be retrieved, ∑ k n k is the total number of words in the data segment to be retrieved.

3. The AI-based data classification retrieval method according to claim 2 is characterized in that: After obtaining the word frequency, calculate the inverse document frequency F of word i i : Where N is the total number of documents, df i is the number of documents containing word i; Based on the data segment to be retrieved, calculate the weight W of word i P : W P =∑ i T i ×F i The keyword group is extracted according to the weight and the semantic model recognition result.

4. The AI-based data classification retrieval method according to claim 1, characterized in that: The step of classifying and identifying the keywords, establishing a search index, and searching in a database based on the search index to obtain a search tree specifically includes: Performing pool analysis on the classified word groups, and classifying and marking the keywords of the keyword groups, so as to obtain word groups having at least noun, adjective, and verb classification marks respectively; Filter out keywords with weight values ​​not less than the preset weight value, take the word with the largest weight value as the root node, and split and extend the nodes in order according to the weight value arrangement rule from large to small to build a search index; Based on the search index, the synonyms and / or near-synonyms of the word group are used as root nodes in the database, and the search index is semantically encoded through a semantic model and a matching mechanism. By searching for codes whose semantic encoding similarity is greater than a preset similarity value, a search tree is established in the database according to the code.

5. The AI-based data classification retrieval method according to claim 1, characterized in that: The step of extracting the first feature vector of the search index and the second feature vector of the search tree and calculating the similarity specifically includes: Constructing the first feature vector of the search index: arranging the weight values ​​in the search index from large to small in a regular manner to form a first feature vector; and extracting the second feature vector of the search tree, and calculating the similarity between the first feature vector and the second feature vector: d(u,0)=u d(0,v)=v d(u,v)=min{d(u-1,v)+1,d(u,v-1)+1,d(u-1,v-1)+[a[u]≠b[v]]} Where a is the first eigenvector of the search index, b is the second eigenvector of the search tree, d(u,v) is the edit distance required to convert the first u characters in the first eigenvector into the first v characters in the second eigenvector, a[u] is the u-th character in the first eigenvector, b[v] is the v-th character in the second eigenvector, [a[u]≠b[v]] is the indicator function, if [a[u]≠b[v]] its value is 1, if [a[u]=b[v]] its value is 0; Based on the edit distance, the similarity between the first feature vector and the second feature vector is obtained, and the similarity is arranged based on the similarity to present the search result.

6. The AI-based data classification retrieval method according to claim 1, characterized in that: The step of optimizing based on the similarity arrangement result specifically includes: In the process of pooling and classification of keyword groups, the loss function is: Wherein, c is the number of words, d is the category of the word group, yeg is represented by the one-hot encoding of the true category label of the e-th sample. If sample e belongs to category g, then yeg = 1; if sample e does not belong to category g, then yeg = 0. Represented as a probability vector for each category; Based on the loss function, the preset weight value is dynamically adjusted to reduce the loss of the pooling process.

7. An AI-based data classification retrieval system, the AI-based data classification retrieval system is used to implement the AI-based data classification retrieval method according to any one of claims 1 to 6, the AI-based data classification retrieval system comprising: An extraction module is used to obtain the data segment to be searched and extract the keyword group; Establishing a module for classifying and identifying the keywords, establishing a search index, and searching in a database based on the search index to obtain a search tree; A calculation module, used for extracting the first feature vector of the search index and the second feature vector of the search tree, and calculating the similarity; The optimization module is used to arrange the results based on similarity and perform optimization.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the AI-based data classification and retrieval method as described in any one of claims 1 to 6 is implemented.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, it implements the AI-based data classification and retrieval method as described in any one of claims 1 to 6.

Images