A big data search engine based on deep learning

Abstract

The present invention provides a big data search engine based on deep learning, which belongs to the field of deep learning technology, and includes: a mechanism creation module, which is used to create a search index based on a search database under each search format collected by a big data platform, and analyze the search indexes under all search formats to establish an index mechanism; a search module, which is used to parse the search request input by the user, and analyze the parsing result based on the index mechanism to obtain the initial search result; a log interception module, which is used to intercept the search log of the search format corresponding to each intention index within a specified period with the input time of the user's search request as the entry point; a feature extraction module, which is used to extract features from the search log to correct the initial weight of the corresponding intention index, and output the initial search result after internal sorting according to the correction result. Ensure the accurate determination of the importance of the intention index, and then ensure the orderly output of the results, and improve the output efficiency.

Description

Technical Field

[0001] The present invention relates to the field of deep learning technology, and in particular to a big data search engine based on deep learning. Background Art

[0002] With the development of information technology, especially the rapid rise and popularization of social networks, mobile Internet, Internet of Things, and big data applications, the data generated by human social development has exploded. Faced with such a huge amount of data, how to accurately obtain the required data from the massive data and rank the importance of the data is a challenge we face today.

[0003] The common search principle at present is to use an efficient spider program, starting from the specified URL and following the hyperlinks on the web page, traversing the entire Internet using a depth-first algorithm or a breadth-first algorithm, and crawling web page information into a local database. Then, an indexer is used to index important information units in the database, such as titles, keywords, abstracts, etc. or full text, and finally the index results are directly output. Before the results are output, the current importance of the index will not be judged, resulting in chaotic output results, and the purpose of orderly output of search results cannot be achieved.

[0004] Therefore, the present invention proposes a big data search engine based on deep learning. Summary of the invention

[0005] The present invention provides a big data search engine based on deep learning, which is used to establish an indexing mechanism through search indexes under different search formats, provide a basis for weight setting for subsequent intention indexes, and correct the weights through search logs to ensure the accurate determination of the importance of the intention indexes, thereby ensuring the orderly output of results and improving output efficiency.

[0006] The present invention provides a big data search engine based on deep learning, comprising:

[0007] The mechanism creation module is used to create a search index based on the search database under each search format collected by the big data platform, and to analyze the search indexes under all search formats to establish an index mechanism;

[0008] A search module, used to parse the search request input by the user, and analyze the parsing result based on the index mechanism to obtain an initial search result, wherein the initial search result includes a number of intent indexes and an initial weight of each intent index;

[0009] The log interception module is used to intercept the search log of the search format corresponding to each intention index within a specified period based on the input time of the user's search request as the entry point;

[0010] The feature extraction module is used to extract features from the search log to correct the initial weight of the corresponding intent index, and to internally sort the initial search results according to the correction results and then output them.

[0011] Preferably, the mechanism creation module includes:

[0012] A structure acquisition unit, used for acquiring the format structure of each search format from the format-structure comparison table;

[0013] A result matching unit, used to obtain search terms in each search format, and match the search terms with the search database to obtain search results for each search term, and then obtain a result structure for each search result;

[0014] A comparison analysis unit, used for performing structural clustering on all result structures to obtain a central structure, and performing comparison analysis on the central structure and the format structure, and assigning a comparison coefficient to the corresponding central structure;

[0015] The index setting unit is used to sort the comparison coefficients under each search format, and combine the edit distance between each result structure and the corresponding central structure under each clustering result to sort the search terms involved in each search format, and set the search index to each search term in turn according to the sorting results and the search format.

[0016] Preferably, the mechanism creation module further includes:

[0017] A field acquisition unit is used to extract thread nodes from the search terms and search threads of each search index and search results, and set a judgment coefficient for each thread node;

[0018] ;

[0019] in, represents the judgment coefficient of the corresponding thread node; sim(sx0,sx1) represents the log similarity function between the actual work log sx1 and the standard work log sx0 of the corresponding thread node under the search term; y01 represents the similarity threshold; represents the intersection ratio function of the working parameters of the actual working log sx1 and the standard working log sx0; represents the intersection symbol; represents the union symbol; max represents the maximum value symbol;

[0020] When the determination coefficient is 0, the threads with the highest and second highest matching degree with the corresponding search thread are extracted from the remaining threads, and at the same time, the threads corresponding to the terms with the highest and second highest matching degree with the corresponding search terms are extracted from the remaining terms;

[0021] Calculate the matching coefficients between the corresponding thread node and the four matching threads respectively, and obtain the node with the highest similarity in the thread under the maximum coefficient from the four matching coefficients as a temporary replacement node;

[0022] When the determination coefficient is 0, the threads with the highest and second highest matching degree with the corresponding search thread are extracted from the remaining threads, and at the same time, the threads corresponding to the terms with the highest and second highest matching degree with the corresponding search terms are extracted from the remaining terms;

[0023] Calculate the matching coefficients between the corresponding thread node and the four matching threads respectively, and obtain the node with the highest similarity in the thread under the maximum coefficient from the four matching coefficients as a temporary replacement node;

[0024] ;

[0025] in, represents the matching coefficient between the corresponding thread node and the j1th matching thread; Represents the actual work log sx1 and the standard work log of the j2th thread node in the j1th matching thread Similarity function; sum represents the cumulative sum symbol; Indicates the number of thread nodes existing in the j1th matching thread; Indicates all the j1th matching threads of and; Indicates except The variance of the sum of the remaining 3 matching threads; Indicates all of and;

[0026] Set the node field according to all nodes involved in the corresponding search thread;

[0027] Extract demand features from the original search demand of each search index and determine the demand fields of the corresponding search index;

[0028] A parameter setting unit, used to set a first generation parameter based on the node field, and at the same time, set a second generation parameter based on the requirement field;

[0029] A file writing unit, configured to write a sub-code file corresponding to the search index based on the first generation parameter and the second generation parameter, and to sequentially combine all sub-code files under the same search format according to the edit distance to obtain a single code file;

[0030] The mechanism generation unit is used to input all single code files into the mechanism building model to generate the index mechanism.

[0031] Preferably, the search module includes:

[0032] A traversal unit, configured to perform a global traversal on the parsing result based on the index mechanism to obtain a plurality of initial indexes and a first weight;

[0033] Extension unit, used to search and mine each initial index, and combined with Get the extended index corresponding to the initial index and assign a second weight to each extended index , wherein the search mining result includes: a first index existing together with the initial index under a plurality of input requests, and n0 represents the number of input requests obtained by search mining based on the corresponding initial index; represents the number of occurrences of the i2th first index determined based on the search and mining results of the corresponding initial index, and the number of occurrences of different first indexes under each input request is 1; Indicates the judgment result. When the value is 1, the corresponding first index is regarded as an extended index; when When the value is 0, the corresponding first index is regarded as a non-extended index; Indicates the floor symbol; Indicates that the i2th first index is related to the sum of the total number of all first indexes under each input request; Indicates the preset weight of the input request to which the i3th extended index appears at the i4th time; Indicates the number of occurrences of the i3th extended index; Indicates all The variance of

[0034] an adjusting unit, configured to adjust the corresponding first weight according to the second weight to obtain an initial weight corresponding to the initial index;

[0035] A screening unit, used for screening all extended indexes under the same initial index according to the second weight to obtain a screening index;

[0036] Preferably, the second weight of each screening index is regarded as the initial weight, and the index with the initial weight is regarded as the intended index.

[0037] Preferably, the log interception module includes:

[0038] A cycle acquisition unit, used to acquire the specified cycle corresponding to each search format from the format-cycle comparison table;

[0039] The log extraction unit is used to extract the search logs in the time period corresponding to the entry point and the specified period from the log database under each search format.

[0040] Preferably, the feature extraction module comprises:

[0041] A curve construction unit, used to construct a feature vector corresponding to the search log according to the feature extraction result, and determine the elimination strength of the corresponding feature element according to the representative strength of each feature element in the feature vector to construct an elimination curve;

[0042] a difference calculation unit, configured to calculate an area difference according to the first area of ​​the elimination curve and the second area of ​​the intensity curve constructed by the representative intensity;

[0043] A step length determination unit, used to determine a weight adjustment step length of a corresponding intention index according to a ratio of a retention strength to a representative strength of each feature element;

[0044] The weight correction unit is used to correct the corresponding initial weight according to the weight adjustment step size and in combination with the area difference.

[0045] Preferably, the feature extraction module further includes:

[0046] The result output unit is used to sort all the modified weights involved in the initial search results according to the modified results, and then sort the search indexes in order, and output the search results matching the search indexes in order.

[0047] Compared with the prior art, the present invention has the following beneficial effects:

[0048] An indexing mechanism is established through search indexes under different search formats to provide a basis for weight setting for subsequent intent indexes, and the weights are corrected through search logs to ensure accurate determination of the importance of intent indexes, thereby ensuring orderly output of results and improving output efficiency.

[0049] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the written description and the accompanying drawings.

[0050] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0052] Figure 1 This is a structural diagram of a big data search engine based on deep learning in an embodiment of the present invention. DETAILED DESCRIPTION

[0053] The preferred embodiments of the present invention are described below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0054] The present invention provides a big data search engine based on deep learning, such as Figure 1 As shown, including:

[0055] The mechanism creation module is used to create a search index based on the search database under each search format collected by the big data platform, and to analyze the search indexes under all search formats to establish an index mechanism;

[0056] A search module, used to parse the search request input by the user, and analyze the parsing result based on the index mechanism to obtain an initial search result, wherein the initial search result includes a number of intent indexes and an initial weight of each intent index;

[0057] The log interception module is used to intercept the search log of the search format corresponding to each intention index within a specified period based on the input time of the user's search request as the entry point;

[0058] The feature extraction module is used to extract features from the search log to correct the initial weight of the corresponding intent index, and to internally sort the initial search results according to the correction results and then output them.

[0059] In this embodiment, the big data platform may be similar to a shopping platform or an information retrieval platform. For example, the shopping platform is similar to the Taobao platform, the Pinduoduo platform, etc., and the information retrieval platform is similar to the Baidu platform, the 360 ​​platform, etc.

[0060] In this embodiment, the search format refers to the information format of the user's search information on the platform. For example, the information input by user A1 on the Taobao platform is: middle-aged and elderly fashion suits, suits & middle-aged and elderly, etc. At this time, "middle-aged and elderly fashion suits" are "middle-aged and elderly fashion" as a modifier, "suit" as a noun, and "suit & middle-aged and elderly" are the parallel of noun & modifier, that is, the parallel of modifier noun, noun & modifier, which is regarded as the search format, and the search results under different search formats are different. Therefore, different search results under the same search format can be combined together to form a search database, that is, each search database corresponds to a search format.

[0061] In this embodiment, the search index includes search format, search terms, comparison coefficients, and edit distances.

[0062] In this embodiment, the index mechanism is obtained by analyzing the code file based on the mechanism establishment model, and is mainly implemented by relying on the created search index and code.

[0063] In this embodiment, the search request refers to the search information entered by the user in the search box of the corresponding platform. In order to facilitate the identification of the search information, the platform automatically converts the information into an identification symbol string representing the request. The information conversion of each platform is implemented based on the known conversion rules of the platform, and the standardized expression of information is often expressed using abstract symbols such as numbers and characters to save platform operating space.

[0064] In this embodiment, the request is parsed to obtain the required keyword terms, which are then searched and analyzed by the indexing mechanism to obtain the intent index and initial weight.

[0065] In this embodiment, for example, the input time is time t1, the designated period corresponding to the intention index a1 is T1, and time t1 is used as the end time of the designated period T1, thereby obtaining the corresponding search log.

[0066] In this embodiment, the initial weight correction is to ensure the accuracy of the search based on the intention index and to facilitate the orderly output of subsequent search results.

[0067] In this embodiment, the internal sorting output refers to the sorting of the feedback information corresponding to the initial search results according to the modified weight of the intention index. For example, there are index 1 and index 2. After the weight is corrected, the weight of index 2 is greater than the weight of index 1. At this time, the internal sorting is: the search feedback results of index 2, the search feedback results of index 1, and they are output in this order.

[0068] The beneficial effects of the above technical solution are: establishing an indexing mechanism through search indexes under different search formats, providing a basis for weight setting for subsequent intention indexes, and correcting the weights through search logs to ensure accurate determination of the importance of intention indexes, thereby ensuring orderly output of results and improving output efficiency.

[0069] The present invention provides a big data search engine based on deep learning, wherein the mechanism creation module comprises:

[0070] A structure acquisition unit, used for acquiring the format structure of each search format from the format-structure comparison table;

[0071] A result matching unit, used to obtain search terms in each search format, and match the search terms with the search database to obtain search results for each search term, and then obtain a result structure for each search result;

[0072] A comparison analysis unit, used for performing structural clustering on all result structures to obtain a central structure, and performing comparison analysis on the central structure and the format structure, and assigning a comparison coefficient to the corresponding central structure;

[0073] The index setting unit is used to sort the comparison coefficients under each search format, and combine the edit distance between each result structure and the corresponding central structure under each clustering result to sort the search terms involved in each search format, and set the search index to each search term in turn according to the sorting results and the search format.

[0074] In this embodiment, the format-structure comparison table includes different search formats and structures matching the formats, and the structures are obtained after the formats are standardized and converted. For example, the structure corresponding to the format of the modifier noun is: A+B structure, A represents the modifier and B represents the noun.

[0075] In this embodiment, the search terms are: middle-aged and elderly suits, fashion suits, etc. At this time, the results of extracting the terms from the search database are regarded as search results. Because the search database contains different search results under the same search format, they can be directly extracted.

[0076] In this embodiment, the result structure is obtained by inputting the search results into a result analysis model, which is obtained after deep learning of the model based on different search results and setting the structure of the search results as samples. For example, relevant package results are obtained by searching on the Taobao interface, and the structure of the term description is determined by capturing and counting the term descriptions of each package in the package results and used as the structure of the search result, and the structural result is also in a form similar to A+B.

[0077] In this embodiment, structural clustering is implemented based on the K-means algorithm, and the search structure of the cluster center of the obtained clustering result is regarded as the central structure, which belongs to the prior art.

[0078] In this embodiment, the control coefficient = , where M1 represents the number of components involved in the corresponding format structure; M2 represents the number of components involved in the corresponding center structure; It represents the constituent edit distance between the u1th constituent part under the corresponding format structure and the u2th constituent part under the corresponding center structure.

[0079] In this embodiment, for example: format 1 includes: entry 10, entry 11, and format 2 includes: entry 21. At this time, the comparison coefficient of format 2 is greater than the comparison coefficient of format 1, and the edit distance of entry 11 is greater than the edit distance of entry 10. The sorting result is: format 2: entry 21, format 1: entry 11, entry 10.

[0080] The beneficial effect of the above technical solution is: by obtaining the format results and result structure of the search format, it is easy to determine the central structure from the result structure to obtain the comparison coefficient to achieve sorting in a large direction, and further achieve sorting in a small direction through the editing distance between the result structure and the central structure, ensuring the orderly setting of the search index, and providing a basis for the efficient establishment of subsequent mechanisms and the orderly output of results.

[0081] The present invention provides a big data search engine based on deep learning, wherein the mechanism creation module further includes:

[0082] A field acquisition unit is used to extract thread nodes from the search terms and search threads of each search index and search results, and set a judgment coefficient for each thread node;

[0083] ;

[0084] in, represents the judgment coefficient of the corresponding thread node; sim(sx0,sx1) represents the log similarity function between the actual work log sx1 and the standard work log sx0 of the corresponding thread node under the search term; y01 represents the similarity threshold; represents the intersection ratio function of the working parameters of the actual working log sx1 and the standard working log sx0; represents the intersection symbol; represents the union symbol; max represents the maximum value symbol;

[0085] When the determination coefficient is 0, the threads with the highest and second highest matching degree with the corresponding search thread are extracted from the remaining threads, and at the same time, the threads corresponding to the terms with the highest and second highest matching degree with the corresponding search terms are extracted from the remaining terms;

[0086] Calculate the matching coefficients between the corresponding thread node and the four matching threads respectively, and obtain the node with the highest similarity in the thread under the maximum coefficient from the four matching coefficients as a temporary replacement node;

[0087] ;

[0088] in, represents the matching coefficient between the corresponding thread node and the j1th matching thread; Represents the actual work log sx1 and the standard work log of the j2th thread node in the j1th matching thread Similarity function; sum represents the cumulative sum symbol; Indicates the number of thread nodes existing in the j1th matching thread; Indicates all the j1th matching threads of and; Indicates except The variance of the sum of the remaining 3 matching threads; Indicates all of and;

[0089] When the judgment coefficient is 1, the corresponding thread node is retained;

[0090] Set the node field according to all nodes involved in the corresponding search thread;

[0091] Extract demand features from the original search demand of each search index and determine the demand fields of the corresponding search index;

[0092] A parameter setting unit, used to set a first generation parameter based on the node field, and at the same time, set a second generation parameter based on the requirement field;

[0093] A file writing unit, configured to write a sub-code file corresponding to the search index based on the first generation parameter and the second generation parameter, and to sequentially combine all sub-code files under the same search format according to the edit distance to obtain a single code file;

[0094] The mechanism generation unit is used to input all single code files into the mechanism building model to generate the index mechanism.

[0095] In this embodiment, the input position of the search term is the initial position, and the search result obtained is the final position, that is, a search thread from the initial position to the final position is obtained, and the thread includes several nodes, such as the initial search node, the search process node 1, the search process node 2, and the result feedback node. That is, the search is first performed from the local database A, and then from the local database B, and so on. The nodes involved are sorted in order to form the search thread.

[0096] In this embodiment, each node will generate a corresponding log during the working process. The log is captured based on the log capture tool, and what is captured is generated in the actual work. In addition, each node will also have a standard log during the search for the corresponding entry. It is obtained by testing the node before it is applied to the actual scenario. It is used as a reference, and then compared with the threshold through two comprehensive calculations of the similarity function and the intersection ratio function of the working parameters to improve the accuracy, and the value of y01 is 0.6.

[0097] In this embodiment, for example, there are parameters 1, 2, and 3 for sx1, and there are parameters 1, 2, 4, and 5 for sx0, then, , where 2 is for parameter 1 and parameter 2, and 5 is for parameter 1, parameter 2, parameter 3, parameter 4 and parameter 5.

[0098] In this embodiment, the thread matching degree is calculated based on a similarity function, which is mainly obtained based on a comparative analysis of the node attribute descriptions of all nodes in the thread, and the node attribute description is mainly related to the function of the node search, and the functions, such as: search depth function, search breadth function, search type function.

[0099] In this embodiment, the matching degree of the entries is also calculated based on the similarity function, mainly based on the similarity of the entry descriptions.

[0100] In this embodiment, the node attribute description of the node is matched with the description-field comparison table to obtain the node field, and the comparison table contains different descriptions and matching fields in order to effectively represent and indicate the node and has uniqueness.

[0101] In this embodiment, demand feature extraction is implemented based on PCA (Principal Component Analysis) and LDA (Linear Component Analysis).

[0102] In this embodiment, the parameters are obtained based on matching of a field-parameter comparison table, which includes different fields and parameters matching the fields. It is mainly for determining the functional parameter description for the fields, which can be directly obtained and input into the file generation model to obtain a sub-code file. The file generation model is based on different parameter combinations and the implementation codes of the parameter combinations as samples to train the neural network model. Therefore, the sub-code file can be directly obtained.

[0103] In this embodiment, a single code file is a sub-code file sorted by distance in the same search format.

[0104] In this embodiment, the mechanism establishment model is obtained by training the neural network model for samples based on the implementation search function of different code files and combined code files. Therefore, the index mechanism can be obtained by analyzing all single code files, providing a strong foundation for subsequent request analysis.

[0105] The beneficial effects of the above technical solution are: setting the node judgment coefficient and the classification discussion with the coefficient as 1 or 0, effectively ensuring the reasonable and reliable operation of all nodes in the search thread to improve the search efficiency, and subsequently setting fields and matching parameters for nodes and requirements to facilitate the generation of code files to ensure the comprehensive and effective implementation of the search function, and the generated mechanism provides a strong analysis basis for the analysis of subsequent request indexes.

[0106] The present invention provides a big data search engine based on deep learning, wherein the search module further comprises:

[0107] A traversal unit, configured to perform a global traversal on the parsing result based on the index mechanism to obtain a plurality of initial indexes and a first weight;

[0108] Extension unit, used to search and mine each initial index, and combined with Get the extended index corresponding to the initial index and assign a second weight to each extended index , wherein the search mining result includes: a first index existing together with the initial index under a plurality of input requests, and n0 represents the number of input requests obtained by search mining based on the corresponding initial index; represents the number of occurrences of the i2th first index determined based on the search and mining results of the corresponding initial index, and the number of occurrences of different first indexes under each input request is 1; Indicates the judgment result. When the value is 1, the corresponding first index is regarded as an extended index; when When the value is 0, the corresponding first index is regarded as a non-extended index; Indicates the floor symbol; Indicates that the i2th first index is related to the sum of the total number of all first indexes under each input request; Indicates the preset weight of the input request to which the i3th extended index appears at the i4th time; Indicates the number of occurrences of the i3th extended index; Indicates all The variance of

[0109] an adjusting unit, configured to adjust the corresponding first weight according to the second weight to obtain an initial weight corresponding to the initial index;

[0110] A screening unit, used for screening all extended indexes under the same initial index according to the second weight to obtain a screening index;

[0111] Among them, the second weight of each filtered index is regarded as the initial weight, and the index with the initial weight is regarded as the intended index.

[0112] In this embodiment, the global traversal is to index each keyword entry in the parsing result, and obtain an initial index and a first weight based on each keyword entry. The first weight is directly obtained based on the index mechanism for the initial index, and the sum of the first weights of multiple initial indexes is 1.

[0113] In this embodiment, the search mining is to mine the request corresponding to the index, so as to determine other indexes that appear in the same request together with the index.

[0114] In this embodiment, initial weight = first weight + second weight / .

[0115] In this embodiment, screening refers to sorting the second weights by size and screening the first two indexes as the screening indexes.

[0116] The beneficial effects of the above technical solution are: the initial index and the first weight are obtained by globally traversing the parsing results, and the index is expanded in combination with search mining to ensure the integrity of the index, and the orderly output of subsequent results is guaranteed by correcting the weights.

[0117] The present invention provides a big data search engine based on deep learning, wherein the log interception module comprises:

[0118] A cycle acquisition unit, used to acquire the specified cycle corresponding to each search format from the format-cycle comparison table;

[0119] The log extraction unit is used to extract the search logs in the time period corresponding to the entry point and the specified period from the log database under each search format.

[0120] In this embodiment, the format-period comparison table includes different search formats and periods that match the formats, and the periods corresponding to different formats may be different.

[0121] In this embodiment, the log database includes logs generated during the search process using different search formats, which facilitates the extraction of data in a period.

[0122] The beneficial effect of the above technical solution is: obtaining the specified period from the comparison table, facilitating matching of reliable logs from the database, and providing a basis for subsequent corrections.

[0123] The present invention provides a big data search engine based on deep learning, wherein the feature extraction module comprises:

[0124] A curve construction unit, used to construct a feature vector corresponding to the search log according to the feature extraction result, and determine the elimination strength of the corresponding feature element according to the representative strength of each feature element in the feature vector to construct an elimination curve;

[0125] a difference calculation unit, configured to calculate an area difference according to the first area of ​​the elimination curve and the second area of ​​the intensity curve constructed by the representative intensity;

[0126] A step length determination unit, used to determine a weight adjustment step length of a corresponding intention index according to a ratio of a retention strength to a representative strength of each feature element;

[0127] The weight correction unit is used to correct the corresponding initial weight according to the weight adjustment step size and in combination with the area difference.

[0128] Preferably, the feature extraction module further includes:

[0129] The result output unit is used to sort all the modified weights involved in the initial search results according to the modified results, and then sort the search indexes in order, and output the search results matching the search indexes in order.

[0130] In this embodiment, the construction of the feature vector is based on a series of numerical values ​​obtained by extracting the features of the results, and the feature extraction belongs to the prior art. After the extracted features are numerically normalized, the standard numerical values ​​can be obtained to construct the feature vector, and each standard numerical value in the vector is the representative intensity of the corresponding feature element, for example: feature vector = {feature 1--a1 feature 2--a2}, where a1 and a2 are the representative intensities.

[0131] In this embodiment, elimination strength = one square of the number of feature elements × representative strength × weight of the feature element / (sum of all representative strengths × weight of the feature element) × strength threshold, where the strength threshold ranges from 0 to 1, and different feature elements correspond to different strength thresholds.

[0132] In this embodiment, the abscissa of the elimination curve is the characteristic element, and the ordinate is the elimination intensity.

[0133] In this embodiment, the area difference=second area−first area, the second area is: the closed area between the intensity curve and the x-axis, and the first area is: the closed area between the elimination curve and the x-axis.

[0134] In this embodiment, retention strength=1-elimination strength / representation strength.

[0135] In this embodiment, the weight adjustment step length=the minimum value of the ratios of all the characteristic elements involved / the average value of the ratios of all the characteristic elements involved.

[0136] .

[0137] The beneficial effect of the above technical solution is: by constructing the elimination curve by constructing the vector of the log and combining the area difference of the intensity curve, the step size is determined to ensure the accuracy and reliability of the weight correction, providing a reliable basis for subsequent size sorting.

[0138] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A big data search engine based on deep learning, characterized in that: include: The mechanism creation module is used to create a search index based on the search database under each search format collected by the big data platform, and to analyze the search indexes under all search formats to establish an index mechanism; A search module, used to parse the search request input by the user, and analyze the parsing result based on the index mechanism to obtain an initial search result, wherein the initial search result includes a number of intent indexes and an initial weight of each intent index; The log interception module is used to intercept the search log of the search format corresponding to each intention index within a specified period based on the input time of the user's search request as the entry point; A feature extraction module is used to extract features from the search log to modify the initial weight of the corresponding intent index, and to internally sort the initial search results according to the modified results and then output them; Wherein, the mechanism creation module includes: A structure acquisition unit, used for acquiring the format structure of each search format from the format-structure comparison table; A result matching unit, used to obtain search terms in each search format, and match the search terms with the search database to obtain search results for each search term, and then obtain a result structure for each search result; A comparison analysis unit, used for performing structural clustering on all result structures to obtain a central structure, and performing comparison analysis on the central structure and the format structure, and assigning a comparison coefficient to the corresponding central structure; An index setting unit is used to sort the comparison coefficients under each search format, and to sort the search terms involved in each search format in order based on the edit distance between each result structure and the corresponding center structure under each clustering result, and to set a search index for each search term in order based on the sorting result and the search format; The format-structure comparison table includes different search formats and structures matching the formats, and the structures are obtained after the formats are standardized and transformed; The result structure is obtained by inputting the search results into a result analysis model, which is obtained by deep learning a neural network model based on different search results and setting the structure of the search results as samples; Structural clustering is implemented based on the K-means algorithm, and the search structure of the cluster center of the obtained clustering results is regarded as the central structure; Contrast coefficient = , where M1 represents the number of components involved in the corresponding format structure; M2 represents the number of components involved in the corresponding center structure; represents the composition edit distance between the u1th component under the corresponding format structure and the u2th component under the corresponding center structure; Wherein, the mechanism creation module further includes: A field acquisition unit is used to extract thread nodes from the search terms and search threads of each search index and search results, and set a judgment coefficient for each thread node; ; Wherein, Px represents the judgment coefficient of the corresponding thread node; sim(sx0,sx1) represents the log similarity function between the actual work log sx1 and the standard work log sx0 of the corresponding thread node under the search term; y01 represents the similarity threshold; represents the intersection ratio function of the working parameters of the actual working log sx1 and the standard working log sx0; Indicates the intersection symbol; Represents the union symbol; When the determination coefficient is 0, the threads with the highest and second highest matching degree with the corresponding search thread are extracted from the remaining threads, and at the same time, the threads corresponding to the terms with the highest and second highest matching degree with the corresponding search terms are extracted from the remaining terms; Calculate the matching coefficients between the corresponding thread node and the four matching threads respectively, and obtain the node with the highest similarity in the thread under the maximum coefficient from the four matching coefficients as a temporary replacement node; ; in, represents the matching coefficient between the corresponding thread node and the j1th matching thread; Represents the actual work log sx1 and the standard work log of the j2th thread node in the j1th matching thread Similarity function; sum represents the cumulative sum symbol; Indicates the number of thread nodes existing in the j1th matching thread; Indicates all the j1th matching threads of and; Indicates except The variance of the sum of the remaining 3 matching threads; Indicates all The sum of; max represents the maximum value symbol; When the judgment coefficient is 1, the corresponding thread node is retained; Set the node field according to all nodes involved in the corresponding search thread; Extract demand features from the original search demand of each search index and determine the demand fields of the corresponding search index; A parameter setting unit, used to set a first generation parameter based on the node field, and at the same time, set a second generation parameter based on the requirement field; A file writing unit, configured to write a sub-code file corresponding to the search index based on the first generation parameter and the second generation parameter, and to sequentially combine all sub-code files under the same search format according to the edit distance to obtain a single code file; A mechanism generation unit, used to input all single code files into the mechanism building model to generate an index mechanism; The search thread is composed of the input position of the search term as the initial position and the search result as the final position, and the several nodes contained in the thread are regarded as thread nodes; The standard work log is obtained by testing the corresponding node before it is applied to the actual scenario and is used as a reference; Wherein, the feature extraction module includes: A curve construction unit, used to construct a feature vector corresponding to the search log according to the feature extraction result, and determine the elimination strength of the corresponding feature element according to the representative strength of each feature element in the feature vector to construct an elimination curve; a difference calculation unit, configured to calculate an area difference according to the first area of ​​the elimination curve and the second area of ​​the intensity curve constructed by the representative intensity; A step length determination unit, used to determine a weight adjustment step length of a corresponding intention index according to a ratio of a retention strength to a representative strength of each feature element; A weight correction unit, used to correct the corresponding initial weight according to the weight adjustment step size and in combination with the area difference; Among them, the standard value of each characteristic element in the characteristic vector is the representative intensity; Retention intensity = 1-elimination intensity / representation intensity.

2. The big data search engine based on deep learning according to claim 1, characterized in that: The search module comprises: A traversal unit, configured to perform a global traversal on the parsing result based on the index mechanism to obtain a plurality of initial indexes and a first weight; Extension unit, used to search and mine each initial index, and combined with Get the extended index corresponding to the initial index and assign a second weight to each extended index , wherein the search mining result includes: a first index existing together with the initial index under a plurality of input requests, and n0 represents the number of input requests obtained by search mining based on the corresponding initial index; represents the number of occurrences of the i2th first index determined based on the search and mining results of the corresponding initial index, and the number of occurrences of different first indexes under each input request is 1; Indicates the judgment result. When the value is 1, the corresponding first index is regarded as an extended index; when When the value is 0, the corresponding first index is regarded as a non-extended index; Indicates the floor symbol; Indicates that the i2th first index is related to the sum of the total number of all first indexes under each input request; Indicates the preset weight of the input request to which the i3th extended index appears at the i4th time; Indicates the number of occurrences of the i3th extended index; Indicates all The variance of an adjusting unit, configured to adjust the corresponding first weight according to the second weight to obtain an initial weight corresponding to the initial index; The screening unit is used to screen all the extended indexes under the same initial index according to the second weight to obtain a screened index.

3. The big data search engine based on deep learning according to claim 2, characterized in that: The second weight of each filtered index is considered as the initial weight, and the index with the initial weight is considered as the intended index.

4. The big data search engine based on deep learning according to claim 1, characterized in that: The log interception module includes: A period acquisition unit, used to respectively acquire a specified period corresponding to each search format from the format-period comparison table; The log extraction unit is used to extract the search logs in the time period corresponding to the entry point and the specified period from the log database under each search format.

5. The big data search engine based on deep learning according to claim 1, characterized in that: The feature extraction module further includes: The result output unit is used to sort all the modified weights involved in the initial search results according to the modified results, and then sort the search indexes in order, and output the search results matching the search indexes in order.

Images