Object storage data crawling path tracking system and method

Abstract

The present invention belongs to the technical field of data tracking, and provides a system and method for tracking a data crawling path based on object storage, comprising: screening out a data text candidate set based on subject keyword comparison, processing to obtain a retrieval range value based on the data text's domain of origin and the number of texts, judging whether it is necessary to reduce the index range, and if so, reducing the index range by combining a Boolean operator, thereby improving index matching efficiency; utilizing natural language processing technology to obtain semantic vectors of the data text and semantic vectors of the data retrieval text, and processing to obtain similarity; combining feature data of the data text in the corresponding new data text candidate set, and performing correlation analysis to obtain a correlation value, and combining the similarity between the data text and the data retrieval text, processing to obtain a matching degree, and recommending corresponding data text to a user based on the matching degree, thereby improving the crawling accuracy of stored data.

Description

Technical Field

[0001] The present invention belongs to the technical field of data tracking, and in particular to a system and method for tracking a data capture path based on object storage. Background Art

[0002] With the rapid development of information technology, searching and tracking massive amounts of stored data has become a major challenge. Traditional stored data retrieval methods, such as keyword search, often suffer from low recall and precision when faced with complex and diverse stored data. Furthermore, users often need to incorporate specific contextual information during the retrieval and tracking process to improve the accuracy and relevance of retrieval and tracking. Therefore, the development of crawling path tracking systems and methods for object-based stored data is particularly important.

[0003] In the existing technology, the search and tracking of stored data is achieved by grouping the selected alternative texts, pre-processing them according to common features, and adapting the corresponding similarity algorithm. However, the problem is that there is a lack of evaluation of the scope of its preliminary index tracking, such as the amount of text involved in the alternative data text and the field of attribution involved. The index scope is not analyzed and processed, resulting in a large number of indexed data texts, which increases the difficulty and complexity of index tracking. Secondly, semantic matching and capture are achieved through semantic similarity algorithms, but there is a lack of contextual analysis of user search queries, and it is impossible to achieve more accurate search and tracking of data texts.

[0004] To this end, the present invention provides a system and method for tracing a data crawling path based on object storage. Summary of the Invention

[0005] In order to make up for the deficiencies of the prior art, at least one technical problem raised in the background technology is solved.

[0006] The technical solution adopted by the present invention to solve the technical problem is: a method for tracing a path of data captured based on object storage, comprising:

[0007] Receive the data retrieval text input by the user on its retrieval platform and perform preprocessing, extract the subject keywords in the data retrieval text based on the preprocessed data retrieval text using statistical feature technology, compare the subject keywords in the data retrieval text with the subject keywords of the data texts stored in the retrieval platform, screen out the data texts that meet the subject keywords in the data retrieval text as the data text candidate set, analyze and process the data texts in the data text candidate set according to the domain and the number of texts, and determine whether the index range needs to be reduced according to the retrieval range value. If so, generate a reduction signal;

[0008] Based on the reduction signal, the index scope is reduced according to the topic keywords and combined with Boolean operators, and a new set of data text candidates is obtained;

[0009] Based on the new data text candidate set, the data text in the new data text candidate set and the data retrieval text are converted into vector representations using natural language processing technology, the semantic vectors of the data text and the semantic vectors of the data retrieval text are obtained, and the similarity between the data text and the data retrieval text is calculated using cosine similarity;

[0010] The user's situational feature data is obtained, and combined with the feature data of the data text in the corresponding new data text candidate set, and correlation analysis is performed to obtain the correlation value between the user's situational feature data and the data text, and combined with the similarity between the data text and the data retrieval text, the matching degree is obtained through processing and analysis, and the corresponding data text is recommended to the user based on the matching degree.

[0011] As a further technical solution of the present invention, the process of extracting the subject keywords includes:

[0012] Get the part of speech of each word from all the words, and filter out the topic keywords from all the words based on the part of speech and the key value of the word GJ;

[0013] Specifically, if the part of speech of a word falls within the range of parts of speech commonly used to express a topic word, and the key value GJ of the word is greater than or equal to a preset key threshold, the word is marked as a topic keyword.

[0014] As a further technical solution of the present invention, the key value GJ is obtained as follows:

[0015] The obtained word frequency ratio PL, TF-IDF ratio TB and word span ratio KB are processed by the formula: The key value GJ is obtained, where s1, s2 and s3 are all preset proportional coefficients.

[0016] As a further technical solution of the present invention, the word occurrence frequency ratio PL is obtained by ratioing the word occurrence frequency with the sum of the occurrence frequencies of all words, and the word TF-IDF ratio TB and the word span ratio KB are obtained in the same way as the occurrence frequency ratio PL.

[0017] As a further technical solution of the present invention, the method for obtaining the search range value is:

[0018] Obtaining the attribution domain of each data text in the data text candidate set, removing duplicate attribution domains based on the attribution domain type, obtaining the number of valid attribution domain types, and performing a ratio processing on the number of valid attribution domain types in the data text candidate set and the total number of attribution domain types stored in the retrieval platform to obtain the valid domain ratio of the data text candidate set;

[0019] Count the number of data texts in each type of valid attribution domain in the data text candidate set, sum and average them to obtain the mean number of domain texts, and compare the mean number of domain texts with the total number of data texts stored in the retrieval platform to obtain the text number ratio of the data text candidate set;

[0020] The effective domain ratio and the text quantity ratio of the data text candidate set are summed to obtain the retrieval range value.

[0021] As a further technical solution of the present invention, the method of converting the data text in the new data text candidate set and the data retrieval text into a vector representation by using natural language processing technology includes:

[0022] D1, build a vocabulary containing all unique words from the data text;

[0023] D2, use the pre-trained word embedding model BERT to map each word in the vocabulary into a high-dimensional vector space to obtain the vocabulary vector;

[0024] D3: For each data text, sum the TF-IDF values ​​of all words to obtain the total vector weight of the word, and perform a ratio process on the TF-IDF value of the word and the total vector weight of the word to obtain the vector weight of the word;

[0025] D4: For each data text, all word vectors are multiplied by the vector weights of the corresponding words, and the sum is averaged to obtain the semantic vector of the data text.

[0026] As a further technical solution of the present invention, the matching degree is obtained by summing the association value between the user's scenario feature data and the data text and the similarity between the data text and the data retrieval text.

[0027] As a further technical solution of the present invention, the correlation value is obtained by averaging the scores between the feature data of the data text and the context feature data of the user, specifically:

[0028] All the scores obtained are marked as F1, F2, ..., Fz, where z represents the number of scores;

[0029] Average all scores:

[0030] The correlation value GL is obtained by the formula: GL=c1*F1+c2*F2+...cn*Fn, where c1, c2...cn are weight coefficients.

[0031] As a further technical solution of the present invention, the weight coefficient is obtained as follows:

[0032] Obtain the user's historical query records within the historical query period. If the user's contextual feature data appears in a single historical query record, mark the single historical query record as a query record that appears. If the user's contextual feature data does not appear in a single historical query record, mark the single historical query record as a query record that does not appear.

[0033] Count the number of query records and compare it with the total number of historical query records to obtain the query record ratio, which is marked as CX;

[0034] Obtain the number of scenario feature data appearing in the query record, sum and average the number of scenario feature data in all query records to obtain the mean number of scenario feature data in the query record, and compare it with the total number of scenario feature data of the user to obtain the occurrence feature data ratio, which is marked as TZ;

[0035] The occurrence value CV of the scenario feature data is obtained by the formula: CV = g1*CX+g2*TZ, where g1 and g2 are both preset proportional coefficients;

[0036] The occurrence values ​​CV of all scenario feature data are summed up to obtain the total occurrence value, and the occurrence value CV of the scenario feature data is ratioed to the total occurrence value to obtain the weight coefficient of the corresponding score of the scenario feature data.

[0037] Object storage data crawling path tracking system, including:

[0038] Retrieval tracking and evaluation module: Receives data retrieval texts input by users on their retrieval platform and performs preprocessing. Based on the preprocessed data retrieval texts, statistical feature technology is used to extract subject keywords in the data retrieval texts. The subject keywords in the data retrieval texts are compared with the subject keywords of data texts stored in the retrieval platform. Data texts that meet the subject keywords in the data retrieval texts are selected as data text candidate sets. According to the domain and number of texts in the data text candidate sets, the retrieval range value is obtained through analysis and processing. According to the retrieval range value, it is determined whether the index range needs to be reduced. If so, a reduction signal is generated.

[0039] Retrieval tracking control module: Based on the reduction signal, it reduces the index scope according to the subject keywords and combines Boolean operators to obtain a new set of data text candidates;

[0040] Similarity analysis module: Based on the new data text candidate set, the data text in the new data text candidate set and the data retrieval text are converted into vector representations using natural language processing technology, the semantic vectors of the data text and the semantic vectors of the data retrieval text are obtained, and the similarity between the data text and the data retrieval text is calculated using cosine similarity;

[0041] Association matching processing module: obtains the user's situational feature data, combines it with the feature data of the data text in the corresponding new data text candidate set, and performs association analysis to obtain the association value between the user's situational feature data and the data text, and combines it with the similarity between the data text and the data retrieval text to obtain the matching degree through processing and analysis, and recommends the corresponding data text to the user based on the matching degree.

[0042] The beneficial effects of the present invention are as follows:

[0043] 1. Receive the data retrieval text input by the user on its retrieval platform and preprocess it. Based on the preprocessed data retrieval text, use statistical feature technology to extract the subject keywords in the data retrieval text, compare the subject keywords in the data retrieval text with the subject keywords of the data text stored in the retrieval platform, and screen out the data texts that meet the subject keywords in the data retrieval text as the data text candidate set. According to the belonging field and the number of texts of each data text in the data text candidate set, analyze and process to obtain the retrieval range value, and judge whether the index range needs to be reduced according to the retrieval range value. If so, generate a reduction signal; based on the reduction signal, reduce the index range according to the subject keywords and combined with Boolean operators, and obtain a new data text candidate set. The present invention evaluates the index range of its data text candidate set and uses Boolean operators to reduce the index range when its range is large, which is beneficial to reducing data texts with low index matching relevance and improving crawling efficiency.

[0044] 2. Based on the new data text candidate set, natural language processing technology is used to convert the data text in the new data text candidate set and the data retrieval text into vector representations, and the semantic vector of the data text and the semantic vector of the data retrieval text are obtained, and the similarity between the data text and the data retrieval text is calculated using cosine similarity; the user's scenario feature data is obtained, and combined with the feature data of the data text in the corresponding new data text candidate set, and correlation analysis is performed to obtain the correlation value between the user's scenario feature data and the data text, and combined with the similarity between the data text and the data retrieval text, the matching degree is obtained through processing and analysis, and the corresponding data text is recommended to the user based on the matching degree. The present invention not only calculates the similarity, but also integrates the scenario analysis, and performs data text crawling and tracking through the similarity obtained by semantic analysis and the correlation value obtained by scenario analysis, thereby improving the crawling efficiency of stored data while improving the crawling accuracy of stored data. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The present invention will be further described below with reference to the accompanying drawings.

[0046] Figure 1 This is a flowchart of the steps of the object storage data crawling path tracing method according to an embodiment of the present invention;

[0047] Figure 2 It is a flowchart of the object storage data crawling path tracking system according to an embodiment of the present invention. DETAILED DESCRIPTION

[0048] In order to make the technical means, creative features, objectives and effects achieved by the present invention easier to understand, the present invention is further described below in conjunction with specific implementation methods.

[0049] Example 1

[0050] like Figure 1 As shown, the object storage data crawling path tracking method according to the embodiment of the present invention includes:

[0051] Step 1: Receive the data retrieval text input by the user on its retrieval platform and pre-process it. Based on the pre-processed data retrieval text, use statistical feature technology to extract the subject keywords in the data retrieval text, compare the subject keywords in the data retrieval text with the subject keywords of the data texts stored in the retrieval platform, and screen out the data texts that meet the subject keywords in the data retrieval text as the data text candidate set. According to the belonging field and the number of texts of each data text in the data text candidate set, analyze and process to obtain the retrieval range value, and judge whether the index range needs to be reduced according to the retrieval range value. If so, generate a reduction signal;

[0052] Among them, the process of preprocessing its data retrieval text includes:

[0053] S1, Word Segmentation: For Chinese text, it is first necessary to perform word segmentation to split the sentence into independent words or phrases;

[0054] S2, Stop Word Removal: Stop words refer to words that frequently appear in the text but contribute little to the text theme, such as "de" (的), "le" (了), "zai" (在), etc.;

[0055] S3, Part-of-Speech Tagging: Perform part-of-speech tagging on the words in the text, including but not limited to: nouns, verbs;

[0056] The extraction of topic keywords includes:

[0057] A1, Word Frequency Statistics: Statistically calculate the frequency of occurrence of each word in the preprocessed data retrieval text;

[0058] A2, Calculate the TF-IDF value of the word using the TF-IDF algorithm, where TF (Term Frequency) represents the word frequency and IDF (Inverse Document Frequency) represents the inverse document frequency;

[0059] A3, Measure the word span of the word, where the word span represents the distance between the first occurrence and the last occurrence of the word in the text;

[0060] Based on the word frequency of occurrence, TF-IDF value, and word span, respectively, statistically calculate the word frequency ratio PL, TF-IDF ratio TB, and word span ratio KB of the word;

[0061] Perform data processing on the obtained word frequency ratio PL, TF-IDF ratio TB, and word span ratio KB of the word through the formula: Obtain the key value GJ, where s1, s2, and s3 are all preset proportionality coefficients, s1 takes the value of 1.102, s2 takes the value of 1.134, and s3 takes the value of 1.24;

[0062] It should be noted that there are m groups of historical data, and each group of historical data includes the word frequency ratio PL, TF-IDF ratio TB, and word span ratio KB of the word. Since the word frequency ratio PL, TF-IDF ratio TB, and word span ratio KB of the word will all affect the key degree of the word, according to the key degree of the word, the staff uses an offline model to fit based on the word frequency ratio PL, TF-IDF ratio TB, and word span ratio KB of the word to obtain the key value GJ matched by the word frequency ratio PL, TF-IDF ratio TB, and word span ratio KB of the word;

[0063] Then, based on the frequency ratios PL, TF-IDF ratios TB, and word span ratios KB of multiple word groups, a linear model is used to fit the historical data of the frequency ratios PL, TF-IDF ratios TB, word span ratios KB, and key values ​​GJ of multiple word groups. The prepared historical data is brought into the selected fitting model for fitting to obtain fitting coefficients s1, s2, and s3.

[0064] The word frequency ratio PL is obtained by comparing the word frequency to the sum of all word frequencies. The word TF-IDF ratio TB and word span ratio KB are obtained in the same way as the word frequency ratio PL. For example, in the data retrieval text, the word appears N i , where N represents frequency, i represents word number, i = 1, 2, 3, 4...n; the frequency ratio of the words is:

[0065] Get the part of speech of each word from all the words, and filter out the topic keywords from all the words based on the part of speech and the key value of the word GJ;

[0066] Specifically, if the part of speech of a word falls within the range of parts of speech of commonly expressed subject words, and the key value GJ of the word is greater than or equal to the preset key threshold, the word is marked as a subject keyword;

[0067] If the word's part of speech falls within the range of commonly used expression keywords and the word's key value GJ is less than the preset key threshold, the word will be marked as a non-topic keyword;

[0068] If the part of speech of a word does not fall within the range of common expression keywords, and the keyword value GJ of the word is greater than or equal to the preset keyword threshold, the word is marked as a non-topic keyword;

[0069] If the part of speech of a word does not fall within the range of common expression keywords, and the keyword value GJ of the word is less than the preset keyword threshold, the word will be marked as a non-topic keyword;

[0070] It should be noted that, generally speaking, common parts of speech used to express topics include nouns, verbs, proper nouns, etc. By obtaining the parts of speech of a word and matching it with the common parts of speech used to express the topic, topic keywords can be screened out;

[0071] It can be understood that the key value GJ is obtained by processing the word frequency ratio PL, TF-IDF ratio TB and word span ratio KB. This type of data belongs to statistical data. This type of data can be easily collected through existing statistical monitoring technology and combined with existing mathematical calculation rules to perform numerical operations on the collected data. According to the positive and negative effects of the data, the floating effects of multiple data can be synchronously collected to complete the data processing requirements of the current technical solution steps. In the process of data processing requirements, threshold comparison is involved. The thresholds of the corresponding data are all the experience values ​​of people in this field. Threshold comparison is a conventional data judgment method in the existing technology, and the threshold set for the corresponding threshold comparison is the current data. In the real-time collection scenario, the empirical values ​​proposed by personnel in this field for current data monitoring, for example, the staff calculates the key value GJ based on multiple sets of data, and identifies the criticality of its words based on the key value GJ, and obtains a corresponding relationship between the key value GJ and the criticality of the words. The staff derives and divides its threshold value based on the corresponding relationship between multiple sets of key values ​​and the criticality of the words, thereby obtaining the threshold value. For example, if the key value exceeds the threshold value B in the historical monitoring period, the criticality of the word is high, and the threshold value in the next historical monitoring period can be set to B. On the contrary, when the threshold value in the next period is lower than B, the criticality of the word is also high, and the current threshold value continues to be lowered based on B;

[0072] The way to obtain the retrieval range value is:

[0073] Obtaining the attribution domain of each data text in the data text candidate set, removing duplicate attribution domains based on the attribution domain type, obtaining the number of valid attribution domain types, and performing a ratio processing on the number of valid attribution domain types in the data text candidate set and the total number of attribution domain types stored in the retrieval platform to obtain the valid domain ratio of the data text candidate set;

[0074] Count the number of data texts in each type of valid attribution domain in the data text candidate set, sum and average them to obtain the mean number of domain texts, and compare the mean number of domain texts with the total number of data texts stored in the retrieval platform to obtain the text number ratio of the data text candidate set;

[0075] Sum the effective domain ratio and the text quantity ratio of the data text candidate set to obtain the retrieval range value;

[0076] In some preferred implementations, the search range value is compared with a preset search range threshold, and the specific comparison process is as follows:

[0077] If the search range value is greater than or equal to the preset search range threshold, it means that the number of data texts indexed by the current subject keyword is large and the fields are diverse. The search range is large, which is not conducive to subsequent specific index matching. The index range needs to be reduced, and a reduction signal is generated;

[0078] If the search range value is less than the preset search range threshold, it means that the number of data texts indexed by the current subject keyword is small and the field types are not diverse. The search range is small, which is conducive to subsequent specific index matching and there is no need to reduce the index range;

[0079] It can be understood that the effective domain ratio and text quantity ratio of the data text candidate set are both relevant text data in the actual scene, for example, including the number of types and the number of data texts that obtain the effective belonging field. Such data can be easily collected through existing sensor technology or network monitoring technology and combined with existing mathematical calculation rules to perform numerical operations on the collected data. According to the positive and negative effects of the data, the floating influence of multiple data is achieved and the synchronous collection is achieved to complete the data processing requirements of the current technical solution steps. In the process of data processing requirements, threshold comparison is involved. The thresholds of the corresponding data are all empirical values ​​of those in this field. Threshold comparison is a conventional data judgment method in the prior art, and the threshold set for the corresponding threshold comparison is the empirical value proposed by those in this field for current data monitoring in the real-time collection scenario of the current data, such as the search range value. If the search range value exceeds the threshold A in the historical monitoring period, it means that the search range is large. In the next historical monitoring period, the threshold can be set to A. On the contrary, if the threshold is lower than A in the next period and the search range is large, the current threshold will continue to be lowered based on A.

[0080] Step 2: Based on the reduction signal, reduce the index range according to the topic keywords and combine Boolean operators to obtain a new data text candidate set;

[0081] Specifically, related topic keywords are combined using Boolean operators to obtain combined topic keywords. For example, the related words "apple" and "mobile phone" are combined into "apple" AND "mobile phone", where AND is a Boolean operator.

[0082] Re-comparison is performed between the combined subject keywords and the subject keywords of the data text stored in the retrieval platform to obtain a new candidate set of data texts;

[0083] It should be noted that the search range value corresponding to the new data text candidate set is smaller than the preset search range threshold;

[0084] Step 3: Based on the new data text candidate set, use natural language processing technology to convert the data text in the new data text candidate set and the data retrieval text into vector representations, obtain the semantic vectors of the data text and the semantic vectors of the data retrieval text, and use cosine similarity to calculate the similarity between the data text and the data retrieval text;

[0085] Specifically, natural language processing technology is used to convert the data text in the new data text candidate set and the data retrieval text into vector representations, including:

[0086] D1, build a vocabulary containing all unique words from the data text;

[0087] D2, use the pre-trained word embedding model BERT to map each word in the vocabulary into a high-dimensional vector space to obtain the vocabulary vector;

[0088] D3: For each data text, sum the TF-IDF values ​​(term frequency - inverse document frequency) of all words to obtain the total vector weight of the word. Ratio the TF-IDF value of the word to the total vector weight of the word to obtain the vector weight of the word (v1, v2, v3...vn, where n represents the number of words).

[0089] D4: For each data text, multiply all word vectors by the corresponding word vector weights, then sum and average the results to obtain the semantic vector of the data text;

[0090] Specifically, the semantic vector of the data retrieval text is obtained in the same way as the semantic vector of the data text;

[0091] The cosine similarity is used to calculate the similarity between the data text and the data retrieval text, specifically:

[0092] By formula: Obtain the similarity cosθ between the data text and the data retrieval text, where A represents the semantic vector of the data retrieval text, B represents the semantic vector of the data text, and ||A|| and ||B|| represent the modulus lengths of vectors A and B, respectively (i.e., the length of the vector, which can be obtained by taking the square root of the sum of the squares of the elements of the vector);

[0093] Step 4: Obtain the user's contextual feature data, combine it with the feature data of the data text in the corresponding new data text candidate set, and perform correlation analysis to obtain the correlation value between the user's contextual feature data and the data text. Combined with the similarity between the data text and the data search text, the matching degree is obtained through processing and analysis, and the corresponding data text is recommended to the user based on the matching degree.

[0094] Specifically, the user's contextual feature data includes but is not limited to the query time and location, and the corresponding feature data of the data text in the new data text candidate set includes but is not limited to the data text creation time and the location included;

[0095] The specific process of correlation analysis is as follows:

[0096] For the feature data of the data text and the context feature data of the user, the scores between the feature data are calculated respectively, and all the scores are averaged to obtain their correlation values;

[0097] Exemplary:

[0098] Use the time window to compare the data text creation time and the user query time, and combine it with a linear function to calculate the score, specifically:

[0099] First, calculate the time difference (in days) between the data text creation time and the user query time;

[0100] Set a time window (e.g. 7 days);

[0101] Calculate the score:

[0102] Use linear function: score = 1-(time difference / time window length);

[0103] Note that if the time difference is greater than the time window length, the score can be 0 or a small negative number, indicating no correlation or negative correlation.

[0104] Exemplary:

[0105] Get the distance between the data text location and the user's location;

[0106] Calculate the score:

[0107] Use a linear function: score = 1-(distance / maximum distance threshold), where the maximum distance threshold is an empirical value set according to the application scenario;

[0108] All the scores obtained are marked as F1, F2, ..., Fz, where z represents the number of scores;

[0109] Average all scores:

[0110] The correlation value GL is obtained by the formula: GL = c1*F1+c2*F2+...cn*Fn, where c1, c2...cn are weight coefficients;

[0111] The weight coefficient is obtained as follows:

[0112] Obtain the user's historical query records within the historical query period. If the user's contextual feature data appears in a single historical query record, mark the single historical query record as a query record that appears. If the user's contextual feature data does not appear in a single historical query record, mark the single historical query record as a query record that does not appear.

[0113] Count the number of query records and compare it with the total number of historical query records to obtain the query record ratio, which is marked as CX;

[0114] Obtain the number of scenario feature data appearing in the query record, sum and average the number of scenario feature data in all query records to obtain the mean number of scenario feature data in the query record, and compare it with the total number of scenario feature data of the user to obtain the occurrence feature data ratio, which is marked as TZ;

[0115] The occurrence value CV of the scenario feature data is obtained by the formula: CV = g1*CX + g2*TZ, where g1 and g2 are both preset proportional coefficients, where g1 is 1.064 and g2 is 1.031;

[0116] It should be noted that there are m groups of historical data, each of which includes the query record ratio CX and the occurrence feature data ratio TZ. Since the query record ratio CX and the occurrence feature data ratio TZ will affect the appearance of the scenario feature data, according to the appearance degree of the scenario feature data, the staff uses the offline model to fit the query record ratio CX and the occurrence feature data ratio TZ to obtain the appearance value CV of the scenario feature data matched by the query record ratio CX and the occurrence feature data ratio TZ;

[0117] Then, based on multiple sets of query record ratios CX and occurrence feature data ratios TZ, a linear model is used to fit the historical data of multiple sets of query record ratios CX, occurrence feature data ratios TZ, and occurrence value CV of scenario feature data. The prepared historical data is brought into the selected fitting model for fitting to obtain fitting coefficients g1 and g2;

[0118] Sum up the occurrence values ​​CV of all scenario feature data to obtain the total occurrence value, perform ratio processing on the occurrence value CV of scenario feature data and the total occurrence value to obtain the weight coefficient of the corresponding score of the scenario feature data;

[0119] Sum the obtained correlation value between the user's context feature data and the data text, and the similarity between the data text and the data search text to obtain the matching degree;

[0120] Recommend corresponding data texts to users in descending order of matching degree;

[0121] The technical solution of the embodiment of the present invention is: receiving the data retrieval text input by the user on its retrieval platform and preprocessing it, based on the preprocessed data retrieval text, using statistical feature technology to extract the subject keywords in the data retrieval text, comparing the subject keywords in the data retrieval text with the subject keywords of the data text stored in the retrieval platform, screening out the data texts that meet the subject keywords in the data retrieval text as the data text candidate set, analyzing and processing the retrieval range value according to the belonging field and the number of texts of each data text in the data text candidate set, judging whether the index range needs to be reduced according to the retrieval range value, and if so, generating a reduction signal; based on the reduction signal, reducing the index range according to the subject keywords and combining with Boolean operators, and obtaining a new data text candidate set. The present invention evaluates the index range of its data text candidate set and uses Boolean operators to reduce the index range when its range is large, which is conducive to reducing the number of index matching correlations with low relevance. According to the text, it is beneficial to improve the crawling efficiency; based on the new data text candidate set, the data text in the new data text candidate set and the data retrieval text are converted into vector representations using natural language processing technology to obtain the semantic vector of the data text and the semantic vector of the data retrieval text, and the cosine similarity is used to calculate the similarity between the data text and the data retrieval text; the user's situational feature data is obtained, and combined with the feature data of the data text in the corresponding new data text candidate set, and a correlation analysis is performed to obtain the correlation value between the user's situational feature data and the data text, and combined with the similarity between the data text and the data retrieval text, the matching degree is obtained through processing and analysis, and the corresponding data text is recommended to the user according to the matching degree. The present invention not only calculates the similarity, but also integrates the situational analysis, and crawls and tracks the data text through the similarity obtained by semantic analysis and the correlation value obtained by situation analysis, thereby improving the crawling efficiency of stored data while improving the crawling accuracy of stored data.

[0122] Example 2

[0123] like Figure 2 As shown, the object storage data crawling path tracking system according to an embodiment of the present invention includes:

[0124] Retrieval tracking and evaluation module: Receives data retrieval texts input by users on their retrieval platform and performs preprocessing. Based on the preprocessed data retrieval texts, statistical feature technology is used to extract subject keywords in the data retrieval texts. The subject keywords in the data retrieval texts are compared with the subject keywords of data texts stored in the retrieval platform. Data texts that meet the subject keywords in the data retrieval texts are selected as data text candidate sets. According to the domain and number of texts in the data text candidate sets, the retrieval range value is obtained through analysis and processing. According to the retrieval range value, it is determined whether the index range needs to be reduced. If so, a reduction signal is generated.

[0125] Retrieval tracking control module: Based on the reduction signal, it reduces the index scope according to the subject keywords and combines Boolean operators to obtain a new set of data text candidates;

[0126] Similarity analysis module: Based on the new data text candidate set, the data text in the new data text candidate set and the data retrieval text are converted into vector representations using natural language processing technology, the semantic vectors of the data text and the semantic vectors of the data retrieval text are obtained, and the similarity between the data text and the data retrieval text is calculated using cosine similarity;

[0127] Association matching processing module: obtains the user's situational feature data, combines it with the feature data of the data text in the corresponding new data text candidate set, and performs association analysis to obtain the association value between the user's situational feature data and the data text, and combines it with the similarity between the data text and the data retrieval text to obtain the matching degree through processing and analysis, and recommends the corresponding data text to the user based on the matching degree.

[0128] The basic principles, main features, and advantages of the present invention are shown and described above. Those skilled in the art should understand that the present invention is not limited to the foregoing embodiments. The foregoing embodiments and descriptions are merely illustrative of the principles of the present invention. Various changes and modifications may be made to the present invention without departing from the spirit and scope of the present invention. Such changes and modifications are intended to fall within the scope of the present invention. The scope of protection claimed in the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for tracing a data crawling path based on an object storage system, characterized by: include: Receive the data retrieval text input by the user on its retrieval platform and perform preprocessing, extract the subject keywords in the data retrieval text based on the preprocessed data retrieval text using statistical feature technology, compare the subject keywords in the data retrieval text with the subject keywords of the data texts stored in the retrieval platform, screen out the data texts that meet the subject keywords in the data retrieval text as the data text candidate set, analyze and process the data texts in the data text candidate set according to the domain and the number of texts, and determine whether the index range needs to be reduced according to the retrieval range value. If so, generate a reduction signal; Based on the reduction signal, the index scope is reduced according to the topic keywords and combined with Boolean operators, and a new set of data text candidates is obtained; Based on the new data text candidate set, the data text in the new data text candidate set and the data retrieval text are converted into vector representations using natural language processing technology, the semantic vectors of the data text and the semantic vectors of the data retrieval text are obtained, and the similarity between the data text and the data retrieval text is calculated using cosine similarity; The user's situational feature data is obtained, and combined with the feature data of the data text in the corresponding new data text candidate set, and correlation analysis is performed to obtain the correlation value between the user's situational feature data and the data text, and combined with the similarity between the data text and the data retrieval text, the matching degree is obtained through processing and analysis, and the corresponding data text is recommended to the user based on the matching degree.

2. The object storage data crawling path tracing method according to claim 1, characterized in that: The extraction process of the subject keywords includes: Get the part of speech of each word from all the words, and filter out the topic keywords from all the words based on the part of speech and the key value of the word GJ; Specifically, if the part of speech of a word falls within the range of parts of speech commonly used to express a topic word, and the key value GJ of the word is greater than or equal to a preset key threshold, the word is marked as a topic keyword.

3. The object storage data crawling path tracing method according to claim 2, characterized in that: The key value GJ is obtained as follows: The obtained word frequency ratio PL, TF-IDF ratio TB and word span ratio KB are processed by the formula: The key value GJ is obtained, where s1, s2 and s3 are all preset proportional coefficients.

4. The object storage data crawling path tracing method according to claim 3, characterized in that: The word frequency ratio PL is obtained by ratioing the word frequency to the sum of the frequency of all words. The TF-IDF ratio TB and the word span ratio KB of the word are obtained in the same way as the frequency ratio PL.

5. The object storage data crawling path tracing method according to claim 1, characterized in that: The method for obtaining the search range value is as follows: Obtaining the attribution domain of each data text in the data text candidate set, removing duplicate attribution domains based on the attribution domain type, obtaining the number of valid attribution domain types, and performing a ratio processing on the number of valid attribution domain types in the data text candidate set and the total number of attribution domain types stored in the retrieval platform to obtain the valid domain ratio of the data text candidate set; Count the number of data texts in each type of valid attribution domain in the data text candidate set, sum and average them to obtain the mean number of domain texts, and compare the mean number of domain texts with the total number of data texts stored in the retrieval platform to obtain the text number ratio of the data text candidate set; The effective domain ratio and the text quantity ratio of the data text candidate set are summed to obtain the retrieval range value.

6. The object storage data crawling path tracing method according to claim 1, characterized in that: The method of converting the data text in the new data text candidate set and the data retrieval text into a vector representation by using natural language processing technology includes: D1, build a vocabulary containing all unique words from the data text; D2, use the pre-trained word embedding model BERT to map each word in the vocabulary into a high-dimensional vector space to obtain the vocabulary vector; D3: For each data text, sum the TF-IDF values ​​of all words to obtain the total vector weight of the word, and perform a ratio process on the TF-IDF value of the word and the total vector weight of the word to obtain the vector weight of the word; D4: For each data text, all word vectors are multiplied by the vector weights of the corresponding words, and the sum is averaged to obtain the semantic vector of the data text.

7. The object storage data crawling path tracing method according to claim 1, characterized in that: The matching degree is obtained by summing the association value between the user's scenario feature data and the data text and the similarity between the data text and the data search text.

8. The object storage data crawling path tracing method according to claim 7, characterized in that: The correlation value is obtained by averaging the scores between the feature data of the data text and the context feature data of the user, specifically: All the scores obtained are marked as F1, F2, ..., Fz, where z represents the number of scores; Average all scores: The correlation value GL is obtained by the formula: GL=c1*F1+c2*F2+...cn*Fn, where c1, c2...cn are weight coefficients.

9. The object storage data crawling path tracing method according to claim 8, characterized in that: The weight coefficient is obtained as follows: Obtain the user's historical query records within the historical query period. If the user's contextual feature data appears in a single historical query record, mark the single historical query record as a query record that appears. If the user's contextual feature data does not appear in a single historical query record, mark the single historical query record as a query record that does not appear. Count the number of query records and compare it with the total number of historical query records to obtain the query record ratio, which is marked as CX; Obtain the number of scenario feature data appearing in the query record, sum and average the number of scenario feature data in all query records to obtain the mean number of scenario feature data in the query record, and compare it with the total number of scenario feature data of the user to obtain the occurrence feature data ratio, which is marked as TZ; The occurrence value CV of the scenario feature data is obtained by the formula: CV = g1*CX+g2*TZ, where g1 and g2 are both preset proportional coefficients; The occurrence values ​​CV of all scenario feature data are summed up to obtain the total occurrence value, and the occurrence value CV of the scenario feature data is ratioed to the total occurrence value to obtain the weight coefficient of the corresponding score of the scenario feature data.

10. A system for tracking a crawling path based on object storage data, the system being used to implement the tracking method according to any one of claims 1 to 9, characterized in that: include: Retrieval tracking and evaluation module: Receives data retrieval texts input by users on their retrieval platform and performs preprocessing. Based on the preprocessed data retrieval texts, statistical feature technology is used to extract subject keywords in the data retrieval texts. The subject keywords in the data retrieval texts are compared with the subject keywords of data texts stored in the retrieval platform. Data texts that meet the subject keywords in the data retrieval texts are selected as data text candidate sets. According to the domain and number of texts in the data text candidate sets, the retrieval range value is obtained through analysis and processing. According to the retrieval range value, it is determined whether the index range needs to be reduced. If so, a reduction signal is generated. Retrieval tracking control module: Based on the reduction signal, it reduces the index scope according to the subject keywords and combines Boolean operators to obtain a new set of data text candidates; Similarity analysis module: Based on the new data text candidate set, the data text in the new data text candidate set and the data retrieval text are converted into vector representations using natural language processing technology, the semantic vectors of the data text and the semantic vectors of the data retrieval text are obtained, and the similarity between the data text and the data retrieval text is calculated using cosine similarity; Association matching processing module: obtains the user's situational feature data, combines it with the feature data of the data text in the corresponding new data text candidate set, and performs association analysis to obtain the association value between the user's situational feature data and the data text, and combines it with the similarity between the data text and the data retrieval text to obtain the matching degree through processing and analysis, and recommends the corresponding data text to the user based on the matching degree.

Images