A method and system for capturing internet data

Abstract

The present invention provides a method and system for capturing Internet data, belonging to the field of Internet technology, and the method includes: capturing the search information of users on the Internet platform according to the capture tool set by the Internet platform, analyzing the search information based on the preset semantic model set by the Internet platform, and obtaining several search parameters in combination with the parameter database; combining the search parameters according to the preset search rules of the Internet platform, setting the combination weight for each combination and estimating the resource response of each combination; generating resource capture tasks based on the combination weight and resource response, and then performing reliable data capture based on the Internet platform. The reliability of data capture is improved.

Description

Technical Field

[0001] The present invention relates to the field of Internet technology, and in particular to a method and system for capturing Internet data. Background Art

[0002] With the rapid development of the Internet, all kinds of data are growing rapidly. At this time, it is particularly important to capture reliable data.

[0003] Common platform-based data crawling mainly relies on the existence of each word in the search information in the resource (that is, the search results can contain a single or multiple search words), and the crawling results are massive resource results obtained by combining the resource information of a single word or multiple words. This process revolves around the words in the search information itself. Due to the green nature of the current network environment and the abundance of resources, if the search information contains words that are not relevant to the user's needs or words with bad descriptions during this process, it will interfere with the crawling of resources to a certain extent, thereby reducing the reliability of the overall crawling results.

[0004] Therefore, the present invention proposes a method and system for capturing Internet data. Summary of the invention

[0005] The present invention provides a method and system for crawling Internet data, which are used to analyze retrieval information through a preset semantic model and obtain retrieval parameters in combination with a parameter database, effectively avoiding interference in the crawling process, and ensuring the rationality of crawling task generation by analyzing parameter weights and resource responses, thereby improving the reliability of data crawling.

[0006] The present invention provides a method for crawling Internet data, comprising:

[0007] Step 1: Capture the user's search information on the Internet platform using a capture tool set by the Internet platform, analyze the search information based on a preset semantic model set by the Internet platform, and obtain several search parameters in combination with a parameter database;

[0008] Step 2: Combining the search parameters according to the preset search rules of the Internet platform, setting a combination weight for each combination, and estimating the resource response of each combination;

[0009] Step 3: Generate a resource crawling task based on the combined weights and resource response conditions, and then perform reliable data crawling based on the Internet platform.

[0010] Preferably, several search parameters are obtained in combination with the parameter database, including:

[0011] Reliably parsing the search information based on a preset semantic model set by the Internet platform to obtain information reliability, and at the same time, performing vocabulary parsing on the search information based on the information parsing model to obtain reliable vocabulary and unreliable vocabulary;

[0012] Locking the input window of the search information based on the Internet platform;

[0013] Retrieving historical input information based on the input window from a historical database, and determining a reliable influence coefficient of the window business environment of the input window on each historical input information;

[0014] All reliable influence coefficients are processed by mean and variance to obtain correction factors;

[0015] Matching the unreliable vocabulary with the vocabulary-coefficient comparison table to obtain the initial reliability coefficient of the corresponding unreliable vocabulary, and combining the correction factor, the semantic similarity factor between the corresponding unreliable vocabulary and each reliable vocabulary, and the information reliability to determine the suspicious coefficient of the corresponding unreliable vocabulary;

[0016] When the suspicious coefficient is greater than or equal to the preset reliability coefficient, the corresponding unreliable word is changed to a reliable word, and then all the reliable words are compared with the parameter database in turn to obtain a number of search parameters.

[0017] Preferably, determining the reliable influence coefficient of the window business environment of the input window on each historical input information includes:

[0018] Obtain historical input information of the input window in the current unit time period T1 and m unit time periods before the current unit time period, and respectively calculate the historical reliability slope of each historical input information, wherein the historical reliability slope is obtained by analyzing before the historical input information is input into the input window and after the historical input information is input into the input window;

[0019] Construct the first curve based on all historical reliable slopes based on the sequence of unit periods;

[0020] Construct the basis function of each curve point and each remaining point in the first curve , determine the influence factor of each basic function on the corresponding curve point from the function-factor comparison table, where, Indicates the information correlation between the historical input information of the i-th curve point and the historical input information of the j-th remaining point; represents the slope difference between the historically reliable slope of the i-th curve point and the historically reliable slope of the j-th remaining point;

[0021] The reliable influence coefficient of the corresponding historical input information is determined based on all influencing factors of the same curve point, the reliable ratio of the historical input information of the corresponding curve point before it is input into the input window, and the reliable ratio based on the reliable evaluation result after it is input into the input window.

[0022] Preferably, setting a combination weight for each combination includes:

[0023] Obtain the first historical search frequency of each search parameter in the corresponding combination, and at the same time, obtain the second historical search frequency and the third historical search frequency of the corresponding combination, wherein the second historical search frequency is the number of searches performed strictly in accordance with the order of the search parameters in the corresponding combination, and the third historical frequency is the number of searches performed not strictly in accordance with the order of the search parameters in the corresponding combination;

[0024] Setting a preliminary weight to a corresponding combination depending on the first historical search frequency, the second historical search frequency, and the third search frequency;

[0025] Acquire from the search database a first search history with the highest matching degree for all search parameters involved in the search information and a second search history with the second highest matching degree, and obtain a first number of matches between the first search history and the parameters in the corresponding combination and a second number of matches between the second search history and the parameters in the corresponding combination;

[0026] According to the first quantity and the second quantity, the corresponding preliminary weights are adjusted to obtain the combined weight.

[0027] Preferably, the resource response of each combination is estimated, including:

[0028] Retrieving a resource instruction set based on each combination from the Internet platform, and obtaining the calling function of each resource instruction in the resource instruction set in the corresponding resource calling process;

[0029] The resource calling processes involved in the resource instruction set are sorted in order according to the calling function, and the response events of the corresponding combinations in the corresponding resource calling process are determined in turn according to the sorting results, and then the response order of all response events under the same resource calling process is combined to obtain the resource response situation.

[0030] Preferably, generating a resource crawling task based on the combined weight and the resource response situation includes:

[0031] Determine a resource response range based on the calling effect and the combination weight under each combination, and extract the involved events in the required situation from the resource response situation according to the resource response range;

[0032] Extracting repeated events from the involved events under all combinations, setting a first value for each repeated event, and setting a second value for non-repeated events from the involved events under all combinations;

[0033] The first value and the second value are respectively assigned to corresponding events to generate event vectors, and are input into an event analysis model to obtain a resource capture task.

[0034] Preferably, the resource response range is determined based on the calling function and the combination weight under each combination, including:

[0035] Determine whether the second historical search frequency is 0, and if so, randomly select a historical search result from the search history under the third historical frequency;

[0036] Otherwise, randomly select a historical search result from the search history under the second historical search frequency;

[0037] Obtain the process elements of each resource retrieval process under each combination and perform similarity analysis with the historical resource elements of the historical search results to obtain a similar array under the corresponding combination;

[0038] Locking the to-be-used values ​​in the similarity array that are greater than the preset similarity value, and determining a preliminary response range for the resource retrieval process of the to-be-used values ​​based on the position of the ordering results;

[0039] Perform standardized numerical transformation on each call-up effect involved in the corresponding combination, and compare it with the combination weight under the corresponding combination to obtain a control array based on the preliminary response range, so as to construct a fitting line and a control curve based on the control array;

[0040] Determine whether the fitting line and the control curve have an intersection;

[0041] If there is an intersection, the response range under the curve segment obtained based on the intersection is used as the resource response range;

[0042] If there is no intersection, the initial response range is used as the resource response range.

[0043] Preferably, the response range under the curve segment obtained based on the intersection point is used as the resource response range, including:

[0044] If there is one intersection, lock the first midpoint of the line segment corresponding to the intersection and the leftmost point of the reference curve, and lock the second midpoint of the line segment corresponding to the intersection and the rightmost point of the reference curve, and obtain the curve segment formed by the first midpoint and the second midpoint;

[0045] If there are two intersection points, determine the third distance between the two intersection points;

[0046] If the third distance is greater than or equal to the preset distance, extracting the curve segment under the two intersection points from the reference curve;

[0047] If the third distance is less than the preset distance, obtain a third midpoint of the line segment corresponding to the leftmost point and the intersection point of the two intersection points close to the leftmost point of the reference curve, and obtain a fourth midpoint of the line segment corresponding to the rightmost point and the intersection point of the two intersection points close to the rightmost point of the reference curve, and obtain a curve segment formed by the third midpoint and the fourth midpoint;

[0048] If there are multiple intersection points, the first absolute value of the difference between each intersection point and the nearest peak point and the second absolute value of the difference between each intersection point and the nearest valley point are determined respectively; preferably, a combination weight is set for each combination, including:

[0049] Obtain the first historical search frequency of each search parameter in the corresponding combination, and at the same time, obtain the second historical search frequency and the third historical search frequency of the corresponding combination, wherein the second historical search frequency is the number of searches performed strictly in accordance with the order of the search parameters in the corresponding combination, and the third historical frequency is the number of searches performed not strictly in accordance with the order of the search parameters in the corresponding combination;

[0050] Setting a preliminary weight to a corresponding combination depending on the first historical search frequency, the second historical search frequency, and the third search frequency;

[0051] Acquire from the search database a first search history with the highest matching degree for all search parameters involved in the search information and a second search history with the second highest matching degree, and obtain a first number of matches between the first search history and the parameters in the corresponding combination and a second number of matches between the second search history and the parameters in the corresponding combination;

[0052] According to the first quantity and the second quantity, the corresponding preliminary weights are adjusted to obtain the combined weight.

[0053] Preferably, the resource response of each combination is estimated, including:

[0054] Retrieving a resource instruction set based on each combination from the Internet platform, and obtaining the calling function of each resource instruction in the resource instruction set in the corresponding resource calling process;

[0055] The resource calling processes involved in the resource instruction set are sorted in order according to the calling function, and the response events of the corresponding combinations in the corresponding resource calling process are determined in turn according to the sorting results, and then the response order of all response events under the same resource calling process is combined to obtain the resource response situation.

[0056] Preferably, generating a resource crawling task based on the combined weight and the resource response situation includes:

[0057] Determine a resource response range based on the calling effect and the combination weight under each combination, and extract the involved events in the required situation from the resource response situation according to the resource response range;

[0058] Extracting repeated events from the involved events under all combinations, setting a first value for each repeated event, and setting a second value for non-repeated events from the involved events under all combinations;

[0059] The first value and the second value are respectively assigned to corresponding events to generate event vectors, and are input into an event analysis model to obtain a resource capture task.

[0060] Preferably, the resource response range is determined based on the calling function and the combination weight under each combination, including:

[0061] Determine whether the second historical search frequency is 0, and if so, randomly select a historical search result from the search history under the third historical frequency;

[0062] Otherwise, randomly select a historical search result from the search history under the second historical search frequency;

[0063] Obtain the process elements of each resource retrieval process under each combination and perform similarity analysis with the historical resource elements of the historical search results to obtain a similar array under the corresponding combination;

[0064] Locking the to-be-used values ​​in the similarity array that are greater than the preset similarity value, and determining a preliminary response range for the resource retrieval process of the to-be-used values ​​based on the position of the ordering results;

[0065] Perform standardized numerical transformation on each call-up effect involved in the corresponding combination, and compare it with the combination weight under the corresponding combination to obtain a control array based on the preliminary response range, so as to construct a fitting line and a control curve based on the control array;

[0066] Determine whether the fitting line and the control curve have an intersection;

[0067] If there is an intersection, the response range under the curve segment obtained based on the intersection is used as the resource response range;

[0068] If there is no intersection, the initial response range is used as the resource response range.

[0069] Preferably, the response range under the curve segment obtained based on the intersection point is used as the resource response range, including:

[0070] If there is one intersection, lock the first midpoint of the line segment corresponding to the intersection and the leftmost point of the reference curve, and lock the second midpoint of the line segment corresponding to the intersection and the rightmost point of the reference curve, and obtain the curve segment formed by the first midpoint and the second midpoint;

[0071] If there are two intersection points, determine the third distance between the two intersection points;

[0072] If the third distance is greater than or equal to the preset distance, extracting the curve segment under the two intersection points from the reference curve;

[0073] If the third distance is less than the preset distance, obtain a third midpoint of the line segment corresponding to the leftmost point and the intersection point of the two intersection points close to the leftmost point of the reference curve, and obtain a fourth midpoint of the line segment corresponding to the rightmost point and the intersection point of the two intersection points close to the rightmost point of the reference curve, and obtain a curve segment formed by the third midpoint and the fourth midpoint;

[0074] If there are multiple intersection points, determine the first absolute value of the difference between each intersection point and the nearest peak point and the second absolute value of the difference between each intersection point and the nearest valley point respectively;

[0075] in accordance with Lock the last two intersection points, where Indicates the minimum value symbol; represents the absolute value of the j1th first difference; represents the j2th second difference absolute value; m1 represents the number of first difference absolute values; m2 represents the number of second difference absolute values;

[0076] Expanding the range of the last two intersection points according to the fourth distance between each of the last two intersection points and the nearest intersection point;

[0077] ;in, Indicates the extended distance based on the left intersection of the last two intersections; Indicates the extended distance based on the right intersection of the last two intersections; represents the fourth distance corresponding to the left intersection point of the last two intersection points; represents the fourth distance corresponding to the rightmost intersection point of the last two intersection points; It represents the distance between the leftmost intersection point of the last two intersection points and the leftmost point in the control curve; It represents the distance between the right intersection point among the last two intersection points and the rightmost point in the control curve; C0 represents the absolute value of the difference between the last two intersection points determined based on the control curve; C1 represents the absolute value of the difference between the left intersection point among the last two intersection points determined based on the control curve and the leftmost point in the control curve; C2 represents the absolute value of the difference between the right intersection point among the last two intersection points determined based on the control curve and the rightmost point in the control curve.

[0078] The present invention provides a system for capturing Internet data, comprising:

[0079] A parameter determination module, which is used to capture the user's search information on the Internet platform according to the capture tool set by the Internet platform, analyze the search information based on the preset semantic model set by the Internet platform, and obtain a number of search parameters in combination with the parameter database;

[0080] A combination analysis module, used to combine the search parameters according to the preset search rules of the Internet platform, set a combination weight for each combination, and estimate the resource response of each combination;

[0081] The data capture module is used to generate resource capture tasks based on the combined weights and resource response conditions, and then perform reliable data capture based on the Internet platform.

[0082] Compared with the prior art, the present invention has the following beneficial effects:

[0083] The retrieval information is analyzed by the preset semantic model and the retrieval parameters are obtained by combining with the parameter database, which effectively avoids interference in the crawling process. The rationality of crawling task generation is ensured by analyzing the weights of parameters and resource responses, thereby improving the reliability of data crawling.

[0084] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the written description and the accompanying drawings.

[0085] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0086] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0087] Figure 1 This is a flow chart of a method for crawling Internet data in an embodiment of the present invention;

[0088] Figure 2 This is a structural diagram of a system for capturing Internet data in an embodiment of the present invention;

[0089] Figure 3 This is a structural diagram of the scope expansion in an embodiment of the present invention. DETAILED DESCRIPTION

[0090] The preferred embodiments of the present invention are described below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0091] The present invention provides a method for capturing Internet data. Figure 1 As shown, including:

[0092] Step 1: Capture the user's search information on the Internet platform using a capture tool set by the Internet platform, analyze the search information based on a preset semantic model set by the Internet platform, and obtain several search parameters in combination with a parameter database;

[0093] Step 2: Combining the search parameters according to the preset search rules of the Internet platform, setting a combination weight for each combination, and estimating the resource response of each combination;

[0094] Step 3: Generate a resource crawling task based on the combined weights and resource response conditions, and then perform reliable data crawling based on the Internet platform.

[0095] In this embodiment, the Internet platform is similar to a browser-related platform such as Baidu and UC that can perform information retrieval.

[0096] In this embodiment, the capture tool refers to a tool set up on the platform, which is mainly used to record search information, and the search information is the search term entered by the user in the browser. The search term can be a long sentence description, a short sentence description, etc., for example, descriptions such as "how to build a web page".

[0097] In this embodiment, the preset semantic model performs reliability analysis on the retrieval information, and mainly focuses on the fact that the reliability of the semantic meaning is more negative, and the corresponding reliability is lower. The preset semantic model is obtained by training a neural network model based on semantic combinations of different reliabilities as samples, and the information reliability of different retrieval information can be directly obtained, and the value range is (0, 1).

[0098] In this embodiment, the parameter database includes different reliable words and search parameters that match the reliable words. After the reliable words are determined, the search parameters can be obtained by directly matching them with the database. The search parameters are search symbols corresponding to the reliable words. In order to unify the search format and facilitate calculation execution, each search symbol is unique.

[0099] In this embodiment, the preset search rule refers to combining any two search parameters, and the combination includes: any combination of at least two search parameters, and the results corresponding to each combined parameter can be regarded as a combination.

[0100] In this embodiment, the combination weight is related to the historical search frequency.

[0101] Resource response conditions are associated with each combined resource instruction set.

[0102] In this embodiment, the resource crawling task is generated based on the model, mainly for resource crawling, such as resources for the web page building steps.

[0103] The beneficial effects of the above technical solution are: by analyzing the retrieval information through a preset semantic model and combining the parameter database to obtain the retrieval parameters, the interference in the crawling process is effectively avoided, and the rationality of the crawling task generation is ensured by analyzing the weights of the parameters and the resource responses, thereby improving the reliability of data crawling.

[0104] The present invention provides a method for crawling Internet data, which combines a parameter database to obtain a number of search parameters, including:

[0105] Reliably parsing the search information based on a preset semantic model set by the Internet platform to obtain information reliability, and at the same time, performing vocabulary parsing on the search information based on the information parsing model to obtain reliable vocabulary and unreliable vocabulary;

[0106] Locking the input window of the search information based on the Internet platform;

[0107] Retrieving historical input information based on the input window from a historical database, and determining a reliable influence coefficient of the window business environment of the input window on each historical input information;

[0108] All reliable influence coefficients are processed by mean and variance to obtain correction factors;

[0109] Matching the unreliable vocabulary with the vocabulary-coefficient comparison table to obtain the initial reliability coefficient of the corresponding unreliable vocabulary, and combining the correction factor, the semantic similarity factor between the corresponding unreliable vocabulary and each reliable vocabulary, and the information reliability to determine the suspicious coefficient of the corresponding unreliable vocabulary;

[0110] When the suspicious coefficient is greater than or equal to the preset reliability coefficient, the corresponding unreliable word is changed to a reliable word, and then all the reliable words are compared with the parameter database in turn to obtain a number of search parameters.

[0111] In this embodiment, the preset semantic model is obtained by training the neural network model based on samples of historical retrieval information and the measurement results of the information reliability of the historical retrieval information by experts in the semantics, and the measurement results are: the sum of the reliability values ​​of different experts for the retrieval information multiplied by the measurement weights of the corresponding experts, and the value range of the information reliability is 0 to 1. The closer it is to 1, the more reliable the information is, which means that the number of irrelevant semantics and bad semantics in the retrieval information is smaller.

[0112] In this embodiment, the information parsing model is obtained by training a neural network model based on vocabulary combinations under different retrieval information and calibration results of reliable and unreliable words in the vocabulary combinations as samples. Therefore, reliable and unreliable words can be directly obtained according to the model.

[0113] In this embodiment, for example, the search information is "how to enrich the color of the web page to make the page look beautiful", the reliable words obtained by vocabulary analysis are: enrich, web page, build color, beautiful, and the unreliable words are: look;

[0114] The input window refers to a search box for inputting search information.

[0115] In this embodiment, the history database contains all the historical search information based on the input window on the Internet platform, that is, the historical input information.

[0116] In this embodiment, the window business environment refers to the reliability assessment environment of the input window for the retrieval information, and the reliability ratio is obtained by reversely inferring the retrieval results obtained after the retrieval information is input into the input window (because the reliability ratios corresponding to different window business environments after input into the window will be different, mainly to determine the reliability influence coefficient), and the reliability ratio is determined by the platform background before the retrieval information is input into the input window.

[0117] In this embodiment, the correction factor=the average value of all reliable influence coefficients+the variance of all reliable influence coefficients.

[0118] In this embodiment, the vocabulary-coefficient comparison table includes different unreliable vocabulary and initial reliability coefficients matching the unreliable vocabulary, for example, unreliable vocabulary: vocabulary 1, vocabulary 2, reliable vocabulary: vocabulary 3, vocabulary 4, vocabulary 5. At this time, the comparison table includes an initial reliability coefficient of 0.1 corresponding to vocabulary 1 and an initial reliability coefficient of 0.15 corresponding to vocabulary 2, and the value range of the initial reliability coefficient of the unreliable vocabulary is 0 to 0.3. It should be noted that an unreliable vocabulary will be defined as unreliable when analyzed alone, but may be defined as reliable when used in combination with other vocabulary. Therefore, it is necessary to determine the suspicious coefficient of the unreliable vocabulary.

[0119] The specific calculation method for the suspicious coefficient is as follows:

[0120] ;in, is the corresponding initial reliability coefficient; represents the semantic similarity factor between the corresponding unreliable word and the i1th reliable word; n1 represents the number of reliable words.

[0121] In this embodiment, the word meaning similarity factor is obtained based on the vocabulary combination-factor comparison table, and the similarity values ​​of the word meanings of different vocabulary combinations are included in the table. The table is pre-set and convenient for direct retrieval and use.

[0122] In this embodiment, the preset reliability coefficient is 0.31.

[0123] The beneficial effects of the above technical solution are: first, semantic models and parsing models are used to parse information and vocabulary to obtain the reliability of the entire information and the reliable and unreliable vocabulary in the entire information; secondly, historical input information is used to determine the reliable influence coefficient involved in the input window itself to obtain the correction factor, and then the suspicious coefficient is determined from three aspects: the correction factor, the word meaning similarity factor, and the information reliability, to achieve effective and reliable attribution of the unreliable coefficient again, to ensure the comprehensiveness of the vocabulary as much as possible, and then to ensure all possibilities of subsequent retrieval, while ensuring the reliability of the retrieval results and their integrity.

[0124] The present invention provides a method for capturing Internet data, which determines the reliable influence coefficient of the window business environment of the input window on each historical input information, including:

[0125] Obtain historical input information of the input window in the current unit time period T1 and m unit time periods before the current unit time period, and respectively calculate the historical reliability slope of each historical input information, wherein the historical reliability slope is obtained by analyzing before the historical input information is input into the input window and after the historical input information is input into the input window;

[0126] Construct the first curve based on all historical reliable slopes based on the sequence of unit periods;

[0127] Construct the basis function of each curve point and each remaining point in the first curve , determine the influence factor of each basic function on the corresponding curve point from the function-factor comparison table, where, Indicates the information correlation between the historical input information of the i-th curve point and the historical input information of the j-th remaining point; represents the slope difference between the historically reliable slope of the i-th curve point and the historically reliable slope of the j-th remaining point;

[0128] The reliable influence coefficient of the corresponding historical input information is determined based on all influencing factors of the same curve point, the reliable ratio of the historical input information of the corresponding curve point before it is input into the input window, and the reliable ratio based on the reliable evaluation result after it is input into the input window.

[0129] In this embodiment, T1 is a unit time period randomly specified based on the input window and is regarded as the current unit time period, and the m unit time periods are the m unit time periods immediately before the current unit time period T1, that is, historical input information in m+1 unit time periods is obtained.

[0130] In this embodiment, because the reliability ratios of historical input information evaluated before and after input based on the input window are slightly different, the historical reliability slope = (reliability ratio after input - reliability ratio before input) / (1 + reliability ratio before input).

[0131] In this embodiment, the reliability ratio before input = the number of reliable words / the total number of reliable and unreliable words;

[0132] Reliability ratio after input = number of reliable words obtained from search result feedback / total number of reliable and unreliable words, where the number of reliable words obtained from search result feedback refers to the number of words in the search results that are exactly the same as the relevant words in the historical input information. For example, the historical input information determined before input includes: word 1, word 2, word 3, word 4, where word 1, word 2, word 3 are reliable words, and word 4 is an unreliable word. The ratio at this time is 3 / 4.

[0133] After the input, the completely identical words determined according to the search results are: word 1, word 3. At this time, word 2 and word 4 are unreliable words, and word 1 and word 3 are reliable words. At this time, the ratio is: 2 / 4.

[0134] In this embodiment, each historical input information has a historical reliable slope, and each historical input information has its corresponding input time, so the first curve can be constructed based on the chronological order, that is, the horizontal axis of the first curve is the time axis, and the vertical axis is the historical reliable slope axis.

[0135] In this embodiment, the function-factor comparison table includes different basic functions and influencing factors matching the basic functions, and the value range of the influencing factors is (0, 1), wherein each curve point corresponds to a historical input information.

[0136] In this embodiment, the reliability influence coefficient is:

[0137] If the reliability ratio of the same historical input information based on the reliability evaluation result is greater than or equal to the reliability ratio before the historical input information is input into the input window, the reliability influence coefficient is regarded as 0;

[0138] If the reliability ratio of the same historical input information based on the reliability evaluation result is less than the reliability ratio before the historical input information is input into the input window, the reliability influence coefficient is calculated according to the following formula:

[0139] The average value of all influencing factors of the same curve point×the absolute value of the difference between the reliability ratio based on the reliability evaluation result and the reliability ratio before the historical input information is input into the input window.

[0140] The beneficial effects of the above technical solution are: by obtaining historical input information of m+1 unit time periods to determine the sample quantity to ensure the data integrity of curve construction, and obtaining the influence factor based on the slope comparison result of each point and the basic function constructed based on information correlation, to ensure the accuracy of the reliable influence coefficient, thereby providing a basis for calculating the suspicious coefficient.

[0141] The present invention provides a method for crawling Internet data, which sets a combination weight for each combination, including:

[0142] Obtain the first historical search frequency of each search parameter in the corresponding combination, and at the same time, obtain the second historical search frequency and the third historical search frequency of the corresponding combination, wherein the second historical search frequency is the number of searches performed strictly in accordance with the order of the search parameters in the corresponding combination, and the third historical frequency is the number of searches performed not strictly in accordance with the order of the search parameters in the corresponding combination;

[0143] Setting a preliminary weight to a corresponding combination depending on the first historical search frequency, the second historical search frequency, and the third search frequency;

[0144] Acquire from the search database a first search history with the highest matching degree for all search parameters involved in the search information and a second search history with the second highest matching degree, and obtain a first number of matches between the first search history and the parameters in the corresponding combination and a second number of matches between the second search history and the parameters in the corresponding combination;

[0145] According to the first quantity and the second quantity, the corresponding preliminary weights are adjusted to obtain the combined weight.

[0146] In this embodiment, the corresponding combination includes: parameter 1, parameter 2, that is, through the historical search records of the Internet platform, the search frequency involving parameter 1 and the search frequency involving parameter 2 can be directly obtained, that is, the first historical search frequency is regarded as the number of searches plus 1 as long as the search parameter exists in the current search information, the second historical search number is the number of searches performed in the manner of parameter 1 & parameter 2, and the third historical search number is the number of searches performed in the manner of parameter 2 & parameter 1.

[0147] In this embodiment, .

[0148] In this embodiment, the search database contains different historical input information, wherein the first search history and the second search history are two histories in the search database.

[0149] In this embodiment, the parameter matching degree is calculated based on a similarity function, which belongs to the prior art.

[0150] In this embodiment, combination weight=adjustment weight of corresponding combination / sum of adjustment weights of all combinations;

[0151] The adjusted weight of the corresponding combination = the initial weight of the corresponding combination × (first quantity + second quantity) / (2 × the number of parameters in the corresponding combination).

[0152] The beneficial effect of the above technical solution is: preliminarily estimating the weight through the individual retrieval frequency of the parameters, the retrieval frequency in strict order and in non-strict order, and then adjusting the weight through the number of parameter matches to obtain the combined weight, providing a basis for obtaining resource response conditions.

[0153] The present invention provides a method for crawling Internet data, estimating the resource response of each combination, including:

[0154] Retrieving a resource instruction set based on each combination from the Internet platform, and obtaining the calling function of each resource instruction in the resource instruction set in the corresponding resource calling process;

[0155] The resource calling processes involved in the resource instruction set are sorted in order according to the calling function, and the response events of the corresponding combinations in the corresponding resource calling process are determined in turn according to the sorting results, and then the response order of all response events under the same resource calling process is combined to obtain the resource response situation.

[0156] In this embodiment, the resource instruction set refers to the instructions including the combination. For example, entry 1, entry 2 and entry 3 all contain the same combination. At this time, entry 1, entry 2 and entry 3 each correspond to a resource scheduling process because resource scheduling needs to be performed for different entries.

[0157] In this embodiment, the resource instruction set refers to the calling instruction corresponding to the combination involved in the corresponding entry. For example, entry 1 involves parameter 1 and parameter 2, but the resource instruction based on entry 1 is a matching resource for dark colors, and the resource instruction based on entry 2 is a matching resource for light colors, etc. The calling effect refers to the role played by the corresponding resource instruction in the process, and the greater the role played, the higher the corresponding sorting result, and the effect size of the calling effect is expressed by a numerical value ranging from 0 to 1. The larger the numerical value, the greater the effect, and the role played by each instruction in the response resource calling process is pre-set, so the numerical value of the corresponding effect size can also be directly determined.

[0158] In this embodiment, the order of sorting is first sorted by the size of the effect, for example: instruction 1, instruction 2, instruction 3, and then sorted by the size of the effect: the resource call process under instruction 2 - the resource call process under instruction 1 - the resource call process under instruction 3.

[0159] For example, response events are color response events for dark colors and light colors. Since one call quality may involve more than one response event in the corresponding resource call process, it is necessary to sort the response order of the response events. Therefore, the resource response situation of each combination = {corresponding combination: response event b2 under instruction 2--response event a1 response event a2 under instruction 1 response event b3 under instruction 3--response event b1}.

[0160] The beneficial effect of the above technical solution is: by calling the resource instruction set to determine the calling effect and then by sorting the effect and response sequence, the resource response situation is effectively obtained, providing a basis for the orderly resource grabbing task.

[0161] The present invention provides a method for crawling Internet data, which generates a resource crawling task based on the combined weight and resource response status, including:

[0162] Determine a resource response range based on the calling effect and the combination weight under each combination, and extract the involved events in the required situation from the resource response situation according to the resource response range;

[0163] Extracting repeated events from the involved events under all combinations, setting a first value for each repeated event, and setting a second value for non-repeated events from the involved events under all combinations;

[0164] The first value and the second value are respectively assigned to corresponding events to generate event vectors, and are input into an event analysis model to obtain a resource capture task.

[0165] In this embodiment, the resource corresponding range refers to determining the left and right boundaries extracted from the resource response situation.

[0166] In this embodiment, the involved event refers to the response event mentioned in the required situation.

[0167] In this embodiment, the first value=the average value of the sum of the event weights of the corresponding repeated events in the resource retrieval process involved in the corresponding combination×2.

[0168] The second value = the event weight of the corresponding non-repeating event in the resource retrieval process involved in the corresponding combination.

[0169] In this embodiment, event vector = {related events assigned values}.

[0170] In this embodiment, the event analysis model is obtained by training a neural network model based on samples of response events with different combinations of assigned values ​​and set tasks for the combined response events. Therefore, the resource crawling task can be directly obtained based on the model. For example, the first search is performed using parameter 1 & parameter 3 & parameter 4, and then the second search is performed using parameter 1% parameter 3, with the first search result in front and the second search result in the back.

[0171] The beneficial effect of the above technical solution is: by extracting the settings of events and values ​​involved from the resource response situation, it is convenient to construct an event vector and then obtain a resource crawling task, thereby realizing reliable and effective data crawling.

[0172] The present invention provides a method for crawling Internet data, which determines the resource response range based on the retrieval function under each combination and the combination weight, including:

[0173] Determine whether the second historical search frequency is 0, and if so, randomly select a historical search result from the search history under the third historical frequency;

[0174] Otherwise, randomly select a historical search result from the search history under the second historical search frequency;

[0175] Obtain the process elements of each resource retrieval process under each combination and perform similarity analysis with the historical resource elements of the historical search results to obtain a similar array under the corresponding combination;

[0176] Locking the to-be-used values ​​in the similarity array that are greater than the preset similarity value, and determining a preliminary response range for the resource retrieval process of the to-be-used values ​​based on the position of the ordering results;

[0177] Perform standardized numerical transformation on each call-up effect involved in the corresponding combination, and compare it with the combination weight under the corresponding combination to obtain a control array based on the preliminary response range, so as to construct a fitting line and a control curve based on the control array;

[0178] Determine whether the fitting line and the control curve have an intersection;

[0179] If there is an intersection, the response range under the curve segment obtained based on the intersection is used as the resource response range;

[0180] If there is no intersection, the initial response range is used as the resource response range.

[0181] In this embodiment, the elements are related to the resource retrieval elements involved in the retrieval process, and are the participating roles involved in the resource retrieval process, such as retrieval control elements, retrieval planning elements, etc., wherein the elements involved in different resource retrieval processes are all known.

[0182] In this embodiment, the similarity analysis is calculated based on a similarity function, which belongs to the prior art, and the similarity array = {similarity analysis values ​​of different resource retrieval processes and corresponding historical resource elements under the same combination}.

[0183] In this embodiment, the preset similarity value is 0.7, and the value to be used refers to a value in the similarity array whose similarity analysis value is greater than 0.7, and each resource retrieval process corresponds to a similarity analysis value.

[0184] In this embodiment, the order of sorting results refers to the order of sorting results based on the resource call process, for example: the resource response situation of each combination = {corresponding combination: response event b2 under instruction 2--response event a1 response event a2 under instruction 1 response event b3 under instruction 3--response event b1}, wherein the similarity analysis value of the resource call process corresponding to instructions 1 and 3 is greater than 0.7. At this time, the response events involved in instructions 1 and 3 are taken as the preliminary response range, namely: {corresponding combination: response event a2 under instruction 1 response event b3 under instruction 3--response event b1}.

[0185] In this embodiment, the resource retrieval process of the value to be used is determined based on the position of the sorting result in order to further lock the response event and then determine the range. At this time, what is generally obtained is the range formed by the instructions corresponding to all the values ​​to be used according to the outermost position instructions corresponding to the sorting result.

[0186] In this embodiment, the standardized numerical conversion is to represent the calling effect with a numerical value between 0 and 1.

[0187] In this embodiment, the standardized numerical conversion result and the control value of the combination weight are obtained based on the matching of a dual value-control mapping table. The mapping table contains standardized numerical conversion results of different combinations, combination weights, and matching control values, which are pre-planned. For example, the standardized numerical conversion result is 0.9, the combination weight is 0.8, and the matching control value is 0.8.

[0188] The comparison array includes a number of different comparison values, and the comparison values ​​involved in the comparison array correspond one-to-one to the resource retrieval processes involved in the initial response scope.

[0189] In this embodiment, a curve is drawn for the control values ​​in the control array to obtain a control curve, and a fitting process is performed on the curve to obtain a fitting line, and the fitting process is a linear fitting process.

[0190] In this embodiment, the lines are aligned to determine whether there is an intersection.

[0191] In this embodiment, the resource response range is a valid resource call process for the corresponding combination obtained by determining the extraction boundary based on the resource response situation, and the determination of the extraction boundary is to lock the resource call process under the two boundaries for the resource response situation, that is, these two resource boundaries and all the resource call processes within them are the process results corresponding to the resource response range.

[0192] The beneficial effect of the above technical solution is: by conducting similarity analysis on resource elements to construct a similarity array and then locking the values ​​to ensure the reliability of the initial range locking, and subsequently further determining the resource response range through comparative analysis of weights and retrieval effects to ensure the reliability of resource retrieval.

[0193] The present invention provides a method for capturing Internet data, which uses a response range under a curve segment obtained based on an intersection as a resource response range, including:

[0194] If there is one intersection, lock the first midpoint of the line segment corresponding to the intersection and the leftmost point of the reference curve, and lock the second midpoint of the line segment corresponding to the intersection and the rightmost point of the reference curve, and obtain the curve segment formed by the first midpoint and the second midpoint;

[0195] If there are two intersection points, determine the third distance between the two intersection points;

[0196] If the third distance is greater than or equal to the preset distance, extracting the curve segment under the two intersection points from the reference curve;

[0197] If the third distance is less than the preset distance, obtain a third midpoint of the line segment corresponding to the leftmost point and the intersection point of the two intersection points close to the leftmost point of the reference curve, and obtain a fourth midpoint of the line segment corresponding to the rightmost point and the intersection point of the two intersection points close to the rightmost point of the reference curve, and obtain a curve segment formed by the third midpoint and the fourth midpoint;

[0198] If there are multiple intersection points, determine the first absolute value of the difference between each intersection point and the nearest peak point and the second absolute value of the difference between each intersection point and the nearest valley point respectively;

[0199] in accordance with Lock the last two intersection points, where Indicates the minimum value symbol; represents the absolute value of the j1th first difference; represents the j2th second difference absolute value; m1 represents the number of first difference absolute values; m2 represents the number of second difference absolute values;

[0200] Expanding the range of the last two intersection points according to the fourth distance between each of the last two intersection points and the nearest intersection point;

[0201] ;in, Indicates the extended distance based on the left intersection of the last two intersections; Indicates the extended distance based on the right intersection of the last two intersections; represents the fourth distance corresponding to the left intersection point of the last two intersection points; represents the fourth distance corresponding to the rightmost intersection point of the last two intersection points; It represents the distance between the leftmost intersection point of the last two intersection points and the leftmost point in the control curve; It represents the distance between the right intersection point among the last two intersection points and the rightmost point in the control curve; C0 represents the absolute value of the difference between the last two intersection points determined based on the control curve; C1 represents the absolute value of the difference between the left intersection point among the last two intersection points determined based on the control curve and the leftmost point in the control curve; C2 represents the absolute value of the difference between the right intersection point among the last two intersection points determined based on the control curve and the rightmost point in the control curve.

[0202] In this embodiment, the leftmost point is the first curve point in the control curve, and the rightmost point is the last curve point in the control curve.

[0203] In this embodiment, the first midpoint, the second midpoint, the third midpoint and the fourth midpoint are points at the middle positions of the corresponding line segments.

[0204] In this embodiment, the preset distance is pre-set and its value is half the length of two adjacent curve points in the control curve, and the length of two adjacent curve points is set to 1 cm, which is convenient for curve drawing and subsequent distance adjustment calculation.

[0205] It should be noted that if the expansion boundary point corresponding to the range expansion is on the curve segment corresponding to two adjacent points, the outer points of the two adjacent points are locked as the range boundary point.

[0206] If the expanded boundary point corresponding to the range expansion happens to be on the curve point of the curve based on which the reference curve is drawn, then the expanded boundary point is used as the range boundary point.

[0207] For example, Figure 3 As shown, segment Kc is the expansion distance based on curve point 3, and point h1 is located between curve point 2 and curve point 3. At this time, point h1 is expanded for the left side, and curve point 2 is regarded as the range boundary point on the left side.

[0208] In this embodiment, the peak point refers to a point corresponding to a peak value, and the valley point refers to a point corresponding to a valley value.

[0209] The beneficial effect of the above technical solution is: by classifying and discussing the number of intersections, the range conditions under different numbers of intersections can be effectively determined, which not only ensures the rationality of the range but also ensures the reliability of resource retrieval.

[0210] The present invention provides a system for capturing Internet data. Figure 2 As shown, including:

[0211] A parameter determination module, which is used to capture the user's search information on the Internet platform according to the capture tool set by the Internet platform, analyze the search information based on the preset semantic model set by the Internet platform, and obtain a number of search parameters in combination with the parameter database;

[0212] A combination analysis module, used to combine the search parameters according to the preset search rules of the Internet platform, set a combination weight for each combination, and estimate the resource response of each combination;

[0213] The data capture module is used to generate resource capture tasks based on the combined weights and resource response conditions, and then perform reliable data capture based on the Internet platform.

[0214] The beneficial effects of the above technical solution are: by analyzing the retrieval information through a preset semantic model and combining the parameter database to obtain the retrieval parameters, the interference in the crawling process is effectively avoided, and the rationality of the crawling task generation is ensured by analyzing the weights of the parameters and the resource responses, thereby improving the reliability of data crawling.

[0215] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A method for capturing Internet data, characterized in that: include: Step 1: Capture the user's search information on the Internet platform using a capture tool set by the Internet platform, analyze the search information based on a preset semantic model set by the Internet platform, and obtain several search parameters in combination with a parameter database; Step 2: Combining the search parameters according to the preset search rules of the Internet platform, setting a combination weight for each combination, and estimating the resource response of each combination; Step 3: Generate a resource crawling task based on the combined weight and resource response, and then perform reliable data crawling on the Internet platform; Among them, several search parameters are obtained by combining the parameter database, including: Reliably parsing the search information based on a preset semantic model set by the Internet platform to obtain information reliability, and at the same time, performing vocabulary parsing on the search information based on the information parsing model to obtain reliable vocabulary and unreliable vocabulary; Locking the input window of the search information based on the Internet platform; Retrieving historical input information based on the input window from a historical database, and determining a reliable influence coefficient of the window business environment of the input window on each historical input information; All reliable influence coefficients are processed by mean and variance to obtain correction factors; Matching the unreliable vocabulary with the vocabulary-coefficient comparison table to obtain the initial reliability coefficient of the corresponding unreliable vocabulary, and combining the correction factor, the semantic similarity factor between the corresponding unreliable vocabulary and each reliable vocabulary, and the information reliability to determine the suspicious coefficient of the corresponding unreliable vocabulary; When the suspicious coefficient is greater than or equal to the preset reliability coefficient, the corresponding unreliable word is changed to a reliable word, and then all the reliable words are compared with the parameter database in turn to obtain a number of search parameters; Wherein, determining the reliable influence coefficient of the window business environment of the input window on each historical input information includes: Obtain historical input information of the input window in the current unit time period T1 and m unit time periods before the current unit time period, and respectively calculate the historical reliability slope of each historical input information, wherein the historical reliability slope is obtained by analyzing before the historical input information is input into the input window and after the historical input information is input into the input window; Construct the first curve based on all historical reliable slopes based on the sequence of unit periods; Construct the basis function of each curve point and each remaining point in the first curve , determine the influence factor of each basic function on the corresponding curve point from the function-factor comparison table, where, Indicates the information correlation between the historical input information of the i-th curve point and the historical input information of the j-th remaining point; represents the slope difference between the historically reliable slope of the i-th curve point and the historically reliable slope of the j-th remaining point; The reliable influence coefficient of the corresponding historical input information is determined based on all influencing factors of the same curve point, the reliable ratio of the historical input information of the corresponding curve point before it is input into the input window, and the reliable ratio based on the reliable evaluation result after it is input into the input window.

2. The method for capturing Internet data according to claim 1, characterized in that: Set portfolio weights for each portfolio, including: Obtain the first historical search frequency of each search parameter in the corresponding combination, and at the same time, obtain the second historical search frequency and the third historical search frequency of the corresponding combination, wherein the second historical search frequency is the number of searches strictly in accordance with the order of the search parameters in the corresponding combination, and the third historical frequency is the number of searches not strictly in accordance with the order of the search parameters in the corresponding combination; Setting a preliminary weight to a corresponding combination depending on the first historical search frequency, the second historical search frequency, and the third search frequency; Acquire from the search database a first search history with the highest matching degree for all search parameters involved in the search information and a second search history with the second highest matching degree, and obtain a first number of matches between the first search history and the parameters in the corresponding combination and a second number of matches between the second search history and the parameters in the corresponding combination; According to the first quantity and the second quantity, the corresponding preliminary weights are adjusted to obtain the combined weight.

3. The method for capturing Internet data according to claim 2, characterized in that: Estimate resource response for each combination, including: Retrieving a resource instruction set based on each combination from the Internet platform, and obtaining the calling function of each resource instruction in the resource instruction set in the corresponding resource calling process; The resource calling processes involved in the resource instruction set are sorted in order according to the calling function, and the response events of the corresponding combinations in the corresponding resource calling process are determined in turn according to the sorting results, and then the response order of all response events under the same resource calling process is combined to obtain the resource response situation.

4. The method for capturing Internet data according to claim 3, characterized in that: Generate a resource crawling task based on the combined weight and resource response, including: Determine a resource response range based on the calling effect and the combination weight under each combination, and extract the involved events in the required situation from the resource response situation according to the resource response range; Extracting repeated events from the involved events under all combinations, setting a first value for each repeated event, and setting a second value for non-repeated events from the involved events under all combinations; The first value and the second value are respectively assigned to corresponding events to generate event vectors, and are input into an event analysis model to obtain a resource capture task.

5. The method for capturing Internet data according to claim 4, characterized in that: The resource response range is determined based on the call effect and combination weight under each combination, including: Determine whether the second historical search frequency is 0, and if so, randomly select a historical search result from the search history under the third historical frequency; Otherwise, randomly select a historical search result from the search history under the second historical search frequency; Obtain the process elements of each resource retrieval process under each combination and perform similarity analysis with the historical resource elements of the historical search results to obtain a similar array under the corresponding combination; Locking the to-be-used values ​​in the similarity array that are greater than the preset similarity value, and determining a preliminary response range for the resource retrieval process of the to-be-used values ​​based on the position of the ordering results; Perform standardized numerical transformation on each call-up effect involved in the corresponding combination, and compare it with the combination weight under the corresponding combination to obtain a control array based on the preliminary response range, so as to construct a fitting line and a control curve based on the control array; Determine whether the fitting line and the control curve have an intersection; If there is an intersection, the response range under the curve segment obtained based on the intersection is used as the resource response range; If there is no intersection, the preliminary response range is used as the resource response range.

6. The method for capturing Internet data according to claim 5, characterized in that: The response range under the curve segment obtained based on the intersection point is used as the resource response range, including: If there is one intersection, lock the first midpoint of the line segment corresponding to the intersection and the leftmost point of the reference curve, and lock the second midpoint of the line segment corresponding to the intersection and the rightmost point of the reference curve, and obtain the curve segment formed by the first midpoint and the second midpoint; If there are two intersection points, determine the third distance between the two intersection points; If the third distance is greater than or equal to the preset distance, extracting the curve segment under the two intersection points from the reference curve; If the third distance is less than the preset distance, obtain a third midpoint of the line segment corresponding to the leftmost point and the intersection point of the two intersection points close to the leftmost point of the reference curve, and obtain a fourth midpoint of the line segment corresponding to the rightmost point and the intersection point of the two intersection points close to the rightmost point of the reference curve, and obtain a curve segment formed by the third midpoint and the fourth midpoint; If there are multiple intersection points, determine the first absolute value of the difference between each intersection point and the nearest peak point and the second absolute value of the difference between each intersection point and the nearest valley point respectively; in accordance with as well as Lock the last two intersection points, where min represents the minimum value symbol; represents the absolute value of the j1th first difference; represents the j2th second difference absolute value; m1 represents the number of first difference absolute values; m2 represents the number of second difference absolute values; Expanding the range of the last two intersection points according to the fourth distance between each of the last two intersection points and the nearest intersection point; ； in, Indicates the extended distance based on the left intersection of the last two intersections; Indicates the extended distance based on the right intersection of the last two intersections; represents the fourth distance corresponding to the left intersection point of the last two intersection points; represents the fourth distance corresponding to the rightmost intersection point of the last two intersection points; It represents the distance between the leftmost intersection point of the last two intersection points and the leftmost point in the control curve; It represents the distance between the right intersection point among the last two intersection points and the rightmost point in the control curve; C0 represents the absolute value of the difference between the last two intersection points determined based on the control curve; C1 represents the absolute value of the difference between the left intersection point among the last two intersection points determined based on the control curve and the leftmost point in the control curve; C2 represents the absolute value of the difference between the right intersection point among the last two intersection points determined based on the control curve and the rightmost point in the control curve.

7. A system for capturing Internet data, characterized in that: include: A parameter determination module, which is used to capture the user's search information on the Internet platform according to the capture tool set by the Internet platform, analyze the search information based on the preset semantic model set by the Internet platform, and obtain a number of search parameters in combination with the parameter database; A combination analysis module, used to combine the search parameters according to the preset search rules of the Internet platform, set a combination weight for each combination, and estimate the resource response of each combination; A data capture module, used to generate a resource capture task based on the combined weight and resource response status, and then perform reliable data capture based on the Internet platform; The parameter determination module is used to: Reliably parsing the search information based on a preset semantic model set by the Internet platform to obtain information reliability, and at the same time, performing vocabulary parsing on the search information based on the information parsing model to obtain reliable vocabulary and unreliable vocabulary; Locking the input window of the search information based on the Internet platform; Retrieving historical input information based on the input window from a historical database, and determining a reliable influence coefficient of the window business environment of the input window on each historical input information; All reliable influence coefficients are processed by mean and variance to obtain correction factors; Matching the unreliable vocabulary with the vocabulary-coefficient comparison table to obtain the initial reliability coefficient of the corresponding unreliable vocabulary, and combining the correction factor, the semantic similarity factor between the corresponding unreliable vocabulary and each reliable vocabulary, and the information reliability to determine the suspicious coefficient of the corresponding unreliable vocabulary; When the suspicious coefficient is greater than or equal to the preset reliability coefficient, the corresponding unreliable word is changed to a reliable word, and then all the reliable words are compared with the parameter database in turn to obtain a number of search parameters; Wherein, determining the reliable influence coefficient of the window business environment of the input window on each historical input information includes: Obtain historical input information of the input window in the current unit time period T1 and m unit time periods before the current unit time period, and respectively calculate the historical reliability slope of each historical input information, wherein the historical reliability slope is obtained by analyzing before the historical input information is input into the input window and after the historical input information is input into the input window; Construct the first curve based on all historical reliable slopes based on the sequence of unit periods; Construct the basis function of each curve point and each remaining point in the first curve , determine the influence factor of each basic function on the corresponding curve point from the function-factor comparison table, where, Indicates the information correlation between the historical input information of the i-th curve point and the historical input information of the j-th remaining point; represents the slope difference between the historically reliable slope of the i-th curve point and the historically reliable slope of the j-th remaining point; The reliable influence coefficient of the corresponding historical input information is determined based on all influencing factors of the same curve point, the reliable ratio of the historical input information of the corresponding curve point before it is input into the input window, and the reliable ratio based on the reliable evaluation result after it is input into the input window.

Images