A website search method and apparatus

Abstract

This application provides a website retrieval method and apparatus that can improve the accuracy of search engine retrieval results and increase the efficiency of obtaining target websites. The method includes: determining at least one key subtext in the initial search text based on its part-of-speech tagging; performing semantic-based transformation processing on each key subtext according to its semantics to generate representative text corresponding to each key subtext and related to the identification information of the website to be retrieved; determining a first search text based on the position of each key subtext in the initial search text and the representative text corresponding to each key subtext; and retrieving the first search text using a search engine to obtain a first target website.

Description

Technical Field

[0001] This application relates to the field of website retrieval technology, and in particular to a website retrieval method and apparatus. Background Technology

[0002] With the development of computer networks, the number of websites has reached billions. Through the internet, users can easily obtain data from various types of websites through website retrieval, with search engines being the key technology for achieving this.

[0003] Since different websites offer different content, users usually enter keywords into a search engine, which then determines the target website the user wants to visit based on those keywords.

[0004] Because there are so many websites that offer similar content, if the keywords that users enter into the search engine are not precise enough, the search engine will have difficulty accurately identifying the target website that the user really wants to visit from billions of websites. Summary of the Invention

[0005] This application provides a website retrieval method and apparatus that can improve the accuracy of search engine retrieval results and increase the efficiency of obtaining target websites.

[0006] Firstly, a website retrieval method is provided, including:

[0007] Based on the part-of-speech tag of the initial search text, at least one key subtext in the initial search text is identified;

[0008] Based on the semantics of each key subtext in at least one key subtext and the identification information of the website to be retrieved, each key subtext is subjected to semantic-based transformation processing to generate a representative text corresponding to each key subtext that is related to the identification information of the website to be retrieved.

[0009] Based on the position of each key subtext in the initial search text, the first search text is determined according to the representative text corresponding to each key subtext.

[0010] The search engine retrieves the first text to be searched and obtains the first target website.

[0011] In one feasible design, a search engine is used to retrieve the first target text and obtain the first target website, including:

[0012] The search engine retrieves the first search text and obtains a list of the first websites.

[0013] Using a neural network model, the first website list is sorted according to the similarity between each website name in the first website list and the first text to be searched, so as to obtain an updated first website list.

[0014] Based on the updated list of first websites, obtain the first target website.

[0015] In a feasible design, the neural network model includes:

[0016] The input layer is used to represent the hypothesis statement and at least two inference statements into vectors to generate a sentence vector matrix for each statement. The at least two inference statements include a positive inference statement and at least one negative inference statement. Each inference statement and the hypothesis statement form a sentence pair. The hypothesis statement and the positive inference statement are semantically related, while the hypothesis statement and the negative inference statement are semantically unrelated.

[0017] The interaction layer is used to generate the first weighted sentence vector matrix corresponding to each sentence in each sentence pair after weighting the similarity between the two sentences in each sentence pair, based on the similarity matrix corresponding to the vector matrix of each sentence in each sentence pair.

[0018] The feature selection layer is used to process the multiple first-weighted sentence vector matrices corresponding to the hypothetical statement to generate the second-weighted sentence vector matrix corresponding to the hypothetical statement.

[0019] The matching layer is used to determine the corresponding similarity value of each sentence pair based on the first weighted sentence vector matrix and the second weighted sentence vector matrix corresponding to each inference statement, and to calculate the loss value of the loss function based on each similarity value;

[0020] The output layer is used to output the similarity value corresponding to each inference statement, as well as a sorted list of at least two inference statements.

[0021] In one feasible design, before determining at least one key subtext in the initial search text based on its part-of-speech tag, the method further includes:

[0022] Identification information of various websites on the Internet is obtained through big data statistics;

[0023] Determine at least one representative text based on the identification information of each website;

[0024] A first association relationship is determined based on at least one representative text and at least one text group, wherein the first association relationship is used to associate at least one text group and at least one representative text, the text group including one text or multiple semantically related texts, and the representative text is determined based on the semantics of each text in the corresponding text group;

[0025] Perform semantic-based transformation processing on each key subtext, including:

[0026] Each key subtext is transformed based on the first association relationship.

[0027] In one feasible design, a search engine is used to retrieve the first text to be searched, and a first list of websites is obtained, including:

[0028] The search engine retrieves the first search text and obtains a list of the first websites.

[0029] Update the first website list based on the search requirements and the domain name of each website in the first website list.

[0030] In one feasible design, after obtaining the first target website, the method also includes:

[0031] Determine the corresponding first parent text based on the first text to be searched;

[0032] If, based on the first parent text and the second association, it is determined that the first parent text corresponds to at least one child text, a link to the first target website is triggered. The second association is used to associate at least one parent text and at least one child text. The object represented by the parent text is at a higher level than the object represented by the associated child text. At least one parent text includes the first parent text.

[0033] Determine the navigation name corresponding to the search request;

[0034] Based on the navigation name, retrieve the list of second websites corresponding to the navigation name in the first target website.

[0035] In a feasible design, the method also includes:

[0036] Collect data from the first target website;

[0037] Based on the second website list, click events are triggered on each website in the second website list to collect data from each website after entering the website.

[0038] In a feasible design, the method also includes:

[0039] Based on at least one subordinate text corresponding to the first superior text and a second website list, determine the target subordinate text to be searched, where the website corresponding to the target subordinate text to be searched is not included in the second website list;

[0040] Use a search engine to find the second target website corresponding to the subtext of the target search;

[0041] Collect data from the second target website.

[0042] In one feasible design, a search engine is used to obtain the second target website corresponding to the target text to be searched, including:

[0043] The target text to be searched is supplemented according to the search requirements to generate a second text to be searched.

[0044] The second target website is obtained by retrieving the second text to be searched through a search engine.

[0045] Secondly, a website retrieval device is provided, comprising:

[0046] The key subtext determination module is used to determine at least one key subtext in the initial search text based on the initial search text.

[0047] The conversion module is used to perform semantic-based conversion processing on each key subtext based on the semantics of each key subtext in at least one key subtext and the identification information of the website to be retrieved, so as to generate a representative text corresponding to each key subtext that is related to the identification information of the website to be retrieved.

[0048] The search text determination module is used to determine the first search text based on the position of each key subtext in the initial search text and the representative text corresponding to each key subtext.

[0049] The target website acquisition module is used to retrieve the first target website by searching the first text to be searched through a search engine.

[0050] Because different users have varying language organization abilities, the initial search text may include subtexts that are useless or even interfere with the search engine's retrieval. In this embodiment, by determining at least one useful key subtext in the initial search text based on its part of speech, the search engine can delete subtexts that are useless or even interfere with the search engine's retrieval, so that the search engine can perform retrieval based on the valid search text, thereby improving the accuracy of the retrieval.

[0051] Furthermore, since the same object may correspond to multiple names, websites typically use only one name for that object. However, when users search for websites, they usually input the text to be searched into the search engine based on their dialect or speaking habits. If the input text differs significantly from the name of the website the user wants to search for, the search engine cannot retrieve the website based on text matching. Therefore, in the above embodiments of this application, each key subtext is subjected to semantic-based transformation processing according to the semantics of each key subtext and the identification information of the website to be searched, to generate a representative text corresponding to each key subtext that is related to the identification information of the website to be searched. Since the semantics of the representative text are related to the identification information of the website to be searched, that is, the representative text matches the website to be searched, thus improving the matching degree between the first search text determined based on the representative text and the first target website in subsequent processes, thereby improving the accuracy of the search engine's retrieval results and increasing the efficiency of obtaining the first target website.

[0052] Furthermore, if a user needs to access a series of websites belonging to the same category, the website retrieval method provided in this application can accurately obtain the main site of that series of websites, i.e., the first target website. Since the first target website, as the main site, includes that series of websites, the user achieves a comprehensive retrieval of the series of websites associated with the first target website while retrieving the first target website, thus improving the comprehensiveness of the series website retrieval. Attached Figure Description

[0053] To more clearly illustrate the technical solution of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0054] Figure 1 This is a schematic diagram of an exemplary website retrieval method provided in an exemplary embodiment of this application;

[0055] Figure 2 This is a schematic diagram of an example neural network model structure provided in an exemplary embodiment of this application;

[0056] Figure 3 This is a schematic diagram illustrating an example of adding the added information vector and the word vector bitwise, provided by an exemplary embodiment of this application;

[0057] Figure 4 This is a schematic diagram illustrating an example of connecting an added information vector to a word vector, provided by an exemplary embodiment of this application.

[0058] Figure 5 This is a schematic diagram of a horizontal matrix splicing provided in an exemplary embodiment of this application;

[0059] Figure 6 This is a schematic diagram illustrating an example of feature selection using a convolutional neural network provided in an exemplary embodiment of this application;

[0060] Figure 7 This is a schematic diagram illustrating the difference between shared and non-shared weights when using bidirectional LSTM for feature extraction, provided by an exemplary embodiment of this application.

[0061] Figure 8 This is a schematic diagram of an example bidirectional LSTM structure provided in an exemplary embodiment of this application;

[0062] Figure 9 This is an example website illustration provided in an exemplary embodiment of this application;

[0063] Figure 10 This is a schematic diagram of the structure of an exemplary website retrieval device provided in an exemplary embodiment of this application. Detailed Implementation

[0064] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application. It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of this application can be combined with each other.

[0065] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this application means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or wireless coupling. The term “and / or” as used herein includes all or any units and all combinations of one or more associated listed items.

[0066] Figure 1 This is a schematic diagram of an exemplary website retrieval method provided in an exemplary embodiment of this application, which can be applied to electronic devices. The method includes:

[0067] S110, Based on the part-of-speech tag of the initial search text, determine at least one key subtext in the initial search text.

[0068] In one feasible design, the method of determining at least one key subtext in the initial search text based on its part-of-speech tag includes:

[0069] Perform word segmentation on the initial text to be searched to obtain at least one subtext corresponding to the initial text to be searched.

[0070] Based on the part-of-speech tag of each subtext in at least one subtext, select at least one key subtext from at least one subtext.

[0071] Taking Chinese as an example, the parts of speech in a text include nouns, verbs, adjectives, prepositions, adverbs, conjunctions, auxiliary words, interjections, onomatopoeia, and so on.

[0072] For example, based on the part-of-speech tag of each subtext in at least one subtext, key subtexts that meet the part-of-speech tag criteria are filtered out from at least one subtext. These criteria can be set according to actual needs, and this application does not limit them.

[0073] As further exemplified, the method of selecting at least one key subtext from at least one subtext based on the part-of-speech tag of each subtext in at least one subtext includes:

[0074] Identify non-key subtexts from at least one subtext whose part of speech is any of the following: preposition, adverb, conjunction, auxiliary word, interjection, or onomatopoeia;

[0075] Identify at least one key subtext as a subtext other than a key subtext.

[0076] For example, if the initial search text is "A City's fruit purchasing website", the subtext after word segmentation includes "A City", "of", "fruit", "purchase", and "website". After removing "of" from the subtext, the remaining subtext is used as the key subtext.

[0077] It can be seen that key subtexts, which are texts whose parts of speech are not function words, can be used by search engines to retrieve the target website based on their semantics.

[0078] Since different users have varying language organization abilities, the initial text to be searched may include subtexts that are useless or even interfere with the search engine's retrieval. In the above embodiment, at least one key subtext is obtained from at least one subtext based on the part of speech of each subtext. This can remove subtexts that are useless or even interfere with the search engine's retrieval in the initial text to be searched, so that the search engine can perform retrieval based on the effective search text and improve the accuracy of the retrieval.

[0079] S120, based on the semantics of each key subtext in at least one key subtext and the identification information of the website to be retrieved, perform semantic-based transformation processing on each key subtext to generate a representative text corresponding to each key subtext that is related to the identification information of the website to be retrieved.

[0080] For example, the website's identification information is the website's name.

[0081] In one feasible design, before determining at least one key subtext in the initial search text based on its part-of-speech tag, the method further includes:

[0082] Identification information of various websites on the Internet is obtained through big data statistics;

[0083] Determine at least one representative text based on the identification information of each website;

[0084] A first association relationship is determined based on at least one representative text and at least one text group, wherein the first association relationship is used to associate at least one text group and at least one representative text, the text group includes a text or multiple semantically related texts, and the semantics of the representative text is determined based on the semantics of each text in the corresponding text group;

[0085] Perform semantic-based transformation processing on each key subtext, including:

[0086] Each key subtext is transformed based on the first association relationship.

[0087] For example, among multiple semantically related texts, the multiple texts have the same semantics; in other words, the multiple texts represent the same object, such as multiple texts including "longevity fruit" and "peanut".

[0088] Currently, the same object may correspond to multiple names. Therefore, the semantics of these multiple names are actually related. Websites typically use only one name for the object, choosing one name from multiple options to represent it. For example, "strawberry" can be called "ground fruit," and "peanut" can be called "longevity fruit," etc. Websites related to "strawberry" and "peanut" often use the names most commonly used, such as "strawberry buying website" or "peanut buying website." However, when users search for websites, they usually input the text to the search engine based on their dialect or speaking habits. If the input text differs significantly from the name of the website the user wants to find, the search engine cannot retrieve the website based on text matching.

[0089] In the embodiments described above, website names on the Internet are obtained through big data statistics. First, at least one representative text is determined based on each website name, ensuring a perfect match between the representative text and the corresponding website name. Then, an association is established between the representative text and semantically related text groups. For example, if the representative text is "strawberry," the corresponding text groups include "ground fruit," "ground berry," "red berry," etc. Since each representative text in the first association has the same semantic meaning as each text in the corresponding text group, and the representative text perfectly matches the corresponding website name, converting key subtexts into representative texts based on the first association improves the matching degree between the first search text determined based on the representative text and the first target website in subsequent processes, thereby improving the accuracy of the search engine's retrieval results.

[0090] For example, the first association is stored in a database and can be maintained and updated by staff to keep the first association real-time and accurate.

[0091] S130, based on the position of each key subtext in the initial search text, determine the first search text according to the representative text corresponding to each key subtext.

[0092] In one feasible design, the method for determining the first search text based on the position of each key subtext in the initial search text, and according to the representative text corresponding to each key subtext, includes:

[0093] Obtain the representative text corresponding to each key subtext;

[0094] Based on the position of each keyword text in the initial search text, determine the position of the corresponding representative text in the first search text;

[0095] Based on the position of each representative text within the first text to be searched, the representative texts are reorganized into the first text to be searched.

[0096] Since the position of the representative text in the first search text can affect the semantics of the entire first search text, the above example can accurately determine the position of the corresponding representative text in the first search text based on the position of each keyword text in the initial search text, and thus accurately determine the first search text.

[0097] S140: Use a search engine to retrieve the first text to be searched and obtain the first target website.

[0098] In one feasible design, a search engine is used to retrieve the first target text and obtain the first target website, including:

[0099] The search engine retrieves the first search text and obtains a list of the first websites.

[0100] Using a neural network model, the first website list is sorted according to the similarity between each website name in the first website list and the first text to be searched, so as to obtain an updated first website list.

[0101] Based on the updated list of first websites, obtain the first target website.

[0102] Currently, search engine results can be influenced by algorithms and keyword matching, potentially resulting in inaccurate, incomplete, or irrelevant results. Search engines often return a large number of results, including many irrelevant ones, making it difficult for users to find the most useful information. Due to the vast amount of information on the internet, search engines struggle to filter it, generating inaccurate lists of websites, which may expose users to inappropriate or false information. Some websites and companies use search engine marketing techniques to optimize their website and content rankings, which can cause important information to be buried in the search results, resulting in incomplete search results presented to users and inaccurate website rankings.

[0103] In the example above, after obtaining the first list of websites through a search engine, a neural network model can accurately determine the semantic similarity between each website name in the first list and the first text to be searched. This allows for accurate sorting of the first list of websites, presenting users with an accurate, complete, and semantically relevant list, thus improving user satisfaction with the search engine. Furthermore, based on the accurate first list of websites, the first target website can be precisely identified.

[0104] In a feasible design, such as Figure 2 As shown, the neural network model includes:

[0105] The input layer is used to represent the hypothesis statement and at least two inference statements into vectors to generate a sentence vector matrix for each statement. The at least two inference statements include a positive inference statement and at least one negative inference statement. Each inference statement and the hypothesis statement form a sentence pair. The hypothesis statement and the positive inference statement are semantically related, while the hypothesis statement and the negative inference statement are semantically unrelated.

[0106] The interaction layer is used to generate the first weighted sentence vector matrix corresponding to each sentence in each sentence pair after weighting the similarity between the two sentences in each sentence pair, based on the similarity matrix corresponding to the vector matrix of each sentence in each sentence pair.

[0107] The feature selection layer is used to process the multiple first-weighted sentence vector matrices corresponding to the hypothetical statement to generate the second-weighted sentence vector matrix corresponding to the hypothetical statement.

[0108] The matching layer is used to determine the corresponding similarity value of each sentence pair based on the first weighted sentence vector matrix and the second weighted sentence vector matrix corresponding to each inference statement, and to calculate the loss value of the loss function based on each similarity value;

[0109] The output layer is used to output the similarity value corresponding to each inference statement, as well as a sorted list of at least two inference statements.

[0110] The input layer is illustrated below.

[0111] During model training, existing methods typically input sentence pairs containing only two statements, denoted as sentence A and sentence B, resulting in low matching accuracy. Therefore, this embodiment, in addition to inputting sentence A and sentence B, also inputs several sentences semantically unrelated to sentence A and sentence B. Sentence A and sentence B are used as positive examples (hypothetical statements and positive inference statements), while the semantically unrelated sentences are used as negative examples (negative inference statements). Furthermore, the number of negative examples in this embodiment is not limited; negative examples can be randomly generated samples from other matched sentence pairs.

[0112] It should be noted that since there are at least two inference statements, there are at least two sentence pairs consisting of an inference statement and a hypothesis statement, and these at least two sentence pairs share the hypothesis statement. Consider the following statement as an example:

[0113] Hypothetical statement: Xiaoming is happy.

[0114] Positive reasoning statement: Xiaoming is in a good mood;

[0115] Negative reasoning statement 1: Xiaoming is unhappy;

[0116] Negative inference statement 2: ...

[0117] The above statements can form two sentence pairs: one is "Xiaoming is happy, Xiaoming is in a good mood," and the other is "Xiaoming is happy, Xiaoming is unhappy."

[0118] In one feasible design, the hypothetical statement and at least two inference statements are represented by vectors to generate sentence vector matrices for each statement.

[0119] First, each statement in the hypothesis statement and at least two reasoning statements is segmented into words. Then, a pre-trained WordEmbedding model is used to represent each word segment, generating a word vector for each segment. The WordEmbedding model uses models such as word2vec and glove. Finally, based on the position of each word in the corresponding statement, the word vectors of the words corresponding to each statement are combined to form the sentence vector matrix corresponding to that statement.

[0120] In a feasible design, in order to increase the amount of input information, this embodiment adds some information vectors on the basis of word vectors, including part-of-speech vectors, co-occurrence information vectors, and positional encoding vectors.

[0121] Part-of-speech vectors: Each part of speech is represented by a random vector of fixed length.

[0122] Co-occurrence information vector: Co-occurrence information refers to words that appear together in the hypothetical statement and the inference statement, such as the word "Xiaoming" in the hypothetical statement and the positive inference statement mentioned above. In this embodiment, co-occurrence information has three representations: 0, 1, and 2, where 0 represents... <pad>The added sentence dimension refers to the empty value filled in at this position so that it can be included in the neural network model; 1: represents that the word appears in both the sentence and the word; 2: represents that the word does not appear in both the hypothetical statement and the inferred statement. In this embodiment, the co-occurrence information vector is set to a one-dimensional vector.

[0123] Location encoding vector: Location encoding can be calculated using a formula or represented using a learnable, randomly initialized vector.

[0124] Specifically, the position encoding vector calculated according to the formula can be obtained using the following formulas (1) and (2):

[0125]

[0126]

[0127] In formulas (1) and (2), pos represents the position of the word segment in the input sentence, d1 represents the dimension of the word vector, C is the periodicity coefficient, and PE (pos，2i) PE represents the positional encoding of the 2i-th dimension of the word segment at position pos. (pos，2i+1) This represents the positional encoding of the (2i+1)th dimension of the word segment at position pos.

[0128] Specifically, before combining the word vector matrices of the words corresponding to each sentence into the sentence vector matrix based on the position of each word in the corresponding sentence, the method also includes:

[0129] The part-of-speech vector, co-occurrence information vector, and position encoding vector are added to the word vector. In this embodiment, the initial word vector is named as obtained by WordEmbedding, and the vector obtained after adding the above vectors is called the word vector.

[0130] Specifically, in terms of the addition method, you can choose to add the above vectors to the initial word vectors. Figure 3 This is an example of how the added information vector and the word vector are added bit by bit, as provided in an exemplary embodiment of this application. Alternatively, the above vector can be concatenated after the initial word vector to form a longer vector. Figure 4 This is a schematic diagram illustrating an example of connecting an added information vector to a word vector, provided by an exemplary embodiment of this application.

[0131] For example, since the deep matching model encodes each sentence separately, to increase the amount of data input, this embodiment swaps the roles of sentences A and B and inputs them twice, as follows:

[0132] First, select two semantically related positive example statements, such as sentence A and sentence B, to be used as both hypothesis and positive inference statements. Then, select multiple negative example statements, such as sentence C, to be used as negative inference statements and are semantically unrelated to the positive example statements. Finally, select one positive example statement from each of the two positive example statements as the hypothesis statement and the other as the positive inference statement, and combine them with the negative example statements to form sentence pairs. Thus, the input sentence pairs <hypothesis, inference> include:<A,B> ,<A,C> ,<B,A> ,<B,C> …

[0133] The interaction layer is illustrated below.

[0134] Sentence pair<H,P> For example, based on the similarity matrix corresponding to the vector matrix of each sentence pair, the first weighted sentence vector matrix corresponding to each sentence in each sentence pair after weighting by their similarity is generated, including:

[0135] First, word alignment is performed on the hypothesis statement vector matrix H and the inference statement vector matrix P to make the two vector matrices have the same length; then, the similarity matrix between H and P after word alignment is determined; finally, based on the attention mechanism, the similarity matrix is ​​used and combined with H and P to generate the first weighted sentence vector matrix after weighting their similarity, as shown in the following formulas (3) and (4):

[0136]

[0137]

[0138] In formulas (3) and (4), len(H) is the length of the hypothetical statement in the sentence pair, and len(P) is the length of the inference statement in the sentence pair. This is a vector representation of the original reasoning statement. This is a vector representation of the original hypothetical statement. for A vector representation of the first weighted sentence vector matrix corresponding to the reasoning statements after similarity weighting. for A vector representation of the first weighted sentence vector matrix corresponding to the similarity-weighted hypothetical statement, where e is the weight, obtained from the corresponding value of the similarity matrix. Here, len(H) elements... The first weighted sentence vector matrix corresponding to the hypothetical statement is formed. len(P) The first weighted sentence vector matrix corresponding to the inference statements.

[0139] In one feasible design, the first weighted sentence vector matrix is ​​merged to generate a new first weighted sentence vector matrix.

[0140] Specifically, the first weighted sentence vector matrix corresponding to the hypothetical statement is merged, as shown in the following formula (5):

[0141]

[0142] Where H′ is the new first weighted sentence vector matrix corresponding to the hypothetical statement, - indicates the subtraction of the two corresponding vector matrices, and ⊙ indicates the element-wise multiplication of the two corresponding vector matrices. [] indicates horizontal concatenation of the elements, such as... Figure 5 As shown.

[0143] Similarly, the first weighted sentence vector matrix corresponding to the inference statement is merged as shown in the following formula (6):

[0144]

[0145] Where P′ is the new first weighted sentence vector matrix corresponding to the reasoning statement, - indicates the subtraction of the two corresponding vector matrices, and ⊙ indicates the element-wise multiplication of the two corresponding vector matrices. [] indicates horizontal concatenation of the elements.

[0146] The feature selection layer is illustrated below.

[0147] like Figure 2 As shown, since a vector matrix of N sentence pairs is generated in the interaction layer, N attention representations (i.e., N attention vectors) of the hypothetical statement are produced. Therefore, it is necessary to process the N attention representations of the hypothesis statement to generate a single representation of the hypothesis statement (i.e., a vector matrix representing the hypothesis statement). Here, N is the number of at least two inference statements. In one feasible design, such as... Figure 2 As shown, the methods for generating the second weighted sentence vector matrix corresponding to the hypothetical statement by processing multiple first weighted sentence vector matrices are as follows:

[0148] The N attention representations of the hypothetical statement are averaged, as shown in formula (7):

[0149]

[0150] Among them, H a This is the second weighted sentence vector matrix corresponding to the hypothetical statement. This represents the i-th attention representation of the hypothesis statement.

[0151] In the specific implementation process, in addition to the above methods, a weighted summation method can also be used to sum the model using learnable weights, as shown in formula (8):

[0152]

[0153] Among them, w i Learnable weights initialized randomly.

[0154] Alternatively, feature extraction can be performed using convolutional neural networks, recurrent neural networks, etc. Figure 6 This is a schematic diagram illustrating feature selection using a convolutional neural network, provided in an exemplary embodiment of this application. Figure 6 As shown, multiple sentence vector matrices are concatenated horizontally, then a convolutional neural network is used for convolutional representation, followed by pooling for output.

[0155] The matching layer and the output layer are illustrated below.

[0156] For example, based on the first weighted sentence vector matrix and the second weighted sentence vector matrix corresponding to each inference statement, the corresponding similarity value for each sentence pair is determined. The loss value of the loss function is calculated based on each similarity value, and the similarity value corresponding to each inference statement, as well as a list of at least two sorted inference statements, is output, including:

[0157] By calculating the matching degree values ​​of the sentence vector matrices of the hypothetical statements and the inference statements in each of the sentence pairs obtained above, N output values ​​can be obtained, such as... Figure 2 In the model, Score1, Score2, ..., ScoreN are defined, where N is the number of at least two inference statements, including both positive and negative examples. Then, the loss function can be calculated based on the ranking of the N output values, the model parameters can be adjusted, and training can continue. To reduce computational cost, we can focus only on whether the statement with the highest matching degree is the hypothesis statement or the positive inference statement.

[0158] To better evaluate the matching degree values, this embodiment integrates Pointwise and Listwise ideas. Specifically, it uses a joint loss function composed of the Pointwise loss function and the Listwise loss function to calculate the difference between the matching degree value of each statement vector and the standard value, and adjusts the parameters of the deep matching model based on this difference value. The calculation formula of the Pointwise loss function is as follows (9):

[0159] L p =max(0,ms(r) H ;r P+ )+s(r H r P- )) Formula (9)

[0160] Wherein, s(r H ;r P+ s(r) represents the cosine similarity between the statement vectors corresponding to the hypothetical statement and the positive inference statement. H r P- ) represents the cosine similarity between the statement vectors corresponding to the hypothesis statement and the negative reasoning statement, and m is the preset threshold for determining positive and negative reasoning statements.

[0161] The listwise loss function is shown in the following formula (10):

[0162]

[0163] Where n is the number of samples consisting of positive and negative inference statements.

[0164] To prevent overfitting of the model, this embodiment adds L2 regularization to the loss function, and the final joint loss function is as follows (11):

[0165] loss = L p +L l +L2Re.qularization formula (11)

[0166] In a feasible design, such as Figure 2 As shown, a representation layer is set between the input layer and the interaction layer. The representation layer is used to encode the sentence vector matrix of each sentence in order to extract the features of each sentence vector matrix. That is, in the representation layer, each sentence vector matrix is ​​re-encoded according to the context of each sentence in all sentences to obtain a new sentence vector matrix for each sentence.

[0167] Specifically, this step can be performed using various feature extraction structures, such as Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), and Attention mechanisms. This embodiment still uses a bidirectional LSTM structure. Figure 7 This is a schematic diagram illustrating the difference between shared and non-shared weights when using bidirectional LSTM for feature extraction, as provided in an exemplary embodiment of this application. Figure 7 As shown, during feature extraction, the hypothetical statements and inference statements can share weights or not. In the specific implementation process, the choice can be made based on the training speed requirements and the amount of training data.

[0168] Figure 8 This is a schematic diagram of a bidirectional LSTM structure provided in an exemplary embodiment of this application, and its representation formulas are shown in formulas (12), (13) and (14):

[0169] y t =gVA t +V′A′ t Formula (12)

[0170] A t =f(Ux t +WA t-1 ) Formula (13)

[0171] A′ t =f(U′x t +W′A t-1 ) Formula (14)

[0172] In formulas (12) to (14), V, V', U', U, W, W' are weight matrices, f and g are activation functions, x is the input, A and A' are hidden state parameters, y is the output, and t is the time step.

[0173] In a feasible design, such as Figure 2 As shown, an encoding layer is set between the feature selection layer and the matching layer to encode the first weighted sentence vector matrix and the second weighted sentence vector matrix corresponding to each inference statement again, so as to extract the features of the first weighted sentence vector matrix and the second weighted sentence vector matrix corresponding to each inference statement again.

[0174] Specifically, this embodiment still uses a bidirectional LSTM structure for feature extraction and encoding. Figure 8 A schematic diagram illustrating different output methods used when performing feature extraction using the bidirectional LSTM provided in this application embodiment, such as... Figure 8 As shown, in this embodiment, the hidden state result output to the LSTM structure can be used as the new sentence vector matrix, or the output of the bidirectional LSTM at each time step can be used to calculate the maximum value and the mean value bit by bit and concatenate them to form the new sentence vector matrix.

[0175] In one feasible design, the first target websites are obtained based on the new first list of websites, including:

[0176] Based on the new first website list, the website with the most similar website name will be selected as the first target website.

[0177] The ranking-based neural network model provided in this embodiment includes not only hypothetical statements and positive inference statements in the input text, but also multiple negative inference statements that are semantically unrelated to the hypothetical and positive inference statements. This expands the input from two texts to a text sequence containing both positive and negative examples, increasing the number and types of inputs and thus accelerating the model's fitting speed and enhancing its generalization ability. Furthermore, this embodiment incorporates an attention mechanism to generate sentence vector matrices that weight the similarity between the two statements in each sentence pair. By associating the words between the two statements in each sentence pair, the model's performance is improved, enabling it to more accurately determine the similarity between each website name and the first search text. Finally, this embodiment integrates ranking principles into the model. When adjusting the model parameters using the loss function, the goal is to output the sentence pairs with the highest matching probability as the hypothetical and positive inference statements. This results in higher text matching accuracy after parameter adjustment, enabling precise ranking of the first website list based on textual semantic similarity.

[0178] In one feasible design, a search engine is used to retrieve the first text to be searched, and a first list of websites is obtained, including:

[0179] The search engine retrieves the first search text and obtains a list of the first websites.

[0180] Update the first website list based on the search requirements and the domain name of each website in the first website list.

[0181] Specifically, after obtaining the initial website list, the domain name of the first target website is determined based on the search requirements. Then, based on the domain name of the first target website, the domain names of each website in the initial website list are filtered to remove websites with different domain names, generating a new initial website list. This improves the accuracy of the initial website list. Furthermore, reducing the number of website names in the initial website list improves the efficiency of subsequently determining the first target website from the list. For example, if the search requirement is to find education-related websites, the domain names of the first target websites can be determined to include "edu," etc. Filtering the initial website list based on this domain name can significantly reduce the size of the initial website list.

[0182] Because different websites offer different content, current search engines can only retrieve a single target website based on the keywords entered by the user. However, when a user collects data on a specific type of content on a website, the search engine needs to retrieve a series of target websites corresponding to that type of content, a function that current search engines cannot provide.

[0183] Based on this, after accurately acquiring the first target website, if data collection from the first target website is required, in order to improve the comprehensiveness and completeness of data collection, it is necessary to identify related websites whose content belongs to the same category as the first target website, and then collect data from those related websites as well. Typically, these related websites belong to the same category as the first target website. For example, if the first target website is a website for purchasing fruit in City A, then related websites include "Apple Purchase Website in City A," "Banana Purchase Website in City A," and "Pear Purchase Website in City A."

[0184] In one feasible design, after obtaining the first target website, this application also provides a method for obtaining related websites, including:

[0185] Step a: Determine the corresponding first parent text based on the first text to be searched;

[0186] Step b: If, based on the first parent text and the second association relationship, it is determined that the first parent text corresponds to at least one child text, the link of the first target website is triggered. The second association relationship is used to associate at least one parent text and at least one child text. The object represented by the parent text is at a higher level than the object represented by the associated child text. At least one parent text includes the first parent text.

[0187] Step c: Determine the navigation name corresponding to the search request;

[0188] Step d: Based on the navigation name, obtain the list of second websites corresponding to the navigation name in the first target website. The websites included in the second website list are the subordinate websites of the first target website, that is, the websites included in the second website list are related to the first target website.

[0189] For example, the method further includes:

[0190] Step e: Collect data from the first target website;

[0191] Step f: Based on the second website list, trigger the click event for each website in the second website list to collect data from each website after entering it.

[0192] The collected data includes various forms of data such as text, images, and lists from the website.

[0193] The search requirement can be either a user-input requirement or a keyword extracted from the initial text to be searched.

[0194] For example, determining the corresponding first parent text based on the first text to be searched includes:

[0195] Perform word segmentation on the first text to be searched to obtain the corresponding word segments;

[0196] Iterate through the word segments corresponding to the first text to be searched. During the iteration of each word segment:

[0197] If the current word segment is contained in the parent text of the second association, then the word segment is used as the first parent text corresponding to the first text to be searched.

[0198] For example, if there are multiple first parent texts corresponding to the first text to be searched, steps b, c, and d are performed for each first parent text.

[0199] For example, methods for determining the navigation name corresponding to a search request include:

[0200] Based on the third association and the search requirements, the corresponding navigation name is determined. The third association is used to link at least one search requirement with at least one navigation name, and the at least one navigation name is determined based on the website's navigation names using big data statistics.

[0201] For example, this application also provides a method for determining a second association:

[0202] Based on website names and their corresponding lists of subordinate websites in navigation, a second association is determined. Specifically, the parent text in the second association is determined by the website name, and the subordinate text is determined by the website names in the list of subordinate websites. In this second association, one parent text can be associated with one or more subordinate texts.

[0203] It's important to note that the "level" in "the object represented by the parent text is higher in level than the object represented by the associated child text" can be understood from the nature of the object. For example, if the parent text is "City A," the associated child texts might include: "XX District," "XX County," "XX Town," and "XX Village"; or, if the parent text is "Fruit," the associated child texts might include: "Apple," "Pear," and "Banana." In practical applications, parent and child texts can be defined separately according to the settings of the websites being analyzed. In general, parent and child texts belong to the same category of content, and the websites corresponding to the parent and child texts belong to the same category of websites.

[0204] Combination Figure 9 The example above uses "A City Fruit Purchase Website" as the first target website and "A City Fruit Purchase Website" as the first search term. First, the first parent text is determined to be "fruit". Then, based on the second association, it is determined that "A City Fruit" is associated with the following child texts: "A City Apple", "A City Banana", and "A City Pear". The first target website is then accessed. Based on the third association and the search query "purchase fruit", the navigation name "A City Fruit Purchase Link" is determined. Finally, based on the navigation name "A City Fruit Purchase Link", a click event is triggered at the corresponding location (location Q) in the navigation, which then displays the second website list, including "A City Apple Purchase Website", "A City Banana Purchase Website", and "A City Pear Purchase Website".

[0205] In the example above, the second association relationship can determine whether the first parent text is associated with any child text. If it is, it means that the first target website also has child websites. After entering the first target website, the list of second websites at the corresponding location is obtained based on the navigation name. This allows for the simultaneous collection of data from the first target website and data from websites related to the first target website.

[0206] In a feasible design, the method also includes:

[0207] Based on at least one subordinate text corresponding to the first superior text and a second website list, determine the target subordinate text to be searched, where the website corresponding to the target subordinate text to be searched is not included in the second website list;

[0208] Use a search engine to find the second target website corresponding to the subtext of the target search;

[0209] Collect data from the second target website.

[0210] For example Figure 9 As shown, the first parent text corresponds to at least one child text that includes "durian" in addition to "apple", "banana" and "pear". However, the second website list does not include any websites related to durian. Therefore, the target child text to be searched includes "durian".

[0211] This application takes into account the situation where the navigation of the first target website is incomplete regarding links to subordinate websites. By using at least one subordinate text corresponding to the first parent text and a second website list, the target subordinate text to be searched is determined. This allows the search engine to obtain the second target website corresponding to the target subordinate text, enabling further data collection on the second target website and improving the comprehensiveness of data collection.

[0212] In one feasible design, a search engine is used to obtain the second target website corresponding to the target text to be searched, including:

[0213] The target text to be searched is supplemented according to the search requirements to generate a second text to be searched.

[0214] The second target website is obtained by retrieving the second text to be searched through a search engine.

[0215] For example, if the search query is "buy fruit", the query is segmented into words based on parts of speech, generating "buy" and "fruit". Based on the verb "buy" and the search term "website", the target search term "durian" is supplemented to generate the second search term "buy durian website". Then, a search engine is used to search for "buy durian website" to obtain the second target website.

[0216] The method of retrieving the second target website by searching the second text to be searched through a search engine is described in the above embodiments concerning retrieving the first target website by searching the first text to be searched through a search engine, and will not be repeated here.

[0217] It should be noted that the website retrieval method provided in this application can also be applied to the retrieval of a certain type of website. That is, the website that the user wants to access is not one website, but a series of websites belonging to a certain category.

[0218] For example, a user wants to access sports news websites in city A', meaning they want to access a range of sports-related websites, including sports such as football, basketball, and track and field. In this case, the initial search text entered by the user would be "sports news websites in city A'".

[0219] First, identify at least one key subtext that includes "City A", "Sports", "News", and "Website".

[0220] Then, semantic-based transformation processing is performed on the above key subtexts. "A' City" is an alternative name for "A City", and the representative text of "A' City" after transformation is "A City"; the representative text of "Sports" after transformation is "Sports"; the representative text of "News" after transformation is "News"; and the representative text of "Website" after transformation is "Website".

[0221] Secondly, based on the position of each of the aforementioned key subtexts in the initial search text, the first search text determined according to each representative text is "A City Sports News Website".

[0222] Finally, based on the first search text, the first target website was retrieved. The website identifier for the first target website is "City A Sports News Website." Furthermore, as the main site for sports news websites, the first target website also includes a series of websites related to various sports (accessible via the first target website), such as "City A Basketball News Website," "City A Football News Website," and "City A Track and Field News Website." Therefore, when users retrieve the first target website, they also retrieve a series of related websites belonging to the same category.

[0223] As can be seen, the category corresponding to the first target website is one with multiple sub-categories, such as education, sports, and institutions. Among them, the education category includes subjects such as Chinese and mathematics, while the institutions category includes various departments, committees, and other sub-organizations.

[0224] Because different users have varying language organization abilities, the initial text to be searched may include subtexts that are useless or even interfere with the search engine's retrieval. In this embodiment, by determining at least one useful key subtext in the initial text to be searched, subtexts that are useless or even interfere with the search engine's retrieval can be deleted, so that the search engine can perform retrieval based on the valid search text, thereby improving the accuracy of the retrieval.

[0225] Furthermore, since the same object may correspond to multiple names, websites typically use only one name for that object. However, when users search for websites, they usually input the text to be searched into the search engine based on their dialect or speaking habits. If the input text differs significantly from the name of the website the user wants to search for, the search engine cannot retrieve the website based on text matching. Therefore, in the above embodiments of this application, each key subtext is subjected to semantic-based transformation processing according to the semantics of each key subtext and the identification information of the website to be searched, to generate a representative text corresponding to each key subtext that is related to the identification information of the website to be searched. Since the semantics of the representative text are related to the identification information of the website to be searched, that is, the representative text matches the website to be searched, thus improving the matching degree between the first search text determined based on the representative text and the first target website in subsequent processes, thereby improving the accuracy of the search engine's retrieval results and increasing the efficiency of obtaining the first target website.

[0226] Furthermore, if a user needs to access a series of websites belonging to the same category, the website retrieval method provided in this application can accurately obtain the main site of that series of websites, i.e., the first target website. Since the first target website, as the main site, includes that series of websites, the user, while retrieving the first target website, also achieves a comprehensive retrieval of the series of websites associated with the first target website, thus improving the comprehensiveness of the series website retrieval.

[0227] Figure 10 This is a schematic diagram of an exemplary website retrieval device provided in an exemplary embodiment of this application, as shown below. Figure 10 As shown, the device includes:

[0228] The key subtext determination module is used to determine at least one key subtext in the initial search text based on the initial search text.

[0229] The conversion module is used to perform semantic-based conversion processing on each key subtext based on the semantics of each key subtext in at least one key subtext and the identification information of the website to be retrieved, so as to generate a representative text corresponding to each key subtext that is related to the identification information of the website to be retrieved.

[0230] The search text determination module is used to determine the first search text based on the position of each key subtext in the initial search text and the representative text corresponding to each key subtext.

[0231] The target website acquisition module is used to retrieve the first target website by searching the first text to be searched through a search engine.

[0232] In one feasible design, the target website acquisition module is implemented by retrieving the first target website from the first text to be searched through a search engine:

[0233] The search engine retrieves the first search text and obtains a list of the first websites.

[0234] Using a neural network model, the first website list is sorted according to the similarity between each website name in the first website list and the first text to be searched, so as to obtain an updated first website list.

[0235] Based on the updated list of first websites, obtain the first target website.

[0236] In a feasible design, the neural network model includes:

[0237] The input layer is used to represent the hypothesis statement and at least two inference statements into vectors to generate a sentence vector matrix for each statement. The at least two inference statements include a positive inference statement and at least one negative inference statement. Each inference statement and the hypothesis statement form a sentence pair. The hypothesis statement and the positive inference statement are semantically related, while the hypothesis statement and the negative inference statement are semantically unrelated.

[0238] The interaction layer is used to generate the first weighted sentence vector matrix corresponding to each sentence in each sentence pair after weighting the similarity between the two sentences in each sentence pair, based on the similarity matrix corresponding to the vector matrix of each sentence in each sentence pair.

[0239] The feature selection layer is used to process the multiple first-weighted sentence vector matrices corresponding to the hypothetical statement to generate the second-weighted sentence vector matrix corresponding to the hypothetical statement.

[0240] The matching layer is used to determine the corresponding similarity value of each sentence pair based on the first weighted sentence vector matrix and the second weighted sentence vector matrix corresponding to each inference statement, and to calculate the loss value of the loss function based on each similarity value;

[0241] The output layer is used to output the similarity value corresponding to each inference statement, as well as a sorted list of at least two inference statements.

[0242] In one feasible design, the device further includes an association determination module, which is used to obtain the identification information of each website on the Internet through big data statistics; determine at least one representative text based on the identification information of each website; and determine a first association based on at least one representative text and at least one text group, wherein the first association is used to associate at least one text group and at least one representative text, the text group including one text or multiple semantically related texts, and the representative text determined based on the semantics of each text in the corresponding text group;

[0243] The conversion module performs semantic-based conversion processing on each key subtext in the following way:

[0244] Each key subtext is transformed based on the first association relationship.

[0245] In one feasible design, the target website acquisition module is implemented by retrieving the first search text through a search engine to obtain a list of first websites:

[0246] The search engine retrieves the first search text and obtains a list of the first websites.

[0247] Update the first website list based on the search requirements and the domain name of each website in the first website list.

[0248] In one feasible design, the device further includes a second website list acquisition module, used to determine the corresponding first parent text based on the first text to be searched; if, based on the first parent text and a second association relationship, it is determined that the first parent text corresponds to at least one child text, a link to the first target website is triggered, wherein the second association relationship is used to associate at least one parent text and at least one child text, the object represented by the parent text is at a higher level than the object represented by the associated child text, and at least one parent text includes the first parent text; determine the navigation name corresponding to the search request; and obtain the second website list corresponding to the navigation name in the first target website based on the navigation name.

[0249] In one feasible design, the device also includes a data acquisition module:

[0250] The data acquisition module is used to collect data from the first target website;

[0251] The data acquisition module is also used to trigger click events for each website in the second website list, based on the second website list, so as to collect data from the corresponding website after entering each website.

[0252] In one feasible design, the target website acquisition module is further used to: determine the target searchable sub-text based on at least one sub-text corresponding to the first parent text and a second website list, wherein the website corresponding to the target searchable sub-text is not included in the second website list; acquire the second target website corresponding to the target searchable sub-text through a search engine; and the data acquisition module is further used to collect data from the second target website.

[0253] In one feasible design, the target website acquisition module is implemented by retrieving the second target website corresponding to the target text to be searched through a search engine:

[0254] The target text to be searched is supplemented according to the search requirements to generate a second text to be searched.

[0255] The second target website is obtained by retrieving the second text to be searched through a search engine.

[0256] Other implementation methods and effects of this device can be found in the description of the website retrieval method embodiment, and will not be repeated here.

[0257] The basic principles of this application have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this application are merely examples and not limitations, and should not be considered as essential features of each embodiment of this application. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the application to the necessity of employing the aforementioned specific details for implementation.

[0258] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0259] The block diagrams of devices, apparatuses, devices, and systems involved in this application are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as "comprising," "including," "having," etc., are open-ended terms meaning "including but not limited to," and are used interchangeably with them. The terms "or" and "and" as used herein refer to the terms "and / or," and are used interchangeably with them unless the context clearly indicates otherwise. The term "such as" as used herein refers to the phrase "such as but not limited to," and is used interchangeably with it.

[0260] It should also be noted that in the apparatus, equipment, and methods of this application, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions of this application.

[0261] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein can be applied to other aspects without departing from the scope of this application. Therefore, this application is not intended to be limited to the aspects shown herein, but rather to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0262] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this application to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.< / pad>

Claims

1. A website retrieval method, characterized in that, include: Based on the part-of-speech tag of the initial text to be searched, at least one key subtext in the initial text to be searched is determined; Based on the semantics of each of the at least one key subtexts, a semantic-based transformation process is performed on each key subtext to generate a representative text corresponding to each key subtext that is related to the identification information of the website to be searched. Based on the position of each of the key subtexts in the initial search text, the first search text is determined according to the representative text corresponding to each key subtext; The first target website is obtained by searching the first text using a search engine. After obtaining the first target website, the method further includes: Determine the corresponding first parent text based on the first text to be searched; If, based on the first parent text and the second association, it is determined that the first parent text corresponds to at least one child text, a link to the first target website is triggered. The second association is used to associate at least one parent text and at least one child text, where the parent text represents an object at a higher level than the associated child text. The at least one parent text includes the first parent text. The steps for generating the second association include: Based on the website names and the corresponding list of subordinate websites in the navigation, determine the second association relationship; Determine the navigation name corresponding to the search request; Based on the navigation name, obtain the list of second websites corresponding to the navigation name in the first target website; Collect data from the first target website; Based on the second website list, click events are triggered for each website in the second website list to collect data from each website after entering the website in the second website list; Based on the at least one subordinate text corresponding to the first superior text and the second website list, a target subordinate text to be searched is determined, wherein the website corresponding to the target subordinate text to be searched is not included in the second website list; The search engine is used to obtain a second target website corresponding to the target text to be searched, wherein obtaining the second target website corresponding to the target text to be searched includes: The target text to be searched is supplemented according to the search requirements to generate a second text to be searched; The second target website is obtained by retrieving the second text to be searched using the search engine. Collect data from the second target website.

2. The method according to claim 1, characterized in that, The step of retrieving the first target website from the first text to be searched using a search engine includes: The first search text is retrieved using a search engine to obtain a list of first websites; Using a neural network model, the first website list is sorted according to the similarity between each website name in the first website list and the first search text, so as to obtain an updated first website list. Based on the updated list of the first websites, obtain the first target website.

3. The method according to claim 2, characterized in that, The neural network model includes: An input layer is used to represent the hypothesis statement and at least two inference statements into vectors to generate a sentence vector matrix for each statement. The at least two inference statements include a positive inference statement and at least one negative inference statement. Each inference statement forms a sentence pair with the hypothesis statement. The hypothesis statement is semantically related to the positive inference statement, and the hypothesis statement is semantically unrelated to the negative inference statement. The interaction layer is used to generate the first weighted sentence vector matrix corresponding to each of the two sentences in each sentence pair after weighting their similarity to each other, based on the similarity matrix corresponding to each sentence vector matrix of each sentence pair. The feature selection layer is used to process the multiple first weighted sentence vector matrices corresponding to the hypothetical statement to generate the second weighted sentence vector matrix corresponding to the hypothetical statement. The matching layer is used to determine the corresponding similarity value of each sentence pair based on the first weighted sentence vector matrix and the second weighted sentence vector matrix corresponding to each of the inference statements, and to calculate the loss value of the loss function based on each of the similarity values; The output layer is used to output the similarity value corresponding to each of the inference statements, as well as a sorted list of at least two of the inference statements.

4. The method according to any one of claims 1-3, characterized in that, Before determining at least one key subtext in the initial search text based on its part-of-speech tag, the method further includes: Identification information of various websites on the Internet is obtained through big data statistics; At least one representative text is determined based on the identification information of each website; A first association is determined based on at least one representative text and at least one text group, wherein the first association is used to associate at least one text group and at least one representative text, the text group comprising a text or multiple semantically related texts, and the representative text being determined based on the semantics of each text in the corresponding text group; The semantic-based transformation processing of each of the key sub-texts includes: Each of the key subtexts is transformed based on the first association relationship.

5. The method according to claim 2 or 3, characterized in that, The step of retrieving the first text to be searched through a search engine to obtain a first list of websites includes: The first search text is retrieved using a search engine to obtain a list of first websites; Update the first website list based on the search requirements and the domain name of each website in the first website list.

6. A website retrieval device, characterized in that, include: The key subtext determination module is used to determine at least one key subtext in the initial search text based on the initial search text. The conversion module is used to perform semantic-based conversion processing on each of the key subtexts based on the semantics of each of the key subtexts and the identification information of the website to be retrieved, so as to generate a representative text corresponding to each key subtext that is related to the identification information of the website to be retrieved. The search text determination module is used to determine the first search text based on the position of each key subtext in the initial search text and the representative text corresponding to each key subtext. The target website acquisition module is used to retrieve the first target website by searching the first text to be searched through a search engine; The second website list acquisition module is used to, after acquiring the first target website, determine the corresponding first parent text based on the first search text; if, based on the first parent text and the second association relationship, it is determined that the first parent text corresponds to at least one child text, a link to the first target website is triggered, wherein the second association relationship is used to associate at least one parent text and at least one child text, the object represented by the parent text is at a higher level than the object represented by the associated child text, the at least one parent text includes the first parent text, and the steps for generating the second association relationship include: Based on the website names and the corresponding list of subordinate websites in the navigation based on big data statistics, a second association is determined; the navigation name corresponding to the search request is determined; based on the navigation name, a second website list corresponding to the navigation name in the first target website is obtained; The data acquisition module is used to collect data from the first target website; according to the second website list, it triggers click events for each website in the second website list to collect data from the corresponding website after entering each website in the second website list; The target website acquisition module is further configured to: determine a target sub-text to be searched based on the at least one sub-text corresponding to the first superior text and the second website list, wherein the website corresponding to the target sub-text to be searched is not included in the second website list; and acquire the second target website corresponding to the target sub-text to be searched through the search engine, wherein acquiring the second target website corresponding to the target sub-text to be searched through the search engine includes: The target text to be searched is supplemented according to the search requirements to generate a second text to be searched; The second target website is obtained by retrieving the second text to be searched using the search engine. The data acquisition module is also used to acquire data from the second target website.

Images