Method and device for extracting website information, electronic equipment and storage medium

Abstract

The embodiment of the invention discloses a method and a device for extracting website information, electronic equipment and a storage medium. The method comprises the following steps: acquiring long text contents of a target website and a corresponding sub-level page; splitting the long text content by using the large model to obtain a plurality of knowledge fragments, and labeling the long text content and the knowledge fragments to obtain a first label; performing vector conversion on the knowledge fragment and the first label to obtain a first vector value and a first label vector value; according to the first vector value, performing clustering processing on the knowledge fragments to obtain a knowledge fragment cluster; according to the first label vector value, performing priority ranking on knowledge fragments in the knowledge fragment cluster, and according to the knowledge fragment cluster after priority ranking, constructing an initial mind map of the target website; obtaining a target keyword selected for the target website, and determining a corresponding keyword vector; and updating the initial mind map based on the keyword vector and the first tag vector value to generate a new mind map related to the target keyword.

Description

Technical Field

[0001] The present application relates to the field of data collection technology, and in particular to a method, device, electronic device and storage medium for extracting website information. Background Art

[0002] With the rapid development of the Internet industry and the continuous advancement of big data and artificial intelligence technologies, the amount of content on the Internet has increased dramatically, making it crucial for users to efficiently extract valuable information from massive data.

[0003] In the related art, information is generally searched through search engines. The large model can directly obtain the answer to the question by training the data set of information searched by the search engine. However, when searching for information through the search engine, the website depth is not comprehensive, it is impossible to specify the website and sub-pages, and the search results cannot be intuitively displayed to the user, resulting in low accuracy in extracting website information. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a method, device, electronic device and storage medium for extracting website information, so as to solve the problem of low accuracy in extracting website information.

[0005] To solve the above technical problems, the embodiments of the present application are implemented as follows: In a first aspect, an embodiment of the present application provides a method for extracting website information, comprising: obtaining page content in a web page of a target website and a corresponding sub-page, and extracting long text content in the page content; using a large model to split the long text content to obtain multiple knowledge fragments, and labeling the long text content and the knowledge fragment to obtain a first label; performing vector conversion on the knowledge fragment to obtain a corresponding first vector value, and performing vector conversion on the first label to obtain a first label vector value; clustering the knowledge fragments according to the first vector value to obtain a knowledge fragment cluster; prioritizing the knowledge fragments in the knowledge fragment cluster according to the first label vector value, and constructing an initial mind map of the target website according to the prioritized knowledge fragment cluster; obtaining the target keyword selected for the target website, and determining the keyword vector corresponding to the target keyword; based on the keyword vector and the first label vector value, updating the initial mind map through the large model to generate a new mind map related to the target keyword.

[0006] In a second aspect, an embodiment of the present application provides a device for extracting website information, comprising: an acquisition module, used to acquire page content in web pages of a target website and corresponding sub-pages, and extract long text content in the page content; a splitting module, used to split the long text content using a large model to obtain multiple knowledge fragments, and label the long text content and the knowledge fragments to obtain a first label; a vector conversion module, used to perform vector conversion on the knowledge fragments to obtain corresponding first vector values, and to perform vector conversion on the first label to obtain a first label vector value; a clustering module, used to cluster the knowledge fragments according to the first vector value to obtain a knowledge fragment cluster; according to the first label vector value, prioritize the knowledge fragments in the knowledge fragment cluster, and construct an initial mind map of the target website according to the prioritized knowledge fragment cluster; an extraction module, used to acquire the target keyword selected for the target website, and determine the keyword vector corresponding to the target keyword; based on the keyword vector and the first label vector value, the initial mind map is updated through the large model to generate a new mind map related to the target keyword.

[0007] In a third aspect, an embodiment of the present application provides an electronic device, comprising a processor and a memory electrically connected to the processor, the memory storing a computer program, and the processor being used to call and execute the computer program from the memory to implement the above-mentioned method of extracting website information.

[0008] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium for storing a computer program, wherein the computer program can be executed by a processor to implement the above-mentioned method for extracting website information.

[0009] In a fifth aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the above-mentioned method of extracting website information.

[0010] In a sixth aspect, an embodiment of the present application provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, it implements the above-mentioned method for extracting website information.

[0011] By adopting the technical solution of the embodiment of the present application, the page content in the web page of the target website and the corresponding sub-page is obtained, and the long text content in the page content is extracted, and the long text content is split and processed using a large model to obtain multiple knowledge fragments, and the long text content and the knowledge fragment are labeled to obtain the first label corresponding to the knowledge fragment; the knowledge fragment is vectorized to obtain the corresponding first vector value, and the first label is vectorized to obtain the first label vector value. It can be seen that by extracting and splitting the long text content in the page content of the target website and the sub-page, obtaining multiple knowledge fragments, and labeling and vectorizing the long text content and the knowledge fragments, the core knowledge content of the target website and the sub-page can be obtained, and the core knowledge content can be deeply processed. According to the first vector value, the knowledge fragments are clustered to obtain the knowledge fragment clusters; according to the first label vector value, the knowledge fragments in the knowledge fragment clusters are prioritized, and the initial mind map of the target website is constructed based on the prioritized knowledge fragment clusters; the target keywords selected for the target website are obtained, and the keyword vectors corresponding to the target keywords are determined; based on the keyword vectors and the first label vector values, the initial mind map is updated through the large model to generate a new mind map related to the target keywords. Among them, by clustering the knowledge fragments and prioritizing the knowledge fragments in each knowledge fragment cluster to construct the initial mind map of the target website, the user can clearly obtain the association between the target website and each sub-level page. When searching for the target keyword on the target website, a new mind map related to the target keyword can also be generated through the large model based on the keyword vector and the first label vector value. Without manual screening, the complete information of the website can be efficiently extracted to display the core content related to the target keyword, which can improve the accuracy of the extracted website information. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate one or more embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in one or more embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0013] Figure 1 is a structural diagram of a system for extracting website information according to an embodiment of the present application; Figure 2 is a schematic flow chart of a method for extracting website information according to an embodiment of the present application; Figure 3 is a schematic diagram of a single web page data structure according to an embodiment of the present application; Figure 4 is a schematic diagram of page content data stored in a web page stack according to an embodiment of the present application; Figure 5 is a schematic diagram of a single page content according to an embodiment of the present application; Figure 6 is a structural diagram of a knowledge graph of a target website according to an embodiment of the present application; Figure 7 is a structural diagram of an initial mind map according to an embodiment of the present application; Figure 8 is a schematic diagram of a scenario for extracting website information according to an embodiment of the present application; Fig. 9 is a schematic flow chart of a method for extracting website information according to another embodiment of the present application; Fig.10 is a schematic block diagram of a device for extracting website information according to an embodiment of the present application; Fig.11 It is a schematic diagram of the hardware structure of a device for extracting website information according to an embodiment of the present application. DETAILED DESCRIPTION

[0014] The embodiments of the present application provide a method, device, electronic device and storage medium for extracting website information, so as to solve the problem of low accuracy in extracting website information.

[0015] In order to enable those skilled in the art to better understand the technical solutions in this application, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this application.

[0016] The method for extracting website information provided in the embodiment of the present application can be executed by an electronic device, or by software installed in the electronic device. Specifically, the electronic device can be a terminal device or a server device. Among them, the terminal device can include a smart phone, a laptop computer, a smart wearable device, a vehicle terminal, etc., and the server device can include an independent physical server, a server cluster composed of multiple servers, or a cloud server capable of cloud computing.

[0017] In the following, in conjunction with the accompanying drawings, a method for extracting website information provided by an embodiment of the present application is described in detail through specific embodiments and application scenarios.

[0018] Figure 1FIG. 1 is a schematic diagram of a system for extracting website information according to an embodiment of the present application. Figure 1 As shown, it includes: a data collection module, a content acquisition module, a knowledge fragment splitting module, a construction module and a generation module. Specifically, the data collection module is used to collect sub-level pages of a preset depth related to the target website; the content acquisition module is used to obtain the page content of the target website and the sub-level pages; the knowledge fragment splitting module is used to split the long text content in the page content obtained in the content acquisition module to generate multiple knowledge fragments; the construction module is used to construct a mind map corresponding to the target website based on the knowledge fragments; the generation module is used to generate a new mind map related to the target keyword based on the mind map constructed by the construction module.

[0019] Figure 2 A schematic flow chart of a method for extracting website information provided by an embodiment of the present invention is shown, and the method comprises the following steps: S202, obtaining page contents in web pages of the target website and corresponding sub-pages, and extracting long text contents in the page contents.

[0020] The corresponding sub-level pages include the pages corresponding to each link in the target website. Specifically, there are multiple links in the target website, the target website is used as a first-level page, each link corresponds to a page, the page corresponding to the link is used as a second-level page, each second-level page also has multiple links, the page corresponding to the link in the second-level page is used as a third-level page, and so on, to obtain the sub-level pages of the preset depth, and the sub-level pages of the preset depth are used as the sub-level pages corresponding to the target website.

[0021] The page content is the information contained in the web pages of the target website and the corresponding sub-pages, such as the title title tag in the head tag and the content in the meta tag. The content attribute of the meta tag is taken out and stored in the form of an array of the title and keyword information of the target website or sub-page. The keyword information represents the main purpose of the target website or sub-page. All the text content in the body tag is taken out through the search information language (XML Path Language, xpath). The text content represents the main core content of the web page at this level and is taken as the long text content. All the file lists, picture lists, and video lists of the web page at this level are taken out as web page media information.

[0022] The titles, keyword information, long text content and web media information in the target website and sub-pages are used as the page content of the web pages at this level, and the main core content in the page content is used as the long text content.

[0023] S204, using the large model to split the long text content to obtain multiple knowledge fragments, and labeling the long text content and the knowledge fragments to obtain a first label.

[0024] Large models include neural network models with large-scale parameters and complex structures trained through deep learning technology, which are used to process massive data and complex tasks.

[0025] The large model is called to perform a secondary cleaning of the page content to remove irregular tags. The content of the large model's prompt word-1promot-1 is: "Please play the role of an expert in web page data extraction and pass the context <context>The content removes redundant information and Hypertext Markup Language (Html) tags and styles, and outputs the cleaned standard text information. The content information is as follows: <context> Page Contents< / context> ”.

[0026] Finally, after the page contents corresponding to the target website and the sub-pages are extracted, a schematic diagram of a single web page data structure corresponding to the page contents is formed, such as Figure 3 As shown in , it includes title, meta, content, depth, URL, image, file, video, and arrays corresponding to meta tags, images, files, and videos. At the same time, the constructed page content is stored in the web page stack in sequence, such as Figure 4 As shown, it is a schematic diagram of page content data stored in a web page stack, such as web page 1-data data, web page 2-data, web page 3-data, web page 4-data, and web page N-data.

[0027] The long text content in the page content of the target website and the corresponding sub-pages is obtained from the web page stack in sequence, that is, the long text content in the content section of the page content is obtained.

[0028] Then use the big model to split the long text content to obtain multiple knowledge fragments. Specifically, first use the big model to label the long text content. For example, the obtained label can be used as the parent label, and then the long text content is split into multiple knowledge fragments including the core text content. For example, if the long text content is split into paragraphs, about 350 words can be used as a paragraph, and the end of the paragraph is based on the end of the sentence. The page content is split into n paragraphs until the text content is less than 350 words as 1 paragraph, and then it is no longer split. After the long text content is split, multiple knowledge fragments are obtained. Each knowledge fragment is intelligently labeled by the labeling prompt word in the big model prompt word to obtain the label of each knowledge fragment. The label can be used as a sub-label, and finally the sub-label of each knowledge fragment and the parent label of the long text content corresponding to the knowledge fragment can be used as the first label. It can be seen that the first label corresponding to each knowledge fragment after each long text content is split includes not only the label of the knowledge fragment itself, but also the label of the long text content to which it belongs.

[0029] S206, performing vector conversion processing on the knowledge fragment to obtain a corresponding first vector value, and performing vector conversion processing on the first label to obtain a first label vector value.

[0030] The knowledge fragment and the corresponding first label are vectorized through the vector model, wherein the data corresponding to the knowledge fragment is converted to obtain the first vector value; the first label is vectorized to obtain the first label vector value. The page content can also be vectorized, and the vector value corresponding to the content of each section can be obtained after the page content is vectorized. For example, the title and meta in the page content correspond to their respective content and their respective vector values, the knowledge fragment included in the page content corresponds to the first vector value, and the first label generated based on the long text content and the knowledge fragment corresponds to the first label vector value.

[0031] By converting knowledge snippets and first tags in page content into vector patterns, relevant words, sentences, and other data can be converted into numbers that capture their meanings and relationships.

[0032] S208, clustering the knowledge fragments according to the first vector value to obtain a knowledge fragment cluster; prioritizing the knowledge fragments in the knowledge fragment cluster according to the first label vector value, and constructing an initial mind map of the target website according to the knowledge fragment cluster after priority sorting.

[0033] The feature of each knowledge fragment can be extracted according to the first vector value, and based on the feature of each knowledge fragment, the knowledge fragment is clustered to obtain multiple knowledge fragment clusters including multiple knowledge fragments. The clustering process includes: clustering knowledge fragments with similar features, and the knowledge fragments with similar features are clustered to form multiple knowledge fragment clusters.

[0034] The first label vector value is obtained by vector conversion of the first label. The first label is obtained by labeling the long text content in the page content and the knowledge fragments split from the long text content. Therefore, the first label vector value includes the label vector values ​​corresponding to the long text content and the knowledge fragments, that is, the vector values ​​corresponding to the parent label and the child label in the above example. According to the first label vector value, the priority ranking of the knowledge fragments in each knowledge fragment cluster is determined, that is, according to the first label vector value, the association relationship between the knowledge fragments in each knowledge fragment cluster is determined.

[0035] Based on the knowledge fragment clusters after prioritizing the knowledge fragments, the page contents of the target website and sub-pages can be reordered and sorted to obtain an initial mind map with a hierarchical relationship. The initial mind map can visually display the core contents of each level of pages in the target website.

[0036] S210, obtaining the target keyword selected for the target website, and determining the keyword vector corresponding to the target keyword; based on the keyword vector and the first label vector value, updating the initial mind map through the large model to generate a new mind map related to the target keyword.

[0037] The target keyword includes information that is searched on the target website. The target keyword is converted into a keyword vector. According to the relationship between the keyword vector and the first label vector value, multiple knowledge fragments related to the target keyword are determined. The multiple knowledge fragments are integrated and processed through the large model, and the initial mind map is updated to obtain a new mind map related to the target keyword. As an example, the relationship between the keyword vector and the first label vector value is determined by calculating the similarity value between the two. For example, when the similarity value is high, it means that the target keyword and the corresponding knowledge fragment are closely related. The knowledge fragment and the relevant information corresponding to the knowledge fragment are obtained, and the knowledge fragment and the relevant information corresponding to the knowledge fragment are input into the large model. The large model can automatically generate a new mind map based on the relevant information of the knowledge fragment. The new mind map is generated for the update based on the initial mind map, and can include the core content of the page with a hierarchical structure related to the target keyword. For example, the core content of the page can include the summary corresponding to the target knowledge fragment, the title of the page content of the corresponding long text content, and the summary of the page content.

[0038] By adopting the technical solution of the embodiment of the present application, the page content in the web page of the target website and the corresponding sub-page is obtained, and the long text content in the page content is extracted, and the long text content is split and processed using a large model to obtain multiple knowledge fragments, and the long text content and the knowledge fragment are labeled to obtain the first label corresponding to the knowledge fragment; the knowledge fragment is vectorized to obtain the corresponding first vector value, and the first label is vectorized to obtain the first label vector value. It can be seen that by extracting and splitting the long text content in the page content of the target website and the sub-page, obtaining multiple knowledge fragments, and labeling and vectorizing the long text content and the knowledge fragments, the core knowledge content of the target website and the sub-page can be obtained, and the core knowledge content can be deeply processed. According to the first vector value, the knowledge fragments are clustered to obtain the knowledge fragment clusters; according to the first label vector value, the knowledge fragments in the knowledge fragment clusters are prioritized, and the initial mind map of the target website is constructed based on the prioritized knowledge fragment clusters; the target keywords selected for the target website are obtained, and the keyword vectors corresponding to the target keywords are determined; based on the keyword vectors and the first label vector values, the initial mind map is updated through the large model to generate a new mind map related to the target keywords. Among them, by clustering the knowledge fragments and prioritizing the knowledge fragments in each knowledge fragment cluster to construct the initial mind map of the target website, the user can clearly obtain the association between the target website and each sub-level page. When searching for the target keyword on the target website, a new mind map related to the target keyword can also be generated through the large model based on the keyword vector and the first label vector value. Without manual screening, the complete information of the website can be efficiently extracted to display the core content related to the target keyword, which can improve the accuracy of the extracted website information.

[0039] In one embodiment, the long text content and the knowledge fragment are tagged to obtain the first tag (ie, S204), and the following step A may be performed: Step A: label the long text content through the large model to obtain the parent label, label the knowledge fragment to obtain the child label, and use the parent label and the child label as the first label.

[0040] It should be noted that the large model can also be used to label the page content to obtain the parent label corresponding to the long text content in the page content, label the knowledge fragments after the long text content in the page content is split, obtain the corresponding child labels, and obtain labels for other parts of the page content.

[0041] Preferably, the page content summary of the page content is extracted through the large model, and the knowledge fragment summary of the knowledge fragment is extracted, and the page content summary and the knowledge fragment summary are used as the key content summary.

[0042] By calling the big model, you can also extract and summarize the page content, and also extract and summarize the knowledge fragments after splitting the long text content in the page content into paragraphs. Label the long text content in the page content to get the parent label, and label the corresponding knowledge fragments to get the child label; extract and summarize the page content to get the page content summary, and extract and summarize the corresponding knowledge fragments to get the knowledge fragment summary, use the parent label and child label as the first label, and use the page content summary and knowledge fragment summary as the key content summary.

[0043] From this, we can see that the first label corresponding to the knowledge fragment, in addition to the sub-label of the knowledge fragment itself, also includes the parent label of the corresponding long text content, that is, the first label corresponding to the knowledge fragment includes the parent label and the sub-label, which can improve the accuracy when subsequently processing the priority order of the knowledge fragment level.

[0044] The process of splitting long text content and labeling and extracting summaries can be specifically executed by calling the large model: "Please read the page content carefully, and after analyzing the long text content in the page content, automatically label the long text content you read with a preset number of tags (such as 3-5). The order of labeling is sorted according to the degree of fit. The tags should not be repeated, and the content should be summarized with information, which should not exceed 100 words.

[0045] It is important to note that: 1. The generated content must follow the semantics of the original text and must not provide information that is not included in the original text; 2. The output content format is the exchange format (JavaScript Object Notation, JSON), and the structure is: {tags: [tag array], summary: information summary}; 3. Strictly follow the given format for output and do not output other information."

[0046] In this embodiment, by calling the large model to label the page content and knowledge fragments and extract the key content summary, the core content of the target website and sub-pages can be extracted, and the information in each web page can be accurately analyzed to improve the accuracy of the extracted information.

[0047] In one embodiment, the knowledge fragment is subjected to vector conversion processing to obtain the corresponding first vector value, and the first label is subjected to vector conversion processing to obtain the first label vector value (ie, S206), and the following steps B1-B2 may be performed: Step B1, attach the knowledge fragment to the corresponding long text content, use the vector model to vector embed the knowledge fragment, and convert the knowledge fragment into a first vector value, which includes the vector value corresponding to the knowledge fragment.

[0048] Vector model is a model used to convert text data into vector data. The trained vector model can capture the semantic similarities and relationships between words to enable computers to better understand and process text information. Better provide rich feature representation.

[0049] The knowledge fragments are attached to the long text content in each page content, and the knowledge fragments are vector embedded through the vector model. The words, phrases or sentences therein are converted into continuous vector representations to obtain the first vector value. The first vector value can better enable the computer to fully understand the content of the knowledge fragment.

[0050] Step B2, using a vector model to perform vector embedding on the first label, converting the first label into a first label vector value, where the first label vector value includes a parent label vector value and a child label vector value.

[0051] The vector model can also perform vector embedding on the first label to obtain a first label vector value. Since the first label includes a parent label and a child label, the first label vector value includes a parent label vector value and a child label vector value.

[0052] It should be noted that the vector model not only performs vector conversion processing on the knowledge fragment and the first label, but also can perform vector conversion processing on other information in the page content, such as the title of the page content, information in meta, etc.

[0053] like Figure 5 As shown, it is a schematic diagram of the content of a single page, including: the title corresponding to the page content, and the content and vector value of the title; the content and vector value of meta and meta, the content of meta includes N contents, namely, the vector value corresponding to content N; Uniform Resource Locator (URL); the long text content in content, the label corresponding to the long text content, the label is a parent label, corresponding to multiple child labels, the child labels are labels of knowledge fragments, the child labels are such as label 1, label 2...label N, and their corresponding vector values, as well as multiple knowledge fragments and the knowledge fragment content, parent labels, child labels and corresponding vector values ​​corresponding to the knowledge fragments, the vector value includes: the vector value of the knowledge fragment content and the first label vector value (parent label vector value and child label vector value); files, videos, pictures, depth, etc.

[0054] By analogy, construct knowledge fragments of all web pages with N-level links, and attach all knowledge fragments to the long text content of the corresponding page content. It should be noted that the page content corresponding to the target website can be at the same level as the title of the second-level page, and the page content of the second-level page can be at the same level as the title of the third-level page.

[0055] In this embodiment, the knowledge fragment and the first label are vector embedded through the vector model, the knowledge fragment is converted into a first vector value, the first label is converted into a first label vector value, and the knowledge fragment is attached to the long text content, so as to construct a complete single page content. Through the vector model, the page content of the target website and sub-pages is more deeply converted, which facilitates the computer's understanding of the page content.

[0056] In one embodiment, the knowledge fragments are clustered according to the first vector value to obtain a knowledge fragment cluster; the knowledge fragments in the knowledge fragment cluster are prioritized according to the first label vector value, and an initial mind map of the target website is constructed according to the prioritized knowledge fragment cluster (i.e., S208), and the following steps C1-C6 may be performed: Step C1, calculating the distance from the knowledge fragment to the centroid according to the knowledge fragment and the first vector value, and obtaining a plurality of knowledge fragment clusters through clustering processing.

[0057] The centroid includes the mean of all data (i.e., knowledge fragment vector values) in the knowledge fragment cluster.

[0058] Get all knowledge fragments and the first vector value corresponding to the knowledge fragments, and use the clustering algorithm to perform K-means clustering operation on the knowledge fragments according to the similarity between the texts. First, determine the number of knowledge fragment clusters generated when clustering knowledge fragments, such as taking the number of web pages corresponding to the target website and sub-pages as the number of knowledge fragment clusters. When the number of web pages is 1, take the number of labels of the web page, that is, the number of knowledge fragments on the web page as the number of knowledge fragment clusters.

[0059] Specifically, as an example, randomly select K data points as the initial centroids: , data points are pieces of knowledge, for each data point , calculate the distance between the data point and all the centroids and assign it to the knowledge fragment cluster with the nearest centroid. Where n is the number of features, and The data points are and centroid The value of the kth feature.

[0060] Distance from knowledge fragment to centroid The calculation formula can be expressed as:

[0061] For each knowledge fragment cluster , by calculating the mean of all data points in the cluster as the new centroid ,

[0062] Among them∣ ∣ represents a knowledge fragment cluster The above process is repeated until the change of the centroid is less than a threshold or the predetermined maximum number of iterations is reached, and finally the knowledge fragment cluster to which the centroid and the data point belong is obtained.

[0063] It can be seen that by clustering the knowledge fragments, multiple knowledge fragment clusters can be obtained, and the knowledge fragments in each knowledge fragment cluster are knowledge fragments with high similarity.

[0064] Step C2, determining the cosine similarity value between every two knowledge fragments in the knowledge fragment cluster according to the first label vector value of each knowledge fragment in the knowledge fragment cluster.

[0065] The calculation of cosine similarity value is as follows: two vectors A and B are expressed as: A= , B= , then the cosine similarity value between A and B is:

[0066] Among them, AB represents the dot product of vectors A and B, that is, ; A represents the modulus of vector A, that is ; represents the modulus of vector B, that is ; Calculate the cosine similarity value according to the formula ; n is the number of vectors; i is the i-th vector among n vectors, Sum.

[0067] Use cosine similarity to compare: In multidimensional space, text is represented as a vector, and each dimension represents a feature. Cosine similarity evaluates the similarity between two vectors by calculating the angle between them. The closer the cosine similarity value is to 1, the smaller the angle is, indicating that the two vectors are more similar.

[0068] Take the target website as the starting node to construct a directed graph and build a knowledge graph with direction. Specifically, obtain N sub-level pages according to N-level sorting, and build it with the target website as the starting point. Hang the sub-level pages of the same origin under the upper-level graph node according to the level, and be at the same level as the page content summary, knowledge fragment, etc. of the previous level, as the attribute nodes of the graph node.

[0069] Specifically, the knowledge fragments in each knowledge fragment cluster are connected to obtain the first label vectors corresponding to the two connected knowledge fragments, and the first label vector includes a parent label vector and a child label vector. The cosine similarity values ​​of the parent label vector and the cosine similarity values ​​of the child label vector are calculated respectively, and the sum of the cosine similarity values ​​of the parent label vector and the cosine similarity values ​​of the child label vector is taken as the cosine similarity value of the two knowledge fragments.

[0070] Since the cosine similarity value of the parent tag vector and the cosine similarity value of the child tag vector are calculated separately, when the cosine similarity value of the child tag vector is relatively high, but the cosine similarity value of the parent tag vector is relatively low, it can be distinguished whether the page content of the corresponding web page does not include the text of the actual content, such as an explanatory web page, then, even if the cosine similarity value of the child tag vector is relatively high and the cosine similarity value of the parent tag vector is relatively low, the overall cosine similarity value of the two knowledge fragments will not be very high. Calculating the cosine similarity value through the first tag vector value can improve the quality of obtaining the target knowledge fragment and improve the comparison of the similarity between knowledge fragments.

[0071] Step C3, determining the weighted value of the connection between every two knowledge fragments in the knowledge fragment cluster according to the cosine similarity value between every two knowledge fragments.

[0072] According to step C2, the cosine similarity value of each two connected knowledge fragments in the same knowledge fragment cluster is calculated to obtain the weight of each connection between two knowledge fragments in the knowledge fragment cluster, and the sum of the weighted values ​​corresponding to the two knowledge fragments is used as the weighted value of the connection between the knowledge fragments, and the default weight (such as the default weight is 1) is summed to obtain the weighted value of the connection between each two knowledge fragments. Among them, when the cosine similarity values ​​of two knowledge fragments are consistent, the weighted values ​​on the connection between the two knowledge fragments are compared. If the weighted value is greater than the other two unweighted knowledge fragments, it means that the two unweighted knowledge fragments are at the same level; if they are two knowledge fragments at different levels, the connection between the two knowledge fragments has a weighted value. For example, when knowledge fragment B and knowledge fragment C are not at the same level, and knowledge fragment B is greater than the level of knowledge fragment C, then when the cosine similarity value of knowledge fragment A and knowledge fragment B is the same as the cosine similarity value of knowledge fragment A and knowledge fragment C, the weighted value of the connection between knowledge fragment A and knowledge fragment B needs to be greater than the weighted value of the connection between knowledge fragment A and knowledge fragment C. That is, when constructing a knowledge graph, there are weighted values ​​for the connections between knowledge fragments that are not at the same level. If two knowledge fragments are at the same level, the weighted value of the connection between the two knowledge fragments may not be calculated.

[0073] Step C4, according to the weighted values, obtain the association relationship between the knowledge fragments at different levels in the knowledge fragment cluster, and determine the priority ranking of the knowledge fragments in the knowledge fragment cluster based on the association relationship.

[0074] The association relationship includes whether two knowledge fragments are at the same level. For example, if there is a weighted value between the connection between two knowledge fragments in the same knowledge fragment cluster, then the two knowledge fragments are knowledge fragments at different levels.

[0075] Based on the weighted value, the association relationship is determined, and the connection between each knowledge fragment in the knowledge fragment cluster can be obtained, and then the level of the knowledge fragments in each knowledge fragment cluster is determined, and the knowledge fragments are prioritized. Priority sorting is to weight the knowledge fragments that are not at the same level to obtain a knowledge fragment cluster including the priority sorted knowledge fragments.

[0076] Step C5, constructing the knowledge graph of the target website based on the prioritized knowledge fragment clusters.

[0077] According to the number of knowledge fragment clusters and the priority ranking of knowledge fragments in the knowledge fragment clusters, the association relationship of knowledge fragments in each knowledge fragment cluster is determined. According to the knowledge fragment clusters of knowledge fragments with association relationships, the knowledge graph of the target website is constructed. The knowledge graph includes the page content, key content summary, parent tags and child tags of the target website and sub-pages, which is used to store the content in the target website and sub-pages, i.e. the core content, and establish association relationships.

[0078] Repeat the above steps to cluster the knowledge fragments in turn, and calculate the cosine similarity value and weighted value after connecting the knowledge fragments in the same knowledge fragment cluster, and construct a knowledge graph including the relationship between each web page. As an example, Figure 6 The figure shows a structure diagram of a knowledge graph of a target website. Starting from the target URL page, that is, starting from the target website, the knowledge fragments, tags, meta, titles and content (long text content and page content summary corresponding to the target website) of the target website are obtained. Meta may include multiple meta 1, meta 2, meta 3. The target web page corresponds to a secondary page and a tertiary page, wherein the secondary page includes the knowledge fragments, tags, meta, titles and content (long text content and page content summary corresponding to the secondary page) of the secondary page, wherein the knowledge graph may also include the knowledge fragment summary and page content summary of the knowledge fragment. When the knowledge fragments in the target website and the knowledge fragments in the secondary page belong to the same knowledge fragment cluster, that is, the same cluster, the value of the cosine similarity is calculated, and the knowledge fragments are weighted, and the weighted values ​​of the lines between the knowledge fragments are calculated. The knowledge fragments with weighted values ​​have a higher level. Other knowledge fragments in the same knowledge fragment cluster, including knowledge fragments in secondary pages or tertiary pages, if they do not have weighted values, are at a lower level than the knowledge fragments with weighted values. Among them, Figure 6 The page content of the secondary page includes the knowledge fragments of the secondary page. The same cluster / weighted means that the two knowledge fragments are in the same knowledge fragment cluster and at different levels. The connection between the two knowledge fragments needs to be weighted.

[0079] Step C6: Based on the knowledge graph, a visual initial mind map is formed.

[0080] The initial mind map includes: the title corresponding to the long text content, the summary of key content and the label. The title corresponding to the long text content includes the title of the page content where the long text content is located. The summary of key content includes: page content summary and knowledge fragment summary. Labels include parent labels and child labels. It should be noted that the visualized content in the initial mind map can be set by the user, such as only displaying the title and knowledge fragment summary corresponding to the long text content, etc. The mind map mainly generates text content that is easy for users to understand.

[0081] The knowledge graph in step C5 is a knowledge graph formed in the electronic device. In order to facilitate users to intuitively understand the content of the entire target website, an initial mind map is generated based on the titles and key content summaries corresponding to the long text content in the web page content at all levels, so that the relationship and core content of all levels of the target website can be visualized. The generated content format is the lightweight markup language markdown, and the text is converted into a mind map tool markmap to display the visualized content.

[0082] As an example, Figure 7 The figure shows a structural diagram of an initial mind map, including a page content summary of the target website corresponding to the target URL title, and the page content summary has labels 1 and 2. Labels 1 and 2 serve as sub-labels of the knowledge fragments corresponding to the target website, knowledge fragment 1 summary, knowledge fragment 2 summary, and the label of the target website can be used as the parent label of the corresponding knowledge fragment. It also includes a secondary page title, which can correspond to the page content summary of the secondary page, knowledge fragment 3 summary, knowledge fragment 4 summary, and a tertiary page title.

[0083] In this embodiment, the knowledge fragments are clustered by the first vector value corresponding to the knowledge fragments to obtain multiple knowledge fragment clusters, and the priority order between the knowledge fragments in each knowledge fragment cluster is determined by calculating the cosine similarity value between each two knowledge fragments in the knowledge fragment cluster, and according to the weighted value of the connection between the knowledge fragments in the knowledge fragment cluster, the knowledge fragments of different levels are obtained, and then the knowledge graph of the target website is constructed, and a visual initial mind map is generated based on the knowledge graph. The core content of the target website and each sub-level page can be clearly obtained, and the association relationship between them can be determined to generate an initial mind map that can be intuitively seen by the user.

[0084] In one embodiment, based on the keyword vector and the first tag vector value, the initial mind map is updated through the large model to generate a new mind map related to the target keyword (ie, S210), and the following steps D1-D2 may be performed: Step D1, based on the keyword vector and the first label vector value, calculate the similarity value between the keyword vector and the first label vector value, search the knowledge graph for the target knowledge fragment whose similarity value is higher than the preset threshold, and obtain the relevant information corresponding to the target knowledge fragment, the relevant information includes the page content summary and the knowledge fragment summary, and the title corresponding to the long text content.

[0085] Since the user can intuitively understand the overall content of the target website through the initial mind map, they can use keywords to search for core content of the target website. Specifically, keywords are obtained for the target website, and the keywords are vectorized to obtain keyword vectors. The target website is used as the initial node to start querying all knowledge fragments and titles, and the similarity value between the keyword vector and the first label vector value is calculated. The similarity value between the keyword vector and the vector value corresponding to the title can also be calculated. The similarity value includes a cosine similarity value. According to the similarity value, the knowledge fragment with a similarity value higher than a preset threshold is searched in the constructed knowledge graph as the target knowledge fragment, and the relevant information corresponding to the target knowledge fragment is obtained. The relevant information includes: the title of the page content corresponding to the long text content corresponding to the target knowledge fragment, the page content summary and the knowledge fragment summary, and can also include the association relationship between each target knowledge fragment.

[0086] Searching for knowledge fragments with similarity values ​​higher than a preset threshold in the constructed knowledge graph as target knowledge fragments means sorting the similarity values ​​of multiple searched knowledge fragments in descending order, and taking the knowledge fragments with similarity values ​​higher than the preset threshold as target knowledge fragments with high matching degree, or after obtaining the similarity values, sorting the multiple knowledge fragments in descending order according to the similarity values, and taking the obtained preset number of knowledge fragments as knowledge fragments with high matching degree.

[0087] In the process of searching for knowledge fragments in the knowledge graph, when the similarity value between the keyword vector and the first label vector value is very high, the knowledge fragment corresponding to the first label vector value is obtained, and the knowledge fragment nodes of all levels in the same knowledge fragment cluster are searched according to the size of the connection weights between the knowledge fragment and other knowledge fragments, until the nodes of all knowledge fragments are found, the similarity values ​​of the keyword vector and the first label vector values ​​of all the found knowledge fragments are compared respectively. The higher the weighted value, the more similar the two knowledge fragments are. All similarity values ​​are arranged in descending order to obtain multiple knowledge fragments with a hierarchical relationship. Multiple knowledge fragments can be used as target knowledge fragments, and relevant information of the target knowledge fragments can be obtained. Alternatively, a preset number of knowledge fragments can be taken out as target knowledge fragments. The preset number of knowledge fragments are knowledge fragments with similarity values ​​higher than a preset threshold, and the relevant information corresponding to the target knowledge fragments is input into the large model.

[0088] Step D2, input relevant information into the big model, update the initial mind map through the big model, and generate a new mind map related to the target keywords.

[0089] Input relevant information into the big model as reference knowledge, and output the answer after sorting it out by calling the big model, including dynamically updating the initial mind map and generating a new mind map. Specifically, the prompt word-3 promot-3 used in the big model is as follows: "Please read the input content carefully, analyze it, summarize it automatically, and output it in a logical way; It is important to note that: The generated content must follow the semantics of the original text and must not provide information that is not included in the original text."

[0090] Among them, the new mind map generated, the prompt word-4 promot-4 used in the large model is as follows: "Please read the relevant information carefully, analyze it according to the relevant information, automatically generate a title and summarize it, and output the core content and viewpoints in a graded manner. The output format must be markdown; It is important to note that: 1. The generated content must follow the semantics of the original text and must not provide information that is not included in the original text; 2. Strictly follow the given format for output and do not output other information."

[0091] It should be noted that the title and key content summary in the generated new mind map can be consistent with the initial mind map, or a more logical new title and key content summary can be automatically generated, without specific limitation.

[0092] In this embodiment, by calculating the similarity value between the keyword vector and the first label vector, the target knowledge fragment with high similarity to the keyword vector is searched in the knowledge graph corresponding to the constructed target website, without searching from the web pages of the target website one by one, and the relevant information of the target knowledge fragment is input into the large model to dynamically generate a new mind map related to the target keyword. In extracting comprehensive key information of the target website, all relevant content can be extracted through the association relationship of the knowledge fragments and the similarity value between the keyword vector and the first label vector, and the results are sorted and output through the large model, and then a new mind map is output, thereby improving the efficiency of extracting core knowledge related to the keyword.

[0093] In one embodiment, before obtaining the page content of the target website and the corresponding sub-page (ie, S202), the following steps E1-E3 may be performed: Step E1, adding the address of the target website to the system URL address to generate the target URL and the corresponding website fingerprint.

[0094] URL fingerprint is used to characterize the uniqueness of the website and page content corresponding to the URL. The address of the target website is added after the URL address of the system to generate a target URL including the address of the target website. The target URL is a first-level page, and the page corresponding to the link in the first-level page is a second-level page. The URL fingerprint of the target website is also generated.

[0095] Step E2: Based on the target URL, receiving the collection task corresponding to the target URL.

[0096] The acquisition task includes: obtaining sub-level pages of a preset depth with the same web page source as the target URL, and the same web page source is used to represent that the source addresses of the target website and the sub-level pages are the same.

[0097] The preset depth includes the sub-level pages of the target level that need to be obtained according to the actual situation.

[0098] According to the target URL of the target website, the corresponding collection task is obtained. The collection task can be used as a queue to collect the corresponding sub-pages of the preset depth in sequence. Among them, according to the set N-level depth, all link addresses in the target URL are extracted, and it is determined whether they are the same source as the target URL. Link addresses from different sources are discarded, and the same-source addresses are added to the collection task queue until the original content of all N-level homologous web pages is fully acquired.

[0099] Step E3, executing a collection task according to the URL fingerprint to obtain multiple levels of sub-level pages corresponding to the target website.

[0100] Determine whether there are duplicate pages in the target website and sub-level pages based on the URL fingerprint. When the URL fingerprint is the same, it means that the corresponding web pages are the same. Then skip the links with the same URL fingerprint information and only obtain the web page corresponding to one URL fingerprint. Specifically, the URL fingerprint includes the URL fingerprint of the target website. All sub-level pages are different from the URL fingerprint of the target website. In addition, the URL fingerprints of all sub-level pages need to be obtained, and the URL fingerprints of all sub-level pages are also different. By executing the collection task, multiple levels of sub-level pages corresponding to the target website are obtained.

[0101] In this embodiment, based on obtaining the target URL and URL fingerprint of the target website, when executing the collection task, it is possible to obtain sub-pages of a preset level with the same web page source as the target website, and based on the URL fingerprint, eliminate the sub-pages with the same URL fingerprint, and obtain sub-pages with the same source and unique content, which can effectively improve the accuracy and depth of extracting answer information.

[0102] In one embodiment, Figure 8 As shown, it is a schematic diagram of the scenario of extracting website information. The user adds the address of the target website after the system URL of the system to generate the target URL, and extracts the content related to the user input through the large model and data fusion. Based on the system for extracting website information, the following process steps are executed, starting with step 1, obtaining the request content of the target URL, generating a URL fingerprint, and forming a distributed message queue through network requests, wherein the message queue formed based on the request content includes: obtaining the sub-level page corresponding to the target website, specifically, obtaining the sub-level page of the preset depth in a distributed manner, and the output result includes multiple sub-level pages, and outputting the result, the output result includes the web page source content of the target website and the sub-level page, and the web page source content includes the relevant information of the web page. Step 2, extracting data elements from the web page source content, the data elements include web page content, and the web page content includes title, meta, content, and depth. Among them, the source corresponding to the web page content and the request content of the target URL is the same domain N-level link, step 3, parsing or filtering the web page content, obtaining the long text content, splitting the long text content, and constructing knowledge fragments, the knowledge fragments include knowledge fragment 1, knowledge fragment 2...knowledge fragment N. Among them, the knowledge fragments are all derived from the web page source content. Step 4: Data fusion or sorting of knowledge fragments to obtain a knowledge graph. Step 5: Build an initial mind map through the Large Language Model (LLM) and the information retrieval component Reranker.

[0103] When the user enters a search content, question or keyword, the searcher extracts relevant content from the initial mind map, where the searcher searches for answers related to the user's search content in the target website based on the knowledge graph in the large model. The extracted content is sent to the user in the form of a new mind map.

[0104] Fig. 9 is a schematic flow chart of a method for extracting website information according to another embodiment of the present application, such as Fig. 9 As shown, the method comprises the following steps: S901, the user adds the address of the target website to the system URL address, generates the target URL and specifies the level of the sub-level page.

[0105] S902, receiving the collection task queue corresponding to the target URL, and sequentially acquiring the sub-level pages of the specified level.

[0106] Among them, the web page source of the sub-level page and the target website is the same, and the generated URL fingerprints are different.

[0107] S903, obtaining web pages of the target website and sub-pages, extracting page content from the web pages, and obtaining long text content in the page content.

[0108] S904: Use the large model to clean the page content and split the long text content to obtain multiple knowledge fragments.

[0109] S905, label the long text content through the large model to obtain the parent label, label the knowledge fragment to obtain the child label, and use the parent label and the child label as the first label.

[0110] S906, extracting a page content summary of the page content through the large model, and extracting a knowledge fragment summary of the knowledge fragment.

[0111] S907, performing vector conversion processing on the knowledge fragment to obtain a corresponding first vector value, and performing vector conversion processing on the first label to obtain a first label vector value.

[0112] The first label vector value includes a parent label vector value and a child label vector value.

[0113] S908: Cluster the knowledge fragments according to the first vector value to obtain a plurality of knowledge fragment clusters.

[0114] S909, determining the cosine similarity value between every two knowledge fragments and the weighted value of the connection between every two knowledge fragments in the knowledge fragment cluster according to the first label vector value.

[0115] S910, determining the priority ranking of the knowledge fragment levels in the knowledge fragment cluster according to the cosine similarity value and the weighted value.

[0116] S911, build the knowledge graph of the target website based on the knowledge fragment clusters and priority sorting, and build a visual initial mind map based on the knowledge graph.

[0117] Among them, the initial mind map includes the title of the page content, the summary of the page content, the summary of the knowledge fragment, etc.

[0118] S912, obtaining the target keyword selected for the target website, and determining the keyword vector corresponding to the target keyword, and determining the target knowledge fragment by calculating the cosine similarity value and the weighted value between the keyword vector and the first label vector value.

[0119] S913, input relevant information of the target knowledge fragment into the big model, update the initial mind map through the big model, and generate a new mind map related to the target keyword.

[0120] The specific process from S901 to S913 has been described in detail in the above embodiment and will not be repeated here.

[0121] By adopting the technical solution of the embodiment of the present application, the page content in the web page of the target website and the corresponding sub-page is obtained, and the long text content in the page content is extracted, and the long text content is split and processed using a large model to obtain multiple knowledge fragments, and the long text content and the knowledge fragment are labeled to obtain the first label corresponding to the knowledge fragment; the knowledge fragment is vectorized to obtain the corresponding first vector value, and the first label is vectorized to obtain the first label vector value. It can be seen that by extracting and splitting the long text content in the page content of the target website and the sub-page, obtaining multiple knowledge fragments, and labeling and vectorizing the long text content and the knowledge fragments, the core knowledge content of the target website and the sub-page can be obtained, and the core knowledge content can be deeply processed. According to the first vector value, the knowledge fragments are clustered to obtain the knowledge fragment clusters; according to the first label vector value, the knowledge fragments in the knowledge fragment clusters are prioritized, and the initial mind map of the target website is constructed based on the prioritized knowledge fragment clusters; the target keywords selected for the target website are obtained, and the keyword vectors corresponding to the target keywords are determined; based on the keyword vectors and the first label vector values, the initial mind map is updated through the large model to generate a new mind map related to the target keywords. Among them, by clustering the knowledge fragments and prioritizing the knowledge fragments in each knowledge fragment cluster to construct the initial mind map of the target website, the user can clearly obtain the association between the target website and each sub-level page. When searching for the target keyword on the target website, a new mind map related to the target keyword can also be generated through the large model based on the keyword vector and the first label vector value. Without manual screening, the complete information of the website can be efficiently extracted to display the core content related to the target keyword, which can improve the accuracy of the extracted website information.

[0122] In summary, specific embodiments of the present subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recorded in the claims can be performed in a different order and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing can be advantageous.

[0123] The above is a method for extracting website information provided in an embodiment of the present application. Based on the same idea, an embodiment of the present application also provides a device for extracting website information.

[0124] Fig.10 FIG. 1 is a schematic diagram of a device for extracting website information according to an embodiment of the present invention. Fig.10 As shown, the device for extracting website information includes: an acquisition module 101, a splitting module 102, a vector conversion module 103, a clustering module 104, and an extraction module 105: The acquisition module 101 is used to acquire the page content of the target website and the corresponding sub-pages, and extract the long text content in the page content; The splitting module 102 is used to split the long text content using the large model to obtain multiple knowledge fragments, and label the long text content and the knowledge fragments to obtain a first label.

[0125] The vector conversion module 103 is used to perform vector conversion processing on the knowledge fragment to obtain a corresponding first vector value, and to perform vector conversion processing on the first label to obtain a first label vector value; The clustering module 104 is used to cluster the knowledge fragments according to the first vector value to obtain a knowledge fragment cluster; prioritize the knowledge fragments in the knowledge fragment cluster according to the first label vector value, and construct an initial mind map of the target website according to the prioritized knowledge fragment cluster; The extraction module 105 is used to obtain the target keyword selected for the target website and determine the keyword vector corresponding to the target keyword; based on the keyword vector and the first label vector value, the initial mind map is updated through the large model to generate a new mind map related to the target keyword.

[0126] In one embodiment, the splitting module 102 is specifically used to label the long text content through a large model to obtain a parent label, label the knowledge fragment to obtain a sub-label, and use the parent label and the sub-label as the first label.

[0127] In one embodiment, the vector conversion module 103 is specifically used to hang the knowledge fragment into the corresponding long text content, use the vector model to vector embed the knowledge fragment, and convert the knowledge fragment into a first vector value, the first vector value includes the vector value corresponding to the knowledge fragment; use the vector model to vector embed the first label, and convert the first label into a first label vector value, the first label vector value includes the parent label vector value and the child label vector value.

[0128] In one embodiment, the clustering module 104 specifically includes: A calculation unit, used for calculating the distance between the knowledge fragment and the centroid according to the knowledge fragment and the first vector value, and obtaining a plurality of knowledge fragment clusters through clustering processing; A determination unit, used to determine the cosine similarity value between every two knowledge fragments in the knowledge fragment cluster according to the first label vector value of each knowledge fragment in the knowledge fragment cluster; A weighting unit, used for determining a weighted value of a connection between every two knowledge fragments in a knowledge fragment cluster according to a cosine similarity value between every two knowledge fragments; A sorting unit, used to obtain the association relationship between knowledge fragments at different levels in the knowledge fragment cluster according to the weighted value, and determine the priority sorting of the knowledge fragments in the knowledge fragment cluster based on the association relationship; A construction unit, used to construct a knowledge graph of the target website based on the prioritized knowledge fragment clusters; The visualization unit is used to form an initial visual mind map based on the knowledge graph.

[0129] In one embodiment, the extraction module 105 is specifically used to calculate the similarity value between the keyword vector and the first label vector value based on the keyword vector and the first label vector value, search the knowledge graph for the target knowledge fragment whose similarity value is higher than a preset threshold, and obtain the relevant information corresponding to the target knowledge fragment, the relevant information includes the page content summary and the knowledge fragment summary, and the title corresponding to the long text content; input the relevant information into the big model, update the initial mind map through the big model, and generate a new mind map related to the target keyword.

[0130] In one embodiment, the device also includes a collection module, which is specifically used to add the address of the target website to the system URL address to generate the target URL and the corresponding URL fingerprint; based on the target URL, receiving the collection task corresponding to the target URL, the collection task includes: obtaining sub-pages of a preset depth with the same web page source as the target URL, the same web page source is used to characterize that the source address of the target website and the sub-page is the same; executing the collection task according to the URL fingerprint to obtain sub-pages of multiple levels corresponding to the target website.

[0131] By adopting the technical solution of the embodiment of the present application, the page content in the web page of the target website and the corresponding sub-page is obtained, and the long text content in the page content is extracted, and the long text content is split and processed using a large model to obtain multiple knowledge fragments, and the long text content and the knowledge fragment are labeled to obtain the first label corresponding to the knowledge fragment; the knowledge fragment is vectorized to obtain the corresponding first vector value, and the first label is vectorized to obtain the first label vector value. It can be seen that by extracting and splitting the long text content in the page content of the target website and the sub-page, obtaining multiple knowledge fragments, and labeling and vectorizing the long text content and the knowledge fragments, the core knowledge content of the target website and the sub-page can be obtained, and the core knowledge content can be deeply processed. According to the first vector value, the knowledge fragments are clustered to obtain the knowledge fragment clusters; according to the first label vector value, the knowledge fragments in the knowledge fragment clusters are prioritized, and the initial mind map of the target website is constructed based on the prioritized knowledge fragment clusters; the target keywords selected for the target website are obtained, and the keyword vectors corresponding to the target keywords are determined; based on the keyword vectors and the first label vector values, the initial mind map is updated through the large model to generate a new mind map related to the target keywords. Among them, by clustering the knowledge fragments and prioritizing the knowledge fragments in each knowledge fragment cluster to construct the initial mind map of the target website, the user can clearly obtain the association between the target website and each sub-level page. When searching for the target keyword on the target website, a new mind map related to the target keyword can also be generated through the large model based on the keyword vector and the first label vector value. Without manual screening, the complete information of the website can be efficiently extracted to display the core content related to the target keyword, which can improve the accuracy of the extracted website information.

[0132] Those skilled in the art should understand that Fig.10 The device for extracting website information can be used to implement the method for extracting website information described above, and the detailed description thereof should be similar to the description of the method part above. To avoid tediousness, it will not be repeated here.

[0133] Based on the same technical concept, an embodiment of the present application further provides an electronic device, which is used to execute the above-mentioned method for extracting website information. Fig.11 A schematic diagram of the structure of an electronic device for implementing various embodiments of the present application. The electronic device may have relatively large differences due to different configurations or performances, and may include a processor (processor) 1110, a communication interface (Communications Interface) 1120, a memory (memory) 1130 and a communication bus 1140, wherein the processor 1110, the communication interface 1120, and the memory 1130 communicate with each other through the communication bus 1140. The processor 1110 may call a computer program stored in the memory 1130 and executable on the processor 1110 to perform the following steps: Obtain the page content of the target website and the corresponding sub-pages, and extract the long text content in the page content; The large model is used to split the long text content to obtain multiple knowledge fragments, and the long text content and knowledge fragments are labeled to obtain the first label.

[0134] Performing vector conversion processing on the knowledge fragment to obtain a corresponding first vector value, and performing vector conversion processing on the first label to obtain a first label vector value; According to the first vector value, the knowledge fragments are clustered to obtain a knowledge fragment cluster; according to the first label vector value, the knowledge fragments in the knowledge fragment cluster are prioritized, and according to the prioritized knowledge fragment clusters, an initial mind map of the target website is constructed; Obtain the target keyword selected for the target website and determine the keyword vector corresponding to the target keyword; based on the keyword vector and the first label vector value, update the initial mind map through the large model to generate a new mind map related to the target keyword.

[0135] Adopting the technical solution of the embodiment of the present application, the page content in the web page of the target website and the corresponding sub-page, and extracting the long text content in the page content, the long text content is split and processed using a large model to obtain multiple knowledge fragments, and the long text content and the knowledge fragment are labeled to obtain the first label corresponding to the knowledge fragment; the knowledge fragment is vectorized to obtain the corresponding first vector value, and the first label is vectorized to obtain the first label vector value. It can be seen that by extracting and splitting the long text content in the page content of the target website and the sub-page, obtaining multiple knowledge fragments, and labeling and vectorizing the long text content and the knowledge fragment, the core knowledge content of the target website and the sub-page can be obtained, and the core knowledge content can be deeply processed. According to the first vector value, the knowledge fragments are clustered to obtain the knowledge fragment clusters; according to the first label vector value, the knowledge fragments in the knowledge fragment clusters are prioritized, and the initial mind map of the target website is constructed based on the prioritized knowledge fragment clusters; the target keywords selected for the target website are obtained, and the keyword vectors corresponding to the target keywords are determined; based on the keyword vectors and the first label vector values, the initial mind map is updated through the large model to generate a new mind map related to the target keywords. Among them, by clustering the knowledge fragments and prioritizing the knowledge fragments in each knowledge fragment cluster to construct the initial mind map of the target website, the user can clearly obtain the association between the target website and each sub-level page. When searching for the target keyword on the target website, a new mind map related to the target keyword can also be generated through the large model based on the keyword vector and the first label vector value. Without manual screening, the complete information of the website can be efficiently extracted to display the core content related to the target keyword, which can improve the accuracy of the extracted website information.

[0136] The specific execution steps can refer to the various steps of the above-mentioned method for extracting website information, and can achieve the same technical effect. To avoid repetition, they will not be repeated here.

[0137] It should be noted that the electronic devices in the embodiments of the present application include: servers, terminals, or other devices except terminals.

[0138] The above electronic device structure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. For example, the input unit may include a graphics processing unit (GPU) and a microphone, and the display unit may be configured with a display panel in the form of a liquid crystal display, an organic light-emitting diode, etc. The user input unit includes a touch panel and at least one of other input devices. The touch panel is also called a touch screen. Other input devices may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0139] The memory can be used to store software programs and various data. The memory may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory may include a volatile memory or a non-volatile memory, or the memory may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct memory bus random access memory (DRRAM).

[0140] The processor may include one or more processing units; optionally, the processor integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to the operating system, user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor.

[0141] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the above-mentioned method embodiment for extracting website information is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0142] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0143] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned method embodiment for extracting website information, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0144] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0145] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the processor is used to run the program or instructions to implement the various processes of the above-mentioned product recommendation method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0146] It should be noted that, in this article, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved. For example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0147] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0148] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.< / context>

Claims

1. A method for extracting website information, characterized in that: The method comprises: Obtaining page contents of web pages of the target website and corresponding sub-pages, and extracting long text contents from the page contents; Using the large model to split the long text content to obtain multiple knowledge fragments, and labeling the long text content and the knowledge fragments to obtain a first label; Performing vector conversion processing on the knowledge fragment to obtain a corresponding first vector value, and performing vector conversion processing on the first label to obtain a first label vector value; According to the first vector value, clustering the knowledge fragments to obtain a knowledge fragment cluster; according to the first label vector value, prioritizing the knowledge fragments in the knowledge fragment cluster, and constructing an initial mind map of the target website according to the knowledge fragment cluster after priority sorting; Obtain the target keyword selected for the target website, and determine the keyword vector corresponding to the target keyword; based on the keyword vector and the first label vector value, update the initial mind map through the large model to generate a new mind map related to the target keyword.

2. The method according to claim 1, characterized in that The tagging process of the long text content and the knowledge fragment to obtain a first tag includes: The long text content is labeled by the large model to obtain a parent label, the knowledge fragment is labeled to obtain a child label, and the parent label and the child label are used as the first label.

3. The method according to claim 2, characterized in that The performing vector conversion processing on the knowledge fragment to obtain a corresponding first vector value, and performing vector conversion processing on the first label to obtain a first label vector value, include: Hanging the knowledge fragment into the corresponding long text content, using a vector model to perform vector embedding on the knowledge fragment, and converting the knowledge fragment into the first vector value, wherein the first vector value includes the vector value corresponding to the knowledge fragment; The first label is vector-embedded by using the vector model, and the first label is converted into the first label vector value, where the first label vector value includes a parent label vector value and a child label vector value.

4. The method according to claim 1, characterized in that: The step of clustering the knowledge fragments according to the first vector value to obtain a knowledge fragment cluster; prioritizing the knowledge fragments in the knowledge fragment cluster according to the first label vector value, and constructing an initial mind map of the target website according to the knowledge fragment cluster after priority sorting, including: According to the knowledge fragment and the first vector value, the distance from the knowledge fragment to the centroid is calculated, and a plurality of the knowledge fragment clusters are obtained through clustering processing; Determine the cosine similarity value between every two knowledge fragments in the knowledge fragment cluster according to the first label vector value of each knowledge fragment in the knowledge fragment cluster; Determine a weighted value of a line connecting every two of the knowledge fragments in the knowledge fragment cluster according to the cosine similarity value of every two of the knowledge fragments; According to the weighted values, the association relationship between the knowledge fragments at different levels in the knowledge fragment cluster is obtained, and based on the association relationship, the priority ranking of the knowledge fragments in the knowledge fragment cluster is determined; Constructing a knowledge graph of the target website according to the priority-sorted knowledge fragment clusters; Based on the knowledge graph, a visualized initial mind map is formed.

5. The method according to claim 4, characterized in that The updating of the initial mind map by the large model based on the keyword vector and the first label vector value to generate a new mind map related to the target keyword includes: Calculate the similarity between the keyword vector and the first label vector value according to the keyword vector and the first label vector value, search the knowledge graph for a target knowledge fragment whose similarity value is higher than a preset threshold, and obtain relevant information corresponding to the target knowledge fragment, wherein the relevant information includes a page content summary and a knowledge fragment summary, and a title corresponding to the long text content; The relevant information is input into the large model, and the initial mind map is updated through the large model to generate the new mind map related to the target keyword.

6. The method according to claim 1, characterized in that Before obtaining the page contents of the target website and the corresponding sub-pages, the method includes: Add the address of the target website to the system uniform resource locator URL address to generate the target URL and the corresponding website fingerprint; Based on the target URL, receiving a collection task corresponding to the target URL, the collection task comprising: acquiring the sub-level page of a preset depth having the same webpage source as the target URL, the same webpage source being used to indicate that the source address of the target website and the sub-level page is the same; The collection task is performed according to the URL fingerprint to obtain the sub-level pages of multiple levels corresponding to the target website.

7. A device for extracting website information, characterized in that: The device comprises: An acquisition module, used to acquire page contents in web pages of a target website and corresponding sub-pages, and extract long text contents in the page contents; A splitting module is used to split the long text content using a large model to obtain multiple knowledge fragments, and to label the long text content and the knowledge fragments to obtain a first label; A vector conversion module, configured to perform vector conversion processing on the knowledge fragment to obtain a corresponding first vector value, and to perform vector conversion processing on the first label to obtain a first label vector value; A clustering module is used to cluster the knowledge fragments according to the first vector value to obtain a knowledge fragment cluster; prioritize the knowledge fragments in the knowledge fragment cluster according to the first label vector value, and construct an initial mind map of the target website according to the knowledge fragment cluster after priority sorting; An extraction module is used to obtain the target keyword selected for the target website and determine the keyword vector corresponding to the target keyword; based on the keyword vector and the first label vector value, the initial mind map is updated through the large model to generate a new mind map related to the target keyword.

8. An electronic device, characterized in that: The method comprises a processor and a memory electrically connected to the processor, wherein the memory stores a computer program, and the processor is used to call and execute the computer program from the memory to implement a method for extracting website information as claimed in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The storage medium is used to store a computer program, and the computer program can be executed by a processor to implement a method for extracting website information as claimed in any one of claims 1 to 6.

10. A computer program product, characterized in that The invention comprises a computer program, which, when executed by a processor, implements a method for extracting website information as claimed in any one of claims 1 to 6.

Images