Distributed open source intelligence extraction method and device, equipment and storage medium

Abstract

The invention provides a distributed open source intelligence extraction method and device, equipment and a storage medium, and the method comprises the steps: firstly obtaining connection configuration, and inputting data source information according to the connection configuration; thirdly, analyzing an HTML (Hypertext Markup Language) structure of the main webpage on the basis of the data source information, constructing a document object model tree on the basis of an analysis result to identify distribution information of contents of the main webpage, calling a pre-trained large language model to perform semantic understanding on the distribution information of the contents of the main webpage, and generating list data for the understood contents; sequentially accessing each detail page link of the list data, extracting HTML (Hypertext Markup Language) content of each detail page, generating a regional feature vector of each detail page, calling a pre-trained large language model to process the regional feature vector of each detail page, and generating structured data of a list and details. The problem that extraction errors occur when information extraction is carried out on the webpage is solved.

Description

Technical Field

[0001] The present invention relates to the field of information extraction, and in particular to a method, device, equipment and storage medium for extracting distributed open source intelligence. Background Art

[0002] In the prior art, the main method of extracting the body of a web page is based on the text and symbol density of HTML. For example, the open source component GNE uses a method of extracting the body of a web page based on the text and symbol density. This method completely relies on the tag structure in the HTML web page to find the body of the web page. However, when the body of the web page is short and the comments or ancillary content is long, the extraction often fails. Or when there are too many HTML tags in the web page, it is easy to cause errors in identifying the position of the text. In addition, non-text content such as copyright information is often mistakenly extracted.

[0003] In view of this, this application is filed. Summary of the invention

[0004] The present invention discloses a method, device, equipment and storage medium for extracting distributed open source intelligence, aiming to solve the problem of extraction errors occurring when extracting information from web pages.

[0005] The first embodiment of the present invention provides a method for extracting distributed open source intelligence, including: Obtain connection configuration, and enter data source information according to the connection configuration; Analyze the HTML structure of the main webpage based on the data source information, and build a document object model tree based on the analysis result to identify the distribution information of the main webpage content, call the pre-trained large language model to perform semantic understanding on the distribution information of the main webpage content, and generate list data for the understood content; Access each detail page link of the list data in turn, extract the HTML content of each detail page and generate a regional feature vector for each detail page, call the pre-trained large language model to process the regional feature vector of each detail page, and generate structured data of the list and details.

[0006] Preferably, the connection configuration includes: interface address, interface request mode, interface parameters and request header.

[0007] Preferably, the analyzing the HTML structure of the main webpage based on the data source information and constructing a document object model tree based on the analysis result to identify the distribution information of the main webpage content is specifically: Parsing the original HTML code of the main webpage into structured data, and constructing a document object model tree according to the structured data; wherein the structured data includes a start tag, an end tag, attributes and text content, and each HTML element becomes a node in the document object model tree; Analyze the document object model tree and generate analysis results, wherein the analysis results include HTML tag density in different areas, the ratio of the number of text characters to the number of tags in different DOM nodes, the distribution of different types of elements in the page, and the depth of nodes in the DOM tree; The regional feature vector of the main webpage is determined according to the analysis structure.

[0008] Preferably, the calling of the pre-trained large language model to semantically understand the distribution information of the main webpage content and generating list data for the understood content is specifically as follows: Convert the distribution information of the main webpage content into a vector representation , based on the position index of each word in the text, using the formula , encode the position information into the word vector and generate a vector representation containing the context position information, where is the word embedding vector, is the word embedding matrix, is the index of a word in the vocabulary, is the bias vector, is the position encoding vector, is the position index, s and c are parameters; The query vector Q, key vector K and value vector V are constructed through the self-attention mechanism word embedding vector, and according to the formula , calculate the contextual relevance between each word vector, perform self-attention calculations in parallel in different subspaces, and use the formula , merge the outputs of multiple attention heads to generate list data, where is the dimension of the key vector, is the single-head attention, h is the number of heads of multi-head attention, are the parameters of the linear layer.

[0009] The second embodiment of the present invention provides a distributed open source intelligence extraction device, including: A data source information input unit, used to obtain a connection configuration and input data source information according to the connection configuration; A list data generating unit, configured to analyze the HTML structure of the main webpage based on the data source information, and to construct a document object model tree based on the analysis result to identify the distribution information of the main webpage content, to call a pre-trained large language model to perform semantic understanding on the distribution information of the main webpage content, and to generate list data for the understood content; The structured data generation unit is used to sequentially access each detail page link of the list data, extract the HTML content of each detail page and generate a regional feature vector for each detail page, call a pre-trained large language model to process the regional feature vector of each detail page, and generate structured data of the list and details.

[0010] Preferably, the connection configuration includes: interface address, interface request mode, interface parameters and request header.

[0011] Preferably, the analyzing the HTML structure of the main webpage based on the data source information and constructing a document object model tree based on the analysis result to identify the distribution information of the main webpage content is specifically: Parsing the original HTML code of the main webpage into structured data, and constructing a document object model tree according to the structured data; wherein the structured data includes a start tag, an end tag, attributes and text content, and each HTML element becomes a node in the document object model tree; Analyze the document object model tree and generate analysis results, wherein the analysis results include HTML tag density in different areas, the ratio of the number of text characters to the number of tags in different DOM nodes, the distribution of different types of elements in the page, and the depth of nodes in the DOM tree; The regional feature vector of the main webpage is determined according to the analysis structure.

[0012] Preferably, the calling of the pre-trained large language model to semantically understand the distribution information of the main webpage content and generating list data for the understood content is specifically as follows: Convert the distribution information of the main webpage content into a vector representation , based on the position index of each word in the text, using the formula , encode the position information into the word vector and generate a vector representation containing the context position information, where is the word embedding vector, is the word embedding matrix, is the index of a word in the vocabulary, is the bias vector, is the position encoding vector, is the position index, s and c are parameters; The query vector Q, key vector K and value vector V are constructed through the self-attention mechanism word embedding vector, and according to the formula , calculate the contextual relevance between each word vector, perform self-attention calculations in parallel in different subspaces, and use the formula , merge the outputs of multiple attention heads to generate list data, where is the dimension of the key vector, is the single-head attention, h is the number of heads of multi-head attention, are the parameters of the linear layer.

[0013] A third embodiment of the present invention provides a distributed open source intelligence extraction device, including a memory and a processor, wherein the memory stores a computer program, and the computer program can be executed by the processor to implement a distributed open source intelligence extraction method as described in any one of the above items.

[0014] The fourth embodiment of the present invention provides a computer-readable storage medium, characterized in that a computer program is stored therein, and the computer program can be executed by a processor of a device where the computer-readable storage medium is located to implement a method for extracting distributed open source intelligence as described in any one of the above items.

[0015] A distributed open source intelligence extraction method, device, equipment and storage medium provided by the present invention first obtains the connection configuration and enters the data source information according to the connection configuration; then, the HTML structure of the main webpage is analyzed based on the data source information, and a document object model tree is constructed based on the analysis result to identify the distribution information of the main webpage content, and a pre-trained large language model is called to semantically understand the distribution information of the main webpage content, and list data is generated for the understood content; then, each detail page link of the list data is accessed in turn, the HTML content of each detail page is extracted and the regional feature vector of each detail page is generated, and the pre-trained large language model is called to process the regional feature vector of each detail page, and structured data of the list and details is generated. The problem of extraction errors when extracting information from webpages is solved. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a flowchart of a method for extracting distributed open source intelligence provided by the first embodiment of the present invention; Figure 2 It is a module diagram of a distributed open source intelligence extraction device provided by the second embodiment of the present invention. DETAILED DESCRIPTION

[0017] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0018] In order to better understand the technical solution of the present invention, the embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0019] It should be clear that the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0020] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms "a", "said" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0021] It should be understood that the term "and / or" used in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0022] The word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.

[0023] The "first\second" mentioned in the embodiments is only to distinguish similar objects, and does not represent a specific order for the objects. It is understandable that the "first\second" can be interchanged with the specific order or sequence where permitted. It should be understood that the objects distinguished by "first\second" can be interchanged where appropriate, so that the embodiments described herein can be implemented in an order other than those illustrated or described herein.

[0024] The specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0025] The present invention discloses a method, device, equipment and storage medium for extracting distributed open source intelligence, aiming to solve the problem of extraction errors occurring when extracting information from web pages.

[0026] See also Figure 1 The first embodiment of the present invention provides a distributed open source intelligence extraction method, which can be executed by a distributed open source intelligence extraction device (hereinafter referred to as the extraction device), and in particular, is executed by one or more processors in the extraction device to implement at least the following steps: S101, obtaining a connection configuration, and entering data source information according to the connection configuration, wherein the connection configuration includes: an interface address, an interface request mode, an interface parameter, and a request header; In this embodiment, the extraction device can be a terminal with data processing capabilities such as a server, workstation, desktop computer, laptop computer, etc. The extraction device can be installed with a corresponding operating system and application software, and the functions required by this embodiment can be implemented through the combination of the operating system and application software.

[0027] It should be noted that, in this embodiment, the extraction device first constructs an intelligent configuration module, which can automatically read the pre-defined connection configuration information, and also supports obtaining the latest configuration information through a dynamic interface. The connection configuration information not only includes the interface address of the target data source, but also specifies in detail the interface request method (such as GET or POST), the parameters required for the request, and the corresponding request header information. The extraction device loads the configuration file at startup, performs format verification and data integrity check on each configuration read, and ensures the legitimacy of the interface address and the accuracy of the request parameters. Then, the extraction device initiates a network request through a secure encrypted channel according to the verified configuration, and dynamically monitors the response status during the data transmission process, so as to record the error log and trigger the retry mechanism in time when encountering abnormal situations. More importantly, the intelligent configuration module has an adaptive adjustment function, which automatically optimizes the interface request method and parameter settings through feedback on historical interaction data and real-time responses during the collection process, thereby achieving efficient adaptation to different data sources. When the connection is successfully established, the extraction device enters the relevant information extracted from the data source into a distributed database, and performs version management and archiving on these data.

[0028] S102, analyzing the HTML structure of the main webpage based on the data source information, and constructing a document object model tree based on the analysis result to identify the distribution information of the main webpage content, calling a pre-trained large language model to perform semantic understanding on the distribution information of the main webpage content, and generating list data for the understood content; In this embodiment, the extraction device first performs a comprehensive analysis of the original HTML code of the main web page and converts it into structured data, which not only includes each start tag, end tag, attribute and text content, but also maps each HTML element to an independent node in the document object model (DOM) tree. The entire parsing process can use a grammar analyzer to ensure that the semantic information of each element in the page can be accurately captured even when facing complex nested and dynamically generated HTML structures. Next, the extraction device conducts an in-depth analysis of the constructed DOM tree, by counting the HTML tag density contained in each node, calculating the ratio between the number of text characters and the number of tags in different DOM nodes, and evaluating the distribution of various types of elements in the page, and even recording the depth information of each node in the DOM tree in detail. These statistical results not only reveal the structural characteristics of each area of ​​the web page, but also accurately distinguish the main content from peripheral interference elements, such as navigation bars, advertisements and comment areas. It is worth noting that in the statistical analysis process, by dynamically adjusting the parsing strategy, it is possible to adapt to various non-standardized or chaotic HTML coding styles, thereby improving the accuracy of content recognition. Based on the above analysis structure, the extraction device further uses these data to construct a regional feature vector of the main web page. The regional feature vector of the main web page can describe the spatial distribution and hierarchical structure of the web page content.

[0029] Furthermore, in this embodiment, the extraction device uses a pre-trained large language model to perform deep semantic understanding of the distribution information of the main webpage content and generate structured list data. The extraction device converts the main webpage content distribution information identified by statistical methods into a vector representation, which depends on the position information of each word in the text. Specifically, the extraction device first converts each word of the original text into a vector through a word embedding module, where the formula (in, is the word embedding vector, is the word embedding matrix, is the index of a word in the vocabulary, is the bias vector), and obtains the basic word embedding vector; then, the extraction device uses the formula according to the position index of each word in the text ,(in, is the position encoding vector, is the position index, s and c are parameters) to encode the position information into the word vector, thereby generating a vector representation containing the context position information. This allows the semantic information of each word to be fully preserved, and its relative position in the text is also clearly expressed. Next, the extraction device uses these vectors to construct the query vector Q, key vector K and value vector V, and processes these vectors with the help of the self-attention mechanism, and calculates the formula , realizing the calculation of contextual relevance between words. In order to further improve the model's ability to capture complex semantic relationships, the extraction device performs self-attention calculations in parallel in different subspaces, and then combines the outputs of multiple attention heads through the formula To integrate, is the dimension of the key vector, is the single-head attention, h is the number of heads of multi-head attention, is the parameter of the linear layer. The multi-head attention mechanism enables the model to capture semantic features from multiple perspectives when processing the same text and effectively filter out noise information.

[0030] After the attention module is processed, the residual connection is used to fuse the output to alleviate the gradient vanishing problem in deep network training, using the formula H=X+f(X). Among them, X is the module input, and f(X) is the processing result after self-attention and multi-head attention; at the same time, the predicted probability distribution and the true distribution of the model output are measured by the cross entropy loss function, and the formula is: ; To achieve automatic optimization of model parameters.

[0031] Where N is the number of samples, is the true label of the ith sample (usually a one-hot encoded vector), is the predicted probability distribution of the ith sample (obtained from the model output).

[0032] S103, access each detail page link of the list data in turn, extract the HTML content of each detail page and generate a regional feature vector for each detail page, call a pre-trained large language model to process the regional feature vector of each detail page, and generate structured data of the list and details.

[0033] In this embodiment, the extraction device first accesses the link of each detail page in the list data in turn, and automatically extracts the HTML content of each detail page. Unlike the more scattered content of the main web page, the detail page usually presents the article body and related information in a concentrated manner. When the extraction device parses the detail page, it converts the HTML code of the page into structured data and constructs a corresponding document object model tree. In this process, the extraction device generates a regional feature vector that reflects the overall layout of the page, and extracts the article body, key images, and core metadata such as author and publication time. The generated regional feature vector is then input into the pre-trained large language model. The model uses its built-in self-attention and multi-head attention mechanisms to deeply analyze the semantic information of each part, and automatically identifies and integrates structured data with high semantic accuracy.

[0034] See also Figure 2The second embodiment of the present invention provides a distributed open source intelligence extraction device, including: Data source information entry form 201, used to obtain connection configuration and enter data source information according to the connection configuration; The list data generating unit 202 is used to analyze the HTML structure of the main webpage based on the data source information, and to construct a document object model tree based on the analysis result to identify the distribution information of the main webpage content, to call the pre-trained large language model to perform semantic understanding on the distribution information of the main webpage content, and to generate list data for the understood content; The structured data generation unit 203 is used to access each detail page link of the list data in sequence, extract the HTML content of each detail page and generate a regional feature vector for each detail page, call a pre-trained large language model to process the regional feature vector of each detail page, and generate structured data of the list and details.

[0035] Preferably, the connection configuration includes: interface address, interface request mode, interface parameters and request header.

[0036] Preferably, the analyzing the HTML structure of the main webpage based on the data source information and constructing a document object model tree based on the analysis result to identify the distribution information of the main webpage content is specifically: Parsing the original HTML code of the main webpage into structured data, and constructing a document object model tree according to the structured data; wherein the structured data includes a start tag, an end tag, attributes and text content, and each HTML element becomes a node in the document object model tree; Analyze the document object model tree and generate analysis results, wherein the analysis results include HTML tag density in different areas, the ratio of the number of text characters to the number of tags in different DOM nodes, the distribution of different types of elements in the page, and the depth of nodes in the DOM tree; The regional feature vector of the main webpage is determined according to the analysis structure.

[0037] Preferably, the calling of the pre-trained large language model to semantically understand the distribution information of the main webpage content and generating list data for the understood content is specifically as follows: Convert the distribution information of the main webpage content into a vector representation , based on the position index of each word in the text, using the formula , encode the position information into the word vector and generate a vector representation containing the context position information, where is the word embedding vector, is the word embedding matrix, is the index of a word in the vocabulary, is the bias vector, is the position encoding vector, is the position index, s and c are parameters; The query vector Q, key vector K and value vector V are constructed through the self-attention mechanism word embedding vector, and according to the formula , calculate the contextual relevance between each word vector, perform self-attention calculations in parallel in different subspaces, and use the formula , merge the outputs of multiple attention heads to generate list data, where is the dimension of the key vector, is the single-head attention, h is the number of heads of multi-head attention, are the parameters of the linear layer.

[0038] A third embodiment of the present invention provides a distributed open source intelligence extraction device, including a memory and a processor, wherein the memory stores a computer program, and the computer program can be executed by the processor to implement a distributed open source intelligence extraction method as described in any one of the above items.

[0039] The fourth embodiment of the present invention provides a computer-readable storage medium, characterized in that a computer program is stored therein, and the computer program can be executed by a processor of a device where the computer-readable storage medium is located to implement a method for extracting distributed open source intelligence as described in any one of the above items.

[0040] A distributed open source intelligence extraction method, device, equipment and storage medium provided by the present invention first obtains the connection configuration and enters the data source information according to the connection configuration; then, the HTML structure of the main webpage is analyzed based on the data source information, and a document object model tree is constructed based on the analysis result to identify the distribution information of the main webpage content, and a pre-trained large language model is called to semantically understand the distribution information of the main webpage content, and list data is generated for the understood content; then, each detail page link of the list data is accessed in turn, the HTML content of each detail page is extracted and the regional feature vector of each detail page is generated, and the pre-trained large language model is called to process the regional feature vector of each detail page, and structured data of the list and details is generated. The problem of extraction errors when extracting information from webpages is solved.

[0041] Exemplarily, the computer program described in the third and fourth embodiments of the present invention may be divided into one or more modules, and the one or more modules are stored in the memory and executed by the processor to complete the present invention. The one or more modules may be a series of computer program instruction segments capable of completing specific functions, and the instruction segments are used to describe the execution process of the computer program in the device for extracting distributed open source intelligence. For example, the device described in the second embodiment of the present invention.

[0042] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the method for extracting distributed open source intelligence, and uses various interfaces and lines to connect the various parts of the method for extracting distributed open source intelligence.

[0043] The memory can be used to store the computer program and / or module, and the processor realizes various functions of a distributed open source intelligence extraction method by running or executing the computer program and / or module stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, a text conversion function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as audio data, text message data, etc.), etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0044] Wherein, if the implemented module is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Wherein, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0045] It should be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the accompanying drawings of the device embodiments provided by the present invention, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines. A person of ordinary skill in the art may understand and implement it without paying any creative effort.

[0046] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A method for extracting distributed open source intelligence, characterized in that: include: Obtain connection configuration, and enter data source information according to the connection configuration; Analyze the HTML structure of the main webpage based on the data source information, and construct a document object model tree based on the analysis result to identify the distribution information of the main webpage content, call the pre-trained large language model to perform semantic understanding on the distribution information of the main webpage content, and generate list data for the understood content; Access each detail page link of the list data in turn, extract the HTML content of each detail page and generate a regional feature vector for each detail page, call the pre-trained large language model to process the regional feature vector of each detail page, and generate structured data of the list and details.

2. According to claim 1, a distributed open source intelligence extraction method is characterized in that: The connection configuration includes: interface address, interface request mode, interface parameters and request header.

3. The method for extracting distributed open source intelligence according to claim 1, characterized in that: The step of analyzing the HTML structure of the main webpage based on the data source information and constructing a document object model tree based on the analysis result to identify the distribution information of the main webpage content is as follows: Parsing the original HTML code of the main webpage into structured data, and constructing a document object model tree according to the structured data; wherein the structured data includes a start tag, an end tag, attributes and text content, and each HTML element becomes a node in the document object model tree; Analyze the document object model tree and generate analysis results, wherein the analysis results include HTML tag density in different areas, the ratio of the number of text characters to the number of tags in different DOM nodes, the distribution of different types of elements in the page, and the depth of nodes in the DOM tree; The regional feature vector of the main webpage is determined according to the analysis structure.

4. The method for extracting distributed open source intelligence according to claim 1, characterized in that: The calling of the pre-trained large language model performs semantic understanding on the distribution information of the main webpage content, and generates list data for the understood content, specifically: Convert the distribution information of the main webpage content into a vector representation , according to the position index of each word in the text, using the formula , encode the position information into the word vector and generate a vector representation containing the context position information, where is the word embedding vector, is the word embedding matrix, is the index of a word in the vocabulary, is the bias vector, is the position encoding vector, is the position index, s and c are parameters; The query vector Q, key vector K and value vector V are constructed through the self-attention mechanism word embedding vector, and according to the formula , calculate the contextual relevance between each word vector, perform self-attention calculations in parallel in different subspaces, and use the formula , merge the outputs of multiple attention heads to generate list data, where is the dimension of the key vector, is the single-head attention, h is the number of heads of multi-head attention, are the parameters of the linear layer.

5. A distributed open source intelligence extraction device, characterized in that: include: A data source information input unit, used to obtain a connection configuration and input data source information according to the connection configuration; A list data generating unit, configured to analyze the HTML structure of the main webpage based on the data source information, and to construct a document object model tree based on the analysis result to identify the distribution information of the main webpage content, to call a pre-trained large language model to perform semantic understanding on the distribution information of the main webpage content, and to generate list data for the understood content; The structured data generation unit is used to sequentially access each detail page link of the list data, extract the HTML content of each detail page and generate a regional feature vector for each detail page, call a pre-trained large language model to process the regional feature vector of each detail page, and generate structured data of the list and details.

6. A distributed open source intelligence extraction device according to claim 5, characterized in that: The connection configuration includes: interface address, interface request mode, interface parameters and request header.

7. The distributed open source intelligence extraction device according to claim 5, characterized in that: The step of analyzing the HTML structure of the main webpage based on the data source information and constructing a document object model tree based on the analysis result to identify the distribution information of the main webpage content is as follows: Parsing the original HTML code of the main webpage into structured data, and constructing a document object model tree according to the structured data; wherein the structured data includes a start tag, an end tag, attributes and text content, and each HTML element becomes a node in the document object model tree; Analyze the document object model tree and generate analysis results, wherein the analysis results include HTML tag density in different areas, the ratio of the number of text characters to the number of tags in different DOM nodes, the distribution of different types of elements in the page, and the depth of nodes in the DOM tree; The regional feature vector of the main webpage is determined according to the analysis structure.

8. The distributed open source intelligence extraction device according to claim 5, characterized in that: The calling of the pre-trained large language model performs semantic understanding on the distribution information of the main webpage content, and generates list data for the understood content, specifically: Convert the distribution information of the main webpage content into a vector representation , according to the position index of each word in the text, using the formula , encode the position information into the word vector and generate a vector representation containing the context position information, where is the word embedding vector, is the word embedding matrix, is the index of a word in the vocabulary, is the bias vector, is the position encoding vector, is the position index, s and c are parameters; The query vector Q, key vector K and value vector V are constructed through the self-attention mechanism word embedding vector, and according to the formula , calculate the contextual relevance between each word vector, perform self-attention calculations in parallel in different subspaces, and use the formula , merge the outputs of multiple attention heads to generate list data, where is the dimension of the key vector, is the single-head attention, h is the number of heads of multi-head attention, are the parameters of the linear layer.

9. A distributed open source intelligence extraction device, characterized in that: It includes a memory and a processor, wherein the memory stores a computer program, and the computer program can be executed by the processor to implement a method for extracting distributed open source intelligence as described in any one of claims 1 to 4.

10. A computer-readable storage medium, characterized in that: A computer program is stored, and the computer program can be executed by a processor of the device where the computer-readable storage medium is located to implement a method for extracting distributed open source intelligence as described in any one of claims 1 to 4.

Images