Figure information extraction method and device, computer equipment and computer readable storage medium

Abstract

The invention provides a character information extraction method and device, computer equipment and a computer readable storage medium. The method comprises the steps that a target webpage text containing character content is acquired; performing webpage node analysis on the target webpage text to judge whether the target webpage text contains a single-person description tag or not; and if the target webpage text contains the single-person description tag, performing named entity identification on the target webpage text based on the single-person description tag to obtain character information corresponding to each character. By adopting the method and the device, the webpage label and the text information are combined, analyzed and divided into the single-person description label, and the person information in the page can be extracted with low cost and high precision.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of computer technology, and in particular to a person information extraction method and device, computer equipment and a computer readable storage medium. BACKGROUND

[0002] Extracting person information from a webpage has strong practical value and application prospects in many fields. For example, user experience can be enhanced and personalized content recommendation can be achieved by extracting person information. For another example, a relationship network of a specific person can be analyzed, the connection between persons and their influence can be understood, and a relationship network can be generated.

[0003] However, no matter in which scenario, the number of pages to be processed is extremely large, and because the webpage format is diverse and there is a large amount of unstructured text, if a traditional webpage analysis tool is used, specific extraction rules need to be written for each page; and if a large model analysis is used, relationship extraction and information extraction need to be done together, both of which undoubtedly face the problem of uncontrollable cost.

[0004] Therefore, how to extract relevant information of page persons at low cost and high precision is a great challenge at present. SUMMARY

[0005] The purpose of the present application is to provide a person information extraction method and device, computer equipment and a computer readable storage medium, which make full use of the label information of the webpage, combine the webpage label and the text information, and cut the multi-person description text into single-person text, so as to ensure the extraction quality of the person relationship, reduce the recognition difficulty of the subsequent model, and save the information extraction cost.

[0006] In a first aspect, the present application provides a person information extraction method, comprising: obtaining a target webpage text containing person content; performing webpage node analysis on the target webpage text to determine whether the target webpage text contains a single-person description label; If the target webpage text contains a single-person description label, performing named entity recognition on the target webpage text based on the single-person description label to obtain person information corresponding to each person.

[0007] In some embodiments of the present application, performing webpage node analysis on the target webpage text to determine whether the target webpage text contains a single-person description label comprises: performing webpage node analysis on the target webpage text based on a breadth-first search algorithm to determine node attribute information of each webpage node in the target webpage text; and determining whether the target webpage text contains a single-person description label based on the node attribute information.

[0008] In some embodiments of the present application, based on a breadth-first search algorithm, webpage node analysis is performed on the target webpage text to determine node attribute information of each webpage node in the target webpage text, including: initializing a node queue and enqueuing a root node of the target webpage text; in the case that the queue is not empty, determining the number of valid person names and the number of valid child nodes corresponding to each webpage node in the queue; based on the number of valid person names and the number of valid child nodes, determining the node attribute information of each webpage node.

[0009] In some embodiments of the present application, in the case that the queue is not empty, the number of valid person names and the number of valid child nodes corresponding to each webpage node in the queue are determined, including: in the case that the queue is not empty, the total number of valid person name information is counted to determine the number of valid person names corresponding to each webpage node in the queue; wherein the valid person name information is person name information whose character length meets a preset condition; for each webpage node, the child nodes carrying valid person name information are screened out and counted to obtain the number of valid child nodes.

[0010] In some embodiments of the present application, based on the number of valid person names and the number of valid child nodes, the node attribute information of each webpage node is determined, including: calculating the ratio of the number of valid person names to the number of valid child nodes as a target value; if the target value is between a first preset threshold and a second preset threshold, it is determined that the node attribute information of the corresponding webpage node is a single-person description label; if the target value is greater than the second preset threshold, it is determined that the node attribute information of the corresponding webpage node is a multi-person description label; wherein the second preset threshold is greater than the first preset threshold.

[0011] In some embodiments of the present application, the target webpage text containing character content is obtained, including: obtaining a webpage text to be extracted for character information; inputting the webpage text into a trained text classification model to output webpage category information of the webpage text; wherein the text classification model includes a pre-trained model DistilBERT; determining the webpage text with the webpage category information as a character page as the target webpage text.

[0012] In some embodiments of the present application, the character information extraction method further includes: if the target webpage text does not contain a single-person description label, performing relationship matching and character information extraction on the target webpage text based on a preset large model to obtain the character information.

[0013] In a second aspect, the present application provides a character information extraction device, including: a webpage obtaining module for obtaining a target webpage text containing character content; a node analysis module for performing webpage node analysis on the target webpage text to determine whether the target webpage text contains a single-person description label; The information extraction module is configured to, if the target webpage text contains a single-person description label, perform named entity recognition on the target webpage text based on the single-person description label to obtain the information of each person.

[0014] In a third aspect, the present application also provides a computer device, comprising: one or more processors; a memory; and one or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the processor to implement the above-mentioned person information extraction method.

[0015] In a fourth aspect, the present application also provides a computer readable storage medium, which stores a computer program. The computer program is loaded by a processor to execute the steps of the person information extraction method.

[0016] In a fifth aspect, the present application provides a computer program product or a computer program, which comprises computer instructions stored in a computer readable storage medium. A processor of a computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to enable the computer device to execute the method provided in the first aspect.

[0017] The above-mentioned person information extraction method, device, computer device and computer readable storage medium can be used to determine whether the target webpage text contains a single-person description label by obtaining the target webpage text containing the person content and performing webpage node analysis on the target webpage text. When the target webpage text contains the single-person description label, the target webpage text is subjected to named entity recognition based on the single-person description label to obtain the information of each person. In this way, the single-person description label is located by using the webpage node analysis before the person information is extracted, and the person information is accurately extracted by using the single-person description label, which can significantly reduce the time cost and ensure the accuracy of the extracted person information. BRIEF DESCRIPTION OF DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0019] Figure 1 The scene diagram of the person information extraction method in the embodiments of the present application; Figure 2 The flowchart of the person information extraction method in the embodiments of the present application; Figure 3 A flowchart of a webpage node analysis step in an embodiment of the present application; Figure 4 An interface diagram of a person page in an embodiment of the present application; Figure 5 A webpage node diagram of a label positioning step in an embodiment of the present application; Figure 6 A structure diagram of a person information extraction device in an embodiment of the present application; Figure 7 A structure diagram of a computer device in an embodiment of the present application. DETAILED DESCRIPTION

[0020] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0021] It should be noted that, in the description of the present application, the terms "first", "second" are only used for the purpose of description, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include one or more of the features. In the description of the present application, the meaning of "multiple" is two or more, unless otherwise specifically limited.

[0022] Meanwhile, the person information extraction method provided by the embodiments of the present application can be applied to, for example, Figure 1The shown person information extraction system. Among them, the person information extraction system includes a client 102 and a server 104. The client 102 can be a device that includes receiving and transmitting hardware, that is, a device with receiving and transmitting hardware capable of performing bidirectional communication on a bidirectional communication link. Such a device can include a cellular or other communication device with a single-line display or a multi-line display or a cellular or other communication device without a multi-line display. The client 102 can be a desktop terminal or a mobile terminal, and the client 102 can also be one of a mobile phone, a tablet computer, and a notebook computer. The server 104 can be a stand-alone server, or a server network or server cluster composed of servers, including but not limited to a computer, a network host, a single network server, a plurality of network server sets, or a cloud server composed of a plurality of servers. Among them, the cloud server is composed of a large number of computers or network servers based on cloud computing (Cloud Computing). In addition, the client 102 and the server 104 establish a communication connection through a network, and the network can be any one of a wide area network, a local area network, and a metropolitan area network.

[0023] In addition, those skilled in the art can understand that Figure 1 The application environment shown in the above is only one application scenario applicable to the scheme of the present application, and does not constitute a limitation on the application scenarios of the scheme of the present application. Other application environments can include more or fewer devices than those shown in the above. Figure 1 For example, Figure 1 Only one server is shown in the above. It can be understood that the person information extraction system can also include one or more other devices, which are not limited specifically herein. In addition, the person information extraction system can also include a memory for storing data, such as storing person information.

[0024] Of course, Figure 1 The scenario diagram of the person information extraction system shown in the above is only one example, and the person information extraction system and the scenario described in the embodiments of the present application are used to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art can know that, with the evolution of the person information extraction system and the appearance of new business scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0025] In the current technical practice, the extraction of person information from a web page usually follows a two-stage process: first, accurate positioning of the person description, and second, detailed extraction of the person information. Since most person pages often contain descriptions of multiple roles, the primary task is to accurately locate the description area of a single person and then perform targeted information extraction to avoid information confusion and mismatch.

[0026] In order to realize the accurate positioning of the character description information, and in order to cope with the diversity of different web page structures, we usually use web page analysis tools such as XPath or BeautifulSoup to parse the page content and understand the web page structure, so as to extract the required character description part. In the character information extraction stage, the commonly used methods in the industry include dictionary matching, rule engine and deep learning technology. However, no matter which technical means is adopted, if the positioning work in the early stage deviates, it will have a negative impact on the accuracy of the final information extraction.

[0027] Therefore, ensuring the accuracy of the character description positioning is the key prerequisite to improve the accuracy of character information extraction. The accuracy of this link is directly related to the success or failure of the subsequent information extraction work, and is the core link that cannot be ignored in the whole process. Next, the focus of the present application, i.e. how to realize the accurate positioning of the single character description information, will be described in detail.

[0028] Referring to Figure 2 , a flowchart of a character information extraction method provided by the embodiment of the present application, the embodiment mainly takes the method applied to the server 104 in the above Figure 1 as an example to illustrate, which includes steps S201 to S203, as follows: S201, obtaining a target web page text containing character content.

[0029] Among them, the character content can refer to the description text of personal information, and the personal information includes but is not limited to: name, gender, education, position, school, etc.

[0030] Among them, the web page text can refer to the text written in HTML (Hypertext Markup Language) and CSS (Cascading Style Sheets) languages, which is used to build the structure, style and layout of the web page. The target web page text is the web page text determined as the character page.

[0031] In specific implementation, as the number of pages to be monitored increases, the server 104 can adopt one of the following compliance methods to obtain the web page text: (1) use the requests library of Python to send HTTP request to obtain the web page source code; (2) use the urllib library of Python to send HTTP request to obtain the web page source code; (3) use third-party tools such as curl command line tool or browser developer tool to view the web page source code.

[0032] Further, after the server 104 obtains the web page text of the preset monitoring range content, it can use a classification model to filter out the target web page text as the basis for subsequent character description positioning analysis.

[0033] In one embodiment, step S201 comprises: obtaining webpage text of which the information of the person to be extracted is to be extracted; inputting the webpage text into a trained text classification model to output webpage category information of the webpage text; wherein the text classification model comprises a pre-trained model DistilBERT; and determining the webpage text of which the webpage category information is a webpage of a person as target webpage text.

[0034] wherein DistilBERT is a lightweight BERT model suitable for scenarios with high requirements for model size and inference speed.

[0035] In a specific implementation, the server 104 can obtain all subpages under the same domain name as the homepage of a certain target enterprise, and then determine whether each subpage is a webpage of a person through a trained text classification model. If it is determined to be a webpage of a person, the page can be used as a target webpage for subsequent node analysis by the server 104. It can be understood that the target enterprise can be determined according to actual application requirements.

[0036] Here, determining the available text classification model needs to consider the characteristics of the data (such as whether it contains text, images or other types of data), the complexity of the model, the computing resources, and the accuracy and generalization ability of the model. Generally, the most suitable model for a specific task needs to be determined through experiments, so the text classification model selected by the present application is the best structure selected after comparing experimental data.

[0037] S202, performing webpage node analysis on the target webpage text to determine whether the target webpage text contains a single-person description label.

[0038] In a specific implementation, after the server 104 obtains the target webpage text, it can perform node analysis on the target webpage text by performing a breadth-first search (BFS) on the webpage DOM (Document Object Model) tree, and then determine whether the target webpage text contains a single-person description label. Here, the DOM tree is a hierarchical structure in which each HTML tag is considered as a node in the tree, and these nodes are connected to each other in a parent-child relationship. Breadth-first search is a graph traversal algorithm that first visits all direct child nodes (i.e. nodes at the same level) of the starting node, and then moves to the child nodes of these child nodes, and so on. In the context of the DOM tree, this means that the algorithm first visits the root element of the document (usually the tag), then visits all direct child elements of the root element, then visits the child elements of these child elements, and so on, until all reachable nodes are visited.

[0039] Further, in order to improve the accuracy of the character information extraction, the present application proposes to complete the webpage cleaning step before the webpage node analysis, so as to remove noise and unnecessary information, and make the text more pure and easy to process.

[0040] Specifically, the webpage cleaning method can include at least one of the following: (1) removing HTML tags, such as using regular expressions or a special HTML parsing library (such as BeautifulSoup) to remove HTML tags in the webpage, and only keeping the text content; (2) removing special characters, such as removing special characters, symbols and non-printing characters in the text to keep the text clear; (3) removing white space characters, such as removing extra spaces, tabs and line breaks to make the text format neat and uniform; (4) removing advertisements and navigation information, such as identifying and removing non-target information such as advertisement content, navigation bar, footnotes, etc. in the webpage; (5) converting case, such as converting the text to lowercase or uppercase to facilitate unified processing; (6) removing stop words, such as for natural language processing tasks, common stop words (such as "of", "is", "in", etc.) can be removed to reduce noise and improve the weight of key information; (7) normalizing text, such as normalizing text containing specific types of information such as dates, times, numbers, etc., such as unifying date formats, currency symbols, units of measurement, etc.

[0041] In one embodiment, step S202 includes: based on a breadth-first search algorithm, performing webpage node analysis on the target webpage text to determine node attribute information of each webpage node in the target webpage text; based on the node attribute information, determining whether the target webpage text contains a single-person description label.

[0042] In a specific implementation, the node attribute information can include a multi-person description label and a single-person description label. The multi-person description label refers to a node label whose direct child node carries N person description information, N≥2; the single-person description label refers to a node label whose direct child node carries only one person description information. Using breadth-first search, the single-person description positioning is realized by combining webpage labels and text information, which reduces the difficulty of subsequent model processing.

[0043] In one embodiment, based on a breadth-first search algorithm, performing webpage node analysis on the target webpage text to determine node attribute information of each webpage node in the target webpage text includes: initializing a node queue and enqueuing the root node of the target webpage text; in the case that the queue is not empty, determining the number of valid names and the number of valid child nodes corresponding to each webpage node in the queue; based on the number of valid names and the number of valid child nodes, determining the node attribute information of each webpage node.

[0044] In a specific implementation, reference can be made to Figure 3The steps of applying BFS on the webpage DOM tree by the server 104 include: (1) initialization: starting from the root node of the DOM tree (generally a tag); (2) queue establishment: using a queue to manage the access process. Initially, the root node is placed into the queue; (3) traversing nodes: taking the first element in the queue, then accessing the element, i.e., judging whether it is a single-person description tag, if yes, storing it into the result list, if not, accessing the child elements of the element, repeating the process until the queue is empty. In this way, after traversing the entire webpage nodes, the single-person description tags filtered out are stored in the result list.

[0045] Specifically, the basis for judging whether a node is a single-person description tag is the number of valid person names and the number of valid child nodes corresponding to each webpage node, and the manner of obtaining the number of valid person names and the number of valid child nodes will be described in detail below.

[0046] In one embodiment, when the queue is not empty, the number of valid person names and the number of valid child nodes corresponding to each webpage node in the queue are determined, including: when the queue is not empty, the total number of valid person name information is counted to determine the number of valid person names corresponding to each webpage node; wherein the valid person name information is person name information with a character length satisfying a preset condition; for each webpage node, the child nodes carrying valid person name information are filtered out and counted to obtain the number of valid child nodes.

[0047] The valid person name information is person name information with a character length satisfying a preset condition, such as an English person name with a character length less than "30" or a Chinese person name with a character length less than "15".

[0048] In a specific implementation, the scheme described in the previous embodiment is used to determine the node attribute information of each webpage node, and the number of valid person names corresponding to each webpage node is determined. The number of valid person names is determined based on the number of valid person name information, so when analyzing the number of valid person names corresponding to each webpage node, the person name information under the direct child nodes corresponding to the webpage node is first filtered out, and then the character type of the person name information is determined according to the preset condition threshold to further filter out the person name information satisfying the preset condition as valid person name information, so as to count the number.

[0049] Further, the number of valid child nodes mentioned in the embodiments of the present application refers to the total number of valid child nodes, and the valid child node refers to a node carrying person name information and must be valid person name information.

[0050] For example, refer to Figure 4For a target webpage text visual interface diagram, the diagram contains multiple person description information, in order to improve the accuracy of person information extraction, first need to carry out single person description label positioning, so can use webpage label to cut multiple person description text into single person, in order to facilitate subsequent information extraction. The logic of positioning single person description label can refer to the matrix diagram on the left side of Figure 5 , Figure 5 The matrix diagram on the left side can be regarded as a page, which corresponds to Figure 4 It is known that it contains nine person description information, and the whole page is the root node "A" on the right side. The nodes "a, b, c" represent three rows of information in the page. If the nodes "a1, a2, a3" are all person description labels, and the corresponding name information contained is valid name information, it means that the first row of the page has three persons. The remaining nodes "b1, b2, b3, c1, c2, c3" are the same. Therefore, the number of valid names of node "A" is "9", and the number of valid child nodes of node "A" is "3". The number of valid names of node "a" is "3", and the number of valid child nodes of node "a" is "3".

[0051] In one embodiment, based on the number of valid names and the number of valid child nodes, the node attribute information of each webpage node is determined, including: calculating the ratio of the number of valid names to the number of valid child nodes as a target value; if the target value is between the first preset threshold and the second preset threshold, it is determined that the node attribute information of the corresponding webpage node is a single person description label; if the target value is greater than the second preset threshold, it is determined that the node attribute information of the corresponding webpage node is a multiple person description label; wherein the second preset threshold is greater than the first preset threshold.

[0052] In a specific implementation, whether the child node is a single person description label is determined by using the fact that when the label changes from multiple persons to single person, the ratio of the number of names to the number of child nodes will sharply decrease, and in an ideal case, the ratio is close to "1". Through testing, when the first preset threshold is set to "0.7" and the second preset threshold is set to "1.2", the accuracy of person information extraction is relatively higher. Therefore, the embodiment of the present application proposes that when the ratio is greater than "1.2", it can be determined that the corresponding node is a multiple person description label, and when the ratio is between [0.7, 1.2], it can be determined that the corresponding node is a single person description label, and the text in a node is the description text of a person.

[0053] Specifically, as shown in Figure 5 Taking node "A" as an example, it can be analyzed that the number of valid names of node "A" is "9" and the number of valid child nodes is "3", so the target value is 9 / 3=3. Since 3>1.2, node "A" can be determined as a multiple person description label.

[0054] For example, taking node "a" as an example, analysis shows that the number of valid names of node "a" is "3", and the number of valid child nodes is also "3", so the target value is 3 / 3 = 1, and since 1 ∈ [0.7, 1.2], node "a" can be determined as a single-person description label.

[0055] In S203, if the target web page text contains a single-person description label, the target web page text is subjected to named entity recognition based on the single-person description label to obtain the information of each person.

[0056] In a specific implementation, the server 104 can use an entity recognition model to perform target field recognition on the target web page text, such as name, gender, education, position, and school, which can be used as the information of the person.

[0057] In one embodiment, the method for extracting information of a person further includes: if the target web page text does not contain a single-person description label, performing relationship matching and information extraction on the target web page text based on a preset large model to obtain the information of the person.

[0058] In a specific implementation, the preset large model includes but is not limited to one of the following: BERT (Bidirectional Encoder Representations from Transformers), RoBERTa (A Robustly Optimized BERT Pre-Training Approach), DistilBERT (Distilling BERT for Natural Language Understanding), and XGBoost.

[0059] The method for extracting information of a person in the above embodiments can determine whether the target web page text contains a single-person description label by obtaining the target web page text containing the information of the person and performing web node analysis on the target web page text, and when the target web page text contains a single-person description label, performing named entity recognition on the target web page text based on the single-person description label to obtain the information of each person. In this way, the single-person description label is located by using web node analysis before the information of the person is extracted, and the information of the person is accurately extracted by using the single-person description label, which can significantly reduce the time cost and ensure the accuracy of the extraction of the information of the person.

[0060] It should be understood that, although Figure 2 The steps in the flowchart of the method for extracting information of a person are displayed in sequence according to the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other orders. Moreover,Figure 2 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0061] To better implement the character information extraction method provided in the embodiments of this application, based on the character information extraction method proposed in the embodiments of this application, the embodiments of this application also provide a character information extraction device, such as... Figure 6 As shown, the person information extraction device 600 includes: The webpage acquisition module 610 is used to acquire the text of a target webpage containing content about people; The node analysis module 620 is used to perform webpage node analysis on the target webpage text to determine whether the target webpage text contains a single person description tag. The information extraction module 630 is used to perform named entity recognition on the target webpage text based on the individual description tag if the target webpage text contains the individual description tag, so as to obtain the person information corresponding to each person.

[0062] In one embodiment, the node analysis module 620 is further configured to perform webpage node analysis on the target webpage text based on a breadth-first search algorithm to determine the node attribute information of each webpage node in the target webpage text; and based on the node attribute information, determine whether the target webpage text contains a single-person description tag.

[0063] In one embodiment, the node analysis module 620 is further configured to initialize a node queue and enqueue the root node of the target webpage text; if the queue is not empty, determine the number of valid names and the number of valid child nodes corresponding to each webpage node in the queue; and determine the node attribute information of each webpage node based on the number of valid names and the number of valid child nodes.

[0064] In one embodiment, the node analysis module 620 is further configured to count the total number of valid name information when the queue is not empty, so as to determine the number of valid names corresponding to each web page node in the queue; wherein, the valid name information is the name information whose character length meets the preset condition; for each web page node, the child nodes carrying valid name information are selected and counted to obtain the number of valid child nodes.

[0065] In an embodiment, the node analysis module 620 is further configured to calculate a ratio of the number of valid person names to the number of valid sub-nodes as a target value; if the target value is between a first preset threshold and a second preset threshold, determine that the node attribute information of the corresponding webpage node is a single-person description label; if the target value is greater than the second preset threshold, determine that the node attribute information of the corresponding webpage node is a multi-person description label; and the second preset threshold is greater than the first preset threshold.

[0066] In an embodiment, the webpage obtaining module 610 is further configured to obtain webpage text of to-be-extracted person information; input the webpage text into the trained text classification model to output webpage category information of the webpage text; and determine the webpage text of which the webpage category information is a person page as target webpage text.

[0067] In an embodiment, the person information extraction apparatus 600 further includes a model extraction module configured to, if the target webpage text does not contain a single-person description label, perform relationship matching and person information extraction on the target webpage text based on a preset large model to obtain the person information.

[0068] In the above embodiments, the server first locates a single-person description label by using webpage node analysis before extracting person information, and then completes accurate extraction of person information by using the single-person description label, which can significantly reduce time cost while ensuring the accuracy of person information extraction.

[0069] It should be noted that the specific limitations of the person information extraction apparatus can be referred to the limitations of the person information extraction method described above, and will not be repeated here. Each module in the above person information extraction apparatus can be realized by software, hardware and their combinations in whole or in part. The above modules can be embedded in or independent of the processor in the electronic device in hardware form, or stored in the memory in the electronic device in software form, so as to be called and executed by the processor to perform the operations corresponding to each module.

[0070] In some embodiments of the present application, the person information extraction apparatus 600 can be implemented in the form of a computer program, which can run on a computer device as shown in Figure 7 The memory of the computer device can store various program modules constituting the person information extraction apparatus 600, such as the webpage obtaining module 610, the node analysis module 620, and the information extraction module 630 as shown in Figure 6 The computer program constituted by each program module makes the processor execute the steps in the person information extraction method of each embodiment of the present application described in the specification. For example, Figure 7 The computer device as shown in Figure 6The webpage obtaining module 610 in the person information extraction device 600 shown performs step S201. The computer device can perform step S202 through the node analysis module 620. The computer device can perform step S203 through the information extraction module 630. The computer device includes a processor, a memory and a network interface connected through a system bus. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The network interface of the computer device is configured to communicate with external computer devices through network connection. The computer program is executed by the processor to implement a person information extraction method.

[0071] Those skilled in the art can understand that, Figure 7 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0072] In some embodiments of the present application, a computer device is provided, including one or more processors, a memory, and one or more application programs, wherein the one or more application programs are stored in the memory and configured to execute the steps of the person information extraction method by the processor. The steps of the person information extraction method can be the steps of the person information extraction method in each of the above embodiments.

[0073] In some embodiments of the present application, a computer readable storage medium is provided, which stores a computer program. The computer program is loaded by the processor, so that the processor executes the steps of the person information extraction method. The steps of the person information extraction method can be the steps of the person information extraction method in each of the above embodiments.

[0074] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when executed, can include the processes of the above-mentioned embodiment methods. Any reference to memory, storage, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0075] Any combination of the technical features of the above embodiments can be made. In order to make the description simple, all possible combinations of the technical features in the above embodiments are not described, however, as long as the combination of the technical features does not exist, it should be considered as the scope of the present application.

[0076] The above provides a detailed introduction to the person information extraction method, device, computer equipment and computer readable storage medium provided by the embodiment of the present application. The principle and implementation mode of the present application are described in this paper. The above embodiment is only used to help understand the method and core idea of the present application; at the same time, for those skilled in the art, according to the idea of the present application, the specific implementation mode and application range will be changed; according to the above, the content of the specification should not be understood as the limitation of the present application.

Claims

1. A method for extracting personal information, characterized in that, include: Retrieve the text of the target webpage containing information about people; Perform webpage node analysis on the target webpage text to determine whether the target webpage text contains a single-person description tag; If the target webpage text contains the individual description tag, then based on the individual description tag, named entity recognition is performed on the target webpage text to obtain the individual information corresponding to each individual.

2. The method as described in claim 1, characterized in that, The step of performing webpage node analysis on the target webpage text to determine whether the target webpage text contains a single-person description tag includes: Based on the breadth-first search algorithm, the target webpage text is analyzed to determine the node attribute information of each webpage node in the target webpage text. Based on the node attribute information, determine whether the target webpage text contains a single-person description tag.

3. The method as described in claim 2, characterized in that, The breadth-first search algorithm is used to analyze the target webpage text to determine the node attribute information of each webpage node in the target webpage text, including: Initialize the node queue and enqueue the root node of the target webpage text; If the queue is not empty, determine the number of valid names and the number of valid child nodes corresponding to each webpage node in the queue. Based on the number of valid names and the number of valid child nodes, the node attribute information of each webpage node is determined.

4. The method as described in claim 3, characterized in that, The step of determining the number of valid names and valid child nodes corresponding to each webpage node in the queue when the queue is not empty includes: When the queue is not empty, the total number of valid personal name information is counted to determine the number of valid personal names corresponding to each webpage node in the queue; wherein, the valid personal name information is personal name information whose character length meets the preset condition; For each webpage node, the child nodes carrying the aforementioned valid personal name information are selected and counted to obtain the number of valid child nodes.

5. The method as described in claim 3, characterized in that, The process of determining the node attribute information of each webpage node based on the number of valid names and the number of valid child nodes includes: Calculate the ratio of the number of valid names to the number of valid child nodes, and use it as the target value; If the target value is between the first preset threshold and the second preset threshold, then the node attribute information corresponding to the webpage node is determined to be a single-person description tag; If the target value is greater than the second preset threshold, then the node attribute information of the corresponding web page node is determined to be a multi-person description tag; wherein, the second preset threshold is greater than the first preset threshold.

6. The method as described in claim 1, characterized in that, The process of obtaining the target webpage text containing content about people includes: Retrieve the webpage text from which the information of the person to be extracted is extracted; The web page text is input into a trained text classification model, which outputs the web page category information of the web page text; wherein, the text classification model includes the pre-trained model DistilBERT; The webpage text that is determined to be a "person" page in the webpage category information is used as the target webpage text.

7. The method according to any one of claims 1 to 6, characterized in that, The method further includes: If the target webpage text does not contain the individual description tag, then based on the preset large model, relation matching and person information extraction are performed on the target webpage text to obtain the person information.

8. A device for extracting personal information, characterized in that, include: The webpage acquisition module is used to acquire the text of target webpages containing content about people; The node analysis module is used to perform webpage node analysis on the target webpage text to determine whether the target webpage text contains a single person description tag; The information extraction module is used to perform named entity recognition on the target webpage text based on the individual description tag if the target webpage text contains the individual description tag, so as to obtain the person information corresponding to each person.

9. A computer device, characterized in that, The computer device includes: One or more processors; The memory; and one or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the processor to implement the person information extraction method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, It stores a computer program, which is loaded by a processor to execute the steps of the person information extraction method according to any one of claims 1 to 7.

Images