Information extraction method and device, computer equipment, readable storage medium and program product

Abstract

The invention relates to an information extraction method and device, computer equipment, a computer readable storage medium and a computer program product. Comprising the steps of extracting key information from target webpage data based on a webpage extraction strategy; querying in a preset knowledge graph based on the key information, and determining a first confidence coefficient according to a query result; carrying out content identification on a picture in the key information to obtain an image identification result, carrying out text analysis on text information in the key information to obtain a text analysis result, and determining a second confidence coefficient according to a consistency judgment result of the image identification result and the text analysis result; determining a third confidence coefficient according to the rationality verification result; and based on the first confidence coefficient, the second confidence coefficient, the third confidence coefficient and a fourth confidence coefficient corresponding to the webpage extraction strategy, determining a target confidence coefficient corresponding to the key information, and outputting the key information and the target confidence coefficient. By adopting the method, the key information identification and filtering efficiency can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to an information extraction method, apparatus, computer equipment, readable storage medium, and program product. Background Art

[0002] With the development of big data technology, more and more product providers are analyzing massive product information on the Internet to understand market dynamics, competitor situations and consumer needs.

[0003] In related technologies, technicians typically customize different information extraction rules for pages on different platforms. Using these rules, they extract key information from the webpage data of the corresponding webpage, allowing for subsequent product information analysis based on this key information. Due to factors such as the rapid pace of webpage updates and the reliance on manual configuration and updating of information extraction rules, the extracted key information may be inaccurate and unreliable. Therefore, manual identification and filtering of key information is necessary to obtain valid key information. However, manual identification and filtering methods are costly and inefficient. Summary of the Invention

[0004] Based on this, it is necessary to provide an information extraction method, apparatus, computer equipment, readable storage medium and program product to address the above technical problems.

[0005] In a first aspect, the present application provides an information extraction method, the method comprising:

[0006] Extract key information from target web page data based on web page extraction strategy;

[0007] Performing a query in a preset knowledge graph based on the key information, and determining a first confidence level according to the query result;

[0008] Performing content recognition on the image in the key information to obtain an image recognition result, and performing text parsing on the text information in the key information to obtain a text parsing result, and determining a second confidence level based on a consistency judgment result between the image recognition result and the text parsing result;

[0009] Performing a rationality check on the key information, and determining a third confidence level based on the rationality check result;

[0010] Based on the first confidence, the second confidence, the third confidence, and a fourth confidence corresponding to the web page extraction strategy, a target confidence corresponding to the key information is determined, and the key information and the target confidence are output.

[0011] In one embodiment, performing a query in a preset knowledge graph based on the key information and determining a first confidence level according to the query result includes:

[0012] identifying the entities included in the key information;

[0013] Querying the number of entities contained in a preset knowledge graph, and determining a first weight according to the number;

[0014] Determine, based on the knowledge graph, a consistency recognition result between the association relationship of the entity and the key information, and determine a second weight based on the consistency recognition result;

[0015] Determining an industry constraint discrimination result based on the parameters of each entity included in the key information and preset industry constraint conditions, and determining a third weight based on the industry constraint discrimination result;

[0016] A first confidence level is determined based on the first weight, the second weight, and the third weight.

[0017] In one embodiment, determining the second confidence level based on the consistency determination result between the image recognition result and the text parsing result includes:

[0018] Determining a first entity set included in the image recognition result and a second entity set included in the text parsing result;

[0019] Determining a first consistency determination result based on the first entity set and the second entity set, and determining a second consistency determination result based on parameters of each entity in the first entity set and parameters of each entity in the second entity set;

[0020] Performing a semantic consistency check on the image recognition result and the text parsing result to obtain a third consistency determination result;

[0021] A second confidence level is determined according to the first consistency determination result, the second consistency determination result, and the third consistency determination result.

[0022] In one embodiment, the target web page data includes a plurality of content blocks; the method further includes:

[0023] For each content block, generating a feature vector corresponding to the content block according to key information contained in the content block;

[0024] performing clustering processing on each of the content blocks based on the feature vector corresponding to each of the content blocks to obtain at least one cluster;

[0025] In the cluster, a target content block whose distance to the cluster center meets a preset proximity condition is determined, and a content extraction template is generated based on the target content block; the content extraction template is used to extract key information contained in the web page data.

[0026] In one embodiment, the webpage extraction strategy includes a content extraction template, and the method further includes:

[0027] If the target confidence is greater than or equal to a preset threshold, executing the step of outputting the key information and the target confidence;

[0028] If the target confidence is less than a preset threshold, a content extraction template is retrieved from a preset content extraction template library, and based on the retrieved content extraction template, the web page extraction strategy is executed to extract key information from the target web page data.

[0029] In one embodiment, the method further comprises:

[0030] Obtaining initial web page data of the target web page;

[0031] The initial web page data is subjected to tag filtering, and the initial web page data after tag filtering is subjected to text normalization processing to obtain target web page data.

[0032] In a second aspect, the present application further provides an information extraction device, comprising:

[0033] The extraction module is used to extract key information from the target web page data based on the web page extraction strategy;

[0034] A first determination module is configured to perform a query in a preset knowledge graph based on the key information and determine a first confidence level according to the query result;

[0035] a second determination module, configured to perform content recognition on the image in the key information to obtain an image recognition result, perform text parsing on the text information in the key information to obtain a text parsing result, and determine a second confidence level based on a consistency judgment result between the image recognition result and the text parsing result;

[0036] a third determination module, configured to perform a rationality check on the key information and determine a third confidence level based on a rationality check result;

[0037] The fourth determination module is configured to determine a target confidence corresponding to the key information based on the first confidence, the second confidence, the third confidence, and a fourth confidence corresponding to the web page extraction strategy, and output the key information and the target confidence.

[0038] In one embodiment, the first determining module is specifically configured to:

[0039] identifying the entities included in the key information;

[0040] Querying the number of entities contained in a preset knowledge graph, and determining a first weight according to the number;

[0041] Determine, based on the knowledge graph, a consistency recognition result between the association relationship of the entity and the key information, and determine a second weight based on the consistency recognition result;

[0042] Determining an industry constraint discrimination result based on the parameters of each entity included in the key information and preset industry constraint conditions, and determining a third weight based on the industry constraint discrimination result;

[0043] A first confidence level is determined based on the first weight, the second weight, and the third weight.

[0044] In one embodiment, the second determining module is specifically configured to:

[0045] Determining a first entity set included in the image recognition result and a second entity set included in the text parsing result;

[0046] Determining a first consistency determination result based on the first entity set and the second entity set, and determining a second consistency determination result based on parameters of each entity in the first entity set and parameters of each entity in the second entity set;

[0047] Performing a semantic consistency check on the image recognition result and the text parsing result to obtain a third consistency determination result;

[0048] A second confidence level is determined according to the first consistency determination result, the second consistency determination result, and the third consistency determination result.

[0049] In one embodiment, the target web page data includes a plurality of content blocks; the apparatus further includes:

[0050] A generating module, configured to generate, for each content block, a feature vector corresponding to the content block according to key information contained in the content block;

[0051] A clustering module, configured to perform clustering processing on each of the content blocks based on a feature vector corresponding to each of the content blocks to obtain at least one cluster;

[0052] The fifth determination module is used to determine, in the cluster cluster, a target content block whose distance to the cluster center meets a preset proximity condition, and generate a content extraction template based on the target content block; the content extraction template is used to extract key information contained in the web page data.

[0053] In one embodiment, the webpage extraction strategy includes a content extraction template, and the apparatus further includes:

[0054] A first processing module, configured to execute the step of outputting the key information and the target confidence if the target confidence is greater than or equal to a preset threshold;

[0055] The second processing module is used to re-acquire the content extraction template from the preset content extraction template library if the target confidence is less than the preset threshold, and execute the web page extraction strategy based on the re-acquired content extraction template to extract key information from the target web page data.

[0056] In one embodiment, the apparatus further comprises:

[0057] An acquisition module, used to acquire the initial web page data of the target web page;

[0058] The filtering module is used to perform tag filtering on the initial web page data and perform text normalization processing on the tag-filtered initial web page data to obtain target web page data.

[0059] In a third aspect, the present application further provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the method steps of the first aspect when executing the computer program.

[0060] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which implements the method steps of the first aspect when the computer program is executed by a processor.

[0061] In a fifth aspect, the present application further provides a computer program product, comprising a computer program, which implements the method steps of the first aspect when executed by a processor.

[0062] The above-mentioned information extraction method, apparatus, computer equipment, readable storage medium and program product can extract key information from target web page data based on a web page extraction strategy, then perform a query in a preset knowledge graph based on the key information, determine a first confidence level based on the query results, perform content recognition on the image in the key information to obtain an image recognition result, and perform text parsing on the text information in the key information to obtain a text parsing result, determine a second confidence level based on the consistency judgment result between the image recognition result and the text parsing result, perform a rationality check on the key information, determine a third confidence level based on the rationality check result, and then, based on the first confidence level, the second confidence level, the third confidence level and the fourth confidence level corresponding to the web page extraction strategy, determine the target confidence level corresponding to the key information, and output the key information and the target confidence level. Using this solution, the target confidence level of the key information can be calculated and output while extracting the key information. The target confidence level can reflect the reliability of the key information without the need for manual identification. In this way, technical personnel can filter key information based on the target confidence corresponding to the actual business needs without manual filtering, thereby effectively saving labor costs and improving the efficiency of key information identification and filtering. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following briefly introduces the drawings required for use in the embodiments of the present application or related technical descriptions. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying any creative work.

[0064] Figure 1 is an application environment diagram of an information extraction method in one embodiment;

[0065] Figure 2 A flowchart illustrating an example of an information extraction method in one embodiment;

[0066] Figure 3 is a structural block diagram of an information extraction device in one embodiment;

[0067] Figure 4 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0068] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0069] The embodiments of this application provide an information extraction method. This embodiment uses the method applied to a terminal as an example for illustration. It is understood that the method can also be applied to a server, or to a system including a terminal and a server, and implemented through interaction between the terminal and the server. It is understood that the method can be applied to any terminal with data processing capabilities, and the embodiments of this application are not limiting.

[0070] The information extraction method provided by this application will be described in detail below in conjunction with specific implementation methods. Figure 1 As shown, the method includes the following steps:

[0071] Step 102: extract key information from the target web page data based on the web page extraction strategy.

[0072] In an embodiment of the present application, a terminal can use a preset web data acquisition tool to retrieve web data from multiple platforms on the internet to obtain target web page data. The target web page data can be web page data for product introduction pages. For example, a distributed crawler cluster can be used to crawl web page data from target product introduction pages on platforms such as JD.com, Tmall, and brand websites in real time. The target products can be products in a preset category (e.g., fast-moving consumer goods such as alcohol and fast food). Alternatively, the terminal can retrieve target web page data based on a URL (Uniform Resource Locator) directory classification strategy; or alternatively, strategies such as dynamic User-Agent rotation, IP (Internet Protocol) proxy pools, and request interval randomization can be employed.

[0073] After the terminal obtains the target web page data, it can extract key information from the target web page data based on the web page extraction strategy. The web page extraction strategy can be a pre-configured extraction rule (such as a regular expression, etc.) or a content extraction template. Different web page types have different web page extraction strategies. For example, different platforms have different web page implementation methods, and different web page extraction strategies can be configured for different platforms. Key information can be content features used to reflect product introduction information. Taking wine products as an example, key information can include wine product specification information, promotion rule information, alcohol content, identification information (such as brand information, manufacturer information), etc. It can be understood that the specific content of key information can be determined according to actual business needs. The above content is only used as an example, and this embodiment does not limit this.

[0074] Optionally, the method further includes: obtaining initial web page data of the target web page; performing tag filtering on the initial web page data, and performing text normalization processing on the tag-filtered initial web page data to obtain the target web page data.

[0075] In an embodiment of the present application, a terminal can utilize a distributed crawler cluster to crawl webpage data from various platforms in real time to obtain initial webpage data. Alternatively, the terminal can obtain initial webpage data based on a URL directory classification strategy (e.g., jd.com / beer / huarun-snow). Alternatively, the terminal can utilize dynamic User-Agent rotation, IP proxy pools, request interval randomization, and other mechanisms to obtain initial webpage data from various platforms. After obtaining the initial webpage data, the terminal can pre-process the initial webpage data. Specifically, the terminal can filter the initial webpage data for advertising tags. For example, it can remove interfering tags such as script, style, and advertisement based on XPath rules. The terminal can then perform HTML (HyperText Markup Language) structure repair, such as using an LXML parser to automatically complete missing tags and normalize the DOM (Document Object Model) tree structure. Furthermore, the terminal can perform text normalization on the processed initial webpage data, such as converting special characters (e.g., to spaces) and normalizing multilingual mixed text, to obtain the target webpage data. In this way, preprocessing the initial web page data can filter out invalid information and make the web page data more standardized and complete for subsequent processing.

[0076] Step 104: perform a query in a preset knowledge graph based on the key information, and determine a first confidence level according to the query result.

[0077] In an embodiment of the present application, a knowledge graph can be established for the introduction information of the target object. For example, a knowledge graph for beer products can be established based on information such as professional terminology, brand product line information, and promotion rules in the beer field. The entities in the knowledge graph can be company names, brand names, product parameters, etc., and the association relationship between the above information is reflected through the association relationship between each entity in the knowledge graph. The terminal device can query in the preset knowledge graph based on the extracted key information, obtain the query results, and calculate the first confidence based on the query results and the knowledge graph confidence calculation rules. The first confidence can reflect the confidence of the extracted key information relative to the knowledge graph.

[0078] Step 106: Perform content recognition on the image in the key information to obtain an image recognition result, and perform text analysis on the text information in the key information to obtain a text analysis result. Determine a second confidence level based on the consistency judgment result between the image recognition result and the text analysis result.

[0079] In an embodiment of the present application, the key information may include various types of content, such as pictures, text, etc., wherein the pictures may be static pictures, dynamic pictures, or screenshots obtained from videos, etc. The terminal may perform content recognition on the picture to obtain an image recognition result. For example, the terminal may use the YOLOv5 algorithm to detect the wine label area, and extract information such as the brand name and alcohol content through OCR (Optical Character Recognition). The terminal may also parse the text information through a pre-trained model to obtain a text parsing result. For example, the text content may be parsed through the BERT model to obtain valid information contained in the text, such as promotional rules information. In addition, the terminal may also filter the text recognition results through the knowledge graph, filter out text recognition results that do not exist in the knowledge graph (i.e., interference information), and obtain valid text recognition results.

[0080] The terminal may compare the image recognition result with the text parsing result, determine the degree of consistency between the two based on the comparison result, obtain a consistency determination result, and determine a second confidence level based on the consistency determination result. It is understood that the consistency determination result is used to indicate the degree of consistency between the content of the image recognition result and the text parsing result. The higher the consistency determination result, the more consistent the content of the two, and the higher the second confidence level, and vice versa.

[0081] Step 108: Perform a rationality check on the key information and determine a third confidence level based on the rationality check result.

[0082] In an embodiment of the present application, the terminal can also perform a rationality check on the key information and determine a third confidence level based on the rationality check result. The rationality check result can reflect whether the key information complies with preset common sense information or industry standards, etc. The higher the rationality check result, the more reasonable the key information is, and the higher the third confidence level, and vice versa. For example, if the unit of beer in the key information is "can", "bottle" or "can", a preset score is obtained; if the unit is "box" or "double", no score is obtained; if the alcohol content in the key information is within a preset numerical range, a preset score is obtained, and if it is not within the preset numerical range, no score is obtained. The terminal can perform rationality checks from multiple dimensions, obtain the verification results of each dimension, and sum the scores corresponding to each verification result to obtain the third confidence level.

[0083] Step 110 : Determine a target confidence level corresponding to the key information based on the first confidence level, the second confidence level, the third confidence level, and the fourth confidence level corresponding to the webpage extraction strategy, and output the key information and the target confidence level.

[0084] In an embodiment of the present application, the weights corresponding to the confidences of different dimensions may be different. The terminal may pre-store the weights corresponding to the confidences of each dimension. For example, the weights may be in descending order of the first confidence, the second confidence, the fourth confidence, and the third confidence. The terminal may perform a weighted summation of the first confidence, the second confidence, the third confidence, and the fourth confidence based on the pre-configured weights to obtain a target confidence. The target confidence may reflect the accuracy and reliability of the key information extracted by the web page extraction strategy. The fourth confidence corresponding to the web page extraction strategy may be determined based on the reuse rate of the web page extraction strategy. Specifically, the fourth confidence may be positively correlated with the reuse rate of the web page extraction strategy; alternatively, the fourth confidence may be determined based on the average confidence of the key information extracted by the web page extraction strategy during the historical extraction process, i.e., positively correlated with the average confidence. The terminal may then output the key information and the target confidence corresponding to the key information.

[0085] This solution allows us to simultaneously extract key information and calculate and output its target confidence level. This level reflects the reliability of key information, eliminating the need for manual identification. This allows technicians to filter key information based on its corresponding target confidence level according to actual business needs, obtaining high-confidence key information without the need for manual filtering. This effectively saves labor costs and improves the efficiency of key information identification and filtering.

[0086] Optionally, a query is performed in a preset knowledge graph based on the key information, and a first confidence level is determined according to the query results, including: identifying entities contained in the key information; querying the number of entities contained in the preset knowledge graph, and determining a first weight based on the number; determining the consistency identification result of the association relationship between the entities and the key information based on the knowledge graph, and determining a second weight based on the consistency identification result; determining the industry constraint judgment result based on the parameters of each entity contained in the key information and the preset industry constraints, and determining a third weight based on the industry constraint judgment result; determining a first confidence level based on the first weight, the second weight, and the third weight.

[0087] In an embodiment of the present application, the terminal can parse and identify the key information to obtain the entities contained in the key information, which can be used as the first entity set. Then, based on the first entity, the knowledge graph preset by the query is combined to determine the number of entities in the first entity set contained in the knowledge graph, and the first weight is determined according to the preset calculation rules and the number. In one example, the first weight can be determined based on the correspondence between the number and the weight. For example, the identified entities include "beer brand A", "500ml×24 cans" and "full discount". If all three entities exist in the knowledge graph, the highest weight is obtained. If there are two entities, the middle weight corresponds to the middle weight. If there is one entity or no entity exists, the lowest weight corresponds to the lowest weight. In another example, the ratio of the number to the total number of entities contained in the first entity set can be calculated to determine the first weight. The first weight and the ratio are positively correlated.

[0088] The terminal can also use the knowledge graph to determine the consistency between the entity association and key information, and determine a second weight based on the consistency identification results. For example, if the entity information is sub-brand A1, brand A, and the activity rule "amount discount", the knowledge graph can be used to determine whether there is a direct or indirect association between the three, obtain a consistency identification result, and then determine the second weight based on the consistency identification result. Specifically, the weights for each entity pair with a correct association can be added together to obtain the second weight. For example, if A1 and A are associated, the brand association is correct, resulting in a weight of 0.2. If brand A is associated with the activity rule "amount discount", the rule matches, resulting in a weight of 0.5. The two are added together to obtain the second weight. Optionally, more granular matching rules can be used for activity rules. For example, differentiating between activity type and activity parameter value. If both match, the corresponding weight is x. If only one of the two matches, the corresponding weight is y. If neither matches, the corresponding weight is z, with x > y > z.

[0089] The terminal can also determine whether the parameters of each entity included in the key information meet the preset industry constraints, obtain the industry constraint judgment result, and then determine the third weight based on the industry constraint judgment result. For each parameter, if the parameter meets the preset industry constraints, the preset weight is obtained; otherwise, the weight is 0. The terminal can then calculate the sum of the weights corresponding to each parameter to obtain the third weight.

[0090] The terminal may directly sum the first weight, the second weight, and the third weight to obtain the first confidence level. Alternatively, the terminal may perform a weighted sum of the first weight, the second weight, and the third weight to obtain the first confidence level.

[0091] Through this solution, the first confidence level of the key information relative to the knowledge graph can be determined based on the association relationships between entities and entity components, as well as industry constraint conditions. This can reflect the comprehensive reliability of the key information relative to the knowledge graph and effectively improve the accuracy of the first confidence level.

[0092] Optionally, determining a second confidence level according to the consistency discrimination result between the image recognition result and the text parsing result includes: determining a first entity set included in the image recognition result and a second entity set included in the text parsing result; determining a first consistency discrimination result based on the first entity set and the second entity set, and determining a second consistency discrimination result based on the parameters of each entity in the first entity set and the parameters of each entity in the second entity set; performing semantic consistency verification on the image recognition result and the text parsing result to obtain a third consistency discrimination result; and determining the second confidence level based on the first consistency discrimination result, the second consistency discrimination result, and the third consistency discrimination result.

[0093] In the embodiments of this application, the terminal can extract each entity set included in the image recognition result to obtain a first entity set, and the terminal can also extract each entity included in the text parsing result to obtain a second entity set. For example, the types of each entity include: sub-brand, brand, activity rule, alcohol content. The terminal can compare the first entity set and the second entity set to obtain a first consistency discrimination result. The first consistency discrimination result can represent the matching degree of the entities included in the first entity set and the second entity set. The terminal can determine a first consistency score based on the first consistency discrimination result. For example, each entity type corresponds to a score value and a deduction value. When the entities of this type match, the score value corresponding to this type is added to the current score (the initial score is 0). When the entities of this type do not match, the current score is subtracted by the deduction value corresponding to the type. For example, if the image recognition result includes the entity "Snowflake勇闯天涯superX" and the entity included in the text parsing result is the same, the current score is increased by 1 point; if the entity included in the image recognition result is the specification unit "tin" and does not match the entity "bottle" included in the text parsing result, the current score is subtracted by 0.3; if the entity "alcohol concentration" included in the image recognition result does not match the entity "juice concentration" included in the text parsing result, the current score is subtracted by 0.2. In another example, the first entity set and the second entity set can be input into a pre-trained artificial intelligence model to obtain a first consistency discrimination result, and the value of this first consistency discrimination result is the first consistency score.

[0094] For entities with parameter values, such as alcohol concentration, discount strength, etc., the terminal can determine the second consistency discrimination result based on the parameters of each entity in the first entity set and the parameters of each entity in the second entity set. Specifically, if the parameter values are the same, a score x is obtained; if the parameter values are different, a score of 0 is obtained; if it only exists in one recognition result, a score y is obtained. Among them, x > y. For example, if the parameter value of alcohol concentration in the image recognition result is 5%, and the alcohol concentration in the text parsing result is 3%, the second consistency score is 0; if they are the same, the second consistency score is 0.2; if the alcohol concentration does not exist in the text parsing result, the second consistency score is 0.1. It can be understood that if there are multiple entities with parameter values, the sum of the second consistency scores corresponding to each entity can be used as the final second consistency score.

[0095] The terminal can also perform semantic consistency verification on the image recognition result and the text parsing result through a semantic recognition model to obtain the third consistency discrimination result. The third consistency discrimination result represents the similarity degree of their semantics. The higher the semantic similarity, the higher the third consistency score. For example, semantic matching can be performed on the promotion rules. If the image recognition result and the text parsing result belong to the same type, such as (full reduction, gift, discount), the third consistency discrimination result is consistent. For example, if the image recognition result is "Reduce 20 when reaching 199" and the text parsing result is "Full reduction activity", the third consistency score corresponding to the third consistency discrimination result is a preset value (non-zero value); if the image recognition result and the text parsing result belong to different types, such as the image recognition result is "Send a customized wine glass" and the text parsing result is "Limited-time discount", then they do not match, and the third consistency score corresponding to the third consistency discrimination result is 0. Optionally, the semantic recognition model can also be used to identify the association relationship between each entity in the image recognition result and the text parsing result, and determine the industry knowledge association consistency based on the recognition result and the knowledge graph. For example, the product name needs to be correctly associated with the product line in the knowledge graph. For example, "Super X of YCWTY" is associated with "young people positioning". If "Super X of YCWTY" is associated with "middle-aged people" in the recognition result, the association is incorrect. The terminal can adjust the third consistency score according to the discrimination result of the industry knowledge association consistency. For example, if the association is correct, the third consistency score is increased; if the association is incorrect, the third consistency score is decreased.

[0096] The terminal can calculate the second confidence based on the first consistency score, the second consistency score, and the third consistency score. For example, the first consistency score, the second consistency score, and the third consistency score can be weighted and summed to obtain the second confidence.

[0097] Through this solution, the consistency judgment results of image recognition results and text parsing results can be identified from multiple dimensions such as entity consistency, entity parameter consistency and semantic consistency, and then the second confidence level can be determined based on the consistency judgment results, which can improve the accuracy of the second confidence level.

[0098] Optionally, the target web page data contains multiple content blocks; the method also includes: for each content block, generating a feature vector corresponding to the content block based on the key information contained in the content block; clustering each content block based on the feature vector corresponding to each content block to obtain at least one cluster; in the cluster, determining a target content block whose distance to the cluster center meets a preset proximity condition, and generating a content extraction template based on the target content block; the content extraction template is used to extract the key information contained in the web page data.

[0099] In an embodiment of the present application, for each content block, the terminal can generate a feature vector corresponding to the content block based on the key information contained in the content block. For example, the entity types include: brand, campaign rules, alcohol content, and product specifications. Accordingly, the feature vector can be a multidimensional vector containing each of the aforementioned entity types, and weights can be assigned to each type. For example, product specifications have a weight of 0.3, campaign rules have a weight of 0.25, alcohol content has a weight of 0.2, and brand has a weight of 0.15. The terminal can perform a weighted distance calculation based on the feature vectors and weights corresponding to each content block. Then, based on the weighted distance and a dynamic K value, the terminal uses a K-means clustering algorithm to cluster each content block, generating at least one cluster. The terminal can then identify target content blocks within the cluster whose distance to the cluster center meets a preset proximity condition. For example, a preset number of target content blocks with the closest distance to the cluster center can be selected. Furthermore, a content extraction template is generated based on the target content blocks. The content extraction template is used to extract key information contained in the webpage data. Optionally, the dynamic K value may be determined based on the elbow rule. For example, the elbow rule may be used to calculate an initial value, and then dynamically adjusted according to configuration information of a technician.

[0100] Through this solution, on the basis of traditional K-means clustering, through the feature weighting mechanism and dynamic threshold adjustment, the similarity measurement of data points is optimized according to the characteristics of the beer industry, and the accurate clustering of content blocks is achieved. At the same time, the weights of the features of each dimension are set, and the influence of key features on the kmeans clustering results is strengthened, which can further improve the accuracy of clustering, thereby obtaining effective target content blocks, and further improving the accuracy and effectiveness of content extraction templates.

[0101] Optionally, the web page extraction strategy includes a content extraction template, and the method further includes: if the target confidence is greater than or equal to a preset threshold, executing the step of outputting key information and target confidence; if the target confidence is less than the preset threshold, re-acquiring the content extraction template in the preset content extraction template library, and based on the re-acquired content extraction template, executing the step of extracting key information from the target web page data based on the web page extraction strategy.

[0102] In this embodiment of the present application, the terminal can determine whether the target confidence is less than a preset threshold. If the target confidence is greater than or equal to the preset threshold, the terminal outputs key information and the target confidence step. If the target confidence is less than the preset threshold, the terminal retrieves a content extraction template from a preset content extraction template library and then, based on the retrieved content extraction template, executes step 102 to extract key information from the target web page data based on the web page extraction strategy, as well as subsequent related steps. Optionally, the key information and target confidence can be output as structured data and stored in a database, such as in JSON format. If the confidence is low, the key information can be marked as pending review for further manual review.

[0103] Through this solution, when the target confidence is high, key information and target confidence can be output, thereby screening out key information with high confidence; when the target confidence is low, the template is reselected and then the key information is re-extracted, thereby improving the reliability and accuracy of the extracted key information.

[0104] like Figure 2 As shown, the embodiment of the present application provides an example of an information extraction method, including the following steps:

[0105] Step 201: Acquire initial webpage data of a target webpage.

[0106] Step 202 , performing tag filtering on the initial web page data, and performing text normalization processing on the tag-filtered initial web page data to obtain target web page data.

[0107] Step 203: Determine whether there is a content extraction template corresponding to the target webpage.

[0108] If yes, execute step 204 ; otherwise, execute step 205 .

[0109] Step 204: extract key information from the target web page data based on the content extraction template.

[0110] Step 205: extract key information from the target web page data based on the web page extraction rules.

[0111] Step 206: perform a query in a preset knowledge graph based on the key information, and determine a first confidence level according to the query result.

[0112] Step 207: Perform content recognition on the image in the key information to obtain an image recognition result, and perform text analysis on the text information in the key information to obtain a text analysis result. Determine a second confidence level based on the consistency judgment result between the image recognition result and the text analysis result.

[0113] Step 208: Perform a rationality check on the key information and determine a third confidence level based on the rationality check result.

[0114] Step 209: Determine a fourth confidence level based on the reuse rate of the content extraction template.

[0115] If there is no content extraction template, the reuse rate is 0 or the default value.

[0116] Step 210 : Determine a target confidence level corresponding to the key information based on the first confidence level, the second confidence level, the third confidence level, and a fourth confidence level corresponding to the webpage extraction strategy.

[0117] Step 211: determine whether the target confidence is greater than or equal to a preset threshold.

[0118] If yes, execute step 212; otherwise, execute step 213.

[0119] Step 212: Output key information and target confidence.

[0120] Step 213 : Re-acquire the content extraction template from the preset content extraction template library, and execute step 204 based on the re-acquired content extraction template.

[0121] It should be understood that, although the steps in the flowcharts of the above embodiments are shown in sequence as indicated by the arrows, these steps are not necessarily performed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be performed in other orders. Moreover, at least a portion of the steps in the flowcharts of the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily performed at the same time, but can be performed at different times. The execution order of these steps or stages is not necessarily to be performed in sequence, but can be performed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0122] Based on the same inventive concept, embodiments of the present application also provide an information extraction device for implementing the aforementioned information extraction method. The implementation solution provided by this device is similar to the implementation solution described in the aforementioned method. Therefore, the specific limitations of one or more information extraction device embodiments provided below can be found in the above-mentioned limitations of the information extraction method and will not be repeated here.

[0123] In an exemplary embodiment, Figure 3 As shown, an information extraction device is provided, comprising:

[0124] An extraction module 310 is used to extract key information from target web page data based on a web page extraction strategy;

[0125] A first determination module 320 is configured to perform a query in a preset knowledge graph based on the key information and determine a first confidence level according to the query result;

[0126] A second determination module 330 is configured to perform content recognition on the image in the key information to obtain an image recognition result, perform text parsing on the text information in the key information to obtain a text parsing result, and determine a second confidence level based on a consistency determination result between the image recognition result and the text parsing result;

[0127] A third determination module 340 is configured to perform a rationality check on the key information and determine a third confidence level based on the rationality check result;

[0128] The fourth determination module 350 is configured to determine a target confidence corresponding to the key information based on the first confidence, the second confidence, the third confidence, and a fourth confidence corresponding to the web page extraction strategy, and output the key information and the target confidence.

[0129] In one embodiment, the first determining module is specifically configured to:

[0130] identifying the entities included in the key information;

[0131] Querying the number of entities contained in a preset knowledge graph, and determining a first weight according to the number;

[0132] Determine, based on the knowledge graph, a consistency recognition result between the association relationship of the entity and the key information, and determine a second weight based on the consistency recognition result;

[0133] Determining an industry constraint discrimination result based on the parameters of each entity included in the key information and preset industry constraint conditions, and determining a third weight based on the industry constraint discrimination result;

[0134] A first confidence level is determined based on the first weight, the second weight, and the third weight.

[0135] In one embodiment, the second determining module is specifically configured to:

[0136] Determining a first entity set included in the image recognition result and a second entity set included in the text parsing result;

[0137] Determining a first consistency determination result based on the first entity set and the second entity set, and determining a second consistency determination result based on parameters of each entity in the first entity set and parameters of each entity in the second entity set;

[0138] Performing a semantic consistency check on the image recognition result and the text parsing result to obtain a third consistency determination result;

[0139] A second confidence level is determined according to the first consistency determination result, the second consistency determination result, and the third consistency determination result.

[0140] In one embodiment, the target web page data includes a plurality of content blocks; the apparatus further includes:

[0141] A generating module, configured to generate, for each content block, a feature vector corresponding to the content block according to key information contained in the content block;

[0142] A clustering module, configured to perform clustering processing on each of the content blocks based on a feature vector corresponding to each of the content blocks to obtain at least one cluster;

[0143] The fifth determination module is used to determine, in the cluster cluster, a target content block whose distance to the cluster center meets a preset proximity condition, and generate a content extraction template based on the target content block; the content extraction template is used to extract key information contained in the web page data.

[0144] In one embodiment, the webpage extraction strategy includes a content extraction template, and the apparatus further includes:

[0145] A first processing module, configured to execute the step of outputting the key information and the target confidence if the target confidence is greater than or equal to a preset threshold;

[0146] The second processing module is used to re-acquire the content extraction template from the preset content extraction template library if the target confidence is less than the preset threshold, and execute the web page extraction strategy based on the re-acquired content extraction template to extract key information from the target web page data.

[0147] In one embodiment, the apparatus further comprises:

[0148] An acquisition module, used to acquire initial web page data of a target web page;

[0149] The filtering module is used to perform tag filtering on the initial web page data and perform text normalization processing on the tag-filtered initial web page data to obtain target web page data.

[0150] Each module in the above-mentioned information extraction device can be implemented in whole or in part through software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in the computer device in the form of software, so that the processor can call and execute the corresponding operations of each module.

[0151] In an exemplary embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as shown in FIG. Figure 4 As shown. The computer device includes a processor, memory, an input / output interface, a communication interface, a display unit, and an input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with external terminals via wired or wireless means, and the wireless means can be implemented via Wi-Fi, a mobile cellular network, near-field communication (NFC), or other technologies. When executed by the processor, the computer program implements an information extraction method. The display unit of the computer device is used to form a visually visible image, and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device casing, or an external keyboard, touchpad or mouse.

[0152] Those skilled in the art will understand that Figure 4The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0153] In an exemplary embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps of the above-mentioned information extraction method when executing the computer program.

[0154] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned information extraction method are implemented.

[0155] In one embodiment, a computer program product is provided, comprising a computer program, which implements the steps of the above-mentioned information extraction method when executed by a processor.

[0156] It should be noted that the user information (including but not limited to user device identification, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0157] Those skilled in the art will understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. In particular, any reference to memory, database, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The databases involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the various embodiments provided herein may be, but are not limited to, general-purpose processors, central processing units (CPUs), graphics processing units (GPUs), digital signal processors (DSPs), programmable logic devices (PLDs), quantum computing-based data processing logic devices, artificial intelligence (AI) processors, and the like.

[0158] The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0159] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. An information extraction method, characterized in that: The method comprises: Extract key information from target web page data based on web page extraction strategy; Performing a query in a preset knowledge graph based on the key information, and determining a first confidence level according to the query result; Performing content recognition on the image in the key information to obtain an image recognition result, and performing text parsing on the text information in the key information to obtain a text parsing result, and determining a second confidence level based on a consistency judgment result between the image recognition result and the text parsing result; Performing a rationality check on the key information, and determining a third confidence level based on the rationality check result; Based on the first confidence, the second confidence, the third confidence, and a fourth confidence corresponding to the web page extraction strategy, a target confidence corresponding to the key information is determined, and the key information and the target confidence are output.

2. The method according to claim 1, characterized in that The querying in a preset knowledge graph based on the key information and determining a first confidence level according to the query result includes: identifying the entities included in the key information; Querying the number of entities contained in a preset knowledge graph, and determining a first weight according to the number; Determine, based on the knowledge graph, a consistency recognition result between the association relationship of the entity and the key information, and determine a second weight based on the consistency recognition result; Determining an industry constraint discrimination result based on the parameters of each entity included in the key information and preset industry constraint conditions, and determining a third weight based on the industry constraint discrimination result; A first confidence level is determined based on the first weight, the second weight, and the third weight.

3. The method according to claim 1, characterized in that The determining of the second confidence level according to the consistency judgment result between the image recognition result and the text parsing result includes: Determining a first entity set included in the image recognition result and a second entity set included in the text parsing result; Determining a first consistency determination result based on the first entity set and the second entity set, and determining a second consistency determination result based on parameters of each entity in the first entity set and parameters of each entity in the second entity set; Performing a semantic consistency check on the image recognition result and the text parsing result to obtain a third consistency determination result; A second confidence level is determined according to the first consistency determination result, the second consistency determination result, and the third consistency determination result.

4. The method according to claim 1, wherein The target web page data includes multiple content blocks; the method further includes: For each content block, generating a feature vector corresponding to the content block according to key information contained in the content block; performing clustering processing on each of the content blocks based on the feature vector corresponding to each of the content blocks to obtain at least one cluster; In the cluster, a target content block whose distance to the cluster center meets a preset proximity condition is determined, and a content extraction template is generated based on the target content block; the content extraction template is used to extract key information contained in the web page data.

5. The method according to claim 1, wherein The webpage extraction strategy includes a content extraction template, and the method further includes: If the target confidence is greater than or equal to a preset threshold, executing the step of outputting the key information and the target confidence; If the target confidence is less than a preset threshold, a content extraction template is retrieved from a preset content extraction template library, and based on the retrieved content extraction template, the web page extraction strategy is executed to extract key information from the target web page data.

6. The method according to claim 1, characterized in that The method further comprises: Obtaining initial web page data of the target web page; The initial web page data is subjected to tag filtering, and the initial web page data after tag filtering is subjected to text normalization processing to obtain target web page data.

7. An information extraction device, characterized in that: The device comprises: The extraction module is used to extract key information from the target web page data based on the web page extraction strategy; A first determination module is configured to perform a query in a preset knowledge graph based on the key information and determine a first confidence level according to the query result; a second determination module, configured to perform content recognition on the image in the key information to obtain an image recognition result, perform text parsing on the text information in the key information to obtain a text parsing result, and determine a second confidence level based on a consistency judgment result between the image recognition result and the text parsing result; a third determination module, configured to perform a rationality check on the key information and determine a third confidence level based on a rationality check result; The fourth determination module is configured to determine a target confidence corresponding to the key information based on the first confidence, the second confidence, the third confidence, and a fourth confidence corresponding to the web page extraction strategy, and output the key information and the target confidence.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images