Multi-source parameter page parallel acquisition and quality fusion terminal attribute image construction method

Abstract

The application discloses a terminal attribute portrait construction method for multi-source parameter page parallel collection and quality fusion, receives a terminal identifier and a terminal type, loads an attribute field set and a semantic extraction template, recalls candidate parameter pages and filters out target parameter pages, collects the target parameter pages in parallel, obtains cleaned parameter text through parameter page navigation and content cleaning, and evaluates a page quality score; the semantic extraction template and the cleaned parameter text are input into a semantic extraction model, a target configuration item is determined from an extraction result as a target attribute result, a comprehensive quality score is calculated according to the page quality score and a field filling rate, a highest-score result is taken as a main attribute result, and the remaining results are used to supplement empty fields to obtain a fused attribute result; and the fused attribute result is associated with the terminal identifier, the terminal type, a page address, the comprehensive quality score and a field supplement record to generate a terminal attribute portrait record. The application realizes batch automatic construction of a terminal attribute portrait, improves field completeness and traceability.

Description

Technical Field

[0001] This invention relates to the field of network terminal management and terminal attribute recognition technology, and in particular to a method for constructing a terminal attribute profile by parallel acquisition and quality fusion of multi-source parameter pages. Background Technology

[0002] With the deepening of network operation, terminal capability identification, and user profile construction, there is a need to maintain a network terminal attribute database, which centrally stores multi-dimensional attributes such as brand, model, CPU, memory, operating system, network standard, Wi-Fi protocol, and battery capacity for various terminal models, including mobile phones, routers, and personal computers. The number of network terminal models is large and updates rapidly; a single series often contains multiple configuration versions, making batch completion and continuous maintenance of the terminal attribute database a common requirement. Current terminal attribute maintenance mainly employs methods such as manual querying and input, fixed-site crawling, fixed-template parsing, and single-page semantic extraction. Manual querying involves maintenance personnel searching web pages based on the terminal model and comparing and inputting attributes one by one; fixed-site crawling and fixed-template parsing involve writing dedicated crawling scripts or field mapping rules for the specific web page structure of a particular website; single-page semantic extraction selects a page from the search results, inputs the page text into a semantic model, and extracts attribute fields. These methods are still feasible for a small number of models and a fixed web page structure. However, in scenarios involving the automatic completion of large-scale terminal attribute databases, the above methods suffer from the following problems: First, manual maintenance is inefficient, as different terminal types require vastly different sets of attribute fields, making manual processing of each field costly and preventing batch automatic completion. Second, search results are often mixed with invalid pages such as encyclopedias, Q&A sites, forums, advertisements, document downloads, and official website homepages, resulting in inaccurate parameter page positioning and significant differences in webpage structures, making it difficult to adapt dynamic tabs, parameter subpages, and redirected pages using fixed rules. Third, single-source pages often lack some fields or do not match the target model, and there is a lack of page quality assessment mechanisms, leading to missing or incorrect fields after extraction from low-quality pages containing a lot of noise. Fourth, when the same page contains multiple configuration versions of the same series, it is easy to mistakenly extract non-target configurations as target model results. Fifth, existing methods lack multi-source quality fusion and standardized profile construction mechanisms, and the extraction results lack unified field constraints, making it impossible to select the best from multiple extraction results based on source quality, complete missing fields, and record the source of completion, thus failing to form a traceable and database-compatible terminal attribute profile. Summary of the Invention

[0003] In view of the above problems, a terminal attribute profile construction method based on parallel acquisition and quality fusion of multi-source parameter pages is proposed to overcome or at least partially solve the above problems, including:

[0004] Receive terminal identifier and terminal type, load attribute field set and semantic extraction template based on terminal type; recall candidate parameter pages based on terminal identifier, and filter out target parameter pages from candidate parameter pages;

[0005] The target parameter page is collected in parallel, and parameter page navigation and content cleaning are performed to obtain the cleaned parameter text corresponding to the target parameter page; the page quality of the cleaned parameter text is evaluated to obtain the page quality score corresponding to the cleaned parameter text;

[0006] Input the semantic extraction template and cleaned parameter text into the semantic extraction model to obtain the attribute extraction results; determine the target configuration item that matches the terminal identifier from the attribute extraction results, and use it as the target attribute result corresponding to the target parameter page;

[0007] Based on the page quality score and the field fill rate calculated based on the attribute field set, a comprehensive quality score is calculated for the target attribute results; the target attribute result with the highest comprehensive quality score is determined as the main attribute result, and the field values ​​of the target attribute results other than the main attribute result are added to the missing fields of the main attribute result to obtain the fused attribute result;

[0008] The fusion attribute results are associated with the terminal identifier, terminal type, target parameter page address, comprehensive quality score, and supplementary field records to generate network terminal attribute profile records.

[0009] Optionally, the attribute field set includes attribute fields corresponding to the terminal type. The attribute fields include at least one of the following: brand, model, price, launch date, central processing unit, memory, hard disk, operating system, network standard, Wi-Fi protocol, Wi-Fi frequency band, Bluetooth version, battery capacity, screen size, screen refresh rate, and number of multiple inputs and multiple outputs.

[0010] Semantic extraction templates are used to constrain the semantic extraction model to output structured fields according to the set of attribute fields, and to limit the field format, allowed inference rules, and null value handling rules.

[0011] Optionally, the candidate parameter page recalled based on the terminal identifier includes:

[0012] Generate search requests based on terminal identifiers;

[0013] Use search requests to access search engine results pages, product parameter databases, manufacturer websites, and third-party specification sites, and collect candidate parameter pages from these sources.

[0014] Save the title, address, and source information of the candidate parameter page.

[0015] Optionally, filtering the target parameter page from the candidate parameter pages includes:

[0016] The candidate parameter page is filtered, including blacklist domain filtering, document link filtering, shallow path link filtering, ad redirection link filtering, and duplicate link filtering.

[0017] After filtering, the title and address information of the candidate parameter pages are input into the semantic selection model, and the semantic selection model outputs the priority of the candidate parameter pages.

[0018] Select the highest priority preset number of candidate parameter pages as the target parameter pages.

[0019] Optionally, the parallel acquisition target parameter page includes:

[0020] Create a separate data acquisition task for each target parameter page;

[0021] Parameter page navigation is performed in independent data collection tasks. Parameter page navigation includes direct crawling, parameter link extraction, browser-automated triggering, and fallback crawling.

[0022] When the quality of the text obtained directly from crawling is lower than the first navigation quality threshold, links containing parameter keywords are extracted and crawled a second time.

[0023] When the quality of the text obtained from the secondary crawl is lower than the second navigation quality threshold, the browser will automatically trigger the parameter-related elements on the page.

[0024] When the quality of the text obtained by the browser's automated trigger is lower than the third navigation quality threshold, the text obtained directly from the crawl is used as the fallback parameter content.

[0025] Optional content cleaning includes:

[0026] Extract the page title and page source URL;

[0027] Remove image links, navigation text, footer text, advertising text, customer service text, copyright text, and irrelevant links;

[0028] Locate the beginning position of the parameter content based on the starting keyword of the parameter, and locate the end position of the parameter content based on the ending keyword of the parameter.

[0029] Use the text between the start and end positions of the parameter content as the cleaned parameter text;

[0030] The starting keywords for the parameters include “configuration filter”, “model”, “specification parameters”, “detailed parameters”, “important parameters”, “hardware specifications”, “technical specifications”, “basic parameters”, and “product parameters”.

[0031] The parameters end with keywords including "related recommendations", "same series products", "precautions", "all rights reserved" and "Copyright".

[0032] Optional, page quality assessment includes:

[0033] The content completeness score is calculated based on the effective text length and effective number of lines in the cleaned text.

[0034] The degree of structuring is scored based on the number of table rows, key-value pair rows, or structured rows in the cleaning parameter text.

[0035] The terminal matching score is calculated based on the similarity between the terminal identifier and the cleaning parameter text.

[0036] Page quality scores and page quality levels are generated based on content completeness scores, structure scores, and terminal compatibility scores.

[0037] Page quality ratings include GOOD, WEAK, and BAD.

[0038] When the page quality level is GOOD, the cleaning parameter text will be marked as normal semantic extraction text.

[0039] When the page quality level is WEAK, the cleaning parameter text will be marked as low-confidence semantic extraction text.

[0040] When the page quality level is BAD, retain the cleaning parameter text for backtracking.

[0041] Optionally, a page quality score and page quality level can be generated based on content completeness score, structured content score, and terminal compatibility score, including:

[0042] The page quality score is obtained by weighting and summing the content completeness score, the structure score, and the terminal matching score.

[0043] The weighted summation uses the first weight coefficient, the second weight coefficient, and the third weight coefficient, and the sum of the first weight coefficient, the second weight coefficient, and the third weight coefficient is 1;

[0044] When the content completeness score is lower than the first preset threshold, the structure score is lower than the second preset threshold, or the terminal matching score is lower than the third preset threshold, the page quality level will be set to BAD level.

[0045] When the content completeness score reaches the first preset threshold, the structuring score reaches the second preset threshold, and the terminal matching score reaches the third preset threshold, the page quality level is set to GOOD or WEAK based on the relationship between the page quality score and the fourth preset threshold.

[0046] Optionally, the semantic extraction template and cleaned parameter text are input into the semantic extraction model to obtain attribute extraction results; the target configuration item matching the terminal identifier is determined from the attribute extraction results as the target attribute result corresponding to the target parameter page, including:

[0047] Combine the semantic extraction template and the cleaned parameter text into an extraction request;

[0048] The semantic extraction model outputs attribute extraction results based on the extraction request. The attribute extraction results are an array, and each element in the array represents a configuration item.

[0049] Calculate the text similarity between the terminal identifier and the Chinese name field in the configuration item, the text similarity between the terminal identifier and the detailed model field in the configuration item, and the text similarity between the terminal identifier and the model field in the configuration item;

[0050] The text similarity between the terminal identifier and the configuration item is determined based on the text similarity of the Chinese name field, the detailed model field, and the model field.

[0051] The text similarity between the terminal identifier and the configuration item is corrected by combining the model code matching result and the year matching result, and the corrected text similarity is obtained.

[0052] The configuration item with the highest text similarity after correction is identified as the target configuration item;

[0053] When multiple configuration items have the same text similarity after correction, the target configuration item is determined by sorting them according to field completeness and price information.

[0054] The determined target configuration items are used as the target attribute results.

[0055] Optionally, a comprehensive quality score is calculated for the target attribute results based on the page quality score and the field fill rate calculated based on the attribute field set; the target attribute result with the highest comprehensive quality score is determined as the main attribute result, and the field values ​​from the target attribute results other than the main attribute result are used to fill the missing fields in the main attribute result to obtain the fused attribute result, including:

[0056] The field fill rate is the ratio between the number of non-empty fields in the target attribute result and the total number of fields in the attribute field set;

[0057] The page quality score is normalized to obtain a normalized page quality score.

[0058] The overall quality score is calculated using the following formula:

[0059] Q = a × S + b × F

[0060] Where Q is the overall quality score, S is the normalized page quality score, F is the field fill rate, a and b are weight coefficients, and a+b=1;

[0061] The target attribute with the highest overall quality score is determined as the primary attribute.

[0062] When there are missing fields in the main attribute results, the non-null values ​​of the corresponding fields are searched in descending order of the overall quality scores of the other target attribute results.

[0063] Fill the missing fields in the main attribute result with the found non-null values;

[0064] The source target parameter page, source collection task, source page quality information, and source comprehensive quality score will be added as supplementary field records.

[0065] The supplemented main attribute result is used as the fused attribute result.

[0066] This invention automates the entire process of "multi-source recall, candidate screening, multi-path parallel acquisition, page quality assessment, semantic extraction, quality fusion and profile construction," transforming the completion of the terminal attribute library from manual querying, comparison and entry to batch automatic execution. Different terminal types achieve field constraints by loading corresponding attribute field sets and semantic extraction templates, eliminating the need for individual customization for each model. This significantly reduces operation and maintenance costs and improves the efficiency of terminal attribute library completion.

[0067] This invention employs rule-based filtering and semantic selection of candidate parameter pages. First, it filters out definitively invalid pages using blacklisted domains, documents, shallow paths, ad redirects, and duplicate links. Then, a semantic selection model sorts the remaining candidates and selects multiple target parameter pages. This excludes pages such as encyclopedias, Q&A sites, forums, advertisements, document downloads, and official website homepages from the data collection process, improving the accuracy of parameter page location. Furthermore, for complex webpage structures such as overview pages, parameter subpages, dynamic tabs, and redirect pages, this invention utilizes a multi-level navigation mechanism consisting of direct crawling, parameter link extraction, automated browser triggering, and fallback crawling. Combined with content cleaning based on parameter start and end keywords, it can locate and extract parameter text from pages with different structures, reducing the impact of webpage structure differences on the extraction results.

[0068] This invention performs page quality assessment on the cleaned parameter text before semantic extraction, generating a page quality score based on three dimensions: content completeness, structuring degree, and terminal matching degree. This distinguishes low-quality and low-confidence pages and restricts them from entering the normal extraction process, reducing the generation of erroneous and empty fields from the source. When a parameter page contains multiple configuration versions of the same series, the text similarity between the terminal identifier and each configuration item in the Chinese name, detailed model, and model fields is calculated. This is then corrected by combining model code matching and year matching. If the text similarity is the same after correction, it is further sorted according to field completeness and price information, thereby determining the target configuration item that best matches the input terminal identifier, reducing the risk of misselecting multiple configurations in the same series.

[0069] To address the issue of missing fields from a single source, this invention does not simply use a single page result. Instead, it measures each target attribute result using a comprehensive quality score Q = a × S + b × F. Here, the normalized page quality score S reflects the reliability of the source page, and the field completion rate F reflects the completeness of the extracted results. This weighted average ensures that the main attribute result comes from a higher-quality source, while also supplementing missing fields in the main attribute result using non-empty fields from other sources, in descending order of comprehensive quality score. The source target parameter page, source acquisition task, source page quality information, and source comprehensive quality score are recorded for each supplemented field, making field completion traceable. Finally, the fused attribute result is associated with the terminal identifier, terminal type, target parameter page address, comprehensive quality score, and field supplementation record to form a standardized terminal attribute profile record. This record can be directly written into the terminal attribute database for automatic field completion, or output to downstream processes such as device capability assessment and operator user profile construction, achieving end-to-end traceability of terminal attribute data from collection, extraction, fusion to database storage. Attached Figure Description

[0070] To more clearly illustrate the technical solution of the present invention, the accompanying drawings used in the description of the present invention will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0071] Figure 1 This is a flowchart of the terminal attribute profile construction method based on parallel acquisition and quality fusion of multi-source parameter pages provided in this embodiment of the invention;

[0072] Figure 2 This is a flowchart of parallel acquisition and quality fusion provided in an embodiment of the present invention;

[0073] Figure 3 This is a schematic diagram of the quality scoring and fusion rules provided in an embodiment of the present invention. Detailed Implementation

[0074] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0075] Reference Figures 1 to 3 This invention provides a method for constructing a terminal attribute profile through parallel acquisition and quality fusion of multi-source parameter pages, which may specifically include:

[0076] Receive terminal identifier and terminal type, load attribute field set and semantic extraction template based on terminal type; recall candidate parameter pages based on terminal identifier, and filter out target parameter pages from candidate parameter pages;

[0077] The target parameter page is collected in parallel, and parameter page navigation and content cleaning are performed to obtain the cleaned parameter text corresponding to the target parameter page; the page quality of the cleaned parameter text is evaluated to obtain the page quality score corresponding to the cleaned parameter text;

[0078] Input the semantic extraction template and cleaned parameter text into the semantic extraction model to obtain the attribute extraction results; determine the target configuration item that matches the terminal identifier from the attribute extraction results, and use it as the target attribute result corresponding to the target parameter page;

[0079] Based on the page quality score and the field fill rate calculated based on the attribute field set, a comprehensive quality score is calculated for the target attribute results; the target attribute result with the highest comprehensive quality score is determined as the main attribute result, and the field values ​​of the target attribute results other than the main attribute result are added to the missing fields of the main attribute result to obtain the fused attribute result;

[0080] The fusion attribute results are associated with the terminal identifier, terminal type, target parameter page address, comprehensive quality score, and supplementary field records to generate network terminal attribute profile records.

[0081] This invention provides a method for constructing terminal attribute profiles based on parallel acquisition of multi-source parameter pages, semantic extraction, and quality fusion. This method can be deployed on computing devices within terminal attribute library auto-completion platforms, terminal profile database platforms, device capability assessment platforms, or operator user profile platforms. The method in this embodiment generally includes processing stages such as input and terminal type configuration, candidate parameter page recall and filtering, parallel acquisition of multiple target parameter pages, semantic extraction and multi-configuration matching, quality fusion, and generation of network terminal attribute profile records.

[0082] Specifically, the computing device receives the terminal identifier and terminal type of the network terminal to be processed. The terminal identifier is used to distinguish the network terminal to be processed, and can be a combination of terminal model, product name, brand model, or device code. The terminal type is used to distinguish the category of the terminal device, and can be mobile phone, tablet computer, personal computer, router, smart TV, set-top box, smart speaker, smart camera, etc. The computing device loads the attribute field set and semantic extraction template corresponding to the terminal type based on the terminal type. The attribute field set specifies the attribute fields that need to be extracted for the corresponding terminal type. The attribute field set and semantic extraction template are maintained through a configuration file. When adding a new terminal type or attribute field, configuration items can be added or adjusted in the configuration file without changing the core processing flow. The computing device parses the loaded attribute field set, removes empty fields and maintains the field order. The field order is used to ensure consistent field arrangement when structured output, table export, and profile record storage. The computing device fills the fields in the attribute field set into the semantic extraction template to generate an extraction request template corresponding to the terminal type. The semantic extraction template is used to constrain the semantic extraction model to output structured fields according to the attribute field set.

[0083] After completing the input and terminal type configuration, the computing device recalls candidate parameter pages based on the terminal identifier. Candidate parameter pages refer to web pages on the internet that contain terminal hardware parameters, specifications, technical parameters, product details, network capabilities, or wireless capabilities. These candidate parameter pages can come from manufacturer websites, product parameter databases, third-party specification sites, and search engine results pages. The computing device generates a search request based on the terminal identifier and collects candidate parameter pages from multiple network information sources, forming a candidate parameter page set. The computing device then filters out target parameter pages from these candidate parameter pages. Target parameter pages are those candidate parameter pages that are more likely to contain detailed terminal parameter content. The filtering process removes candidate parameter pages that are definitively unsuitable for parameter extraction and performs semantic selection on the remaining candidate parameter pages, determining those more likely to contain detailed terminal parameter content as target parameter pages. The computing device selects multiple target parameter pages, leveraging the complementarity between different sources to reduce the impact of missing fields from a single source or mismatches between page content and terminal identifiers on the final results.

[0084] After identifying the target parameter page, the computing device collects data from it in parallel. The computing device creates an independent collection task for each target parameter page, and multiple collection tasks are executed in parallel. Each collection task uses an independent browser page context, browser instance, or collection session to avoid state conflicts between multiple collection tasks. Each collection task independently performs parameter page navigation, content cleaning, page quality assessment, and semantic extraction, and saves intermediate data such as the target parameter page address, cleaned parameter text, page quality score, and attribute extraction results for the corresponding collection task. In batch processing scenarios, the computing device can perform concurrent processing at the terminal identifier level, with multiple terminal identifiers entering the processing flow simultaneously, and each terminal identifier further performing parallel collection of multiple target parameter pages.

[0085] Parameter page navigation is used to adapt to parameter pages with different structures, such as overview pages, parameter subpages, dynamic tabs, and jump pages. When the directly obtained page content does not contain sufficient parameter information, the collection task can further obtain the page corresponding to the parameter link, or trigger parameter-related elements on the page, ultimately locating the page containing detailed parameter content. Content cleaning is used to remove information irrelevant to terminal attributes, such as image links, navigation content, footer content, advertising content, and copyright content, from the crawled page content, and to locate the parameter content range, using the text within the parameter content range as the cleaned parameter text. Cleaned parameter text refers to the parameter text obtained after content cleaning that is more suitable for semantic extraction. Each collection task obtains the cleaned parameter text corresponding to the target parameter page.

[0086] The computing device performs page quality assessment on the cleaning parameter text, obtaining a page quality score corresponding to the cleaning parameter text. The page quality assessment measures whether the target parameter page contains sufficient parameter information, whether the page content is structured, and whether the page content matches the terminal identifier. The computing device calculates scores from three aspects: the completeness of the cleaning parameter text content, the degree of structure, and the terminal matching degree, and then combines these three scores into a single page quality score. This page quality score serves as one of the bases for calculating the overall quality score during the quality fusion stage.

[0087] The computing device combines the semantic extraction template and the cleaned parameter text into an extraction request, inputs the extraction request into the semantic extraction model, and obtains the attribute extraction results. The semantic extraction model can be a pre-trained semantic model or an information extraction model, capable of extracting structured attribute data from the cleaned parameter text based on the field constraints in the extraction request. The attribute extraction results can be a structured array, where each element represents a configuration item, and each configuration item includes the field values ​​of some or all fields in the attribute field set. When the page content of the target parameter page contains multiple configuration versions of the same series, multiple configuration items will appear in the attribute extraction results. The computing device determines the target configuration item that matches the terminal identifier from the attribute extraction results and uses the target configuration item as the target attribute result corresponding to the target parameter page. When determining the target configuration item, the computing device calculates the similarity between the terminal identifier and the model information of each configuration item, and corrects it by combining the model code matching result and the year matching result, determining the configuration item with the highest corrected text similarity as the target configuration item. When the corrected text similarity of multiple configuration items is the same, the computing device can further determine the target configuration item by combining field completeness and price information. After semantic extraction, each target parameter page corresponds to a target attribute result.

[0088] After obtaining multiple target attribute results, the computing device performs quality fusion. The computing device calculates the field fill rate for each target attribute result based on the attribute field set. The field fill rate refers to the ratio between the number of non-empty fields in the target attribute result and the total number of fields in the attribute field set. The computing device calculates a comprehensive quality score for each target attribute result based on the page quality score of the target parameter page and the field fill rate of the target attribute result. The comprehensive quality score reflects both the credibility of the source page and the completeness of the target attribute result. The computing device determines the target attribute result with the highest comprehensive quality score as the primary attribute result and uses the target attribute results other than the primary attribute result as supplementary sources. When there are missing fields in the primary attribute result, the computing device searches for the corresponding non-empty field values ​​in other target attribute results, in descending order of their comprehensive quality scores, and adds these non-empty field values ​​to the missing fields in the primary attribute result. The source information of each supplemented field is recorded, forming a field supplementation record. The primary attribute result after supplementation is the fused attribute result. Attribute fields that could not be obtained from any target parameter page remain empty in the fused attribute result.

[0089] The processing steps in the association phase are as follows. The computing device associates the fused attribute results with the terminal identifier, terminal type, target parameter page address, overall quality score, and supplementary field records to generate a network terminal attribute profile record. A network terminal attribute profile record is a structured record that organizes information such as the terminal identifier, terminal type, multi-dimensional attributes, source page, quality score, and supplementary field records of a network terminal, used to describe the terminal's capabilities and characteristics.

[0090] The computing device constructs a network terminal attribute profile record using the fused attribute results as the primary data. The network terminal attribute profile record is organized using key-value pairs in a structured data format, including attribute fields and associated information. The attribute fields consist of each attribute field and its corresponding value from the fused attribute results. The field order in the attribute fields section is consistent with the field order maintained in the attribute field set to ensure consistent field placement during structured output, table export, and database entry of the profile record. The associated information section includes a terminal identifier field, a terminal type field, a target parameter page address field, a comprehensive quality score field, and a supplementary record field.

[0091] The Terminal Identifier field records the terminal identifier input in this embodiment, used to identify the network terminal corresponding to the network terminal attribute profile record. The Terminal Type field records the terminal type input in this embodiment, used to identify the terminal category corresponding to the network terminal attribute profile record. The Target Parameter Page Address field records the address of the target parameter page involved in the data collection. When one terminal identifier corresponds to multiple target parameter pages, the value of the Target Parameter Page Address field can be a list of addresses, where each address in the list corresponds to one target parameter page. The Comprehensive Quality Score field records the comprehensive quality score corresponding to the main attribute result, used to characterize the overall credibility of the main part of the attribute fields in the network terminal attribute profile record. The Field Supplement Record field records the source information of each supplemented field in the fused attribute result. The Field Supplement Record allows each supplemented field value to be traced back to its corresponding supplementary source. Through the above association, the source of each field value in the network terminal attribute profile record can be traced back. The field values ​​in the main attribute result can be traced back to the target parameter page corresponding to the main attribute result, and the field values ​​of the supplemented fields can be traced back to the supplementary source recorded in the Field Supplement Record.

[0092] After generating network terminal attribute profile records, the computing device writes these records into the terminal attribute database to automatically complete missing fields. These records can also be output to terminal profile database entry processes, device capability assessment processes, or operator-side terminal user profile processes. When writing network terminal attribute profile records into the terminal attribute database, the attribute fields correspond to the attribute field columns in the database, and the association information corresponds to the association information columns. When processing multiple terminal identifiers in batches, each terminal identifier generates one network terminal attribute profile record. Multiple records form a batch result, which the computing device then writes into the terminal attribute database or outputs to downstream processes in batches.

[0093] Taking a router-type network terminal as an example, after the computing device inputs the terminal identifier and router terminal type of the router, a network terminal attribute profile record is generated. The value of the terminal identifier field is the input terminal identifier, the value of the terminal type field is router, the target parameter page address field records the target parameter page address that was collected, the attribute fields include model, Wi-Fi protocol, Wi-Fi frequency band, Wi-Fi total rate, number of multiple inputs and multiple outputs, number of antennas, release date and other attribute fields and their corresponding values, the comprehensive quality score field records the comprehensive quality score corresponding to the main attribute result, and the field supplement record field records the attribute fields and source information supplemented by other sources in the profile record.

[0094] In one or more embodiments of the present invention, the attribute field set includes attribute fields corresponding to the terminal type. The attribute fields include at least one of the following: brand, model, price, launch date, central processing unit, memory, hard disk, operating system, network standard, Wi-Fi protocol, Wi-Fi frequency band, Bluetooth version, battery capacity, screen size, screen refresh rate, and number of multiple inputs and multiple outputs.

[0095] Semantic extraction templates are used to constrain the semantic extraction model to output structured fields according to the set of attribute fields, and to limit the field format, allowed inference rules, and null value handling rules.

[0096] Different terminal types may have different sets of attribute fields, while different models of the same terminal type may share the same set of attribute fields. The set of attribute fields defines the range of fields that need to be extracted and output for the corresponding terminal type, and serves as the basis for subsequent field fill rate calculations, structured output, and database entry of profile records.

[0097] For personal computers, the attribute field set can include Chinese name, brand, detailed model, price, release date, CPU, CPU clock speed, maximum turbo frequency, number of cores / threads, memory capacity, hard drive capacity, screen size, wireless network card, network speed, Bluetooth, and Wi-Fi band. For mobile phones, the attribute field set can include brand, Chinese name, detailed model, operating system, price, whether it's a foldable screen, number of cameras, main camera pixels, screen size, screen resolution, screen refresh rate, RAM, internal storage, CPU model, network standard, Wi-Fi protocol, maximum bandwidth, number of MIMOs, maximum Wi-Fi speed, whether fast charging is supported, battery capacity, number of SIM card slots, release date, and Wi-Fi band. For routers, the attribute field set can include model, WAN port speed, LAN port speed, Wi-Fi protocol, Wi-Fi band, number of MIMOs, number of antennas, and release date. The above field examples are used to illustrate the differences in fields for different terminal types. During implementation, attribute fields can be added or deleted according to business needs. The order of fields in the attribute field set is maintained after parsing to ensure consistent field positions when structured output, table export, and profile record entry into the database.

[0098] The attribute field set and semantic extraction template are maintained in pairs in the configuration file. The configuration file is indexed by terminal type, with each terminal type corresponding to one attribute field set record and one semantic extraction template record. Each attribute field in the attribute field set record can carry field format constraints, allowed estimation rules, and null value handling rules. The specific content of these three rules is uniformly written into the semantic extraction template and takes effect during the extraction phase. When adding a new terminal type or a new attribute field, a terminal type record is added to the configuration file or an attribute field item is added to the attribute field set record for that terminal type; there is no need to modify the recall, collection, evaluation, and fusion processing flow.

[0099] The semantic extraction template is used to constrain the semantic extraction model to output structured fields according to the attribute field set, and to limit the field format, allowed inference rules, and null value handling rules. The semantic extraction template can contain two parts: fixed instruction text and replaceable placeholders. The fixed instruction text describes the output requirements, while the replaceable placeholders are replaced with the names of each attribute field in the attribute field set, field format constraints, allowed inference rules, and null value handling rules when the attribute field set is loaded. This replacement forms the extraction request template corresponding to the current terminal type. The extraction request template, combined with the cleansing parameter text, is input into the semantic extraction model, which then outputs structured fields according to the output requirements in the extraction request template.

[0100] Field format constraints specify the output format for each attribute field. Text fields output strings, such as brand, model, and operating system. Numeric and unit fields output both the value and unit; for example, rate fields output the value and megabits per second, screen size outputs the value and inches, and battery capacity outputs the value and milliampere-hours. Date fields output a uniform date format, such as release date or release year or month. Decision fields output yes or no, such as whether it's a foldable screen or supports fast charging. Field format constraints can also specify that price fields output values ​​while retaining the currency unit, and that quantities with multiple inputs and outputs output integers. The semantic extraction model outputs each field according to the field format constraints. If the output value does not conform to the field format constraints, it is treated as a format error during the parsing phase, and the original output is retained for backtracking.

[0101] The allowed inference rules are used to limit which other fields the semantic extraction model can use to infer the value of a certain attribute field when the page text does not directly provide the value, and how the inference is performed. Allowed inference rules are configured by field; only attribute fields explicitly configured to allow inference in the configuration file can be inferred. Attribute fields without configured allowed inference rules will not be inferred. As a configurable form of allowed inference rules, when the cleaning parameter text provides the 2G and 5G wireless fidelity band rates but not the total wireless fidelity rate, the total wireless fidelity rate can be inferred as the sum of the two band rates. When the cleaning parameter text provides both the 2G and 5G wireless fidelity band rates, the wireless fidelity band can be inferred as dual-band. The 2G wireless fidelity band refers to the 2.4 GHz band, and the 5G wireless fidelity band refers to the 5 GHz band. For attribute fields such as brand, model, price, launch date, CPU, memory, hard drive, and operating system, if no allowed inference rules are configured in the configuration file, the semantic extraction model will not perform inference. The inference rules can also stipulate that the inference results are output in the same way as the field values ​​directly given on the page, and the inference source is output as a field label along with the inference results to facilitate subsequent traceability.

[0102] Null value handling rules are used to limit the output method of the semantic extraction model when no corresponding information for a certain attribute field is found in the page text and the field does not meet the allowed inference rules. Null value handling rules can specify that the field outputs an empty string, outputs a null value marker, or omits the field entirely, ensuring consistency in the null value output method across all fields within the same attribute field set. Null value handling rules can also specify that the semantic extraction model should not fill null value fields with default values, common values ​​from similar terminals, or custom values, and should not fill null value fields of the current configuration item with field values ​​from other configuration items on the same page or from other pages. Through null value handling rules, null fields and actual fields in the attribute extraction results can be clearly distinguished during the parsing phase, and the number of null fields serves as the basis for subsequent field fill rate calculations.

[0103] The semantic extraction model outputs structured fields based on the extraction request template. The output is a structured array, where each element represents a configuration item. Each configuration item consists of key-value pairs where the field name is the key and the field value is the value. The key name matches the field name in the attribute field set, and the field value conforms to the field format constraints. The computing device parses the structured array to obtain the attribute extraction result corresponding to each configuration item.

[0104] In one or more embodiments of the present invention, the candidate parameter page for recalling based on terminal identifier includes:

[0105] Generate search requests based on terminal identifiers;

[0106] Use search requests to access search engine results pages, product parameter databases, manufacturer websites, and third-party specification sites, and collect candidate parameter pages from these sources.

[0107] Save the title, address, and source information of the candidate parameter page.

[0108] After the attribute field set and semantic extraction template are loaded, the computing device recalls candidate parameter pages based on the terminal identifier. The computing device generates at least one parameter page search request based on the terminal identifier. Search requests can include two types: general search requests and targeted search requests. General search requests are generated by concatenating the terminal identifier with parameter keywords, retaining a separator between the terminal identifier and parameter keywords during concatenation, resulting in a query statement in the form of "terminal identifier parameters". When the terminal identifier is a combination of brand and model, the general search request directly uses this combination to concatenate with parameter keywords; when the terminal identifier is only the model, the general search request uses this model to concatenate with parameter keywords. Targeted search requests are generated by concatenating the terminal identifier with parameter resource keywords. Parameter resource keywords can include official website parameters, technical specifications, detailed parameters, specification parameters, product parameters, etc. Each targeted search request corresponds to one parameter resource keyword, and the number and content of parameter resource keywords are set in the configuration file. Targeted search requests can also be generated by concatenating the terminal identifier with the name of a parameter resource site and parameter keywords, used to increase the probability of hitting specific parameter database pages. The computing device generates a general search request and multiple targeted search requests in a preset order. The general search request is used to retrieve diverse search results, while the targeted search requests are used to increase the probability of hitting parameter databases, manufacturer websites, or technical specification pages.

[0109] The computing device uses generated search requests to access search engine results pages, product parameter databases, manufacturer websites, and third-party specification sites, collecting candidate parameter pages from these sources. When accessing search engine results pages, the computing device submits the search request as a query to the search engine's query interface, receives the returned result page, and parses the search results entries from the result page. Each search result entry includes a title and a link. When accessing the product parameter database, the computing device converts the search request into search keywords supported by the database, submits it to the database's internal search interface, and receives the returned list of search results. When accessing manufacturer websites, the computing device submits the terminal identifier portion of the search request to the website's internal search entry point and receives the returned internal search results. When accessing third-party specification sites, the computing device submits search keywords in the same way as with the product parameter database and receives the search results. The computing device extracts the titles and links corresponding to the candidate parameter pages from the results returned by the above information sources, forming an initial candidate set.

[0110] Search engine results pages sometimes return redirect links whose addresses differ from the target page's actual address. Directly using these redirect links can lead to subsequent page access during crawling. The computing device parses these redirect links and other redirected links in search results using two methods. The first method reads the target address parameter from the redirect link's address parameters and restores the value to the target page's actual address. The second method sends a network request to the redirect link and follows up with a redirect response, using the final response address as the target page's actual address. For successfully parsed links, the computing device replaces the original link address with the obtained actual address; for links that fail to parse, the computing device retains the original link address and records the parsing status, which is saved along with the candidate parameter page.

[0111] The computing device merges the general search request and the candidate results returned by each information source for each targeted search request, and deduplicates the merged results to form a set of candidate parameter pages. Deduplication is based on the parsed address information. The computing device normalizes the address information, including removing the segment identifier at the end of the address, standardizing the case of hostname characters in the address, and removing session identifier parameters in the address that do not affect page access. After normalization, multiple candidate results with the same address are merged into one candidate parameter page. The title and address information of the candidate parameter page collected earlier are retained, and the source information of each candidate parameter page participating in the merging is merged into the source list of this candidate parameter page. The remaining candidate results are not retained. Candidate results with the same title information but different address information are considered different candidate parameter pages and are not merged.

[0112] The computing device stores the title, address, and source information of candidate parameter pages to provide a basis for subsequent processing. The storage format is one structured record per candidate parameter page, containing a title field, an address field, and a source field. The title field stores the title text of the candidate parameter page; the address field stores the parsed or retained link address; and the source field stores a list of the sources from which the candidate parameter page was collected. Elements in the source list can be from search engine results pages, product parameter databases, manufacturer websites, and third-party specification sites. When a candidate parameter page comes from only one information source, the source list contains only a single value. The computing device writes these structured records into the storage area corresponding to the candidate parameter page set, completing the candidate parameter page retrieval.

[0113] In one or more embodiments of the present invention, filtering the target parameter page from the candidate parameter page includes:

[0114] The candidate parameter page is filtered, including blacklist domain filtering, document link filtering, shallow path link filtering, ad redirection link filtering, and duplicate link filtering.

[0115] After filtering, the title and address information of the candidate parameter pages are input into the semantic selection model, and the semantic selection model outputs the priority of the candidate parameter pages.

[0116] Select the highest priority preset number of candidate parameter pages as the target parameter pages.

[0117] The filtering process is executed in the following order: blacklist domain filtering, document link filtering, shallow path link filtering, ad redirect link filtering, and duplicate link filtering. Each filtering process applies to the remaining candidate parameter pages after the previous filtering process is completed. Filtered candidate parameter pages will no longer participate in subsequent filtering processes and semantic selection.

[0118] Blacklisted domain filtering is used to remove pages such as encyclopedias, Q&A sites, forums, social media platforms, and video platforms that are unsuitable for terminal parameter extraction. The computing device maintains a blacklist of domains in a configuration file, with each entry corresponding to a type of site unsuitable for terminal parameter extraction. When performing blacklisted domain filtering, the computing device extracts the hostname from the address information of the candidate parameter page and compares the hostname with each item in the blacklist. Comparison methods include full comparison and suffix comparison. A full comparison determines a match if the hostname is exactly the same as a domain in the blacklist, while a suffix comparison determines a match if the hostname ends with a domain from the blacklist plus a separator, used to cover subdomains of blacklisted domains. Candidate parameter pages whose hostnames match the blacklist are filtered, while those that do not are retained. Adding a new type of site unsuitable for parameter extraction simply involves adding the corresponding domain entry to the blacklist in the configuration file.

[0119] Document link filtering is used to remove download links for files such as PDF, DOC, DOCX, PPT, PPTX, XLS, and XLSX. The computing device extracts the file extension from the path portion of the candidate parameter page's address information, converting all extensions to lowercase. The extracted extensions are then compared to a set of document extensions. This set includes pdf, doc, docx, ppt, pptx, xls, and xlsx. Candidate parameter pages whose extensions match this set are filtered. Candidate parameter pages whose addresses do not contain a path extension or whose extensions are not in the document extension set are not considered document links and are retained.

[0120] Shallow path link filtering is used to remove pages with shallow path levels, such as official website homepages, category pages, and navigation pages, which typically do not contain detailed parameters. The computing device extracts the path portion from the address information of candidate parameter pages, splits the path portion by delimiters, and counts the number of non-empty path segments. The number of non-empty path segments represents the path level. Candidate parameter pages with a path level below a preset level threshold are identified as shallow path links and filtered. Candidate parameter pages with a path level reaching the preset level threshold are retained. The preset level threshold is set in the configuration file. Candidate parameter pages whose addresses only contain hostnames without paths have a path level of 0, which is below the preset level threshold and is therefore filtered.

[0121] In one or more embodiments of the present invention, shallow path link filtering also includes a parameter database domain name exemption rule. The computing device maintains a parameter database domain name set in a configuration file. The domain names in this set correspond to product parameter database-type websites. The pages on these websites typically carry content directly as parameter pages, and the path hierarchy does not reflect the presence or absence of parameter content. When performing shallow path link filtering, the computing device first checks whether the hostname in the address information of the candidate parameter page matches the parameter database domain name set. Candidate parameter pages whose hostnames match the parameter database domain name set are determined to be shallow path exempted, are not filtered according to path hierarchy, and are directly retained for subsequent filtering. Candidate parameter pages whose hostnames do not match the parameter database domain name set are filtered according to the comparison result between the path hierarchy and a preset hierarchy threshold.

[0122] In one or more embodiments of the present invention, shallow path link filtering can also be implemented as a weight reduction process. Weight reduction means that the computing device does not directly filter candidate parameter pages whose path level is lower than a preset level threshold, but instead reduces the ranking weight of the candidate parameter page in the candidate list. The weight reduction method is set in the configuration file, and can be implemented using either a weight reduction method or a direct filtering method. In the direct filtering method, candidate parameter pages whose path level is lower than the preset level threshold are filtered, but candidate parameter pages whose path level is lower than the preset level threshold and whose hostname matches the parameter database domain name set are still retained according to the shallow path exemption rules.

[0123] In one or more embodiments of the present invention, shallow path link filtering, semantic selection model, and page quality assessment constitute a three-level progressive screening. Pages with deep path levels but no actual parameter content are retained after shallow path link filtering. These pages are rejected by the semantic selection model based on title and address information and given a lower priority. Even if selected as target parameter pages, they are classified as BAD (Bad) in the page quality assessment stage due to low content completeness and structure scores, and are not included in semantic extraction. Pages with shallow path levels but actual parameter content retain a chance to compete under the de-weighting method. Under the direct filtering method, pages belonging to the parameter database domain set are retained according to the shallow path exemption rule. Shallow path link filtering, as an initial screening method, has individual case errors. These errors are absorbed by the subsequent semantic selection model and page quality assessment stages and will not be transmitted to the final extraction result.

[0124] Ad redirect link filtering removes links from ad networks or redirect links that cannot be resolved to a real target address. The computing device maintains a list of ad network domains in its configuration file. When performing ad redirect link filtering, the computing device first extracts the hostname from the address information of the candidate parameter page, compares the hostname with the ad network domain list, and filters candidate parameter pages whose hostnames match the ad network domain list. For candidate parameter pages that do not match the ad network domain list, the computing device further checks whether the address is a redirect link by sending a network request to the address and following a redirect response, using the final response address as the resolution result. When the number of redirects exceeds a preset threshold, the address is determined to be unable to resolve to a real target address, and the corresponding candidate parameter page is filtered. When the final response address is inaccessible or the response status code indicates access failure, resolution is determined to be failed, and the corresponding candidate parameter page is filtered. Candidate parameter pages that resolve successfully but do not match the ad network domain list are retained; the computing device replaces the original address of the candidate parameter page with the final response address before participating in duplicate link filtering. Candidate parameter pages with the same address that appear after being filtered by ad redirect links are removed by duplicate link filtering.

[0125] Duplicate link filtering is used to prevent the same page from being crawled multiple times. The computing device normalizes the address information of the remaining candidate parameter pages after the aforementioned filtering process, according to the normalization method used when saving address information during the candidate parameter page recall phase. After normalization, only the candidate parameter page with the same address is retained based on the one collected earlier in the order; the remaining candidate parameter pages are filtered out.

[0126] After filtering, the computing device inputs the title and address information of the retained candidate parameter pages into the semantic selection model. During input, the computing device concatenates the title and address information of each candidate parameter page into input text. Multiple candidate parameter page input texts form a candidate list, which is then input into the semantic selection model in batches. The semantic selection model determines whether a candidate parameter page is likely to contain detailed hardware parameters based on the title and address information. Key features include whether the title contains keywords such as "parameter," "specification," "configuration," or "technical specifications," and whether the address contains parameter-related path segments. The semantic selection model outputs the priority and resource type of the candidate parameter pages. Priority characterizes the degree to which a candidate parameter page is likely to contain detailed hardware parameters; priority can be expressed as a score, with higher scores indicating a greater likelihood of detailed hardware parameters. Resource type characterizes the source category of the candidate parameter page, including official resources, parameter database resources, and third-party parameter resources. The semantic selection model determines the resource type of the candidate parameter page based on site characteristics in the title and address information.

[0127] The computing device selects a preset number of candidate parameter pages with the highest priority as target parameter pages based on priority. The selection process involves sorting the candidate parameter pages from highest to lowest priority. If priorities are the same, the original order of the candidate parameter pages in the candidate list is maintained. The preset number of candidate parameter pages with the highest priority are selected as target parameter pages from the sorted results. The preset number is set in the configuration file, for example, 3. If the number of candidate parameter pages retained after filtering is less than the preset number, the computing device will use all retained candidate parameter pages as target parameter pages, and subsequent processing stages will execute according to the actual number of target parameter pages.

[0128] In one or more embodiments of the present invention, the parallel acquisition target parameter page includes:

[0129] Create a separate data acquisition task for each target parameter page;

[0130] Parameter page navigation is performed in independent data collection tasks. Parameter page navigation includes direct crawling, parameter link extraction, browser-automated triggering, and fallback crawling.

[0131] When the quality of the text obtained directly from crawling is lower than the first navigation quality threshold, links containing parameter keywords are extracted and crawled a second time.

[0132] When the quality of the text obtained from the secondary crawl is lower than the second navigation quality threshold, the browser will automatically trigger the parameter-related elements on the page.

[0133] When the quality of the text obtained by the browser's automated trigger is lower than the third navigation quality threshold, the text obtained directly from the crawl is used as the fallback parameter content.

[0134] Once the target parameter page is determined, the computing device creates an independent acquisition task for each target parameter page and assigns an acquisition task identifier to each acquisition task. Each acquisition task corresponds to one target parameter page, and multiple acquisition tasks are executed in parallel. Each acquisition task uses an independent browser page context, browser instance, or acquisition session to avoid state conflicts between multiple acquisition tasks. Within the same acquisition task, parameter page navigation, content cleaning, page quality assessment, and semantic extraction are executed sequentially, and different acquisition tasks are independent of each other. Each acquisition task saves intermediate data such as the target parameter page address, final parameter page address, navigation method, cleaned parameter text, page quality score, and attribute extraction results.

[0135] In one or more embodiments of the present invention, parallel acquisition also includes a concurrency control mechanism. The concurrency control mechanism includes a page-level concurrency limit, a model-level concurrency limit, and a same-domain access interval. Page-level concurrency refers to the parallel execution of multiple acquisition tasks within a single terminal identifier processing task. The page-level concurrency limit is equal to the number of target parameter pages for that terminal identifier. In practice, when the number of target parameter pages is 3, the three acquisition tasks are executed simultaneously in parallel. Model-level concurrency refers to the parallel execution of multiple terminal identifiers entering the processing flow simultaneously. The model-level concurrency limit is set in the configuration file, and can be 3 in practice. The same-domain access interval refers to the computing device grouping target parameter pages according to the hostname in the address information. The starting time interval for acquisition tasks within the same hostname group to perform access actions is not less than the domain name access interval duration set in the configuration file. In practice, the domain name access interval duration can be 1 second. Acquisition tasks between different hostname groups are not limited by this interval.

[0136] In one or more embodiments of the present invention, parallel acquisition is further configured with a tiered timeout mechanism and a failure retry mechanism. The tiered timeout mechanism refers to the computing device setting timeout durations for page crawling, navigation rendering, element waiting, click operations, network idle waiting, page crawling service connection, page crawling service reading, semantic extraction model connection, and semantic extraction model reading. In practice, the page loading timeout can be 20 seconds, the navigation rendering timeout can be 30 seconds, the element waiting timeout can be 8 seconds, the click operation timeout can be 5 seconds, the network idle waiting timeout can be 10 seconds, the page crawling service connection timeout can be 10 seconds, the page crawling service reading timeout can be 30 seconds, the semantic extraction model connection timeout can be 10 seconds, and the semantic extraction model reading timeout can be 120 seconds. The computing device also sets an overall timeout for parallel data acquisition. In practice, this timeout can be 300 seconds. If the total execution time of all acquisition tasks for a single terminal exceeds this timeout, any unfinished acquisition tasks will terminate. A retry mechanism is in place to prevent network errors during page crawling or navigation actions. The action will be retried up to the maximum number of retries set in the configuration file. In practice, the maximum number of retries for the page crawling service can be 2. If the action still fails after reaching the maximum number of retries, the acquisition task is marked as failed, and the reason for failure is recorded. If any exception occurs during the execution of an acquisition task, the computing device will terminate the current action of that acquisition task and reclaim the browser page context, browser instance, or acquisition session resources used by the acquisition task. The acquisition task will then be marked as failed, and the reason for failure will be recorded. Failed acquisition tasks do not generate cleaned parameter text or target attribute results and do not participate in quality fusion. The failure of a single acquisition task does not affect the execution of other acquisition tasks. Each acquisition task uses an independent browser page context, browser instance, or acquisition session, and the states of each acquisition task are isolated.

[0137] Before executing parameter page navigation for each data collection task, the computing device determines the text quality criteria for each stage of navigation. Text quality measures whether the page content obtained in a single crawl contains sufficient parameter information. The quality of the text determines whether the data collection task proceeds to the next level of navigation or ends the navigation process. The computing device calculates the effective text length, parameter keyword hit rate, and number of structured lines from the crawled page content. Effective text length refers to the number of characters remaining after converting the page content to text and removing whitespace. Parameter keyword hit rate refers to whether at least one keyword from the following categories appears in the page content: parameter, detailed parameter, specification parameter, configuration, technical specification, specification, and product parameter. The number of structured lines refers to the number of key-value pair lines in the page content. A key-value pair line is a text line that simultaneously contains a field name, a delimiter, and a field value. The delimiter can be a colon, a vertical bar, or a tab. When the text quality simultaneously meets the following conditions: effective text length reaches the length condition, parameter keyword hit rate is achieved, and the number of structured lines reaches the line count condition, the text quality obtained in that crawl is deemed to have reached the navigation quality threshold for the corresponding navigation stage. The length and line count conditions are set in the configuration file.

[0138] In this embodiment, the navigation quality threshold during the direct crawling stage is called the first navigation quality threshold, the navigation quality threshold during the parameter link secondary crawling stage is called the second navigation quality threshold, and the navigation quality threshold during the browser automated triggering stage is called the third navigation quality threshold. These three navigation quality thresholds are independent of the first, second, and third preset thresholds used for page quality level determination, and are configured separately in the configuration file without affecting each other. The text quality criterion is used to determine whether to continue or end the navigation stage, and belongs to a different processing stage from the page quality assessment before semantic extraction; the two are not interchangeable.

[0139] At the start of navigation, the data collection task performs direct crawling of the target parameter page. Direct crawling refers to obtaining the page content of the target parameter page by initiating network requests and converting the page content into text or structured text. During the conversion process, Hypertext Markup Language tags, script code, and style code are removed. The data collection task calculates the text quality of the directly crawled text. When the quality of the directly crawled text reaches the first navigation quality threshold, it is determined that the target parameter page directly contains parameter content, and the directly crawled text is used as the navigation result of this data collection task, at which point the navigation ends. When the quality of the directly crawled text is lower than the first navigation quality threshold, the data collection task extracts parameter links from the page content of the target parameter page. Parameter links are links whose link text or link address contains parameter link keywords. Parameter link keywords include parameters, detailed parameters, specification parameters, configuration, technical specifications, specifications, specifications, and spec. Parameter link keywords are maintained in the configuration file. The keywords used for parameter link keywords and text quality criteria are set separately, and their matching targets differ: the text quality criteria keywords determine whether parameter content prompts appear in the page body text, while the parameter link keywords determine whether the link text and link address are parameter navigation entry points. Therefore, they can be configured into different sets. For example, text quality criteria keywords can include phrases appearing in the page body text such as "product parameters" and "detailed parameters," while parameter link keywords can include common foreign language expressions in link text and link addresses such as "Spec" and "Specifications." Alternatively, they can be configured into the same set and maintained uniformly in the configuration file.

[0140] During extraction, the data collection task iterates through all anchor elements in the directly crawled Hypertext Markup Language (HTML) document. For each anchor element, it checks whether its link text and link address contain parameter link keywords. Anchor elements containing parameter link keywords are identified as parameter links. Parameter link extraction is performed on the HTML document before page content conversion. During direct crawling, both the HTML document and the converted text are retained, and anchor element information is read from the retained HTML document. When multiple parameter links are extracted, the data collection task prioritizes them based on the completeness of the parameter link keywords in the link text and the depth of the path segment in the link address. Parameter links whose link text completely contains parameter-like phrases and whose link address path segment has a deeper level are ranked higher. The data collection task performs secondary crawling on the sorted parameter links sequentially. After each secondary crawling of a parameter link is completed, the text quality of the crawled text is calculated. When the text quality of the secondary crawling of a parameter link reaches the second navigation quality threshold, the text obtained from the secondary crawling is used as the navigation result of the data collection task, and the address of the parameter link is recorded as the final parameter page address. The data collection task then ends its navigation. When the secondary crawling results of all parameter links are lower than the second navigation quality threshold, the data collection task is automatically triggered in the browser.

[0141] Browser-automated triggering refers to a data collection task loading a target parameter page using browser automation tools, triggering page elements related to parameters, specifications, configurations, or technical parameters, and obtaining the parameter page content displayed after the trigger. Triggering objects include parameter labels, specification buttons, technical parameter links, parameter list items, and parameter-related tabs. The data collection task executes the triggering operations sequentially: first, triggering parameter-related links and buttons on the page, then triggering parameter-related tabs, and finally triggering parameter-related list items. After each page element is triggered, the data collection task waits for the page content to update. This waiting can be done in two ways: waiting for a fixed duration or waiting for changes in the page's document object model structure; these two methods are used in combination. After the page content is updated, the data collection task obtains the updated page content and calculates the text quality. When the text quality obtained after a trigger reaches the third navigation quality threshold, the page content obtained after that trigger is used as the navigation result for the data collection task, the navigation method is recorded as browser-automated triggering, and the data collection task ends navigation. If the text quality is still lower than the third navigation quality threshold after all page elements related to the parameters are triggered, the collection task uses the directly crawled text as the fallback parameter content. The navigation result of this collection task is determined as the fallback parameter content, the navigation method is recorded as fallback crawling, and the content of this path is marked as a low confidence source. The marking information is saved together with the cleaned parameter text of this collection task for use in the subsequent page quality assessment and comprehensive quality scoring stages.

[0142] In one or more embodiments of the present invention, the browser's automated triggering waiting strategy is as follows: When the computing device loads the target parameter page, it waits for the page's document object model structure to complete loading. The criterion for completion is the triggering of the page document object model content loading completion event. After each page element is triggered, the computing device prioritizes waiting for the page to be idle. The criterion for idleness is that all page network requests are completed. A timeout period is set for the idle network wait, which can be 10 seconds in practice. The upper limit of the idle network wait timeout is set in the configuration file based on page rendering statistics experience. If the page does not enter an idle network state within the idle network wait timeout period, the computing device degenerates to waiting for a fixed period, which can be 5 seconds in practice. During the waiting period, the computing device periodically checks for changes in the page document object model structure. The detection conditions for changes in the page document object model structure are an increase in the number of parameter-related elements or an increase in the page text length. The waiting ends early when a structural change is detected. After the waiting period ends, the computing device also compares whether the current page address is consistent with the page address before the trigger and obtains a snapshot of the page content. An address change or an update of the content snapshot indicates that the trigger has taken effect.

[0143] In one or more embodiments of the present invention, the abnormal page handling method automatically triggered by the browser is as follows. Before triggering each parameter-related page element, the computing device checks whether the page element is visible by reading its visibility state. If the page element is not visible, it is determined that the page element is obscured. Obscuration sources include login pop-ups, verification code prompts, and advertising masks. The computing device prioritizes closing the obscuring element. After successful closure, it continues to trigger the page element. If the obscuring element cannot be closed, the computing device skips the page element and continues to trigger the next parameter-related page element, recording the abnormal page information in the task record of the collection task. If the text quality is still lower than the third navigation quality threshold after all parameter-related page elements have been triggered, the computing device uses the directly crawled text as the fallback parameter content, as described above. Regardless of whether the triggering failure is due to insufficient waiting or page obscuration, the processing path is determined to converge, that is, skipping the current element and continuing to trigger the next element. After all elements fail to trigger, the fallback crawling begins, and there are no uncertain state branches.

[0144] After each acquisition task completes the parameter page navigation, content cleaning is performed on the navigation results to obtain the cleaned parameter text corresponding to the target parameter page. The content cleaning process and page quality assessment process have been described in the previous implementation method. Each acquisition task processes its own navigation results according to the same content cleaning method and page quality assessment method, and will not be repeated here. The first navigation quality threshold, the second navigation quality threshold, and the third navigation quality threshold are set in the configuration file. The three navigation quality thresholds can be the same or different. In implementation, they can be configured separately according to the requirements of parameter content at different navigation stages. The configuration items of the three navigation quality thresholds and the configuration items of the preset thresholds used for page quality level determination are stored in separate areas in the configuration file to avoid configuration confusion.

[0145] In one or more embodiments of the present invention, content cleaning includes:

[0146] Extract the page title and page source URL;

[0147] Remove image links, navigation text, footer text, advertising text, customer service text, copyright text, and irrelevant links;

[0148] Locate the beginning position of the parameter content based on the starting keyword of the parameter, and locate the end position of the parameter content based on the ending keyword of the parameter.

[0149] Use the text between the start and end positions of the parameter content as the cleaned parameter text;

[0150] The starting keywords for the parameters include “configuration filter”, “model”, “specification parameters”, “detailed parameters”, “important parameters”, “hardware specifications”, “technical specifications”, “basic parameters”, and “product parameters”.

[0151] The parameters end with keywords including "related recommendations", "same series products", "precautions", "all rights reserved" and "Copyright".

[0152] After each acquisition task completes the parameter page navigation, content cleaning is performed on the navigation results. The output of content cleaning is the cleaned parameter text corresponding to the target parameter page. The input for content cleaning is the page content obtained from the parameter page navigation. The page content can be saved in the form of a Hypertext Markup Language document or in the form of converted plain text. In this embodiment, the page content is processed in the form of a text line sequence, and each line retains the original text content and line break relationships of the page.

[0153] The first step in content cleaning is to extract the page title and page source URL. The page title is extracted from the title tags of the page content; the title tag is the heading element in the header of a Hypertext Markup Language (HTML) document. When the page content is obtained automatically by the browser, the page title is extracted from the current page title read by the browser's automation tool. If the page title is empty or the title tag is missing, the computing device uses the first non-empty line in the converted text of the page content as the page title.

[0154] The page source address is the final parameter page address for this data collection task. This address is recorded during the parameter page navigation phase. When direct crawling reaches the first navigation quality threshold, the page source address is the target parameter page address. When secondary crawling via parameter links reaches the second navigation quality threshold, the page source address is the address of that parameter link. When browser automation triggers reach the third navigation quality threshold, the page source address is the target parameter page address. During fallback crawling, the page source address is the target parameter page address. The extracted page title and page source address are written into the task record of this data collection task and saved along with the cleaned parameter text, page quality score, and attribute extraction results for use in the profile recording and association phase.

[0155] The second step in content cleaning is to remove image links, navigation text, footer text, advertising text, customer service text, copyright text, and irrelevant links. The computing device iterates through the converted text of the page content line by line, performing the following deletion checks on each line in sequence.

[0156] Image link lines refer to lines that contain only addresses ending with image file extensions, including jpg, jpeg, png, gif, webp, bmp, and svg. Lines whose entire content consists of image addresses are deleted. When image addresses are mixed with other text in a line, only the image address portion of the line is deleted, and the other text in the line is retained.

[0157] Navigation text refers to the names of columns, menu items, and column links located at the top or side of the page. Navigation text is identified by navigation keywords, which include Home, Product Center, Product Series, Solutions, Service Support, About Us, Login, Register, Back to Top, Share, etc. When a text line begins with a navigation keyword or consists entirely of navigation keywords, the line is deleted. Navigation keywords are maintained in the configuration file and can be adjusted according to different site structures.

[0158] Footer text and copyright text refer to the filing information, copyright statement, and declarative text located at the bottom of the page. When a text line contains content such as "Copyright," "filing number," "privacy policy," or "terms of use," the line is deleted. After the parameter content ends and is positioned, the text in the footer area outside the parameter content range is also deleted entirely.

[0159] Advertising text refers to the text in the advertising area, promotion area, and banner area of ​​the page. If the text line contains words such as "advertisement," "promotion," "partnership," "recommended download," or "buy now," or if the hostname of the link address in the line matches the list of advertising network domains, the line will be deleted.

[0160] Customer service text refers to the guiding text on the page for customer service entry, consultation entry, and contact information. When the text line contains content such as online customer service, human customer service, customer service phone, instant messaging tool consultation, pre-sales consultation, or after-sales consultation, the line is deleted.

[0161] Irrelevant links refer to hyperlink text in the page content that is unrelated to terminal attributes, except for parameter links. When deleting them, the anchor text corresponding to the link is retained, and only the link address is removed; text lines with anchor text in plain address form and without parameter link keywords are deleted entirely.

[0162] The above deletion judgment is executed line by line. The remaining text lines after deletion maintain their original order. Multiple consecutive blank lines are compressed into one blank line. The compressed text line sequence serves as the base text for the content range of the positioning parameter.

[0163] The third step in content cleaning is locating the parameter content range. The computing device first locates the start position of the parameter content on the basic text, and then locates the end position of the parameter content.

[0164] When locating the starting position of parameter content, the computing device uses a set of parameter starting keywords. These keywords include configuration filters, model numbers, specifications, detailed parameters, important parameters, hardware specifications, technical specifications, basic parameters, and product parameters. The computing device scans each line of the basic text line by line, starting from the first line, checking if each line contains any keyword from the parameter starting keyword set. A line starting with a parameter starting keyword is considered a primary hit, while lines containing the same keyword in the middle are considered secondary hits. Primary hits have higher priority than secondary hits.

[0165] After scanning all text lines, the computing device first searches for the primary hit text line. If multiple primary hits exist, the text line with the smallest line number is selected as the starting position of the parameter content. If no primary hit exists, the text line with the smallest line number among the secondary hits is selected as the starting position of the parameter content. When neither the primary nor secondary hits exist, meaning none of the keywords in the parameter starting keyword set appear in the base text, the computing device uses the first line of the base text as the starting position of the parameter content and records the starting positioning method as "missed positioning." When multiple parameter starting keywords appear simultaneously in different text lines, the principle of smallest line number is applied, with the text line with the smallest line number being closer to the beginning of the page parameter area.

[0166] When locating the end position of parameter content, the computing device uses a set of parameter end keywords, which include related recommendations, similar products, precautions, copyright, and related terms. The computing device scans line by line downwards from the line following the start position of the parameter content, checking each line for any keyword from the parameter end keyword set. The line containing the first parameter end keyword is identified as the end position of the parameter content, and the scan stops.

[0167] The line corresponding to the end position of the parameter content is not part of the parameter content itself; the parameter content range is the text lines preceding that line. If no text lines containing the parameter ending keyword are scanned after the line corresponding to the start position of the parameter content, the computing device takes the last line of the basic text as the line corresponding to the end position of the parameter content and records the end positioning method as "missed positioning." If the parameter ending keyword appears before the line corresponding to the start position of the parameter content, it is not used as a basis for determining the end position. Starting scanning after the line corresponding to the start position of the parameter content avoids interference from parameter ending keywords appearing in the top navigation or footer area of ​​the page.

[0168] The fourth step in content cleaning is to extract the cleaning parameter text. The computing device uses all text lines between the start and end positions of the parameter content as the parameter content range. The parameter content range includes the start position line but excludes the end position line.

[0169] The computing device performs a final cleanup on the text lines within the parameter content range. This final cleanup removes any remaining blank lines, lines containing only symbols, and any remaining Hypertext Markup Language (HTML) entities within the lines. The remaining text lines maintain the same order as the base text, resulting in the cleaned parameter text. When the number of remaining text lines after truncation of the parameter content range is zero, the computing device uses the entire base text as the cleaned parameter text and records the truncation method as "whole retention." The cleaned parameter text is saved as a text line sequence in the task record of this acquisition task for use in the page quality assessment and semantic extraction stages.

[0170] Taking a router-type target parameter page as an example, after text line conversion, the page content includes a navigation bar and login / registration entry at the top, a parameter table titled "Specifications" in the middle, and related recommendations and copyright information below the parameter table. During content cleaning, the page title is extracted from the title tag, the page source address is recorded as the final parameter page address for this collection task, the navigation bar line, login / registration line, and image address line are deleted, the line containing the specifications is first detected during basic text scanning, and the line containing related recommendations is detected for the first time in subsequent scans. The parameter table text lines between the starting and ending lines of the parameter content are truncated, and blank lines are deleted to obtain the cleaned parameter text for the target parameter page.

[0171] In one or more embodiments of the present invention, page quality assessment includes:

[0172] The content completeness score is calculated based on the effective text length and effective number of lines in the cleaned text.

[0173] The degree of structuring is scored based on the number of table rows, key-value pair rows, or structured rows in the cleaning parameter text.

[0174] The terminal matching score is calculated based on the similarity between the terminal identifier and the cleaning parameter text.

[0175] Page quality scores and page quality levels are generated based on content completeness scores, structure scores, and terminal compatibility scores.

[0176] Page quality ratings include GOOD, WEAK, and BAD.

[0177] When the page quality level is GOOD, the cleaning parameter text will be marked as normal semantic extraction text.

[0178] When the page quality level is WEAK, the cleaning parameter text will be marked as low-confidence semantic extraction text.

[0179] When the page quality level is BAD, retain the cleaning parameter text for backtracking.

[0180] After each acquisition task obtains the cleaned parameter text, the computing device performs a page quality assessment on the cleaned parameter text. The input to the page quality assessment is the cleaned parameter text, and the output is a page quality score and a page quality level. The page quality score and page quality level are written to the task record of the acquisition task along with the cleaned parameter text. The page quality assessment is used to determine whether the target parameter page contains sufficient parameter information, whether the page content is structured, and whether the page content matches the terminal identifier before semantic extraction, so that low-quality pages can be distinguished before entering semantic extraction. The page quality assessment and the text quality criteria in the parameter page navigation stage belong to different processing stages. The text quality criteria are used to determine whether navigation continues, while the page quality assessment is used to determine how the navigation results enter semantic extraction. The two are not interchangeable.

[0181] The first score in the page quality assessment is the content completeness score, which is calculated based on the effective text length and the number of effective lines in the cleaned parameter text. Effective text length refers to the number of characters remaining after removing whitespace, newline characters, and tabs from the cleaned parameter text; a larger character count indicates more parameter description content. The number of effective lines refers to the number of effective text lines in the cleaned parameter text. Effective text lines are defined as lines with remaining characters after removing leading and trailing whitespace, and where not all remaining characters are punctuation marks or separators.

[0182] The computing device maps the effective text length and the number of effective lines to scores between 0 and 1, respectively. Linear interpolation is used for the mapping, and the mapping parameters are set in the configuration file. These parameters include a lower limit for effective text length, an upper limit for effective text length, a lower limit for effective line count, and an upper limit for effective line count. When the effective text length is not higher than the lower limit, the mapping score is 0; when the effective text length reaches the upper limit, the mapping score is 1; and when it falls between these limits, a linear interpolation score is obtained. The number of effective lines is mapped in the same way. The computing device then weights the effective text length mapping score and the effective line count mapping score to obtain a content completeness score. The weighting coefficient is set in the configuration file, and in practice, each mapping score can be set to 0.5.

[0183] The second score in the page quality assessment is the structure score, which is calculated based on the number of table rows, key-value pair rows, or structured rows in the cleaned parameter text. The cleaned parameter text is extracted from the text line sequence after the page content is converted, retaining some page structure information. When the page content is converted into a text line sequence, table row markers are added to the text lines corresponding to the row elements contained in the table elements. After content cleansing and extraction, the text lines in the cleaned parameter text with table row markers are counted as table rows. When only text is retained in the page content, a table row refers to multiple consecutive rows of key-value pair rows. These multiple rows of key-value pair rows use the same delimiter and each row has a different field name; each row is counted as a table row.

[0184] Key-value pair lines refer to text lines that simultaneously contain field names, delimiters, and field values. Delimiters can be colons, vertical bars, or tabs. Their identification method is consistent with the key-value pair line identification method in the previous text quality criterion. Structured lines refer to the union of table lines and key-value pair lines; a single text line belonging to both a table line and a key-value pair line is counted only once. The calculation device counts the number of structured lines in the text containing the cleaning parameters and maps this number to a score between 0 and 1. The mapping method is the same as the content completeness score mapping method, and the mapping parameters include a lower limit and an upper limit for the number of structured lines. Pages containing multiple tables or a large number of field-value pair content have a higher number of structured lines and a correspondingly higher structured score.

[0185] The third score in the page quality assessment is the terminal matching score, which is calculated based on the similarity between the terminal identifier and the cleaned parameter text. The computing device first segments the terminal identifier into three text units: brand, model, and complete identifier. The brand and model parts are split according to the boundaries between Chinese characters and alphanumeric characters in the terminal identifier. The model part is the remaining alphanumeric string after splitting. The complete identifier is the original string of the terminal identifier.

[0186] The computing device calculates character fragment matching similarity, fuzzy matching similarity, and word segmentation similarity on the cleaning parameter text. Character fragment matching similarity refers to the degree to which the complete identifier or model part appears as a continuous character fragment in the cleaning parameter text. When the complete identifier appears as a continuous character fragment, the character fragment matching similarity is set to 1. When the model part appears as a continuous character fragment, the character fragment matching similarity is set to a preset high value in the configuration file. In practice, the preset high value can be 0.8. When neither appears, the value is 0.

[0187] Fuzzy matching similarity refers to the degree of similarity between the character sequences of the complete identifier and each line of text in the cleaned parameter text. It is calculated by calculating the edit distance similarity between the complete identifier and each line of text line by line. The edit distance similarity represents the degree of difference between the two character sequences. The smaller the difference, the higher the similarity. The maximum value among all lines of text is taken as the fuzzy matching similarity.

[0188] Word segmentation similarity refers to the degree of overlap between the brand and model parts and the word segmentation results of the cleaned parameter text. It is calculated as the ratio of the intersection of the word set composed of the brand and model parts and the word set appearing in the cleaned parameter text to the size of the former. The computing device weights and sums the character fragment matching similarity, fuzzy matching similarity, and word segmentation similarity according to the proportion coefficients in the configuration file, obtaining a terminal matching score between 0 and 1, with the sum of the three proportions being 1. Using the maximum fuzzy matching similarity at the text line level in the calculation avoids dilution of the overall similarity when the cleaned parameter text is long, ensuring that the terminal matching score reflects the content in the cleaned parameter text that is closest to the terminal identifier.

[0189] The computing device generates a page quality score and page quality level based on content completeness score, structuredness score, and terminal matching score. The page quality score is obtained by weighted summation of the three scores, with the sum of the weight coefficients of the three scores in the page quality score being 1. The weight coefficients and weighting method are set in the configuration file.

[0190] Page quality levels include GOOD, WEAK, and BAD. A GOOD level indicates that the page content structure is relatively rich and matches the terminal identifier, making it suitable for direct semantic extraction. A WEAK level indicates that the page content has some parameter information but lacks completeness or matching degree, allowing for low-confidence extraction. A BAD level indicates that the page content is sparse, poorly structured, or significantly mismatched with the terminal identifier, and semantic extraction is not performed. The page quality score and page quality level calculation results are written into the task record of this data collection task. The page quality score serves as one of the bases for calculating the overall quality score during the quality fusion phase.

[0191] After the page quality level is determined, the computing device marks the cleaned parameter text according to the page quality level. When the page quality level is GOOD, the computing device marks the cleaned parameter text as normal semantic extraction text, and the normal semantic extraction text enters the semantic extraction process, and its attribute extraction results participate in quality fusion.

[0192] When the page quality level is WEAK, the computing device marks the cleaned parameter text as low-confidence semantic extraction text. The low-confidence semantic extraction text enters the semantic extraction process. Its page quality score already reflects the situation of insufficient completeness or matching degree. When the attribute extraction result of this path participates in quality fusion, it participates in the sorting according to the comprehensive quality score corresponding to its page quality score.

[0193] When a page quality level is BAD, the computing device does not perform semantic extraction on the cleaning parameter text; it only retains the cleaning parameter text for backtracking. The retained content includes the cleaning parameter text, page quality score, page quality level, and target parameter page address. This data collection task does not generate attribute extraction results and does not participate in quality fusion. Maintenance personnel can retrieve the retained content for BAD-level pages to check why the target parameter page was judged as BAD, and adjust the parameter page navigation configuration, content cleaning configuration, or target parameter page selection configuration accordingly.

[0194] The low-confidence source marker and page quality level marker added to the fallback content during the parameter page navigation phase are different dimensions of marker information and can coexist in the same collection task. The low-confidence source marker records the navigation method, and the page quality level marker records the evaluation result. The two are stored separately in the task record.

[0195] In one or more embodiments of the present invention, a page quality score and page quality level are generated based on a content completeness score, a structured score, and a terminal matching score, including:

[0196] The page quality score is obtained by weighting and summing the content completeness score, the structure score, and the terminal matching score.

[0197] The weighted summation uses the first weight coefficient, the second weight coefficient, and the third weight coefficient, and the sum of the first weight coefficient, the second weight coefficient, and the third weight coefficient is 1;

[0198] When the content completeness score is lower than the first preset threshold, the structure score is lower than the second preset threshold, or the terminal matching score is lower than the third preset threshold, the page quality level will be set to BAD level.

[0199] When the content completeness score reaches the first preset threshold, the structuring score reaches the second preset threshold, and the terminal matching score reaches the third preset threshold, the page quality level is set to GOOD or WEAK based on the relationship between the page quality score and the fourth preset threshold.

[0200] The specific process by which the computing device generates page quality scores and page quality levels is as follows. The content completeness score, structured content score, and terminal matching score calculated during the page quality assessment phase are all values ​​between 0 and 1. These three scores serve as inputs for generating the page quality score and page quality level in this embodiment. The page quality score is obtained by a weighted sum of the content completeness score, structured content score, and terminal matching score, with each score corresponding to a weight coefficient.

[0201] The weighted summation uses a first weight coefficient, a second weight coefficient, and a third weight coefficient. The first weight coefficient corresponds to the content completeness score, the second weight coefficient corresponds to the structuredness score, and the third weight coefficient corresponds to the terminal matching score. The sum of the three weight coefficients is 1. The three weight coefficients are set in the configuration file. In practice, a configuration of 0.4 for the first weight coefficient, 0.3 for the second weight coefficient, and 0.3 for the third weight coefficient can be used. Alternatively, the values ​​of the three weight coefficients can be adjusted according to the business focus. For example, when there is interference from cross-series models with the same name on the parameter page, the value of the third weight coefficient can be increased to increase the proportion of terminal matching in the page quality score, making it more difficult for pages with a low degree of matching with the terminal identifier to enter the normal semantic extraction process.

[0202] The page quality score is obtained by weighted summation. Since all three scores are values ​​between 0 and 1, and the sum of the three weight coefficients is 1, the page quality score also ranges from 0 to 1, with a higher value indicating higher page quality. Within the same processing batch, the page quality scores of each collection task are calculated using the same set of weight coefficients, ensuring comparability between page quality scores from multiple collection results and providing a consistent calculation method for sorting by comprehensive quality score during the quality fusion stage. After the weight coefficients are adjusted, the new weight coefficient values ​​in the configuration file take effect from the next processing batch, while the page quality scores saved in the already generated task records remain unchanged.

[0203] The generation of page quality rating involves two levels of judgment. The first level is the BAD rating judgment. The computing device compares the content completeness score with a first preset threshold, the structured score with a second preset threshold, and the terminal matching score with a third preset threshold. The three comparison results are ORed. That is, if any one of the following three conditions is met, the page quality rating is set to BAD.

[0204] The first-level judgment is executed before the page quality score ranking. The reason for using an OR relationship is that if any one of the content completeness, structuring degree, or terminal matching degree has a serious defect, the other two scores cannot compensate for the defect. For example, a page with a high structuring degree score but a very low terminal matching degree score indicates that the page structure is regular but corresponds to a different type of terminal. Its content should not be used for attribute extraction of the target terminal identifier, and the page should be directly judged as BAD level and not participate in the subsequent ranking. The first, second, and third preset thresholds are set in the configuration file. In implementation, the first preset threshold can be set to 0.4, the second preset threshold to 0.3, and the third preset threshold to 0.3. Alternatively, they can be adjusted according to the common form of parameter pages under different terminal types.

[0205] In this embodiment, the first preset threshold, the second preset threshold, the third preset threshold, and the fourth preset threshold used for determining the page quality level are independent of the first navigation quality threshold, the second navigation quality threshold, and the third navigation quality threshold in the parameter page navigation stage. Each threshold is configured in a configuration file, and the configuration items for both are stored in separate areas. Adjusting the threshold in any stage does not affect the determination in the other stage.

[0206] When all three conditions are met simultaneously—content completeness score reaching the first preset threshold, structuring score reaching the second preset threshold, and terminal matching score reaching the third preset threshold—the computing device executes a second-level judgment. The second-level judgment classifies the page quality level based on the relationship between the page quality score and the fourth preset threshold. When the page quality score reaches the fourth preset threshold, the page quality level is set to GOOD; when the page quality score is below the fourth preset threshold, the page quality level is set to WEAK.

[0207] The fourth preset threshold is set in the configuration file. During implementation, a setting of 0.6 can be used, or it can be adjusted according to the stringency of the text quality requirements for normal semantic extraction. In the second-level judgment, the comparison between the page quality score and the fourth preset threshold is divided into two levels: reaching and falling below. If the page quality score equals the fourth preset threshold, it is treated as reaching the threshold, corresponding to the GOOD level. Pages that pass all three score threshold checks in the second-level judgment are further divided into GOOD and WEAK levels. Pages with balanced scores across all three categories and a high total score enter the normal semantic extraction process, while pages where all three scores reach their respective thresholds but the total score is insufficient enter the low-confidence semantic extraction process.

[0208] The page quality score and page quality level are generated and stored corresponding to the cleaned parameter text. After the page quality level is generated, the computing device marks the cleaned parameter text according to the method described in the previous embodiment. The GOOD level corresponds to the normal semantic extraction text, the WEAK level corresponds to the low-confidence semantic extraction text, and the cleaned parameter text corresponding to the BAD level is retained for backtracking. The marking process will not be repeated here. The page quality score is written into the task record of this acquisition task along with the page quality level. The page quality score in the task record serves as one of the bases for calculating the comprehensive quality score of this channel during the quality fusion stage.

[0209] In one or more embodiments of the present invention, a semantic extraction template and cleaned parameter text are input into a semantic extraction model to obtain attribute extraction results; a target configuration item matching the terminal identifier is determined from the attribute extraction results as the target attribute result corresponding to the target parameter page, including:

[0210] Combine the semantic extraction template and the cleaned parameter text into an extraction request;

[0211] The semantic extraction model outputs attribute extraction results based on the extraction request. The attribute extraction results are an array, and each element in the array represents a configuration item.

[0212] Calculate the text similarity between the terminal identifier and the Chinese name field in the configuration item, the text similarity between the terminal identifier and the detailed model field in the configuration item, and the text similarity between the terminal identifier and the model field in the configuration item;

[0213] The text similarity between the terminal identifier and the configuration item is determined based on the text similarity of the Chinese name field, the detailed model field, and the model field.

[0214] The text similarity between the terminal identifier and the configuration item is corrected by combining the model code matching result and the year matching result, and the corrected text similarity is obtained.

[0215] The configuration item with the highest text similarity after correction is identified as the target configuration item;

[0216] When multiple configuration items have the same text similarity after correction, the target configuration item is determined by sorting them according to field completeness and price information.

[0217] The determined target configuration items are used as the target attribute results.

[0218] The computing device combines the semantic extraction template and the cleaning parameter text into an extraction request. The generation and loading process of the semantic extraction template has been described in the previous embodiments. Here, when combining them, the computing device places the text of the semantic extraction template first and the cleaning parameter text second, inserting a newline character and a fixed separator between them. The fixed separator is used to enable the semantic extraction model to distinguish between the constraint description part and the text to be extracted.

[0219] In each data collection task, the cleaned parameter text with a page quality level of GOOD or WEAK is combined with the same semantic extraction template to form its own extraction request. The cleaned parameter text is the text extracted from the parameter content range, and its length is usually within the upper limit of the input length of the semantic extraction model. When the length of the cleaned parameter text exceeds the upper limit of the input length of the semantic extraction model, the computing device divides the cleaned parameter text into multiple segments according to the text lines. During the segmentation, it ensures that each key-value pair line is completely located in the same segment. Each segment is combined with the semantic extraction template to form a segmented extraction request, which is then input into the semantic extraction model to obtain the segmented attribute extraction results for each segment. Finally, the segmented attribute extraction results are merged by field. During the parsing phase, the attribute extraction results for each segment have undergone format validation according to field format constraints. During merging, field values ​​appearing in multiple segments for the same field are handled as follows: Non-empty field values ​​that conform to the field format constraints are prioritized; if multiple non-empty field values ​​conform to the field format constraints exist, the first non-empty field value is retained; if the same field does not conform to the field format constraints in any segment, the first non-empty field value is retained, and format error information is recorded. Format error information is retained along with the segment merging information for backtracking. After merging, the computing device performs format validation again on each field value in the merged attribute extraction results. Field values ​​with format errors are treated as format errors during the parsing phase, and the original output is retained.

[0220] The computing device inputs the extraction request into the semantic extraction model, which then outputs attribute extraction results based on the request. The semantic extraction model outputs structured fields according to the field format constraints, allowed inference rules, and null value handling rules in the semantic extraction template. The output process is consistent with the output method defined by the semantic extraction template in the previous embodiment. The attribute extraction results are arrays, where each element represents a configuration item. Each configuration item consists of key-value pairs where the field name is the key and the field value is the value. The attribute extraction results can be represented using JSON arrays or other structured formats.

[0221] The computing device parses the output of the semantic extraction model. During parsing, it first checks whether the output contains array start and end symbols. If the output contains only a structured array, it directly performs structured text parsing on that array. If the output contains additional descriptive text besides the structured array, the computing device locates the array start and end symbols, extracts the structured array portion between them, and parses it; the additional descriptive text is not included in the parsing. When the output is a single configuration item object instead of an array, the computing device treats that single configuration item object as an array containing only one element.

[0222] If parsing is successful, a list of configuration items is obtained. If parsing fails, the computing device records the error message and the original output, marks the acquisition task as an extraction failure, and the acquisition task does not produce target attribute results or participate in quality fusion. The error message and original output are retained for backtracking. If the parsed list of configuration items is empty, it is treated as a parsing failure.

[0223] When the attribute extraction result contains only one configuration item, the computing device directly identifies that configuration item as the target attribute result corresponding to the target parameter page, without performing a configuration item matching process. When the attribute extraction result contains multiple configuration items, such as multiple configuration versions within the same series in the target parameter page, the computing device performs a configuration item matching process to determine the target configuration item that matches the terminal identifier from among the multiple configuration items.

[0224] The first step in the configuration item matching process is to calculate the text similarity of the three fields. For each configuration item in the attribute extraction results, the computing device extracts the values ​​of the Chinese name field, the detailed model field, and the model field, and then calculates the text similarity between the terminal identifier and the Chinese name field, the terminal identifier and the detailed model field, and the terminal identifier and the model field, respectively. Before calculation, the computing device performs normalization preprocessing on the terminal identifier and field values. Normalization preprocessing includes unifying letter case, removing spaces, hyphens, underscores, parentheses, and other non-alphanumeric characters. The normalized string is used as the object for similarity calculation. Fields whose values ​​are empty after normalization are not included in the similarity calculation for that field.

[0225] The similarity calculation for the three fields follows the same methods described in the previous embodiments for character fragment matching similarity, fuzzy matching similarity, and word segmentation similarity. Fuzzy matching similarity is calculated using edit distance, with the calculation object replaced by the normalized terminal identifier and the normalized configuration item field value. After calculating the three similarities for each field, the computing device weights the three similarities according to the similarity weights set in the configuration file to obtain the text similarity between the terminal identifier and that field. The similarity weights are set in the same way as the proportion coefficients for the terminal matching score in the previous embodiments. The model field typically consists of alphanumeric codes and directly corresponds to the model code in the terminal identifier; the detailed model field may also include configuration suffixes such as memory and storage in addition to the model code; the Chinese name field may include the full brand name, series name, and marketing name. The text similarity of all three fields is between 0 and 1.

[0226] The second step in the configuration item matching process is to determine the text similarity between the terminal identifier and the configuration item. The computing device calculates the text similarity of the Chinese name field, the detailed model field, and the model field according to the field weights set in the configuration file, and then sums them to obtain the text similarity between the terminal identifier and the configuration item. The sum of the field weights is 1. In practice, a configuration of 0.5 for the model field, 0.3 for the detailed model field, and 0.2 for the Chinese name field can be used, or it can be adjusted in the configuration file.

[0227] When some fields of a configuration item are empty and do not participate in the calculation, the computing device will exclude the field weights corresponding to the empty fields from the weight sum, and then normalize the field weights of the remaining fields participating in the calculation according to the original ratio before summing them again, so as to avoid the text similarity of the configuration item being lowered due to missing fields.

[0228] The third step in the configuration item matching process is to correct the text similarity between the terminal identifier and the configuration item by combining the model code matching results and the year matching results. The computing device extracts a continuous sequence of numbers from the normalized terminal identifier, using each extracted sequence as a model code. Sequences with a length of 4 digits and values ​​falling within the year range are not used as model codes and are handled by year recognition. When any model code in the terminal identifier appears completely in the normalized field value of the model field in the configuration item, the model code of that configuration item is considered to have matched successfully. The computing device adds a code bonus value to the text similarity between the terminal identifier and the configuration item. This code bonus value is set in the configuration file; in practice, it can be 0.1. If the value exceeds 1 after adding the code bonus value, it is treated as 1. If the terminal identifier does not contain a number sequence or the model field of the configuration item is empty, no model code correction is performed, and no similarity is deducted.

[0229] For year matching, the computing device extracts a continuous four-digit sequence from the normalized terminal identifier. When the extracted four-digit sequence is a separate segment and falls within the year range set in the configuration file, it is identified as the year in the terminal identifier. If the launch time or launch date field of a configuration item contains a year, the computing device extracts that year and compares it with the year in the terminal identifier. If they match, the computing device adds a year bonus to the text similarity score, which can be 0.05 in practice; if they do not match, the computing device subtracts a year penalty from the text similarity score, which can also be 0.05 in practice. A score below 0 is treated as 0, and a score above 1 is treated as 1, resulting in the corrected text similarity score. If the terminal identifier does not contain a four-digit sequence matching the year range, or if both the launch time and launch date fields of the configuration item are empty, no year correction is performed.

[0230] In one or more embodiments of the present invention, model code extraction and year identification are performed mutually exclusively. When the computing device extracts a continuous sequence of numbers from the normalized terminal identifier, it extracts the model code as a sequence of numbers with a length greater than or equal to 3 digits and a length less than or equal to 4 digits, and a sequence of numbers with a length of 4 digits but whose values ​​are not within the year range set in the configuration file. The model code is not extracted as a sequence of numbers with a length of 4 digits and whose values ​​fall within the year range; this sequence of numbers is handled by the year identification process. Through mutual exclusion, the same sequence of numbers will not be interpreted as both a model code and a year at the same time. The four digits in the model code do not participate in the model code correction because they fall within the year range, and the mismatched configuration item will not receive a code bonus value due to model code matching.

[0231] In one or more embodiments of the present invention, year recognition also performs word boundary discrimination. After the computing device extracts a continuous four-digit sequence from the normalized terminal identifier, it checks whether the preceding and following characters of the four-digit sequence are both non-numeric characters, including the case where the preceding or following character is absent. If both the preceding and following characters of the four-digit sequence are non-numeric characters, that is, when the four-digit sequence is an independent segment in the terminal identifier, the four-digit sequence is considered as a candidate for year recognition. If either the preceding or following character of the four-digit sequence is a numeric character, it indicates that the four-digit sequence belongs to a longer continuous numeric sequence, and the four-digit sequence is not considered as a candidate for year recognition. When the four-digit sequence of a candidate for year recognition falls into the year range set in the configuration file, it is recognized as the year in the terminal identifier. Through word boundary discrimination, four-digit sequences that are part of a longer numeric sequence are not recognized as years; only four-digit sequences that are independent segments and fall into the year range participate in year matching correction. The year bonus and year penalty values ​​are configured symmetrically, both using a value of 0.05. A score below 0 is treated as 0, and a score above 1 is treated as 1. Year correction is triggered only when a four-digit sequence in the terminal identifier satisfies word boundary criteria and falls within the year range, and the launch time or launch date field value of the configuration item contains the year. If the triggering condition is not met, year correction is not performed. Through mutual exclusion extraction of model code and year, word boundary criteria, symmetrical finite correction values, and truncation processing, the four digits in the model code will not be incorrectly identified as the year, and year correction will not conversely disrupt the configuration item matching results.

[0232] The fourth step in the configuration item matching process is to determine the target configuration item. The computing device compares the corrected text similarity of each configuration item and identifies the configuration item with the highest corrected text similarity as the target configuration item. When multiple configuration items have the same corrected text similarity and are all at the highest level, the computing device sorts them based on field completeness and price information. During sorting, field completeness is compared first; field completeness refers to the number of non-empty fields in a configuration item, and configuration items with more non-empty fields are ranked higher.

[0233] When field completeness is the same, price information is compared, and configuration items with non-empty price fields are ranked before those with empty price fields. If multiple configuration items have non-empty price fields and different prices, the computing device searches the cleaning parameter text for price values ​​near the location where the terminal identifier appears, extracts the numerical portion of the price field value, and ranks configuration items whose numerical portion matches the nearest price value first. If no price value near the location where the terminal identifier appears is available for comparison in the cleaning parameter text, the original order of the configuration items in the array is maintained, and the price information is recorded in the task record as not participating in the decisive sorting.

[0234] The configuration item ranked first after the above sorting is determined as the target configuration item. When the corrected text similarity and field completeness of each configuration item are all 0, the first configuration item is taken as the target configuration item according to the original order of the array. In this case, the field values ​​of the target attribute result are mainly empty, and its field fill rate is 0, resulting in a correspondingly low overall quality score in the quality fusion stage.

[0235] The computing device writes the determined target configuration item as the target attribute result corresponding to the target parameter page into the task record of the acquisition task. In addition to the target attribute result, the task record also stores the position index of the target configuration item in the attribute extraction result array, the corrected text similarity of the target configuration item, the model code matching result, the year matching result, and the parallel sorting result. This matching process information is used for supplementary field recording in the profile record and for subsequent backtracking. Each acquisition task, after semantic extraction and configuration item matching, yields one target attribute result. Multiple target attribute results then enter the quality fusion stage.

[0236] In one or more embodiments of the present invention, the terminal matching score in the page quality assessment stage and the text similarity score in the configuration item matching stage are independent of each other. They are sequential, functionally orthogonal, and do not substitute for each other. There is no priority or mutual correction relationship between them. Their independence is set in the same way as the independence of the navigation quality threshold and the preset threshold for page quality level determination in the parameter page navigation stage. The terminal matching score is calculated on the entire cleaned parameter text and is used in the page quality assessment stage to determine the degree of matching between the target parameter page content and the terminal identifier, deciding whether the content should enter semantic extraction; this is page-level filtering. The text similarity score in the configuration item matching stage is calculated based on the structured field values ​​in the attribute extraction results and is used to determine the target configuration item among multiple configuration items on the same page; this is field-level selection. The terminal matching score's function ends when the page quality assessment is completed and it does not undertake the configuration item adjudication function. When the attribute extraction result contains multiple configuration items, and the corrected text similarity of the multiple configuration items is the same, the computing device processes them according to the previously recorded decision chain. That is, it first compares the field completeness; if the field completeness is also the same, it compares the price information; if the price information cannot be distinguished, it maintains the original order of the configuration items in the array, and records the decision process information in the task record. In the case of parallel items, it is not necessary to combine the terminal matching score; the terminal matching score in the page quality assessment stage is not used as the basis for sorting configuration items.

[0237] In one or more embodiments of the present invention, a comprehensive quality score is calculated for the target attribute results based on the page quality score and the field fill rate calculated based on the attribute field set; the target attribute result with the highest comprehensive quality score is determined as the main attribute result, and the field values ​​from the target attribute results other than the main attribute result are added to the missing fields in the main attribute result to obtain the fused attribute result, including:

[0238] The field fill rate is the ratio between the number of non-empty fields in the target attribute result and the total number of fields in the attribute field set;

[0239] The page quality score is normalized to obtain a normalized page quality score.

[0240] The overall quality score is calculated using the following formula:

[0241] Q = a × S + b × F

[0242] Where Q is the overall quality score, S is the normalized page quality score, F is the field fill rate, a and b are weight coefficients, and a+b=1;

[0243] The target attribute with the highest overall quality score is determined as the primary attribute.

[0244] When there are missing fields in the main attribute results, the non-null values ​​of the corresponding fields are searched in descending order of the overall quality scores of the other target attribute results.

[0245] Fill the missing fields in the main attribute result with the found non-null values;

[0246] The source target parameter page, source collection task, source page quality information, and source comprehensive quality score will be added as supplementary field records.

[0247] The supplemented main attribute result is used as the fused attribute result.

[0248] After the multi-path target attribute results are generated, the computing device enters the quality fusion stage. The quality fusion stage includes processes such as field fill rate calculation, page quality score normalization, comprehensive quality score calculation, main attribute result determination, missing field supplementation, and fused attribute result generation. The input to quality fusion is the target attribute results and page quality scores saved in the task records of each acquisition task. The output is the fused attribute results, main source information, and field supplement records. The fused attribute results, along with the main source information and field supplement records, enter the profile record association stage.

[0249] The computing device first calculates the field fill rate for each target attribute result. The field fill rate F represents the ratio between the number of non-empty fields in the target attribute result and the total number of fields in the attribute field set. Before calculation, the computing device reads the list of field names in the attribute field set, and the total number of fields is based on the number of fields in the attribute field set. When determining non-empty fields, the computing device traverses each field in the attribute field set, checks whether the key corresponding to the field exists in the target attribute result, and ensures that the field value is not an empty string, not a null value marker, or not an empty object.

[0250] Fields with a value of 0 or a Boolean "No" are considered non-empty fields. These two types of field values ​​represent valid attribute information and are not counted as empty fields. Empty fields output during the semantic extraction phase according to the null value handling rules, whose values ​​meet the null value judgment criteria, are counted as missing fields. After judgment, the computing device counts the number of non-empty fields and divides this number by the total number of fields to obtain the field fill rate. The field fill rate ranges from 0 to 1, and the computing device records the field fill rate to two decimal places in the task record for that acquisition task. The field fill rate reflects the completeness of the target attribute result relative to the attribute field set, while the page quality score reflects the credibility of the source page. These two are different dimensions but are used together as input for the comprehensive quality score.

[0251] In one or more embodiments of the present invention, the non-empty field determination also excludes placeholder values. Placeholder values ​​include unknown, N / A, n / a, null, None, empty string, null value marker, and empty object. When a field value in the target attribute result is a placeholder value, the field is treated as an empty field and is not counted in the number of non-empty fields. The placeholder values ​​correspond to the default values, placeholder values, and speculative values ​​that are prohibited by the null value handling rules in the semantic extraction template. By excluding placeholder values, invalid field values ​​output by the semantic extraction model in the form of placeholder values ​​will not increase the field fill rate.

[0252] In one or more embodiments of the present invention, the computing device performs field value backtracking verification on non-empty field values ​​determined by placeholder exclusion. Field value backtracking verification is used to determine whether the field value has a basis in the cleaning parameter text. The verification method is as follows: the computing device checks whether the key components of the field value appear in the cleaning parameter text of the acquisition task. Key components include the numeric sequence, numerical unit, model code, and brand text in the field value, and checks whether the field value has a source estimation label. If at least one key component of the field value appears in the cleaning parameter text, or if the field value has a source estimation label, the field value backtracking verification is deemed to have passed, and it is counted as a non-empty field. If none of the key components of the field value appear in the cleaning parameter text, and the field value does not have a source estimation label, the field value backtracking verification is deemed to have failed, and the field value is downgraded to a low-confidence field. Low-confidence fields are included in the field fill rate calculation using a low-confidence field discount factor, which can be 0.5 in practice. The semantic extraction model's inferred values ​​for fields not actually included in the page will not be fully counted in the non-empty field count if the backtracking verification fails because key components cannot be located in the cleaned parameter text. The field fill rate will not be artificially inflated due to the inferred values. The results of the field value backtracking verification are saved with the task record for review during profile record backtracking.

[0253] The computing device normalizes the page quality score to obtain a normalized page quality score. The page quality score is obtained by a weighted sum of the content completeness score, the structured content score, and the terminal matching score. In this embodiment, all three scores are values ​​between 0 and 1, and the sum of the three weight coefficients is 1. The page quality score ranges from 0 to 1. During normalization, the computing device reads the page quality score from the task record of the data collection task, divides the page quality score by the theoretical maximum value of the page quality score (1), and obtains the normalized page quality score. The normalized page quality score is retained to two decimal places and written back to the task record.

[0254] With the page quality score configured to range from 0 to 1, the values ​​before and after normalization are the same. The normalization step serves to unify the calculation criteria, ensuring that the normalized page quality score and field fill rate in the comprehensive quality score formula fall within the same value range. When the calculation method for the page quality score is adjusted so that its upper limit is no longer 1, the normalization process still maps the page quality score to the 0-1 range, maintaining consistency in the calculation criteria of the comprehensive quality score formula. The computing device performs normalization processing separately for each acquisition task that generates target attribute results.

[0255] The computing device calculates a comprehensive quality score for each target attribute result based on the normalized page quality score and the field fill rate. The comprehensive quality score is calculated using the formula Q = a × S + b × F, where Q represents the comprehensive quality score, S represents the normalized page quality score, F represents the field fill rate, and a and b are weighting coefficients, with a + b = 1. In practice, a is set to 0.3 and b to 0.7. The weighting coefficient for the field fill rate is greater than that for the normalized page quality score because the direct goal of quality fusion is to improve the field completeness of the final attribute profile.

[0256] The values ​​of 'a' and 'b' are set in the configuration file. All collection tasks within the same processing batch use the same set of weighting coefficients to ensure comparability between multi-path comprehensive quality scores. The comprehensive quality score ranges from 0 to 1, and the computing device writes the comprehensive quality score to two decimal places in the task record. The calculation of the comprehensive quality score is limited to collection tasks that produce target attribute results. Collection tasks with a page quality level of BAD do not produce target attribute results and are not included in the comprehensive quality score calculation. Collection tasks that fail to parse during the semantic extraction stage do not produce target attribute results and are not included in the comprehensive quality score calculation. Collection tasks with an empty configuration item list after parsing are treated as parsing failures and are not included in the comprehensive quality score calculation. The low-confidence source marker record navigation method added to the fallback crawled content during the parameter page navigation stage is not included in the calculation of page quality score and comprehensive quality score. The fallback crawled content, as a navigation result, still needs to undergo content cleaning and page quality assessment. Its normalized page quality score is calculated by the page quality assessment stage based on the cleaned parameter text. When the page quality level of the fallback crawled content is WEAK, this crawling task performs semantic extraction and generates target attribute results, which participate in quality fusion. Its overall quality score is calculated according to a formula, with lower source page quality reflected in the value of the normalized page quality score S. When the page quality level of the fallback crawled content is BAD, this crawling task does not generate target attribute results and does not participate in quality fusion. The low-confidence source marker is only used to indicate that the results were obtained through a fallback method, for use in profile record backtracking.

[0257] In one or more embodiments of the present invention, the navigation quality threshold in the parameter page navigation stage and the page quality level in the page quality evaluation stage belong to different stages and are not interchangeable. The navigation quality threshold is used to determine whether a single crawl has located the parameter content, while the page quality evaluation is used to determine whether the parameter information in the navigation result is sufficient. If the fallback crawled content does not reach either navigation quality threshold, it still needs to pass the page quality evaluation before it can enter semantic extraction and quality fusion. When the fallback crawled content is determined to be at the BAD level, no target attribute result is generated, and it does not participate in quality fusion, as previously stated.

[0258] In one or more embodiments of the present invention, after the catch-all crawled content generates target attribute results, its participation in quality fusion is distinguished based on whether there are target attribute results from other navigation methods. When there are target attribute results from other navigation methods besides catch-all crawling, the target attribute results corresponding to the catch-all crawled content do not participate in the main attribute result competition, but only participate as candidate supplementary results in filling missing fields. The order of candidate supplementary results is still executed according to the comprehensive quality score from high to low. When there are no other target attribute results from other navigation methods besides catch-all crawling, that is, when all target attribute results are generated by catch-all crawling, the main attribute result is determined from the target attribute results corresponding to each catch-all crawled content according to the comprehensive quality score. Through the above distinction, catch-all crawled content will not become the main attribute result when there is a normal navigation source. The content from the normal navigation source is given priority as the main data of the attribute profile, and the catch-all crawled content only undertakes the function of field supplementation.

[0259] In one or more embodiments of the present invention, the fields supplemented by fallback crawled content are forcibly marked with low-confidence source annotations in the field supplementation records. After the low-confidence source annotation is written into the source page quality information of the field's source record sub-item, it is stored separately from the source page quality information. When the fallback crawled content becomes the main attribute result, the low-confidence source annotation is added to the field supplementation records in the profile record association stage, and the main data of the profile record is obtained through the fallback method. The low-confidence source annotation is written into the terminal attribute library along with the profile record, and downstream processes can use the profile records hierarchically based on the low-confidence source annotation.

[0260] The computing device determines the target attribute result with the highest overall quality score as the primary attribute result. The computing device compares the overall quality scores of each target attribute result, takes the target attribute result with the highest overall quality score as the primary attribute result, and records the corresponding target parameter page address, acquisition task identifier, page quality score, field fill rate, and overall quality score as the primary source information.

[0261] When the overall quality scores of multiple target attribute results are the same and all are the highest, the computing device compares the normalized page quality scores of the corresponding parallel paths, and the one with the higher normalized page quality score is determined as the primary attribute result. If the normalized page quality scores are still the same, the computing device selects the path created earliest according to the acquisition task creation order and determines it as the primary attribute result, and writes the parallel determination information into the batch processing record. After the primary attribute result is determined, the target attribute results other than the primary attribute result are used as candidate supplementary results. The candidate supplementary results are sorted from high to low according to the overall quality score. If the overall quality scores are the same, they are arranged according to the acquisition task creation order. The sorting results are used in the supplementation process.

[0262] The computing device checks for empty fields in the main attribute results. The criteria for determining empty fields follow the same standard used in the field fill rate calculation phase: fields whose keys are non-existent, or whose values ​​are empty strings, null values, or empty objects are considered empty fields. The computing device iterates through the empty fields sequentially according to the field order maintained in the attribute field set, performing a fill lookup for each empty field.

[0263] For the currently missing field, the computing device searches each path sequentially, starting from the first path of the sorted candidate supplementary results, checking whether the field value corresponding to the currently missing field in the target attribute result of that path is a non-null value. When the first non-null field value is found and passes the field value backtracking verification, the computing device adds the non-null field value to the missing field in the main attribute result, and the search for that missing field ends. The value of that field in subsequent sorted candidate supplementary results is no longer checked. When no non-null value exists for the current field in all candidate supplementary results, the field remains missing, and the computing device adds the field to the list of unsupplemented fields. After the field-by-field supplementation search is completed, each missing field in the main attribute result is either supplemented or recorded in the list of unsupplemented fields, with each supplemented field corresponding to a determined supplementation source.

[0264] In one or more embodiments of the present invention, the supplementary search phase performs field value backtracking verification on non-empty field values ​​in the candidate supplementary results. The verification method is the same as the field value backtracking verification in the field fill rate calculation phase, that is, checking whether the key components of the field value appear in the cleaning parameter text of the acquisition task, or whether the field value has a source inference label. For the current missing field, when the computing device searches path by path starting from the first path of the sorted candidate supplementary results, the field value is only considered as a supplementable value if the field value corresponding to the current missing field in the target attribute result of that path is non-empty and the field value backtracking verification passes. If the field value is non-empty but the field value backtracking verification fails, the field value is skipped, and the search continues for the next path of candidate supplementary results. When the first non-empty field value that passes the verification is found, the computing device supplements the missing field in the main attribute result with the field value.

[0265] In one or more embodiments of the present invention, the supplementary search phase also performs multi-source consistency verification and supplementary source quality threshold control. For the currently missing field, the computing device first collects the corresponding non-empty field values ​​that have passed verification from all candidate supplementary results, and compares whether the collected field values ​​are consistent. When all collected field values ​​are consistent, the field value is added to the missing field in the main attribute result, and multi-source consistency is marked in the source record sub-item of the field. When there is inconsistency in the collected field values, the field value from the candidate supplementary result with the highest comprehensive quality score is added to the missing field in the main attribute result, and a conflict is marked in the source record sub-item of the field. At the same time, the field values ​​of each party and the source target parameter pages of each party are recorded. Supplementary source quality threshold control means that the computing device sets a supplementary quality threshold value in the configuration file. When the comprehensive quality score of the candidate supplementary result is lower than the supplementary quality threshold value, the candidate supplementary result does not participate in the missing field supplementation. The corresponding field remains empty and is added to the list of unsupplemented fields when the comprehensive quality scores of all candidate supplementary results are lower than the supplementary quality threshold value. Through multi-source consistency verification and supplementary source quality threshold control, when a source with a higher overall quality score gives an incorrect value in a specific field while a source with a lower overall quality score gives a correct value, the incorrect value is marked due to the conflict and the values ​​of each party are retained. Furthermore, the field value backtracking verification has excluded field values ​​that are not supported by evidence in the cleaned parameter text. The supplemented fusion attribute results have traceable quality information at the specific field level.

[0266] The computing device generates a source record for each supplemented field, and all source records constitute a field supplement record. Each supplemented field corresponds to a source record sub-item, which includes the supplemented field name, supplemented field value, source target parameter page, source acquisition task, source page quality information, and source comprehensive quality score. The source target parameter page records the address of the target parameter page corresponding to the supplemented field value, the source acquisition task records the acquisition task identifier corresponding to the supplemented field value, the source page quality information includes the page quality score and page quality level of the acquisition task, and the source comprehensive quality score records the comprehensive quality score of the target attribute result.

[0267] The source page quality information also retains the page quality score before normalization processing, facilitating the viewing of the original score during the page quality assessment stage when tracing the source. Supplementary field records are written to the batch processing record in list form and enter the profile record association stage along with the fused attribute results. The profile record association stage uses these supplementary field records as one of the associated information for the network terminal attribute profile record.

[0268] The main attribute result after the supplementary lookup is completed is the merged attribute result. In the merged attribute result, the field values ​​from the main attribute result remain unchanged, the field values ​​of the supplemented fields come from the supplementary source recorded in the field supplementation record, and fields that could not obtain a non-null value from any target attribute result remain empty and are recorded in the list of unsupplemented fields.

[0269] When there is only one target attribute result participating in quality fusion, that target attribute result is directly determined as the primary attribute result. There are no candidate supplementary results, the fused attribute result is the same as the primary attribute result, and the supplementary field record is an empty list. When there are no target attribute results, that is, when all acquisition tasks have not generated target attribute results, the computing device does not generate fused attribute results. The batch processing record identified by this terminal is marked as having no valid source, and all cleaning parameter text and error information are retained for backtracking.

[0270] In one or more embodiments of the present invention, when a processing batch of a terminal identifier has no target attribute results, or the overall quality score corresponding to the main attribute result is lower than the retry threshold set in the configuration file, the terminal identifier enters the retry processing flow. The retry threshold can be 0.6. In the retry processing flow, the computing device writes the terminal identifier into the retry queue and performs automatic retry according to the maximum number of retry attempts set in the configuration file. During retry, the computing device adjusts the recall strategy, including at least one of the following: switching the search engine used for the search request from the first search engine to the second search engine, increasing the number of parameter resource keywords corresponding to the targeted search request, increasing the preset number of target parameter pages, and adjusting the navigation quality threshold configuration in the parameter page navigation stage. The retry processing flow re-executes the candidate parameter page recall, filtering, parallel acquisition, semantic extraction, configuration item matching, and quality fusion for the terminal identifier to obtain the fused attribute results of the retry batch. The computing device compares the overall quality scores of the original processing batch and the retry batch. The batch with the higher overall quality score is written as the final result; if the overall quality scores are the same and the field fill rate of the retry batch is not higher than that of the original processing batch, the results of the original processing batch are retained.

[0271] In one or more embodiments of the present invention, when the processing batch of a terminal identifier still has no target attribute result after the maximum number of retries has been reached, the computing device stops retries, removes the terminal identifier from the retry queue, and outputs a manual review request consisting of the terminal identifier, terminal type, target parameter page address, cleaning parameter text, page quality score, error message, and no valid source label to the manual review queue. The manual review request also includes a failure reason summary, which includes the page quality level, navigation method, failure stage, and failure reason for each acquisition task. After processing the manual review request, maintenance personnel can resubmit the terminal identifier to the input and terminal type configuration stage for reprocessing. Through the retry processing flow and manual review output, terminal identifiers without target attribute results have a complete processing path of labeling, retrying, adjusting recall strategy, and manual review, and the batch processing process will not be interrupted due to the processing failure of individual terminal identifiers.

[0272] When the overall quality scores are the same, selecting the path with the higher normalized page quality score as the primary attribute result will not reduce the field completeness of the final attribute profile. The field completeness of the fused attribute result is composed of the non-empty fields of the primary attribute result and the non-empty fields supplemented by the candidate supplementary results; that is, the union of all non-empty fields of the target attribute results. This union is independent of which path is selected as the primary attribute result from among the parallel paths. Selecting the path with the higher normalized page quality score as the primary attribute result allows the field values ​​retained in the primary attribute result to come from the collection task with higher source page quality, while maintaining the same level of field completeness.

[0273] In one or more embodiments of the present invention, the associated information portion of the network terminal attribute profile record may further include, in addition to the terminal identifier field, terminal type field, target parameter page address field, comprehensive quality score field, and field supplementary record field, a creation time field, a semantic extraction model identifier field, a resource type field, and a quality level field. The creation time field and the semantic extraction model identifier field describe the processing information of the current profile record generation process, while the resource type field and the quality level field describe the source characteristics of the main attribute result source page. These four fields, together with the aforementioned associated information fields, constitute the complete associated information of the network terminal attribute profile record, for retrospective and auditing purposes after the profile record is stored in the database.

[0274] The creation time field records the time when the computing device generates the network terminal attribute profile record. The creation time is written when the computing device completes the generation of the fused attribute results and enters the profile record association stage. The writing format is a time value composed of year, month, day, hour, minute, and second, and the time source is the system time of the computing device executing the terminal identifier processing task. Within the same batch, network terminal attribute profile records corresponding to different terminal identifiers each record their own creation time. The creation time does not change after the profile record is written to the terminal attribute database. When subsequent field updates occur in the profile record, the update operation time and creation time are stored separately, with the creation time remaining unchanged from the initial generation time.

[0275] The semantic extraction model identifier field records the identifier of the semantic extraction model called to generate the main attribute result during this processing. The semantic extraction model identifier is used to distinguish different versions, training batches, or deployment instances of the semantic extraction model. The identifier can be the model name plus a version number, or a unique number assigned to the semantic extraction model in the configuration file. The semantic extraction model identifier is read from the configuration file when the computing device calls the semantic extraction model to perform extraction. Based on the content saved in the aforementioned task record, it is written to the task record of the collection task along with the attribute extraction result. During the profile record association stage, the semantic extraction model identifier saved in the task record of the collection task corresponding to the main attribute result is written to the semantic extraction model identifier field. When the semantic extraction model corresponding to the main attribute result participating in the quality fusion stage is different from the semantic extraction model corresponding to the candidate supplementary result, the semantic extraction model identifier field only records the semantic extraction model identifier corresponding to the main attribute result. The semantic extraction model identifier corresponding to the candidate supplementary result is retained in the field supplementary record and saved together with the source information of each supplementary field for easy traceability.

[0276] The Resource Type field records the resource type of the target parameter page from which the main attribute result originates. The resource type is output by the semantic selection model during the candidate parameter page filtering stage. The resource type values ​​include official resources, parameter database resources, and third-party parameter resources. During the semantic selection stage, the computing device writes the resource type of the candidate parameter page into the structured record of that candidate parameter page. The resource type is saved as a field added to the title, address, and source fields in the structured record. During the acquisition stage, it is written to the task record of the acquisition task along with the target parameter page. During the profile record association stage, the resource type in the task record corresponding to the main attribute result is written into the Resource Type field. The Resource Type field enables the profile record to identify the source category of the main attribute result's main field value. When the target parameter page corresponding to the main attribute result belongs to an official resource, the Resource Type field records it as an official resource. When the target parameter page corresponding to the main attribute result belongs to a parameter database resource or a third-party parameter resource, the Resource Type field records the corresponding source category. The resource type of the candidate supplementary results is retained in the field supplementary record for review during supplementary field tracing.

[0277] The Quality Grade field records the page quality grade of the target parameter page from which the main attribute result originates. The page quality grade is output during the page quality assessment phase and includes three levels: GOOD, WEAK, and BAD. GOOD corresponds to good, WEAK to weak, and BAD to unacceptable. The computing device writes the page quality grade along with the page quality score into the task record of the data collection task. During the profile record association phase, the page quality grade from the task record corresponding to the main attribute result is written into the Quality Grade field. Since BAD-level data collection tasks do not produce attribute extraction results and do not participate in quality fusion, the page quality grade of the data collection task corresponding to the main attribute result can only be GOOD or WEAK, and the Quality Grade field value is correspondingly GOOD or WEAK. The page quality grade of the source page for candidate supplementary results is retained in the source page quality information of the supplementary field record for reference during supplementary field tracing.

[0278] This invention automates the entire process of "multi-source recall, candidate screening, multi-path parallel acquisition, page quality assessment, semantic extraction, quality fusion and profile construction," transforming the completion of the terminal attribute library from manual querying, comparison and entry to batch automatic execution. Different terminal types achieve field constraints by loading corresponding attribute field sets and semantic extraction templates, eliminating the need for individual customization for each model. This significantly reduces operation and maintenance costs and improves the efficiency of terminal attribute library completion.

[0279] This invention employs rule-based filtering and semantic selection of candidate parameter pages. First, it filters out definitively invalid pages using blacklisted domains, documents, shallow paths, ad redirects, and duplicate links. Then, a semantic selection model sorts the remaining candidates and selects multiple target parameter pages. This excludes pages such as encyclopedias, Q&A sites, forums, advertisements, document downloads, and official website homepages from the data collection process, improving the accuracy of parameter page location. Furthermore, for complex webpage structures such as overview pages, parameter subpages, dynamic tabs, and redirect pages, this invention utilizes a multi-level navigation mechanism consisting of direct crawling, parameter link extraction, automated browser triggering, and fallback crawling. Combined with content cleaning based on parameter start and end keywords, it can locate and extract parameter text from pages with different structures, reducing the impact of webpage structure differences on the extraction results.

[0280] This invention performs page quality assessment on the cleaned parameter text before semantic extraction, generating a page quality score based on three dimensions: content completeness, structuring degree, and terminal matching degree. This distinguishes low-quality and low-confidence pages and restricts them from entering the normal extraction process, reducing the generation of erroneous and empty fields from the source. When a parameter page contains multiple configuration versions of the same series, the text similarity between the terminal identifier and each configuration item in the Chinese name, detailed model, and model fields is calculated. This is then corrected by combining model code matching and year matching. If the text similarity is the same after correction, it is further sorted according to field completeness and price information, thereby determining the target configuration item that best matches the input terminal identifier, reducing the risk of misselecting multiple configurations in the same series.

[0281] To address the issue of missing fields from a single source, this invention does not simply use a single page result. Instead, it measures each target attribute result using a comprehensive quality score Q = a × S + b × F. Here, the normalized page quality score S reflects the reliability of the source page, and the field completion rate F reflects the completeness of the extracted results. This weighted average ensures that the main attribute result comes from a higher-quality source, while also supplementing missing fields in the main attribute result using non-empty fields from other sources, in descending order of comprehensive quality score. The source target parameter page, source acquisition task, source page quality information, and source comprehensive quality score are recorded for each supplemented field, making field completion traceable. Finally, the fused attribute result is associated with the terminal identifier, terminal type, target parameter page address, comprehensive quality score, and field supplementation record to form a standardized terminal attribute profile record. This record can be directly written into the terminal attribute database for automatic field completion, or output to downstream processes such as device capability assessment and operator user profile construction, achieving end-to-end traceability of terminal attribute data from collection, extraction, fusion to database storage.

[0282] The above provides a detailed description of the terminal attribute profile construction method based on parallel acquisition and quality fusion of multi-source parameter pages. Specific examples are used in this paper to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for constructing terminal attribute profiles through parallel acquisition and quality fusion of multi-source parameter pages, characterized in that, The method includes: Receive terminal identifier and terminal type, load attribute field set and semantic extraction template based on terminal type; recall candidate parameter pages based on terminal identifier, and filter target parameter pages from candidate parameter pages; The target parameter page is collected in parallel, and parameter page navigation and content cleaning are performed to obtain the cleaned parameter text corresponding to the target parameter page; the page quality of the cleaned parameter text is evaluated to obtain the page quality score corresponding to the cleaned parameter text; The semantic extraction template and the cleaned parameter text are input into the semantic extraction model to obtain the attribute extraction result; the target configuration item that matches the terminal identifier is determined from the attribute extraction result as the target attribute result corresponding to the target parameter page; Based on the page quality score and the field fill rate calculated based on the attribute field set, a comprehensive quality score is calculated for the target attribute results; the target attribute result with the highest comprehensive quality score is determined as the main attribute result, and the field values ​​of the target attribute results other than the main attribute result are added to the missing fields of the main attribute result to obtain the fused attribute result; The fusion attribute results are associated with the terminal identifier, the terminal type, the address of the target parameter page, the comprehensive quality score, and the supplementary field records to generate a network terminal attribute profile record.

2. The method according to claim 1, characterized in that, The attribute field set includes attribute fields corresponding to the terminal type. The attribute fields include at least one of the following: brand, model, price, launch date, central processing unit, memory, hard disk, operating system, network standard, Wi-Fi protocol, Wi-Fi frequency band, Bluetooth version, battery capacity, screen size, screen refresh rate, and number of multiple inputs and multiple outputs. The semantic extraction template is used to constrain the semantic extraction model to output structured fields according to the attribute field set, and to limit the field format, allowed inference rules and null value handling rules.

3. The method according to claim 2, characterized in that, The candidate parameter page based on the terminal identifier includes: A search request is generated based on the terminal identifier; The search request is used to access the search engine results page, product parameter database, manufacturer's official website and third-party specification site, and the candidate parameter page is collected from the search engine results page, product parameter database, manufacturer's official website and third-party specification site; Save the title, address, and source information of the candidate parameter page.

4. The method according to claim 3, characterized in that, The step of filtering the target parameter page from the candidate parameter page includes: The candidate parameter page is subjected to filtering processing, which includes blacklist domain filtering, document link filtering, shallow path link filtering, advertisement jump link filtering, and duplicate link filtering. After filtering, the title and address information of the candidate parameter pages are input into the semantic selection model, and the semantic selection model outputs the priority of the candidate parameter pages; Based on the priority, a preset number of candidate parameter pages with the highest priority are selected as the target parameter pages.

5. The method according to claim 4, characterized in that, The parallel acquisition of the target parameter page includes: Create an independent acquisition task for each of the target parameter pages; The parameter page navigation is executed in the independent data collection task. The parameter page navigation includes direct crawling, parameter link extraction, browser-automated triggering, and fallback crawling. When the quality of the directly crawled text is lower than the first navigation quality threshold, links containing parameter keywords are extracted and crawled a second time. When the quality of the text obtained from the secondary crawling is lower than the second navigation quality threshold, the parameter-related elements on the page are automatically triggered by the browser. When the quality of the text obtained by the browser's automated trigger is lower than the third navigation quality threshold, the text obtained by direct crawling is used as the fallback parameter content.

6. The method according to claim 5, characterized in that, The content cleaning includes: Extract the page title and page source URL; Remove image links, navigation text, footer text, advertising text, customer service text, copyright text, and irrelevant links; Locate the beginning position of the parameter content based on the starting keyword of the parameter, and locate the end position of the parameter content based on the ending keyword of the parameter. The text between the start position and the end position of the parameter content is used as the cleaning parameter text; The starting keywords for the parameters include "configuration filter", "model", "specification parameters", "detailed parameters", "important parameters", "hardware specifications", "technical specifications", "basic parameters" and "product parameters". The parameters end with keywords including "related recommendations", "same series products", "precautions", "all rights reserved" and "Copyright".

7. The method according to claim 6, characterized in that, The page quality assessment includes: The content completeness score is calculated based on the effective text length and effective number of lines of the cleaned parameter text. The degree of structuring is calculated based on the number of table rows, key-value pair rows, or structured rows in the cleaning parameter text. A terminal matching score is calculated based on the similarity between the terminal identifier and the cleaning parameter text. The page quality score and page quality level are generated based on the content completeness score, the structuredness score, and the terminal matching score; The page quality ratings include GOOD, WEAK, and BAD ratings. When the page quality level is GOOD, the cleaning parameter text is marked as normal semantic extraction text; When the page quality level is WEAK, the cleaning parameter text is marked as low-confidence semantic extraction text; When the page quality level is BAD, the cleaning parameter text is retained for backtracking.

8. The method according to claim 7, characterized in that, The process of generating the page quality score and page quality level based on the content completeness score, the structuredness score, and the terminal matching score includes: The page quality score is obtained by weighted summation of the content completeness score, the structured score, and the terminal matching score. The weighted summation uses a first weight coefficient, a second weight coefficient, and a third weight coefficient, the sum of which is 1. When the content completeness score is lower than the first preset threshold, the structuredness score is lower than the second preset threshold, or the terminal matching score is lower than the third preset threshold, the page quality level is set to BAD level. When the content completeness score reaches the first preset threshold, the structuring score reaches the second preset threshold, and the terminal matching score reaches the third preset threshold, the page quality level is set to GOOD level or WEAK level according to the relationship between the page quality score and the fourth preset threshold.

9. The method according to claim 8, characterized in that, The step of inputting the semantic extraction template and the cleaned parameter text into the semantic extraction model to obtain the attribute extraction result; and determining the target configuration item matching the terminal identifier from the attribute extraction result as the target attribute result corresponding to the target parameter page, includes: The semantic extraction template and the cleaning parameter text are combined into an extraction request; The semantic extraction model outputs attribute extraction results according to the extraction request. The attribute extraction results are an array, and each element in the array represents a configuration item. Calculate the text similarity between the terminal identifier and the Chinese name field in the configuration item, the text similarity between the terminal identifier and the detailed model field in the configuration item, and the text similarity between the terminal identifier and the model field in the configuration item; The text similarity between the terminal identifier and the configuration item is determined based on the text similarity of the Chinese name field, the text similarity of the detailed model field, and the text similarity of the model field; The text similarity between the terminal identifier and the configuration item is corrected by combining the model code matching result and the year matching result to obtain the corrected text similarity. The configuration item with the highest text similarity after correction is determined as the target configuration item; When multiple configuration items have the same text similarity after modification, the target configuration item is determined by sorting them according to field completeness and price information. The determined target configuration item is used as the target attribute result.

10. The method according to claim 9, characterized in that, The process involves calculating a comprehensive quality score for the target attribute results based on the page quality score and the field fill rate calculated based on the attribute field set; determining the target attribute result with the highest comprehensive quality score as the primary attribute result; and supplementing the missing fields in the primary attribute result with field values ​​from target attribute results other than the primary attribute result to obtain the fused attribute result, including: The field fill rate is the ratio between the number of non-empty fields in the target attribute result and the total number of fields in the attribute field set; The page quality score is normalized to obtain a normalized page quality score. The overall quality score is calculated using the following formula: Q = a × S + b × F Where Q is the overall quality score, S is the normalized page quality score, F is the field fill rate, a and b are weight coefficients, and a+b=1; The target attribute with the highest overall quality score is determined as the primary attribute. When there are missing fields in the main attribute results, the non-null values ​​of the corresponding fields are searched in descending order of the comprehensive quality scores of the other target attribute results. The found non-empty values ​​are added to the missing fields in the main attribute results; The source target parameter page, source collection task, source page quality information, and source comprehensive quality score will be added as supplementary field records. The supplemented main attribute result is used as the fused attribute result.

Images