Large model-based data acquisition method, medium and equipment

Abstract

The invention relates to the technical field of data acquisition, in particular to a data acquisition method based on a large model, a medium and equipment, and solves the defect of one-step period setting by establishing a correlation model of content characteristics, user requirements and acquisition strategies, and improves data timeliness. Accounts are grouped based on historical acquisition records and data acquisition periods, so that the problems of insufficient high-difficulty task accounts and idle low-frequency task accounts caused by static account allocation are solved; by inputting the multi-element collaborative data collection rule into the pre-trained large model, the efficiency and standardization degree of data collection are improved, the effectiveness of the initial collection text is improved, relevance verification is performed on the initial collection text through the pre-trained large model, and the target collection text is obtained through correction according to the verification result. Finally, the target collection text which is uniform in format, free of contradiction and stable in quality is output, and the efficiency and reliability in the data collection process are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data collection, in particular to a data collection method based on a large model, a medium and equipment. BACKGROUND

[0002] With the explosive growth of Internet data, data collection technology based on large models has become an important means of efficient information acquisition. The existing data collection method based on large models usually adopts fixed and unified collection frequency and rules, which cannot flexibly adapt to the dynamic update characteristics of different websites and the real-time changes of business needs, easily leading to invalid collection of low-frequency update websites or information delay of high-frequency update websites, and failing to fully consider the optimization and risk avoidance of collection accounts, resulting in extensive allocation of collection account resources, low efficiency and high risk, and unable to achieve efficient and reliable data collection goals.

[0003] The Chinese invention patent "Data collection method, device and electronic equipment" (authorization number CN111680203B, authorization date April 18, 2023) adopts a two-stage collection mode combining rough collection and fine collection, and intelligently judges which rough collection data is worth fine collection through a value prediction model, thereby completing the data collection task. The collection frequency and timing of this patent are completely determined by the output of the value prediction model, but it does not consider the historical update frequency and regularity of the target website itself, and focuses on the judgment of data value, completely ignoring the allocation and management of resources needed to perform collection tasks, ignoring the content update characteristics of different websites, user demand differences and dynamic allocation of collection resources, resulting in low collection efficiency, serious resource waste, and difficulty in adapting to real-time acquisition needs of multi-source heterogeneous data.

[0004] Therefore, how to improve the efficiency and reliability in the data collection process has become a problem to be solved. SUMMARY

[0005] To solve the above technical problems, the technical solution adopted by the present application is a data collection method based on a large model, which includes the following steps: S1, according to the historical update frequency and demand degree of each preset title in each target website, obtaining the data collection period corresponding to each preset title, wherein the demand degree is used to represent the matching degree of the content corresponding to the preset title and the user demand.

[0006] S2, according to the historical collection record and the data collection period corresponding to each preset title in each target website, grouping the N data collection accounts, obtaining the account set corresponding to each target website, wherein the historical collection record includes the full amount of records retained in the preset historical period for performing the data collection operation on the preset title, N is a positive integer, and each account set includes a plurality of data collection accounts corresponding to each preset title.

[0007] S3, inputting the data collection rule corresponding to each target website into the pre-trained large model, obtaining the initial collection text corresponding to each preset title in each target website, wherein the data collection rule at least includes the data collection period, the account set, the collection type and the collection level corresponding to each preset title, the collection type is the data type of the content corresponding to the preset title, and the collection level is used to determine the collection depth of the preset title with multiple levels of subpages.

[0008] S4, based on the pre-trained large model, performing relevance verification on the initial collection text corresponding to each preset title, correcting the initial collection text according to the verification result, and obtaining the target collection text.

[0009] The application further provides a non-transitory computer readable storage medium, which stores at least one instruction or at least one program, and the at least one instruction or at least one program is loaded and executed by a processor to implement the above-mentioned data collection method based on a large model.

[0010] The application further provides an electronic device, which comprises a processor and the above-mentioned non-transitory computer readable storage medium.

[0011] The present application has at least the following beneficial effects: by establishing the association model of content characteristics, user demand and collection strategy, the drawbacks of one-size-fits-all period setting in traditional collection methods are solved, which avoids ineffective collection of low-frequency low-demand content, reduces server resource consumption, ensures timely acquisition of high-frequency high-demand content, and improves data timeliness; based on historical collection records reflecting task difficulty and data collection period reflecting task frequency, N data collection accounts are dynamically grouped, and finally a set of accounts dedicated to each target website is formed, so as to solve the problems of insufficient high-difficulty task accounts and idle low-frequency task accounts caused by traditional static account allocation through precise matching of task characteristics and account resources; by inputting the multi-element collaborative data collection rule into the pre-trained large model, the efficiency and standardization of data collection are improved, through the binding of account set and data collection period, the account called each time is an account with matched permissions and good health status, and the collection frequency conforms to the content update rule, which reduces the collection failure rate, through the explicit constraint of collection type and collection level, the collection content is focused on the target data form and page range, avoiding the waste of storage resources caused by irrelevant content collection, improving the effectiveness of the initial collection text, and through the correlation verification of the pre-trained large model on the initial collection text, the initial collection text is modified according to the verification result, and the target collection text is obtained, the contradiction between texts is corrected, and the conflict-free information is integrated, so that the final output target collection text is uniform in format, free of contradictions and stable in quality, thereby improving the efficiency and reliability in the data collection process. BRIEF DESCRIPTION OF DRAWINGS

[0012] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can be obtained without creative labor.

[0013] Figure 1 A flowchart of a data collection method based on a large model provided by the first embodiment of the present application. DETAILED DESCRIPTION

[0014] The technical solutions in the embodiments of the present application will be described clearly and completely in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0015] It should be noted that the terms first, second, etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence. It can be understood that the above-mentioned terms for distinguishing similar objects can be interchanged under appropriate circumstances, so that the present application can also be implemented in other embodiments in addition to the above-illustrated embodiments or described embodiments. In addition, the terms include and have and any variation are intended to cover non-exclusive inclusion, for example, a process, method, system, product or server including a series of steps or units does not necessarily limit to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0016] Embodiment one The present embodiment one provides a large model-based data collection method, as shown in the figure, which includes the following steps: Figure 1 S1, according to the historical update frequency and demand degree of each preset title corresponding to each target website, obtaining the data collection period corresponding to each preset title, wherein the demand degree is used to represent the matching degree of the content corresponding to the preset title and the user demand.

[0017] Among them, the target website is selected according to the user data collection demand, and the specific content needs to be obtained from the network site, which usually includes government official website, industry platform, enterprise portal and other types, and the domain name and accessible content range need to be clear.

[0018] The preset title is a column name or article title corresponding to the content that needs to be collected in the target website, which is used to accurately position the collection range and avoid the increase of invalid data caused by indiscriminate collection, for example, the title of "2024 industrial support policy" under the "policy notice" column in the government service website, "daily morning political summary" under the main column "domestic news" in the news information website, "machine learning top conference paper delivery" under "computer science" in the journal website of academic resource platform, "2025 graduate school admission outline interpretation" under "graduate school admission" in the education website, etc.

[0019] The historical update frequency is the frequent degree of the content corresponding to a preset title in the target website updated in a preset historical time period, which is used to quantitatively represent the timeliness change rule of the content and provide objective basis for subsequent determination of data collection period.

[0020] The demand degree is the matching degree between the content corresponding to a preset title in the target website and the actual data demand of the user, which is used to measure the value priority of the content to the user and provide demand guidance for dynamic adjustment of data collection period.

[0021] ​The data collection period is a time interval for performing a data collection operation on a preset title in a target website, and is used to standardize the collection frequency and balance the data timeliness and resource consumption.

[0022] By establishing the association model of content characteristics, user demand and collection strategy, the drawbacks of the one-size-fits-all period setting in the traditional collection method are solved, that is, the invalid collection of low-frequency and low-demand content is avoided, the server resource consumption is reduced, the timely acquisition of high-frequency and high-demand content is ensured, the data timeliness is improved, and the foundation for the efficiency and accuracy of subsequent data collection is laid.

[0023] In a specific embodiment, S1 includes the following steps: S11, obtaining each update timestamp of each preset title in a preset historical time period, and calculating the time interval between adjacent update timestamps.

[0024] S12, taking the statistical characteristic value of the time interval as the historical update frequency of the corresponding preset title.

[0025] S13, extracting the demand feature vector of the user demand document and the content feature vector of the historical collection content under each preset title.

[0026] S14, obtaining the demand degree corresponding to each preset title according to the similarity between each content feature vector and the demand feature vector.

[0027] S15, inputting the historical update frequency and the demand degree corresponding to each preset title into the period calculation model to obtain the data collection period corresponding to each preset title, wherein the historical update frequency and the demand degree are negatively correlated with the data collection period.

[0028] The update timestamp is a precise time identifier recording the publication and modification of content, usually a Unix timestamp or a standard time format. By calling the background log of the target website, the last update time label of the page or the historical version record, all update timestamps of the content under the preset title in the preset historical time period can be obtained. The time interval between adjacent update timestamps can directly reflect the time span of two content updates. The preset historical time period is a time range set by humans for statistical content update rules. The specific time can be set by the implementer according to the actual situation, for example, 1 year before the current time.

[0029] The single time interval has contingency, such as the interval between two updates is short but the rest of the intervals are long, which cannot accurately represent the overall update rule. The embodiment eliminates accidental errors by statistical characteristic values. The statistical characteristic values are aggregated by a mathematical statistical method to all time intervals, and the discrete interval data is converted into a quantitative index that can reflect the overall update trend, so as to accurately define the historical update frequency. For example, the statistical characteristic values can be the average, median, mode and other numerical values representing the overall update rule in the preset historical period.

[0030] The user demand document is a text file recording specific data needs of users, such as industry policy data required by enterprises and environmental monitoring data required by scientific research, and is the core basis for defining user needs and ensuring the pertinence of demand degree calculation.

[0031] The pre-trained large model can extract the corresponding feature vector by performing semantic encoding on the text. Correspondingly, for the user demand document, the core demand keywords (such as “2024 new energy subsidy policy”) are extracted and converted into a fixed-dimensional vector, i.e., a demand feature vector; for the historical collection content under the preset title, the text theme and core information (such as policy provisions and data indicators) are extracted and converted into a vector of the same dimension, i.e., a content feature vector. The dimensions of the two are consistent, which facilitates the calculation of similarity by vector operation to realize the quantitative association of demand and content.

[0032] The pre-trained large model is a large deep learning model that is pre-trained based on massive general data and has strong generalization ability and complex task processing ability. When facing specific tasks such as feature extraction, rule analysis, text information extraction, text standardization, page structure comparison, rule adjustment strategy generation, and association verification in the embodiment, the model does not need to be retrained. It only needs to be guided by a small amount of task-related data to adapt the large model to the task requirements. Those skilled in the art know that any pre-trained large model in the prior art falls within the protection scope of the present application, such as BERT, Sentence-BERT, etc., which will not be described here.

[0033] The similarity between the content feature vector and the demand feature vector corresponds to a higher demand degree for the preset title. For example, if the similarity between the “demand feature vector” (user needs “new energy subsidy policy”) and the “content feature vector of a certain preset title” (content is “2024 new energy vehicle subsidy details”) is 0.92, it means that the content of the preset title matches the user demand highly, corresponding to “high demand degree”; if the similarity is 0.35, it corresponds to “low demand degree”. By quantifying the “demand degree” by similarity, the subjectivity of manual judgment is avoided. Those skilled in the art know that any similarity calculation method in the prior art falls within the protection scope of the present application, such as cosine similarity, etc., which will not be described here.

[0034] The periodic calculation model is an algorithm model that fuses the "historical update frequency" and "demand degree", such as a linear regression model, a gradient boosting model, wherein the historical update frequency is negatively correlated with the collection period: the higher the update frequency, the faster the content changes, the collection period needs to be shortened to avoid missing new content; the lower the update frequency, the collection period can be extended to reduce resource waste. The demand degree is negatively correlated with the collection period: the higher the demand degree, the more important the content is to the user, the collection period needs to be shortened to ensure timely acquisition; the lower the demand degree, the collection period can be extended to prioritize resource allocation to high-demand content. The periodic calculation model assigns reasonable weights to the two negatively correlated parameters, such as a frequency weight of 0.6 and a demand weight of 0.4, to calculate a unique collection period that takes into account both the "update rule" and the "user demand", achieving dynamic optimization of the collection period.

[0035] Taking the linear regression model as an example, first, the historical update frequency and demand degree input are standardized, converting the historical update frequency F and demand degree D to the [0, 1] interval. The formula of the linear regression model can be set as the collection period C = a × (1 - b × F - c × D), wherein a is the preset maximum collection period, b and c are the weight coefficients of the historical update frequency and demand degree respectively, (1 - b × F - c × D) can be regarded as the period attenuation coefficient, reflecting the negative correlation that the higher the frequency and the stronger the demand, the shorter the period. b > 0, c > 0, and b + c = 1, the specific values of a, b and c can be set by the implementer according to the actual situation, for example, in this embodiment, a can be set to 24 hours, b = 0.6, c = 0.4.

[0036] The above, by acquiring the historical update timestamp of the preset title and calculating the time interval, the subsequent can be based on the real update data rather than the subjective experience statistical update rule, providing objective data support for the collection period; by taking the statistical characteristic value of the time interval as the historical update frequency, the accidental error of a single update interval is eliminated, which can accurately reflect the overall update trend of the content, avoiding the deviation of the collection period setting caused by individual abnormal interval; by converting the user demand document and the historical collection content into the same dimension feature vector, the semantic quantitative association of demand and content is realized, providing a feasible technical path for subsequent similarity calculation, avoiding the inefficiency and subjectivity of manual matching; by determining the demand degree through vector similarity, the fuzzy judgment of high and low is replaced by quantitative numerical value, which can accurately distinguish the matching difference of different preset titles and user demand, providing clear basis for the priority allocation of collection resources; by the periodic calculation model fusing two negatively correlated parameters to output the collection period, the collection period can adapt to the content update speed and match the user demand importance, avoiding the resource waste or information omission caused by the unified period, realizing the dual optimization of collection efficiency and user demand.

[0037] S2, according to the historical collection record and the data collection period corresponding to each preset title in each target website, grouping N data collection accounts, and obtaining an account set corresponding to each target website, wherein the historical collection record includes full records retained in a preset historical period for performing data collection operations on the preset title, N is a positive integer, and each account set includes a plurality of data collection accounts corresponding to each preset title.

[0038] The data collection account is a user account with access to the target website, capable of performing data collection operations, including account ID, login credentials, access permission level, used to simulate normal user access behavior, and ensure that the collection operation is legal, compliant and stable.

[0039] The full record is all data records generated when performing collection operations on a certain preset title in a preset historical period, at least including collection timestamp, account ID performing collection, collection result (success / failure), failure reason (IP ban / verification code interception / data parsing error), collection content volume, access frequency limit, and other information, used to comprehensively reflect the collection difficulty and account adaptation demand of the preset title.

[0040] Based on the historical collection record reflecting the task difficulty and the data collection period reflecting the task frequency, the N data collection accounts are dynamically grouped, and finally an account set exclusive to each target website is formed, so that through the precise matching of task characteristics and account resources, the problems of insufficient high-difficulty task accounts and idle low-frequency task accounts caused by traditional static account allocation are solved, laying a foundation for subsequent efficient collection.

[0041] In a specific embodiment, S2 includes the following steps: S21, analyzing the historical collection record corresponding to each preset title in each target website, and obtaining a historical access difficulty score of each preset title under each target website, wherein the historical access difficulty score is calculated based on the frequency of triggering the verification mechanism, the access request failure rate, and the ratio of collection nodes being limited access in the historical collection record.

[0042] S22, according to the data collection period and the historical access difficulty score of each preset title, obtaining a resource demand weight corresponding to each preset title, wherein the data collection period and the resource demand weight are negatively correlated, and the historical access difficulty score and the resource demand weight are positively correlated.

[0043] S23, according to the resource demand weight of all preset titles, distributing N data collection accounts to each preset title, and obtaining a plurality of data collection accounts corresponding to each preset title.

[0044] S24, according to a plurality of data collection accounts corresponding to each preset title, an account set corresponding to each target website is obtained.

[0045] The frequency of triggering the verification mechanism is the ratio of the number of times of triggering website verification (such as verification code, sliding block verification, and SMS verification) to the total number of collection operations in a preset historical time period, which is used to quantify the anti-collection verification strength of the website for the preset title.

[0046] The access request failure rate is the ratio of the number of times of returning failure (such as 403 forbidden access and 503 service unavailable) of a collection request (such as an HTTP request) to the total number of requests in a preset historical time period, which is used to reflect the probability of being blocked by the collection operation, directly reflects the collection difficulty, and assists in determining the historical access difficulty.

[0047] The ratio of the collection nodes being limited access is the ratio of the number of nodes (such as IP addresses and device IDs) being limited access (such as IP ban and device blacklist) to the total number of collection nodes in a preset historical time period, which is used to measure the degree of limitation of the website on the collection nodes, supplement the collection difficulty, and ensure the comprehensiveness of the difficulty score.

[0048] The frequency of triggering the verification mechanism, the access request failure rate, and the ratio of the collection nodes being limited access in the historical collection record are core indicators reflecting the collection difficulty. The higher the values of the three indicators, the more strict the anti-data collection limitation of the target website for the preset title, the more easily the collection operation is blocked, the higher the corresponding access difficulty, and the higher the historical access difficulty score. The three indicators are integrated into a quantitative score of 0-10 points through weighted calculation, 0 point for no difficulty, and 10 points for extremely high difficulty, which provides a difficulty basis for subsequent resource allocation. The specific values of the weights corresponding to the frequency of triggering the verification mechanism, the access request failure rate, and the ratio of the collection nodes being limited access can be set by the implementer according to the actual situation, for example, the weight corresponding to the frequency of triggering the verification mechanism is 0.4, the weight corresponding to the access request failure rate is 0.4, and the weight corresponding to the ratio of the collection nodes being limited access is 0.3.

[0049] The resource demand weight is a quantitative indicator for measuring the amount of account resources required by the preset title. Specifically, the data collection period and the resource demand weight are negatively related: the shorter the data collection period, the higher the collection frequency, the more accounts are required to rotate to avoid anti-collection, and the higher the weight; the longer the data collection period, the lower the frequency, the less the account demand, and the lower the weight. The historical access difficulty score and the resource demand weight are positively related: the higher the historical access difficulty score, the more strict the anti-collection limitation, the more accounts are required to cope with the risk of ban, and the higher the weight; the lower the historical access difficulty score, the smoother the collection, the less the account demand, and the lower the weight.

[0050] The N accounts are allocated to each preset title by using a "weight proportion allocation method" or a "load balancing algorithm", that is, the more the resource demand weight proportion of a preset title, the more the number of accounts allocated to the preset title, so as to ensure that the number of accounts matches the resource demand, and avoid the problems of insufficient accounts for high demand titles and idle accounts for low demand titles.

[0051] A single target website includes multiple preset titles, and each preset title has allocated a dedicated data collection account. The data collection accounts are integrated according to the hierarchical relationship of target website-preset title (for example, 5 accounts are allocated to the "policy notification" title of website A, and 3 accounts are allocated to the "data statistics" title), to form a unique account set of the target website, and to realize website-level account resource closed-loop management.

[0052] The above extracts three core difficulty indicators from the analysis of historical collection records and calculates the historical access difficulty score, so that the collection difficulty is converted from a vague description to quantitative data, avoiding the subjectivity of manual judgment and providing an objective basis for resource allocation; by establishing the negative correlation between the data collection period and the resource demand weight, and the positive correlation between the historical access difficulty score and the resource demand weight, the resource demand weight can reflect the collection frequency and collection difficulty at the same time, ensuring the comprehensiveness of account demand determination; by allocating accounts according to the resource demand weight proportion, the account resources are tilted towards high-frequency and high-difficulty core preset titles, avoiding resource waste and improving the collection success rate of key tasks; by integrating the account of each preset title to form a target website account set, the account management is bound to the website and title level, reducing the risk of cross-scene account abuse and improving the standardization and controllability of the collection process.

[0053] In a specific embodiment, S23 further includes the following steps: S231, obtaining the collection permission requirement corresponding to each target website.

[0054] S232, obtaining the permission attribute of each data collection account, wherein the permission attribute is used to represent the access permission level that the data collection account can provide.

[0055] S233, for any target website, matching the collection permission requirement of the current target website with the permission attribute of each data collection account, filtering out the data collection accounts whose permission attribute meets the collection permission requirement of the current target website to form a candidate account pool corresponding to the current target website.

[0056] S234, according to the resource demand weight of each preset title in the candidate account pool, allocating data collection accounts for each preset title in the current target website from the candidate account pool corresponding to the current target website to obtain a plurality of data collection accounts corresponding to each preset title.

[0057] Among them, different target websites set different access thresholds for visitors due to content sensitivity and service positioning differences. The collection permission requirement is the permission condition that the target website sets to limit content access and that the visitor needs to meet, specifically including anonymous access permission, registered user access permission, VIP user access permission, and specific geographic area access permission, which is used to clearly define the basic permission standard required to access the target website. By simulating user access process through pre-trained large model or calling access rule description disclosed by the website, the target website's clearly required access permission type can be identified and extracted, providing permission standard for subsequent account screening and avoiding collection failure due to insufficient account permissions.

[0058] The permission attribute is the access permission level that the data collection account itself possesses and can realize in the target website, which is used to represent whether the data collection account can meet the collection permission requirement of the website, and is used to match the collection permission requirement of the target website to determine whether the account has the basic ability to access the website, avoiding invalid accounts participating in distribution.

[0059] The above, by obtaining the collection permission requirement of the target website and the permission attribute of the account, the qualification standard of the account accessing the website is clear, avoiding the lack of permission information leading to subsequent distribution of no permission account; through permission matching screening to build a candidate account pool, only accounts with access ability are retained for distribution, reducing collection failure due to insufficient permissions from the source, and improving collection efficiency; by allocating accounts in the candidate account pool based on resource demand weight, the account resources not only meet the permission requirements, but also can be reasonably distributed according to the task demand priority, balancing permission compliance and resource efficiency.

[0060] In a specific embodiment, if the collection permission requirement of the current target website is anonymous access, all data collection accounts meet the requirement. If the collection permission requirement of the current target website is registered user access, the data collection accounts whose permission attributes have successfully registered and logged in the target website are screened out. If the collection permission requirement of the current target website is VIP user access, the data collection accounts containing VIP identifier in the permission attribute are screened out. If the collection permission requirement of the current target website is specific geographic area access, the data collection accounts whose IP addresses in the permission attribute are located within the specific geographic area range are screened out.

[0061] In a specific embodiment, S234 includes the following steps: S2341, for any preset title, according to the proportion of the resource demand weight of the current preset title in the total resource demand weight of the target website corresponding to the current preset title, and the total number of data collection accounts in the candidate account pool of the target website corresponding to the current preset title, determine the number m of data collection accounts to be allocated for the current preset title, wherein m is a positive integer, m < N.

[0062] S2342, filtering m data collection accounts corresponding to the current preset title from the candidate account pool corresponding to the current preset title.

[0063] S2343, traversing all preset titles to obtain a plurality of data collection accounts corresponding to each preset title.

[0064] Among them, for any preset title, the proportion of the resource demand weight of the current preset title in the total resource demand weight of the target website corresponding to the current preset title is calculated, the product of the proportion and the total number is obtained, and the product result is rounded down to obtain the number m of data collection accounts to be allocated for the current preset title.

[0065] The above determines m through the quantitative calculation of the weight proportion and the total number of candidate accounts, so that the number of accounts is allocated based on objective data rather than subjective judgment, avoiding the problem of insufficient high-weight title accounts and idle low-weight title accounts, and improving the resource allocation accuracy.

[0066] S3, inputting the data collection rule corresponding to each target website into the pre-trained large model to obtain the initial collection text corresponding to each preset title in each target website, wherein the data collection rule at least includes the data collection period, the account set, the collection type and the collection level corresponding to each preset title. The collection type is the data type of the content corresponding to the preset title, and the collection level is used to determine the collection depth of the preset title with multiple levels of subpages.

[0067] Among them, the collection type is the content category to be collected, which is preset according to the data form of the content corresponding to the preset title, such as text, picture, video and the like, which is used to clearly define the range of collected content, filter data forms irrelevant to the demand, and improve the pertinence of collected content.

[0068] The collection level is the content collection depth preset for the preset title with multiple levels of subpages, which is used to control the range of page traversal, avoid missing key subpage content due to insufficient collection depth, or collecting a large amount of redundant information due to excessive depth.

[0069] The above, by inputting the multi-element coordinated data collection rule into the pre-trained large model, the collection operation is no longer dependent on manual intervention, realizing automatic collection according to the rule, improving the efficiency and standardization degree of data collection; through the binding of the account set and the collection period in the collection rule, the account called each time for collection is an account with matched permissions and good health status, and the collection frequency conforms to the content update rule, reducing the collection failure rate; through the clear constraint of the collection type and the collection level, the collection content focuses on the target data form and the page range, avoiding the waste of storage resources caused by collecting irrelevant content, and improving the effectiveness of the initial collection text.

[0070] In a specific embodiment, S3 includes the following steps: S31, based on the pre-trained large model, parsing the data collection rule corresponding to the current target website for any target website, obtaining the data collection script corresponding to the current target website.

[0071] S32, based on the pre-trained large model, calling the data collection account in the account set corresponding to the current target website, executing the data collection script, and obtaining the original collection data corresponding to each preset title in the current target website.

[0072] S33, inputting the original collection data corresponding to each preset title in the current target website into the pre-trained large model, obtaining the initial collection text corresponding to each preset title in the current target website, wherein the pre-trained large model performs abstract generation, key information extraction and format standardization on the original collection data, and wherein the format standardization operation is based on the collection type and collection level corresponding to each preset title.

[0073] Wherein, the data collection script is a code file generated by the pre-trained large model after parsing the data collection rule, which can be directly executed by machine, contains the complete logic of account calling, page collection and content filtering, is used to convert abstract collection rules into executable code, avoids the inefficiency and error of manual script writing, and ensures the standardization of collection operation.

[0074] Abstract generation is an operation of the pre-trained large model to extract core information from long text type original collection data, which extracts key paragraphs based on TextRank algorithm and integrates them into a concise text, which can reduce the reading and processing cost of long text data, help to quickly locate core information, and improve the subsequent data use efficiency.

[0075] Key information extraction is a directional extraction of information by the pre-trained large model according to the collection type, which is used to extract key content under each preset title and eliminate interference information such as preset title format and template. For example, text extraction includes "release time, source institution, core data", attachment extraction includes "attachment name, size, download path", picture extraction includes "alt attribute text, resolution, storage URL", etc., to ensure that key information is not missed.

[0076] The format standardization is an operation of arranging the original collection data into a unified format (such as JSON, table) according to the collection type and collection level by the pre-trained large model, which is used to eliminate the format disorder problem of the original data, ensure the consistency of the initial collection text structure, and facilitate subsequent verification and use. For example, the text is arranged in the JSON format of "title - abstract - key information - original text fragment", the PDF attachment is arranged in the table format of "attachment name - size - download link - text extraction content", and the content of the same collection level is uniformly indented, and finally the initial collection text with unified format and structured information is formed.

[0077] The above, by the pre-trained large model, the collection rule is parsed to generate the data collection script, so that the collection rule is converted from an abstract instruction to an executable code, avoiding the technical threshold and error rate of manual script writing, realizing the automation landing of the collection operation; through the abstract generation and key information extraction, the core information in the original collection data is accurately refined, avoiding the interference of redundant information, reducing the complexity of subsequent data processing, and improving the information acquisition efficiency; through the format standardization operation based on the collection type and level, the initial collection text with different preset titles and different data forms is uniformly formatted, providing a structured basis for subsequent correlation verification, and ensuring the efficiency and accuracy of the verification process.

[0078] In a specific embodiment, S3 further comprises the following steps: S34, after executing the data collection script, obtaining the average failure rate and the number of consecutive failures of the current target website according to the collection result of each data collection account in the account set corresponding to the current target website when collecting data from the current target website, wherein the collection result is success or failure.

[0079] S35, if the average failure rate of the current target website is greater than or equal to a preset failure rate threshold, or the number of consecutive failures is greater than or equal to a preset number threshold, it is determined that the data collection result corresponding to the current target website is abnormal.

[0080] S36, when the data collection result corresponding to each target website is abnormal, inputting the current page structure information, the historical page structure information and the data collection rule of the current target website into the pre-trained large model to obtain the rule adjustment strategy corresponding to the current target website.

[0081] S37, updating the data collection rule corresponding to the current target website according to the rule adjustment strategy, and returning to step S31 to update the initial collection text corresponding to each preset title in the current target website according to the updated data collection rule.

[0082] The average failure rate is the percentage of the number of collection failures in the total number of collections in the current target website account set, and is used to quantify the overall collection failure rate. The consecutive failure number is the maximum number of consecutive "failure" results in the order of collection time, and is used to capture the centralized failure trend.

[0083] The preset failure rate threshold and the preset number threshold are pre-set critical values for determining collection anomalies, providing clear determination criteria for collection anomalies and avoiding misjudgment or missed judgment caused by subjective judgment. The specific values of the preset failure rate threshold and the preset number threshold can be set by the implementer according to the actual situation, such as a preset failure rate threshold of 0.2 and a preset number threshold of 3 times.

[0084] If the average failure rate corresponding to the current target website is greater than or equal to the preset failure rate threshold, or the consecutive failure number is greater than or equal to the preset number threshold, it is determined that the data collection result corresponding to the current target website is abnormal, otherwise, it is determined that the data collection result corresponding to the current target website is normal.

[0085] The essence of data collection is to accurately extract target information from the title of the target website page, and the collection type used to represent what information is extracted, and the collection level used to locate the information extracted from which structural level of the title, are determined by the organization logic of the page structure. The current page structure information is the structured description data of the page structure in the current state of the target website when the data collection result is abnormal, and the historical page structure information is the structured archive data of the page structure of the target website in the historical collection success stage (usually the last collection normal). Therefore, when the data collection result corresponding to the target website is abnormal, by comparing the current page structure information and the historical page structure information, the structural changes leading to the collection anomaly are located, and the collection type and the collection level are adjusted, providing a benchmark for abnormal root attribution, and ensuring the smooth progress of the data collection process.

[0086] In this embodiment, the pre-trained large model compares and analyzes the input current page structure information and historical page structure information, and analyzes the identified structural differences and the input original data collection rule, and outputs a rule adjustment strategy for the collection type and the collection level, and adjusts the adaptability of the data collection rule.

[0087] By statistically averaging the failure rate and the number of consecutive failures, the collection state is converted from a qualitative description to quantitative data, avoiding the subjectivity of manual judgment of collection abnormalities and providing an objective basis for abnormality determination. By double-threshold determination of collection abnormalities, both overall inefficient general abnormalities and concentrated failure sudden abnormalities can be identified, reducing the risk of abnormality omission and ensuring timely discovery of collection problems. By comparing page structure information to generate rule adjustment strategies, the root cause of collection failure can be accurately located, avoiding ineffective attempts caused by blind rule adjustment and improving abnormality repair efficiency. By updating the rules and returning to S31 for re-collection, a closed loop of collection, abnormality determination, rule adjustment, and re-collection is formed, ensuring that even if collection abnormalities occur, effective initial collection text can still be obtained through dynamic adjustment, avoiding collection task interruption due to abnormalities.

[0088] S4, based on the pre-trained large model, performs relevance verification on the initial collection text corresponding to each preset title, modifies the initial collection text according to the verification result, and obtains the target collection text.

[0089] In a specific embodiment, S4 includes the following steps: S401, extracting key entities, event timestamps, and core event descriptions from each initial collection text.

[0090] S402, based on the event timestamp, cross-comparing the key entities and core event descriptions of the initial collection texts from different preset titles to obtain the information comparison result corresponding to each initial collection text, wherein the information comparison result is either contradictory or non-contradictory.

[0091] S403, for any initial collection text, if the information comparison result corresponding to the current initial collection text is contradictory, the current initial collection text and other initial collection texts that have contradictions with the current initial collection text are determined as a group of to-be-verified texts.

[0092] S404, based on the pre-trained large model, analyzing each to-be-verified text to obtain an analysis result corresponding to each to-be-verified text, wherein the analysis result includes information source priority, publication time priority, and logical consistency.

[0093] S405, according to all analysis results corresponding to each group of to-be-verified texts, performing conflict resolution on each group of to-be-verified texts by the pre-trained large model to obtain calibration information corresponding to each group of to-be-verified texts.

[0094] S406, modifying each group of to-be-verified texts according to the verification information to obtain the target collection text.

[0095] Among them, the key entity is the core element with practical significance extracted from the initial collection text, such as place, amount, subject name, and the event timestamp is the event related time information in a unified format extracted from the initial collection text, such as implementation time, release time. The core event description is the structured text segment generated by filtering the highest weight key sentence in the initial collection text through the TextRank algorithm, covering key elements such as "subject, time, behavior, result, attribute". Those skilled in the art know that the entity extraction method and TextRank algorithm in the prior art fall within the protection scope of the present application, and will not be repeated here.

[0096] The key entity, event timestamp and core event description used for correlation comparison are extracted from the initial collection text through the named entity recognition and information extraction capability of the pre-trained large model.

[0097] Based on the event timestamp as the benchmark, the initial collection texts with the same event timestamp are classified into the same comparison group, ensuring that the comparison of different preset title texts is in the same time dimension, and avoiding cross-time information conflict misjudgment. By comparing the key entities of different initial collection texts in the same group, if the entities have essential differences, it is directly determined that there is a contradiction; if the entity category and core meaning are consistent, the entity comparison passes, then the semantic similarity of the core event descriptions of the two groups is calculated using the Sentence-BERT model, and a similarity threshold is set. If the semantic similarity is greater than or equal to the similarity threshold, it is determined that there is no contradiction, that is, the expression difference but the semantic consistency is allowed; if the semantic similarity is less than the similarity threshold, it is determined that there is a contradiction. Those skilled in the art know that any similarity calculation method in the prior art falls within the protection scope of the present application, and will not be repeated here. The specific value of the similarity threshold can be set by the implementer according to the actual situation, for example, the similarity threshold in the present embodiment is 0.9.

[0098] When a certain initial collection text is determined to have a contradiction, the specific initial collection text in conflict with it is determined, the current text and all conflicting texts are classified into a text group to be verified, and the conflict point is labeled, so as to facilitate centralized conflict resolution and avoid one-sidedness caused by isolated processing of a single text.

[0099] Texts from different sources have different credibility (such as official column texts of target websites > third-party reproduced texts), and the information source priority is preset according to the text source. The higher the level, the higher the text credibility. For texts of the same event, the closer the publication time to the event timestamp (or the current collection time), the more likely the information is updated and accurate. The publication time priority is quantified by time difference, and the higher the level, the higher the text credibility. Through a logical verification algorithm, it is verified whether the information in the text is self-consistent. The higher the self-consistency, the higher the text credibility.

[0100] Based on the analysis results of the three dimensions, the unique and accurate calibration information is generated through the decision-making and information fusion ability of the large model to solve the contradictions between texts. Specifically, the comprehensive credibility score of the text to be checked is calculated, such as comprehensive credibility score = source priority × 0.3 + publication time priority × 0.3 + logical consistency × 0.4, and the initial collection text with the highest score is taken as the benchmark text, and its information is preferentially adopted. For the contradiction points between other initial collection texts and the benchmark text, the information of the benchmark text is corrected; the supplementary information (such as the benchmark text lacks "application deadline", and other texts contain "2024-12-31 deadline") in other initial collection texts that has no conflict with the benchmark text is extracted and supplemented to the benchmark text, and finally the calibration information containing the corrected core information and supplementary information is formed. The target collection text is obtained by correcting each group of text to be checked.

[0101] The above, by extracting key entities, event timestamps and core event descriptions, the redundant information in the initial collection text is filtered, and the core elements are accurately extracted, providing high-quality input for subsequent relevance verification, and avoiding irrelevant information interference; By grouping and comparing based on event timestamps, invalid comparisons across time dimensions are excluded, while contradictions in the same time dimension are accurately located, improving comparison efficiency and accuracy; By analyzing the credibility of the text to be checked in multiple dimensions, the conflict resolution is not dependent on subjective judgment, but based on objective quantitative indicators to select the benchmark text, ensuring the fairness and accuracy of the resolution result; By pre-training the large model to perform conflict resolution and integrate supplementary information, not only the contradictions between texts are corrected, but also the non-conflicting information is integrated, avoiding information loss, so that the final output target collection text is uniform in format, without contradictions and stable in quality.

[0102] The above, by establishing the association model of content characteristics, user demand and collection strategy, the disadvantages of one-size-fits-all period setting in the traditional collection method are solved, that is, the invalid collection of low-frequency low-demand content is avoided, the server resource consumption is reduced, the timely acquisition of high-frequency high-demand content is ensured, and the data timeliness is improved; based on the historical collection records reflecting the task difficulty and the data collection period reflecting the task frequency, N data collection accounts are dynamically grouped, and finally a set of accounts dedicated to each target website is formed, so that through the precise matching of task characteristics and account resources, the problems of insufficient accounts for high-difficulty tasks and idle accounts for low-frequency tasks caused by traditional static account allocation are solved; by inputting the multi-element collaborative data collection rule into the pre-trained large model, the efficiency and standardization of data collection are improved, through the binding of the account set and the data collection period, the account called each time is an account with matched permissions and good health status, and the collection frequency conforms to the content update rule, thereby reducing the collection failure rate, through the explicit constraint of the collection type and the collection level, the collection content is focused on the target data form and page range, the storage resource waste caused by irrelevant content collection is avoided, the effectiveness of the initial collection text is improved, through the relevance verification of the pre-trained large model on the initial collection text, the initial collection text is modified according to the verification result, and the target collection text is obtained, the contradiction between texts is corrected, and the conflict-free information is integrated, so that the target collection text output finally has a unified format, no contradiction and stable quality, thereby improving the efficiency and reliability in the data collection process.

[0103] Embodiment two The embodiment two of the present application provides a non-transitory computer readable storage medium, which can be arranged in an electronic device to save at least one instruction or at least one program related to a method in the method embodiment, and the at least one instruction or the at least one program is loaded and executed by the processor to realize the data collection method based on the large model provided in the above embodiment.

[0104] Embodiment three The embodiment three of the present application provides an electronic device, which includes a processor and the non-transitory computer readable storage medium in the embodiment two of the present application.

[0105] The above is only a preferred embodiment of the present application, and does not limit the present application in any form. Although the present application has been disclosed as above with a preferred embodiment, it is not intended to limit the present application. Any skilled person in the art can make some changes or modifications to the above disclosed technical content without departing from the scope of the technical solution of the present application, and any simple modification, equivalent change and modification of the above embodiment according to the technical essence of the present application are still within the scope of the technical solution of the present application.

Claims

1. A data acquisition method based on a large model, characterized in that, The data acquisition method includes the following steps: S1. Based on the historical update frequency and demand level of each preset title in each target website, obtain the data collection period corresponding to each preset title. The demand level is used to characterize the degree of matching between the content corresponding to the preset title and the user's needs. S2, based on the historical collection records and data collection period corresponding to each preset title in each target website, group the N data collection accounts to obtain the account set corresponding to each target website. The historical collection records include the full records of data collection operations performed on the preset title within the preset historical time period. N is a positive integer. Each account set includes several data collection accounts corresponding to each preset title. S3, input the data collection rules corresponding to each target website into the pre-trained large model to obtain the initial collection text corresponding to each preset title in each target website. The data collection rules include at least the data collection period, account set, collection type and collection level corresponding to each preset title. The collection type is the data type of the content corresponding to the preset title. The collection level is used to determine the collection depth of preset titles with multiple levels of subpages. S4. Based on the pre-trained large model, the correlation of the initial collected text corresponding to each preset title is verified, and the initial collected text is corrected according to the verification result to obtain the target collected text.

2. The data acquisition method based on a large model according to claim 1, characterized in that, S1 includes the following steps: S11, obtain the update timestamp of each preset title within a preset historical time period, and calculate the time interval between adjacent update timestamps; S12, the statistical characteristic value of the time interval is used as the historical update frequency of the corresponding preset title; S13, extract the requirement feature vector of the user requirement document and the content feature vector of the historical collected content under each preset title; S14, based on the similarity between each content feature vector and the demand feature vector, obtain the demand level corresponding to each preset title; S15, input the historical update frequency and demand level corresponding to each preset title into the periodic calculation model to obtain the data collection period corresponding to each preset title, wherein the historical update frequency and the demand level are both negatively correlated with the data collection period.

3. The data acquisition method based on a large model according to claim 1, characterized in that, S2 includes the following steps: S21, parse the historical collection records corresponding to each preset title in each target website to obtain the historical access difficulty score of each preset title under each target website. The historical access difficulty score is calculated based on the frequency of triggering verification mechanisms, access request failure rate, and the ratio of access restrictions on collection nodes in the historical collection records. S22, based on the data collection period and historical access difficulty score of each preset title, obtain the resource demand weight corresponding to each preset title, wherein the data collection period is negatively correlated with the resource demand weight, and the historical access difficulty score is positively correlated with the resource demand weight; S23, based on the resource demand weight of all preset titles, assign N data collection accounts to each preset title, and obtain several data collection accounts corresponding to each preset title; S24: Based on the data collection accounts corresponding to each preset title, obtain the account set corresponding to each target website.

4. The data acquisition method based on a large model according to claim 3, characterized in that, S23 also includes the following steps: S231, Obtain the collection permission requirements corresponding to each target website, wherein the collection permission requirements include anonymous access permission, registered user access permission, VIP user access permission and specific geographical area access permission; S232, Obtain the permission attributes of each data collection account, wherein the permission attributes are used to characterize the access permission level that the data collection account can provide; S233, for any target website, match the collection permission requirements of the current target website with the permission attributes of each data collection account, and filter out data collection accounts whose permission attributes meet the collection permission requirements of the current target website to form a candidate account pool corresponding to the current target website; S234, based on the resource requirement weight of each preset title in the candidate account pool, allocate data collection accounts for each preset title in the current target website from the candidate account pool corresponding to the current target website, and obtain several data collection accounts corresponding to each preset title.

5. The data acquisition method based on a large model according to claim 4, characterized in that, S234 includes the following steps: S2341, For any preset title, based on the proportion of the resource demand weight of the current preset title in the total resource demand weight of the target website corresponding to the current preset title, and the total number of data collection accounts in the candidate account pool of the target website corresponding to the current preset title, determine the number of data collection accounts to be allocated for the current preset title, m, where m is a positive integer and m < N. S2342, Select m data collection accounts corresponding to the current preset title from the candidate account pool corresponding to the current preset title; S2343, iterate through all the preset titles and obtain several data collection accounts corresponding to each preset title.

6. The data acquisition method based on a large model according to claim 1, characterized in that, S3 includes the following steps: S31, For any target website, based on the pre-trained large model, analyze the data collection rules corresponding to the current target website and obtain the data collection script corresponding to the current target website; S32, based on the pre-trained large model, call the data collection account in the account set corresponding to the current target website, execute the data collection script, and obtain the original collection data corresponding to each preset title in the current target website; S33, input the original collected data corresponding to each preset title in the current target website into the pre-trained large model to obtain the initial collected text corresponding to each preset title in the current target website. The pre-trained large model performs summary generation, key information extraction and format standardization on the original collected data. The format standardization operation is based on the collection type and collection level settings corresponding to each preset title.

7. The data acquisition method based on a large model according to claim 1, characterized in that, S3 also includes the following steps: S34, after executing the data collection script, obtain the collection results of each data collection account in the account set corresponding to the current target website when collecting data from the current target website, and obtain the average failure rate and the number of consecutive failures corresponding to the current target website, wherein the collection result is either success or failure; S35. If the average failure rate of the current target website is greater than or equal to the preset failure rate threshold, or the number of consecutive failures is greater than or equal to the preset number of failures threshold, then the data collection result of the current target website is determined to be abnormal. S36, when the data collection results for each target website are abnormal, the current page structure information and historical page structure information of the current target website are input into the pre-trained large model to obtain the rule adjustment strategy for the current target website; S37, adjust the strategy according to the rules and update the data collection rules corresponding to the current target website. According to the updated data collection rules, return to step S31 and update the initial collection text corresponding to each preset title in the current target website.

8. The data acquisition method based on a large model according to claim 1, characterized in that, S4 includes the following steps: S401, extract key entities, event timestamps, and core event descriptions from each initial collected text; S402, using the event timestamp as a benchmark, cross-compare the key entities and core event descriptions of the initial collected texts from different preset titles to obtain the information comparison results corresponding to each initial collected text, wherein the information comparison results are either contradictory or not contradictory; S403, for any initial collected text, if the information comparison result corresponding to the current initial collected text is contradictory, then the current initial collected text and other initial collected texts that contradict the current initial collected text are determined as a group of texts to be verified. S404, Analyze each text to be verified based on the pre-trained large model and obtain the analysis results corresponding to each text to be verified. The analysis results include information source priority, publication time priority and logical consistency. S405, Based on all the analysis results corresponding to each group of texts to be verified, the pre-trained large model performs conflict resolution on each group of texts to be verified to obtain the calibration information corresponding to each group of texts to be verified. S406, Based on the verification information, each group of texts to be verified is corrected to obtain the target collected text.

9. A non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores at least one instruction or at least one program segment, characterized in that, The at least one instruction or the at least one program segment is loaded and executed by the processor to implement the large model-based data acquisition method as described in any one of claims 1-8.

10. An electronic device, characterized in that, Includes a processor and the non-transitory computer-readable storage medium as described in claim 9.

Images