Data processing method and device based on collection rules

Abstract

The present application provides a data processing method and device based on collection rules, which relates to the field of data processing technology and solves the technical problem of low accuracy of data collected from websites. The method includes: based on the comparison results of the repeated parts between multiple website domain names of multiple websites to be collected for data and the preset repeated part threshold, identifying at least two target websites to be collected for data corresponding to the same or similar target website format among multiple websites to be collected for data; judging whether there is a target data collection rule corresponding to the mapping of the target website format in the specified collection mapping relationship; if there is a target data collection rule in the specified collection mapping relationship, batch data collection is performed on at least two target websites to be collected for data through the target data collection rule, and the first final data collection results of at least two target websites to be collected for data are obtained.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data processing method and device based on collection rules. Background Art

[0002] For the process of collecting massive information from platforms such as social media and news websites, it is usually necessary for managers to collect data from a certain website. For example, managers download social media and news data from multiple websites, and then extract relevant fields from the content to realize the process of collecting massive information from platforms such as social media and news websites. Sometimes it is necessary to save web page related files locally, such as pictures, attachments, etc. However, in the process of collecting website data, managers often encounter situations where the accuracy of the data collected from the website is low due to different website formats. Summary of the invention

[0003] The purpose of the present invention is to provide a data processing method and device based on collection rules to solve the technical problem of low accuracy of data collected from websites.

[0004] In a first aspect, an embodiment of the present application provides a data processing method based on collection rules, the method comprising:

[0005] Acquire multiple websites to be collected data, wherein the multiple websites to be collected data have different website formats;

[0006] Based on the comparison result of the repeated parts between the multiple website domain names of the multiple websites to be collected data and the preset repeated part threshold, identifying at least two target websites to be collected data corresponding to the same or similar target website formats from the multiple websites to be collected data;

[0007] Determine whether there is a target data collection rule mapped to the target website format in the specified collection mapping relationship; wherein the specified collection mapping relationship includes a mapping relationship between each website format in a plurality of website formats and a data collection rule;

[0008] If the target data collection rule exists in the designated collection mapping relationship, batch data collection is performed on the at least two target websites to be collected data according to the target data collection rule to obtain a first final data collection result of the at least two target websites to be collected data;

[0009] If the target data collection rule does not exist in the specified collection mapping relationship, the data collection rule of the target website format is refined by a collection rule refinement model to obtain a preliminary extraction rule; wherein the collection rule refinement model is generated by model training based on multiple website format data samples with data collection rule labels;

[0010] Perform batch data collection on the at least two target websites to be collected data by using the preliminary extraction rules to obtain preliminary data collection results, and determine whether the preliminary data collection results are correct data;

[0011] If the preliminary data collection result is correct data, determining the preliminary data collection result as the second final data collection result of the at least two target websites to be collected data;

[0012] If the preliminary data collection result is incorrect data or data cannot be collected through the preliminary extraction rules, screenshot data corresponding to the at least two target websites for data collection are obtained, and a third final data collection result of the at least two target websites for data collection is obtained through image recognition based on the screenshot data.

[0013] In a possible implementation, before the step of determining whether there is a target data collection rule corresponding to the target website format in the specified collection mapping relationship, the method further includes:

[0014] Identifying a plurality of first data collection rules with similar extraction rules from a plurality of the data collection rules in the specified collection mapping relationship through a natural language recognition model; wherein the natural language recognition model is generated through a model training process of language interpretation learning;

[0015] The plurality of first website formats mapped corresponding to the plurality of first data collection rules are divided into website formats of the same type.

[0016] In a possible implementation, the target data collection rule includes relevant content of the target field to be collected in the website; batch data collection is performed on the at least two target websites to be collected, including:

[0017] Determine the target subject involved in the target field and the target viewpoint to which the target field belongs with respect to the target subject; wherein the target viewpoint at least includes a positive viewpoint, a negative viewpoint or a neutral viewpoint;

[0018] Distinguish and classify the content data under multiple viewpoints on the target subject in the at least two target websites to be collected for data, and obtain a first viewpoint type corresponding to the positive viewpoint content data on the target subject, a second viewpoint type corresponding to the negative viewpoint content data on the target subject, and a third viewpoint type corresponding to the neutral viewpoint content data on the target subject; the list corresponding to the first viewpoint type records the first website domain name of the website to be collected for data containing the positive viewpoint content data, the list corresponding to the second viewpoint type records the second website domain name of the website to be collected for data containing the negative viewpoint content data, and the list corresponding to the third viewpoint type records the third website domain name of the website to be collected for data containing the neutral viewpoint content data;

[0019] If the target viewpoint is the positive viewpoint content data for the target theme, the target field is selected from the website content corresponding to the first website domain name to obtain a first target field selection result, and batch data collection is performed on the website content corresponding to the first website domain name based on the first target field selection result;

[0020] If the target opinion is the negative opinion content data for the target topic, the target field is selected from the website content corresponding to the second website domain name to obtain a second target field selection result, and batch data collection is performed on the website content corresponding to the second website domain name based on the second target field selection result;

[0021] If the target opinion is the neutral opinion content data for the target topic, the target field is selected from the website content corresponding to the third website domain name to obtain a third target field selection result, and batch data collection is performed on the website content corresponding to the third website domain name based on the third target field selection result.

[0022] In a possible implementation, each of the first viewpoint type, the second viewpoint type and the third viewpoint type includes multiple specific viewpoints; after the step of distinguishing and classifying the content data under multiple viewpoints on the target subject in the at least two target websites to be collected for data, and obtaining the first viewpoint type corresponding to the positive viewpoint content data on the target subject, the second viewpoint type corresponding to the negative viewpoint content data on the target subject, and the third viewpoint type corresponding to the neutral viewpoint content data on the target subject, the method further includes:

[0023] Establishing a viewpoint tree diagram for the target topic based on the multiple specific viewpoints respectively included in the first viewpoint type, the second viewpoint type, and the third viewpoint type; the viewpoint tree diagram includes multiple viewpoint nodes, each of which is used to represent one of the specific viewpoints;

[0024] Determine an association relationship between two viewpoint nodes among the plurality of viewpoint nodes, and determine a connection relationship between the two viewpoint nodes based on the association relationship; wherein the association relationship includes at least one of a mutually contradictory association relationship, an association relationship providing evidence, and an association relationship derived from viewpoints;

[0025] The connection structure of the viewpoint tree diagram is determined based on the multiple connection relationships, and a final viewpoint tree diagram is generated based on the connection structure and the viewpoint nodes.

[0026] In a possible implementation, performing batch data collection on the at least two target websites to be collected data includes:

[0027] Determining a plurality of related data collection websites having web page link relationships with the at least two target data collection websites;

[0028] Comparing the website content of each of the multiple related collection websites with the website content of the at least two target websites to be collected for data, obtaining a comparison result, and determining the website content relevance between the related collection website and the at least two target websites to be collected for data according to the comparison result;

[0029] If there is a situation where the relevance of the contents of the multiple websites corresponding to the multiple related collection websites is lower than a preset content relevance threshold, the target related collection website corresponding to the target website content relevance lower than the preset content relevance threshold is determined as the advertising website noise data mixed in the multiple related collection websites;

[0030] Removing the noise data of the advertising website from the multiple related collection websites to obtain a final related collection website after the removal process;

[0031] Batch data collection is performed on the final relevant collection website and the at least two target websites to be collected data.

[0032] In a possible implementation, the step of comparing the website content of each of the multiple related collection websites with the website content of the at least two target websites for data collection to obtain a comparison result includes:

[0033] Performing a simulated click operation on each of the multiple related collection websites through AI, and acquiring website content in each of the related collection websites based on the simulated click operation;

[0034] The website content of each of the related collection websites is compared with the website content of the at least two target websites to be collected data to obtain a comparison result.

[0035] In a possible implementation, before the step of determining whether there is a target data collection rule corresponding to the target website format in the specified collection mapping relationship, the method further includes:

[0036] Obtaining data samples in different website formats and website content collection methods corresponding to each of the website formats from websites corresponding to the multiple website formats, and learning through AI based on the data samples and the website content collection methods to obtain the specified collection mapping relationship including the data collection rules; or,

[0037] The designated collection mapping relationship including the data collection rule is determined by setting the plug-in.

[0038] In a second aspect, a data processing device based on collection rules is provided, comprising:

[0039] An acquisition module is used to acquire a plurality of websites to be collected data, wherein the plurality of websites to be collected data correspond to different website formats;

[0040] A first identification module is used to identify at least two target websites to be collected for data that have the same or similar target website formats from among the multiple websites to be collected for data based on a comparison result of repeated parts between multiple website domain names of the multiple websites to be collected for data and a preset repeated part threshold;

[0041] A judgment module, used to judge whether there is a target data collection rule corresponding to the target website format in the specified collection mapping relationship; wherein the specified collection mapping relationship includes a mapping relationship between each website format in a plurality of website formats and a data collection rule;

[0042] A first collection module is used for, if the target data collection rule exists in the specified collection mapping relationship, performing batch data collection on the at least two target websites to be collected data according to the target data collection rule to obtain a first final data collection result of the at least two target websites to be collected data;

[0043] A refining module, for refining the data collection rules of the target website format through a collection rule refining model to obtain preliminary extraction rules if the target data collection rules do not exist in the specified collection mapping relationship; wherein the collection rule refining model is generated through model training based on data samples of multiple website formats with data collection rule labels;

[0044] A second collection module is used to perform batch data collection on the at least two target websites to be collected data according to the preliminary extraction rules, obtain preliminary data collection results, and determine whether the preliminary data collection results are correct data;

[0045] A determination module, configured to determine the preliminary data collection result as a second final data collection result of the at least two target websites to be collected data if the preliminary data collection result is correct data;

[0046] The second identification module is used to obtain screenshot data corresponding to the at least two target websites for data collection if the preliminary data collection result is incorrect data or data cannot be collected through the preliminary extraction rules, and obtain a third final data collection result of the at least two target websites for data collection through image recognition based on the screenshot data.

[0047] In a third aspect, an embodiment of the present application provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program that can be executed on the processor, and when the processor executes the computer program, the method described in the first aspect is implemented.

[0048] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to execute the method described in the first aspect above.

[0049] The embodiments of the present application bring the following beneficial effects:

[0050] A data processing method and device based on collection rules provided in an embodiment of the present application can obtain multiple websites to be collected for data, and the multiple websites to be collected for data correspond to different website formats; based on the comparison result of the repeated parts between multiple website domain names of the multiple websites to be collected for data and a preset repeated part threshold, identify at least two target websites to be collected for data that correspond to the same or similar target website formats among the multiple websites to be collected for data; judge whether there is a target data collection rule corresponding to the target website format in the specified collection mapping relationship; wherein the specified collection mapping relationship includes a mapping relationship between each website format in multiple website formats and the data collection rule; if the target data collection rule exists in the specified collection mapping relationship, batch data collection is performed on the at least two target websites to be collected for data by using the target data collection rule to obtain the first final data collection result of the at least two target websites to be collected for data; if the specified collection mapping relationship exists, batch data collection is performed on the at least two target websites to be collected for data by using the target data collection rule to obtain the first final data collection result of the at least two target websites to be collected for data; If the target data collection rule does not exist in the system, the data collection rule of the target website format is refined by a collection rule refining model to obtain a preliminary extraction rule; wherein the collection rule refining model is generated through model training based on a variety of website format data samples with data collection rule labels; batch data collection is performed on the at least two target websites to be collected for data through the preliminary extraction rule to obtain preliminary data collection results, and it is determined whether the preliminary data collection results are correct data; if the preliminary data collection results are correct data, the preliminary data collection results are determined as the second final data collection results of the at least two target websites to be collected for data; if the preliminary data collection results are incorrect data or data cannot be collected through the preliminary extraction rule, the screenshot data corresponding to the at least two target websites to be collected for data is obtained, and the third final data collection results of the at least two target websites to be collected for data are obtained through image recognition based on the screenshot data. In this solution, for website collection tasks with different website formats, websites with at least similar website formats are determined only when there are many repeated parts between website domain names, so that websites with similar website formats can be accurately and efficiently determined, and then the website data can be collected according to accurate data collection rules through the mapping relationship between each website format and the data collection rules. Moreover, for website formats that do not have corresponding data collection rules, the corresponding data collection rules can be extracted through the trained model. Even if the extracted data collection rules are inaccurate, the website data can be collected through screenshots and image recognition. Different data collection processing methods are implemented for different website formats, which improves the accuracy of website data collection for different website formats and solves the technical problem of low accuracy of data collected from websites.

[0051] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the specific implementation methods of the present application or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0053] Figure 1 A flowchart of a data processing method based on collection rules provided in an embodiment of the present application;

[0054] Figure 2 An example of generating a viewpoint tree diagram in the data processing method based on collection rules provided in an embodiment of the present application;

[0055] Figure 3 A schematic diagram of the structure of a data processing device based on collection rules provided in an embodiment of the present application;

[0056] Figure 4 A schematic structural diagram of an electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution of the present application will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present application.

[0058] The terms "including" and "having" and any variations thereof mentioned in the embodiments of the present application are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device including a series of steps or units is not limited to the listed steps or units, but may optionally include other steps or units that are not listed, or may optionally include other steps or units that are inherent to these processes, methods, products or devices.

[0059] In the process of collecting massive amounts of information from the above-mentioned social media, news websites and other platforms, it is usually necessary for managers to collect data from a certain website. However, in the process of collecting website data, it is often the case that the accuracy of the data collected from the website is low due to different website formats.

[0060] Based on this, the embodiment of the present application provides a data processing method and device based on collection rules, through which the technical problem of low accuracy of data collected from websites can be solved.

[0061] The embodiments of the present invention are further described below in conjunction with the accompanying drawings.

[0062] Figure 1 A flow chart of a data processing method based on collection rules provided in an embodiment of the present application. Figure 1 As shown, the method includes:

[0063] Step S110, obtaining multiple websites for data collection.

[0064] As a possible implementation method, multiple websites to be collected data correspond to different website formats. For example, HTML format website, CMS content management system website, static web page website, dynamic web page website, single page application website (SPA) and other different website formats. Another example is a variety of different specific content display formats such as text, pictures and videos.

[0065] Step S120, based on the comparison result of the repeated parts between the multiple website domain names of the multiple websites to be collected data and the preset repeated part threshold, identifying at least two target websites to be collected data corresponding to the same or similar target website formats from the multiple websites to be collected data.

[0066] As for the determination method of the same or similar formats, for example, the similar formats can be initially identified based on whether the repeated portion of the website domain name is greater than a preset repeated portion threshold. For example, if the repeated portion between two website domain names is greater than the preset repeated portion threshold, it is determined that the two websites correspond to the same or similar target website formats; if the repeated portion between the two website domain names is less than or equal to the above-mentioned preset repeated portion threshold, it is determined that the website formats corresponding to the two websites are different.

[0067] As an optional implementation, a process can be constructed to determine at least two target websites to be collected from multiple websites to be collected: first define a website domain name similarity function: use the given two website domain names Di and DJ To calculate the similarity score between them S ( Di , DJ ). Then define the target website format set: find all websites with similarity scores greater than the preset threshold. T website pairs to form a target website format set G Next, these steps can be implemented through a series of algorithmic calculations:

[0068] First, calculate the website domain similarity score: for each pair of website domains Di and DJ , a string similarity algorithm (such as Jaccard similarity coefficient, Levenshtein distance, or other algorithms) can be used to calculate the similarity score between them S ( Di , DJ ). Here, the Jaccard similarity coefficient is used as an example. S ( Di , DJ )=∣ Di ∩ DJ ∣ / ∣ Di ∪ DJ ∣; among them, ∣ Di ∩ DJ ∣ represents the size of the set of characters that two domain names have in common, and ∣ Di ∪ DJ ∣ represents the size of the set of all characters of the two domain names.

[0069] Then, identify the target website format set: you need to find all the similarity scores greater than the preset threshold. T and add them to the target site format set G middle. G ={( i , j )∣ S ( Di , DJ )> T , i , j ∈[1, n ]}.in, G is a set of at least two website pairs, each website pair ( i , j ) refers to the domain name Di and DJ The similarity score is greater than the threshold T .

[0070] It should be noted that Di Indicates i Domain names of websites to be collected, including i =1,2,…, n , n The number of websites for data collection; S ( Di , DJ ) indicates the i and j Similarity scores between website domain names;T Indicates the preset repetition threshold ；G Represents a target website format set, which consists of at least two websites to be collected data with the same or similar target website formats.

[0071] The final target website format set G Will include all similarity scores greater than the preset threshold T If the website is correct. G If there are at least two website pairs in the target website, it means that the system has successfully identified at least two websites to be collected with the same or similar target website format. It should be noted that the formula here is a simplified example based on the Jaccard similarity coefficient. In practical applications, it may be necessary to adjust or select different similarity measurement methods according to specific circumstances.

[0072] Step S130, determining whether there is a target data collection rule corresponding to the target website format in the specified collection mapping relationship.

[0073] The designated collection mapping relationship includes a mapping relationship between each website format in the multiple website formats and a data collection rule.

[0074] For the mapping relationship of preset collection logic, such as the extraction rules corresponding to the collection path of the title text map, in actual applications, different website formats can be set up through plug-ins to set collection rules, or they can be automatically set up through machine learning. For example, a new collection rule can be set up through a plug-in, and then the mapping relationship of the collection rule can be added and improved through machine learning.

[0075] As an optional implementation, in the mapping relationship of extraction rules corresponding to preset formats, similar extraction rules may be searched from the extraction rules, and the formats corresponding to the similar extraction rules may be determined as similar formats.

[0076] Step S140: if there is a target data collection rule in the designated collection mapping relationship, batch data collection is performed on at least two target websites to be collected according to the target data collection rule to obtain first final data collection results of at least two target websites to be collected.

[0077] In a possible implementation manner, there may be multiple specific collection methods for performing batch data collection on at least two target websites to be collected data.

[0078] As an example, the target data collection rule includes relevant content of the target field to be collected in the website; the process of batch data collection for at least two target websites to be collected may specifically include:

[0079] First, determine the target topic involved in the target field and the target viewpoint to which the target field belongs with respect to the target topic, wherein the target viewpoint at least includes a positive viewpoint, a negative viewpoint or a neutral viewpoint;

[0080] Then, the content data under multiple viewpoints on the target topic in at least two target websites to be collected for data are distinguished and classified to obtain a first viewpoint type corresponding to the content data of positive viewpoints on the target topic, a second viewpoint type corresponding to the content data of negative viewpoints on the target topic, and a third viewpoint type corresponding to the content data of neutral viewpoints on the target topic; the list corresponding to the first viewpoint type records the first website domain name of the website to be collected for data containing positive viewpoint content data, the list corresponding to the second viewpoint type records the second website domain name of the website to be collected for data containing negative viewpoint content data, and the list corresponding to the third viewpoint type records the third website domain name of the website to be collected for data containing neutral viewpoint content data;

[0081] If the target viewpoint is positive viewpoint content data for the target topic, a target field is selected from the website content corresponding to the first website domain name to obtain a first target field selection result, and batch data collection is performed on the website content corresponding to the first website domain name based on the first target field selection result;

[0082] If the target opinion is negative opinion content data for the target topic, a target field is selected from the website content corresponding to the second website domain name to obtain a second target field selection result, and batch data collection is performed on the website content corresponding to the second website domain name based on the second target field selection result;

[0083] If the target opinion is neutral opinion content data on the target topic, a target field is selected from the website content corresponding to the third website domain name to obtain a third target field selection result, and batch data collection is performed on the website content corresponding to the third website domain name based on the third target field selection result.

[0084] For example, AI can classify the collected data and briefly summarize and classify its opinions, such as which opinion is inclined to the same subject (positive or negative), generate a chart, summarize and record the domain name, generally including positive or negative opinions, and calculate the probability of positive or negative opinions. In this way, it can first interpret the overall opinions and classify them (positive or negative opinions).

[0085] As another example, batch data collection is performed on at least two target websites to be collected data, including:

[0086] Determine a plurality of related collection websites that have web page link relationships with at least two target websites to be collected for data; compare the website content of each of the plurality of related collection websites with the website content of at least two target websites to be collected for data, obtain a comparison result, and determine the website content relevance between the related collection website and the at least two target websites to be collected for data according to the comparison result;

[0087] If there is a situation where the relevance of the contents of multiple websites corresponding to the multiple related collection websites is lower than the preset content relevance threshold, the target related collection website corresponding to the target website content relevance lower than the preset content relevance threshold is determined as the advertising website noise data mixed in the multiple related collection websites;

[0088] The advertising website noise data in multiple relevant collection websites are removed to obtain a final relevant collection website after the removal; batch data collection is performed on the final relevant collection website and at least two target websites to be collected for data.

[0089] For example, AI can identify advertising data on a page and automatically remove it. Specifically, it can compare with other data of the same type collected, such as those mixed with irrelevant content, identify and learn, and change the collection rules by itself. That is, segment summary: if the content is completely irrelevant, it is determined to be an inserted advertisement, or there is noise data.

[0090] In actual applications, each time a domain name is opened, there is a chance that it will jump to other advertising pages. AI is required to identify whether the new web page is an advertisement and needs to be clicked again to open it, or to read whether the new page is related to the previous web page: if it is related, it is determined that the currently opened web page is not an advertisement (for example, the next page and the previous page are definitely related); if it is completely unrelated, it is determined that the currently opened web page is an advertisement.

[0091] Step S150: If the target data collection rule does not exist in the specified collection mapping relationship, the data collection rule of the target website format is refined by the collection rule refinement model to obtain a preliminary extraction rule.

[0092] Among them, the collection rule extraction model is generated through model training based on data samples in various website formats with data collection rule labels.

[0093] As a possible implementation method, the two websites are preliminarily identified as having similar website formats based on whether their website domain names are repeated; if the domain names are not repeated, the preliminary data collection rules for the website format are refined through the collection rule refinement model. When finding similar extraction rules in the extraction rules, machine learning natural language recognition, i.e. deep learning, can be used.

[0094] Step S160, batch data collection is performed on at least two target websites to be collected data according to the preliminary extraction rules to obtain preliminary data collection results, and it is determined whether the preliminary data collection results are correct data.

[0095] As an example, if the domain names are not repeated, the preliminary collection rules extracted from the preset or model are used to try to see if the correct data can be collected. The formats of the collected news websites are not necessarily consistent. For example, if custom development is required, if different websites have similar formats, batch collection can be performed.

[0096] Step S170: If the preliminary data collection result is correct data, the preliminary data collection result is determined as the second final data collection result of at least two target websites to be collected data.

[0097] In an optional implementation, if correct data can be collected through the preliminary extraction rule, it is determined that the preliminary collection rule can be used directly, and the correct data collected through the preliminary extraction rule is used as the final data collection result.

[0098] Step S180, if the preliminary data collection result is incorrect data or data cannot be collected through the preliminary extraction rules, then obtain screenshot data corresponding to at least two target websites for data collection, and obtain a third final data collection result of at least two target websites for data collection through image recognition based on the screenshot data.

[0099] If the correct data cannot be collected through the preliminary extraction rules, the website data collection process can be performed through screenshots and image recognition to obtain the final data collection results.

[0100] In the embodiment of the present application, for website collection tasks with different website formats, websites with at least similar website formats are determined only when there are many repeated parts between the website domain names, so that websites with similar website formats can be accurately and efficiently determined, and then the website data can be collected according to accurate data collection rules through the mapping relationship between each website format and the data collection rules. Moreover, for website formats that do not have corresponding data collection rules, corresponding data collection rules can be extracted through the trained model. Even if the extracted data collection rules are inaccurate, website data can be collected through screenshots and image recognition. Different data collection processing methods are implemented for different website formats, thereby improving the accuracy of website data collection for different website formats.

[0101] The above steps are described in detail below.

[0102] In some embodiments, before the above step S130, the method may further include the following steps:

[0103] A natural language recognition model is used to identify multiple first data collection rules with similar extraction rules from multiple data collection rules in a specified collection mapping relationship; wherein the natural language recognition model is generated through a model training process of language interpretation learning; and multiple first website formats corresponding to the mapping of the multiple first data collection rules are divided into website formats of the same type.

[0104] In an embodiment of the present application, a natural language recognition model is generated through a model training process of language interpretation learning, and then the natural language recognition model is used to identify multiple first data collection rules with similar extraction rules in multiple data collection rules in a specified collection mapping relationship and classify them into the same type of website formats, so that the classification of different website formats is more accurate and efficient.

[0105] In some embodiments, each of the first opinion type, the second opinion type, and the third opinion type includes multiple specific opinions; after the above process of distinguishing and classifying the content data under multiple opinions on the target topic in at least two target websites to be collected for data, obtaining the first opinion type corresponding to the positive opinion content data on the target topic, the second opinion type corresponding to the negative opinion content data on the target topic, and the third opinion type corresponding to the neutral opinion content data on the target topic, as shown in FIG. Figure 2 As shown, the method may also include the following steps:

[0106] a), establishing a viewpoint tree diagram for the target topic based on the multiple specific viewpoints contained in the first viewpoint type, the second viewpoint type, and the third viewpoint type, wherein the viewpoint tree diagram includes multiple viewpoint nodes, each of which is used to represent a specific viewpoint;

[0107] b) determining an association relationship between two viewpoint nodes among a plurality of viewpoint nodes, and determining a connection relationship between the two viewpoint nodes based on the association relationship, wherein the association relationship includes at least one of a contradictory association relationship, a corroborative association relationship, and an association relationship derived from viewpoints;

[0108] c) determining the connection structure of the viewpoint tree diagram based on multiple connection relationships, and generating a final viewpoint tree diagram based on the connection structure and the viewpoint nodes.

[0109] Exemplarily, viewpoints may be classified into a tree diagram with more specific subdivisions of the relationships. For example, the relationship between two related viewpoints may be an extension of the viewpoints, a relationship in which the viewpoints support each other, or a relationship in which the viewpoints contradict each other. Through these various different specific subdivisions, more specific viewpoints on a certain event or topic may be obtained.

[0110] In some embodiments, the process of comparing the website content of each of the multiple related collection websites with the website content of at least two target websites to be collected data to obtain the comparison result may specifically include the following steps:

[0111] A simulated click operation is performed on each of the multiple relevant collection websites through AI, and the website content in each relevant collection website is obtained based on the simulated click operation; the website content in each relevant collection website is compared with the website content of at least two target websites for data collection to obtain a comparison result.

[0112] As an optional implementation method, AI can be used to simulate human click operations on the collected websites, which can save the cost of manual website visits and improve the accuracy of website content comparison.

[0113] In some embodiments, before step S130, the method may further include the following steps:

[0114] Data samples under different website formats and the website content collection method corresponding to each website format are obtained from websites corresponding to multiple website formats, and AI is used to learn based on the data samples and the website content collection method to obtain a specified collection mapping relationship including data collection rules; or, the specified collection mapping relationship including data collection rules is determined through the setting method of the plug-in.

[0115] In the embodiment of the present application, it is no longer necessary to manually set the collection rules, but AI can select all the content, text, pictures and videos that meet the requirements. That is, AI reads the content, text, pictures and videos (AI is preloaded to read at a high speed), then reads the video viewpoint, and then collects the target viewpoint, thereby improving data collection efficiency.

[0116] Figure 3 A schematic diagram of the structure of a data processing device based on collection rules is provided. Figure 3 As shown, the data processing device 300 based on the collection rule includes:

[0117] An acquisition module 301 is used to acquire a plurality of websites to be collected data, wherein the plurality of websites to be collected data correspond to different website formats;

[0118] The first identification module 302 is used to identify at least two target websites to be collected for data that have the same or similar target website formats from among the multiple websites to be collected for data based on a comparison result of the repeated parts between the multiple website domain names of the multiple websites to be collected for data and a preset repeated part threshold;

[0119] The judgment module 303 is used to judge whether there is a target data collection rule corresponding to the target website format in the specified collection mapping relationship; wherein the specified collection mapping relationship includes a mapping relationship between each website format in a plurality of website formats and a data collection rule;

[0120] A first collection module 304 is configured to, if the target data collection rule exists in the designated collection mapping relationship, perform batch data collection on the at least two target websites to be collected data according to the target data collection rule to obtain a first final data collection result of the at least two target websites to be collected data;

[0121] The refining module 305 is used to refine the data collection rules of the target website format through a collection rule refining model to obtain preliminary extraction rules if the target data collection rules do not exist in the specified collection mapping relationship; wherein the collection rule refining model is generated through model training based on multiple website format data samples with data collection rule labels;

[0122] The second collection module 306 is used to perform batch data collection on the at least two target websites to be collected data according to the preliminary extraction rules, obtain preliminary data collection results, and determine whether the preliminary data collection results are correct data;

[0123] A determination module 307 is configured to determine the preliminary data collection result as a second final data collection result of the at least two target websites to be collected data if the preliminary data collection result is correct data;

[0124] The second identification module 308 is used to obtain screenshot data corresponding to the at least two target websites for data collection if the preliminary data collection result is incorrect data or data cannot be collected through the preliminary extraction rules, and obtain a third final data collection result of the at least two target websites for data collection through image recognition based on the screenshot data.

[0125] In some embodiments, the apparatus further comprises:

[0126] A third identification module is used to identify a plurality of first data collection rules with similar extraction rules from a plurality of data collection rules in the specified collection mapping relationship by a natural language recognition model before determining whether there is a target data collection rule corresponding to the target website format in the specified collection mapping relationship; wherein the natural language recognition model is generated by a model training process of language interpretation learning;

[0127] The division module is used to divide the multiple first website formats corresponding to the multiple first data collection rules into website formats of the same type.

[0128] In some embodiments, the target data collection rule includes relevant content of the target field to be collected in the website; the first collection module is specifically used for:

[0129] Determine the target subject involved in the target field and the target viewpoint to which the target field belongs with respect to the target subject; wherein the target viewpoint at least includes a positive viewpoint, a negative viewpoint or a neutral viewpoint;

[0130] Distinguish and classify the content data under multiple viewpoints on the target subject in the at least two target websites to be collected for data, and obtain a first viewpoint type corresponding to the positive viewpoint content data on the target subject, a second viewpoint type corresponding to the negative viewpoint content data on the target subject, and a third viewpoint type corresponding to the neutral viewpoint content data on the target subject; the list corresponding to the first viewpoint type records the first website domain name of the website to be collected for data containing the positive viewpoint content data, the list corresponding to the second viewpoint type records the second website domain name of the website to be collected for data containing the negative viewpoint content data, and the list corresponding to the third viewpoint type records the third website domain name of the website to be collected for data containing the neutral viewpoint content data;

[0131] If the target viewpoint is the positive viewpoint content data for the target theme, the target field is selected from the website content corresponding to the first website domain name to obtain a first target field selection result, and batch data collection is performed on the website content corresponding to the first website domain name based on the first target field selection result;

[0132] If the target opinion is the negative opinion content data for the target topic, the target field is selected from the website content corresponding to the second website domain name to obtain a second target field selection result, and batch data collection is performed on the website content corresponding to the second website domain name based on the second target field selection result;

[0133] If the target opinion is the neutral opinion content data for the target topic, the target field is selected from the website content corresponding to the third website domain name to obtain a third target field selection result, and batch data collection is performed on the website content corresponding to the third website domain name based on the third target field selection result.

[0134] In some embodiments, each of the first viewpoint type, the second viewpoint type, and the third viewpoint type includes multiple specific viewpoints; the device further includes:

[0135] An establishment module is used to distinguish and classify the content data under multiple viewpoints on the target theme in the at least two target websites to be collected data, and after obtaining a first viewpoint type corresponding to the positive viewpoint content data on the target theme, a second viewpoint type corresponding to the negative viewpoint content data on the target theme, and a third viewpoint type corresponding to the neutral viewpoint content data on the target theme, establish a viewpoint tree diagram for the target theme based on the multiple specific viewpoints respectively included in the first viewpoint type, the second viewpoint type, and the third viewpoint type; the viewpoint tree diagram includes a plurality of viewpoint nodes, each of which is used to represent one of the specific viewpoints;

[0136] A first determination module is used to determine the association relationship between two viewpoint nodes among the plurality of viewpoint nodes, and determine the connection relationship between the two viewpoint nodes based on the association relationship; wherein the association relationship includes at least one of a mutually contradictory association relationship, a corroborative association relationship, and a viewpoint-derived association relationship;

[0137] A generating module is used to determine the connection structure of the viewpoint tree diagram based on the multiple connection relationships, and to generate a final viewpoint tree diagram based on the connection structure and the viewpoint nodes.

[0138] In some embodiments, the first acquisition module is specifically used for:

[0139] Determining a plurality of related data collection websites having web page link relationships with the at least two target data collection websites;

[0140] Comparing the website content of each of the multiple related collection websites with the website content of the at least two target websites to be collected for data, obtaining a comparison result, and determining the website content relevance between the related collection website and the at least two target websites to be collected for data according to the comparison result;

[0141] If there is a situation where the relevance of the contents of the multiple websites corresponding to the multiple related collection websites is lower than a preset content relevance threshold, the target related collection website corresponding to the target website content relevance lower than the preset content relevance threshold is determined as the advertising website noise data mixed in the multiple related collection websites;

[0142] Removing the noise data of the advertising website from the multiple related collection websites to obtain a final related collection website after the removal process;

[0143] Batch data collection is performed on the final relevant collection website and the at least two target websites to be collected data.

[0144] In some embodiments, the first acquisition module is further used for:

[0145] Performing a simulated click operation on each of the multiple related collection websites through AI, and acquiring website content in each of the related collection websites based on the simulated click operation;

[0146] The website content of each of the related collection websites is compared with the website content of the at least two target websites to be collected data to obtain a comparison result.

[0147] In some embodiments, the apparatus further comprises:

[0148] The second determination module is used to obtain data samples under different website formats and website content collection methods corresponding to each website format from websites corresponding to multiple website formats before judging whether there is a target data collection rule mapped to the target website format in the specified collection mapping relationship, and to learn through AI based on the data samples and the website content collection method to obtain the specified collection mapping relationship including the data collection rules; or, to determine the specified collection mapping relationship including the data collection rules through the setting method of the plug-in.

[0149] The data processing device based on collection rules provided in the embodiment of the present application has the same technical features as the data processing method based on collection rules provided in the above embodiment, so it can also solve the same technical problems and achieve the same technical effects.

[0150] An electronic device provided in an embodiment of the present application is Figure 4 As shown, the electronic device 400 includes a processor 402 and a memory 401, wherein the memory stores a computer program that can be run on the processor, and the processor implements the steps of the method provided in the above embodiment when executing the computer program.

[0151] See also Figure 4 The electronic device further includes: a bus 403 and a communication interface 404, a processor 402, a communication interface 404 and a memory 401 are connected via the bus 403; the processor 402 is used to execute an executable module stored in the memory 401, such as a computer program.

[0152] The memory 401 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk storage. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 404 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used.

[0153] The bus 403 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0154] Among them, the memory 401 is used to store programs, and the processor 402 executes the program after receiving the execution instruction. The method executed by the device defined by the process disclosed in any embodiment of the present application can be applied to the processor 402 or implemented by the processor 402.

[0155] The processor 402 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 402. The above processor 402 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware decoding processor to execute, or the hardware and software modules in the decoding processor are combined to execute. The software module may be located in a storage medium mature in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 401, and the processor 402 reads the information in the memory 401 and completes the steps of the above method in combination with its hardware.

[0156] Corresponding to the above-mentioned data processing method based on collection rules, an embodiment of the present application also provides a computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to execute the steps of the above-mentioned data processing method based on collection rules.

[0157] The data processing device based on the collection rules provided in the embodiment of the present application can be specific hardware on the device or software or firmware installed on the device. The device provided in the embodiment of the present application, its implementation principle and the technical effect produced are the same as those in the aforementioned method embodiment. For the sake of brief description, the parts not mentioned in the device embodiment can refer to the corresponding contents in the aforementioned method embodiment. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, devices and units described above can all refer to the corresponding processes in the aforementioned method embodiment, and will not be repeated here.

[0158] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0159] For another example, the flowchart and block diagram in the accompanying drawings show the possible architecture, function and operation of the device, method and computer program product according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of the code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or the flowchart, and the combination of the boxes in the block diagram and / or the flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0160] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0161] In addition, each functional unit in the embodiments provided in the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0162] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the data processing method based on the collection rules described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store program codes.

[0163] It should be noted that similar numbers and letters represent similar items in the following figures. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In addition, the terms "first", "second", "third", etc. are only used to distinguish the description and are not to be understood as indicating or implying relative importance.

[0164] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present application, which are used to illustrate the technical solution of the present application, rather than to limit it. The protection scope of the present application is not limited thereto. Although the present application is described in detail with reference to the aforementioned embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solution recorded in the aforementioned embodiments within the technical scope disclosed in the present application, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solution deviate from the scope of the technical solution of the embodiment of the present application. They should all be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A data processing method based on collection rules, characterized in that: The method comprises: Acquire multiple websites to be collected data, wherein the multiple websites to be collected data have different website formats; Based on the comparison result of the repeated parts between the multiple website domain names of the multiple websites to be collected data and the preset repeated part threshold, identifying at least two target websites to be collected data corresponding to the same or similar target website formats from the multiple websites to be collected data; Determine whether there is a target data collection rule mapped to the target website format in the specified collection mapping relationship; wherein the specified collection mapping relationship includes a mapping relationship between each website format in a plurality of website formats and a data collection rule; If the target data collection rule exists in the designated collection mapping relationship, batch data collection is performed on the at least two target websites to be collected data according to the target data collection rule to obtain a first final data collection result of the at least two target websites to be collected data; If the target data collection rule does not exist in the specified collection mapping relationship, the data collection rule of the target website format is refined by a collection rule refinement model to obtain a preliminary extraction rule; wherein the collection rule refinement model is generated by model training based on multiple website format data samples with data collection rule labels; Perform batch data collection on the at least two target websites to be collected data by using the preliminary extraction rules to obtain preliminary data collection results, and determine whether the preliminary data collection results are correct data; If the preliminary data collection result is correct data, determining the preliminary data collection result as the second final data collection result of the at least two target websites to be collected data; If the preliminary data collection result is incorrect data or data cannot be collected according to the preliminary extraction rule, then obtaining screenshot data corresponding to the at least two target websites to be collected, and obtaining a third final data collection result of the at least two target websites to be collected through image recognition based on the screenshot data; The target data collection rule includes relevant content of the target field to be collected in the website; batch data collection is performed on the at least two target websites to be collected, including: Determine the target subject involved in the target field and the target viewpoint to which the target field belongs with respect to the target subject; wherein the target viewpoint at least includes a positive viewpoint, a negative viewpoint or a neutral viewpoint; Differentiate and classify the content data under multiple viewpoints on the target subject in the at least two target websites to be collected for data, and obtain a first viewpoint type corresponding to the positive viewpoint content data on the target subject, a second viewpoint type corresponding to the negative viewpoint content data on the target subject, and a third viewpoint type corresponding to the neutral viewpoint content data on the target subject; the list corresponding to the first viewpoint type records the first website domain name of the website to be collected for data containing the positive viewpoint content data, the list corresponding to the second viewpoint type records the second website domain name of the website to be collected for data containing the negative viewpoint content data, and the list corresponding to the third viewpoint type records the third website domain name of the website to be collected for data containing the neutral viewpoint content data; each of the first viewpoint type, the second viewpoint type and the third viewpoint type contains multiple specific viewpoints; Establishing a viewpoint tree diagram for the target topic based on the multiple specific viewpoints respectively included in the first viewpoint type, the second viewpoint type, and the third viewpoint type; the viewpoint tree diagram includes multiple viewpoint nodes, each of which is used to represent one of the specific viewpoints; Determine an association relationship between two viewpoint nodes among the plurality of viewpoint nodes, and determine a connection relationship between the two viewpoint nodes based on the association relationship; wherein the association relationship includes at least one of a mutually contradictory association relationship, an association relationship providing evidence, and an association relationship derived from viewpoints; Determine a connection structure of the viewpoint tree diagram based on the multiple connection relationships, and generate a final viewpoint tree diagram based on the connection structure and the viewpoint nodes; If the target viewpoint is the positive viewpoint content data for the target theme, the target field is selected from the website content corresponding to the first website domain name to obtain a first target field selection result, and batch data collection is performed on the website content corresponding to the first website domain name based on the first target field selection result; If the target opinion is the negative opinion content data for the target topic, the target field is selected from the website content corresponding to the second website domain name to obtain a second target field selection result, and batch data collection is performed on the website content corresponding to the second website domain name based on the second target field selection result; If the target opinion is the neutral opinion content data for the target topic, the target field is selected from the website content corresponding to the third website domain name to obtain a third target field selection result, and batch data collection is performed on the website content corresponding to the third website domain name based on the third target field selection result.

2. The method according to claim 1, characterized in that Before the step of determining whether there is a target data collection rule corresponding to the target website format in the specified collection mapping relationship, the method further includes: Identifying a plurality of first data collection rules with similar extraction rules from a plurality of the data collection rules in the specified collection mapping relationship through a natural language recognition model; wherein the natural language recognition model is generated through a model training process of language interpretation learning; The plurality of first website formats mapped corresponding to the plurality of first data collection rules are divided into website formats of the same type.

3. The method according to claim 1, characterized in that Batch data collection is performed on the at least two target websites to be collected data, including: Determining a plurality of related data collection websites having web page link relationships with the at least two target data collection websites; Comparing the website content of each of the multiple related collection websites with the website content of the at least two target websites to be collected for data, obtaining a comparison result, and determining the website content relevance between the related collection website and the at least two target websites to be collected for data according to the comparison result; If there is a situation where the relevance of the contents of the multiple websites corresponding to the multiple related collection websites is lower than a preset content relevance threshold, the target related collection website corresponding to the target website content relevance lower than the preset content relevance threshold is determined as the advertising website noise data mixed in the multiple related collection websites; Removing the noise data of the advertising website from the multiple related collection websites to obtain a final related collection website after the removal process; Batch data collection is performed on the final relevant collection website and the at least two target websites to be collected data.

4. The method according to claim 3, characterized in that The step of comparing the website content of each of the multiple related collection websites with the website content of the at least two target websites to be collected data to obtain a comparison result includes: Performing a simulated click operation on each of the multiple related collection websites through AI, and acquiring website content in each of the related collection websites based on the simulated click operation; The website content of each of the related collection websites is compared with the website content of the at least two target websites to be collected data to obtain a comparison result.

5. The method according to claim 1, characterized in that Before the step of determining whether there is a target data collection rule corresponding to the target website format in the specified collection mapping relationship, the method further includes: Obtaining data samples in different website formats and website content collection methods corresponding to each of the website formats from websites corresponding to the multiple website formats, and learning through AI based on the data samples and the website content collection methods to obtain the specified collection mapping relationship including the data collection rules; or, The designated collection mapping relationship including the data collection rule is determined by setting the plug-in.

6. A data processing device based on collection rules, characterized in that: include: An acquisition module is used to acquire a plurality of websites to be collected data, wherein the plurality of websites to be collected data correspond to different website formats; A first identification module is used to identify at least two target websites to be collected for data that have the same or similar target website formats from among the multiple websites to be collected for data based on a comparison result of repeated parts between multiple website domain names of the multiple websites to be collected for data and a preset repeated part threshold; A judgment module, used to judge whether there is a target data collection rule corresponding to the target website format in the specified collection mapping relationship; wherein the specified collection mapping relationship includes a mapping relationship between each website format in a plurality of website formats and a data collection rule; A first collection module is used for, if the target data collection rule exists in the specified collection mapping relationship, performing batch data collection on the at least two target websites to be collected data according to the target data collection rule to obtain a first final data collection result of the at least two target websites to be collected data; A refining module, for refining the data collection rules of the target website format through a collection rule refining model to obtain preliminary extraction rules if the target data collection rules do not exist in the specified collection mapping relationship; wherein the collection rule refining model is generated through model training based on data samples of multiple website formats with data collection rule labels; A second collection module is used to perform batch data collection on the at least two target websites to be collected data according to the preliminary extraction rules, obtain preliminary data collection results, and determine whether the preliminary data collection results are correct data; A determination module, configured to determine the preliminary data collection result as a second final data collection result of the at least two target websites to be collected data if the preliminary data collection result is correct data; A second recognition module is used to obtain screenshot data corresponding to the at least two target websites to be collected for data, and obtain a third final data collection result of the at least two target websites to be collected for data by image recognition based on the screenshot data if the preliminary data collection result is incorrect data or data cannot be collected according to the preliminary extraction rule; The target data collection rules include relevant content of the target field to be collected on the website; the first collection module is specifically used for: Determine the target subject involved in the target field and the target viewpoint to which the target field belongs with respect to the target subject; wherein the target viewpoint at least includes a positive viewpoint, a negative viewpoint or a neutral viewpoint; Distinguish and classify the content data under multiple viewpoints on the target subject in the at least two target websites to be collected for data, and obtain a first viewpoint type corresponding to the positive viewpoint content data on the target subject, a second viewpoint type corresponding to the negative viewpoint content data on the target subject, and a third viewpoint type corresponding to the neutral viewpoint content data on the target subject; the list corresponding to the first viewpoint type records the first website domain name of the website to be collected for data containing the positive viewpoint content data, the list corresponding to the second viewpoint type records the second website domain name of the website to be collected for data containing the negative viewpoint content data, and the list corresponding to the third viewpoint type records the third website domain name of the website to be collected for data containing the neutral viewpoint content data; If the target viewpoint is the positive viewpoint content data for the target theme, the target field is selected from the website content corresponding to the first website domain name to obtain a first target field selection result, and batch data collection is performed on the website content corresponding to the first website domain name based on the first target field selection result; If the target opinion is the negative opinion content data for the target topic, the target field is selected from the website content corresponding to the second website domain name to obtain a second target field selection result, and batch data collection is performed on the website content corresponding to the second website domain name based on the second target field selection result; If the target viewpoint is the neutral viewpoint content data for the target topic, the target field is selected from the website content corresponding to the third website domain name to obtain a third target field selection result, and batch data collection is performed on the website content corresponding to the third website domain name based on the third target field selection result; Each of the first viewpoint type, the second viewpoint type and the third viewpoint type includes a plurality of specific viewpoints; the device further includes: A building module, used to build a viewpoint tree diagram for the target topic based on the multiple specific viewpoints respectively included in the first viewpoint type, the second viewpoint type, and the third viewpoint type; the viewpoint tree diagram includes multiple viewpoint nodes, each of which is used to represent one of the specific viewpoints; A first determination module is used to determine the association relationship between two viewpoint nodes among the plurality of viewpoint nodes, and determine the connection relationship between the two viewpoint nodes based on the association relationship; wherein the association relationship includes at least one of a mutually contradictory association relationship, a corroborative association relationship, and a viewpoint-derived association relationship; A generating module is used to determine the connection structure of the viewpoint tree diagram based on the multiple connection relationships, and to generate a final viewpoint tree diagram based on the connection structure and the viewpoint nodes.

7. An electronic device comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to execute the method according to any one of claims 1 to 5.

Images