Internet-based data acquisition method and device and storage medium

Abstract

The embodiment of the invention provides a data collection method and device based on the Internet and a storage medium. The method comprises the steps that keywords input by a user and webpage content data corresponding to the keywords are collected; calculating the relevancy between the webpage content data and stored keywords in a preset keyword table; calculating the confidence coefficient of the keyword according to the relevancy; and if the confidence coefficient of the keyword is greater than a first preset value, storing the keyword into the keyword table. According to the technical scheme provided by the invention, repeated or redundant keywords can be deleted, so that the quality of collected keywords is ensured.

Description

Technical Field

[0001] This document relates to the field of data collection technology, and in particular to an Internet-based data collection method, device and storage medium. Background Art

[0002] With the rapid development of mobile Internet, search apps and news apps have occupied most of the time of mobile users. Users will leave a lot of data on mobile apps. For example, the search keywords of users on news apps are: agriculture-related news, and agriculture is a user's interest word. If these interest words can be mined, huge commercial value can be generated.

[0003] In the prior art, the search data and click data of app users are returned to the servers of Internet companies for storage. Then, the operator determines whether the keyword is a valid keyword based on the returned keyword. For example, in an agricultural app, a user searches for "how to cook potatoes". The operator will determine whether the keyword is a valid keyword. If the keyword is not in the operator's database, the operator will include the keyword and use it to promote content for agricultural apps.

[0004] In the above process, the amount of data flowing back is in the tens of millions or even hundreds of millions. The amount of data is too large, and the server needs more storage space and needs to process more data. Under the existing hardware setup, the server may not be able to operate normally, resulting in low server efficiency and server crash. At the same time, in the above process, the server's data calculation amount is too large, so the server will search for keywords with fuzzy matching. If it cannot find the search, it will give up, which will also lead to low search efficiency. In addition, users may enter some meaningless words for retrieval. For example, in order to search for "how to cook potatoes", users enter "potatoes". These meaningless data will also be collected by the server, which will make the keyword library in the server more inaccurate, and the server will misjudge the user's usage behavior.

[0005] In addition, users often search for "potatoes" and "recipes", and the probability of "potatoes" and "recipes" being recommended in the corresponding keyword database will increase, resulting in repeated push of keywords in the keyword database. In addition, when the frequency of "potatoes" and "recipes" is large enough, the server will directly ignore the keywords "potatoes" and "recipes" and will not update the database, resulting in the user's keyword being unable to update. Summary of the invention

[0006] In view of the above analysis, the present application aims to propose an Internet-based data collection method, device and storage medium to solve at least one of the above technical problems.

[0007] In a first aspect, one or more embodiments of this specification provide a data collection method based on the Internet, including:

[0008] Collecting keywords input by users and web page content data corresponding to the keywords;

[0009] Calculating the relevance between the webpage content data and the stored keywords in the preset keyword table;

[0010] Calculating the confidence of the keyword according to the relevance; and

[0011] If the confidence of the keyword is greater than a first preset value, the keyword is stored in the keyword table.

[0012] Further, determining a first probability that the user uses the stored keyword to access the webpage corresponding to the keyword;

[0013] Determine a second probability that the user uses the keyword to access the webpage corresponding to the keyword;

[0014] Calculating the complementary confidence between the stored keyword and the keyword according to the first probability and the second probability;

[0015] Adding the confidence and the complementary confidence to obtain a total confidence of the other keywords and the first keyword; and

[0016] If the total confidence is greater than a first preset value, the keyword is stored in the keyword table.

[0017] Further, obtaining the keywords to be tested from the keyword table;

[0018] Detecting whether the confidence of the keyword to be tested is greater than the first preset value;

[0019] If not, the keyword to be tested is deleted from the keyword table.

[0020] Further, the keyword table includes a plurality of the stored keywords, each of the stored keywords corresponds to a relevance; and

[0021] Calculating the confidence of the keyword according to the relevance includes:

[0022] Selecting a target correlation degree greater than a preset correlation degree from the plurality of correlation degrees;

[0023] The confidence of the keyword is calculated according to the target relevance.

[0024] Further, detecting whether the confidence of the stored keyword is greater than a second preset value; and

[0025] When the confidence of the stored keyword is greater than a second preset value, a credible mark is added, and the search record data corresponding to the stored keyword is deleted.

[0026] Further, detecting whether the confidence of the stored keyword is less than a third preset value; and

[0027] When the confidence of the stored keyword is less than a third preset value, a credible mark is added, and the search record data corresponding to the stored keyword is deleted.

[0028] In a second aspect, one or more embodiments of this specification provide an Internet-based data collection device, including: a collection module, a data processing module, and a storage module;

[0029] The acquisition module is used to acquire the keywords input by the user and the webpage content data corresponding to the keywords;

[0030] The data processing module is used to calculate the relevance between the webpage content data and the stored keywords in the preset keyword table; and calculate the confidence of the keyword based on the relevance; and

[0031] The storage module is used for storing the keyword in the keyword table if the confidence of the keyword is greater than a first preset value.

[0032] Further, the data processing module is further used to determine a first probability that the user uses the stored keyword to access the webpage corresponding to the keyword; determine a second probability that the user uses the keyword to access the webpage corresponding to the keyword; calculate the complementary confidence of the stored keyword and the keyword based on the first probability and the second probability; add the confidence and the complementary confidence to obtain the total confidence of the other keywords and the first keyword; and

[0033] The storage module is used for storing the keyword in the keyword table if the total confidence is greater than a first preset value.

[0034] Furthermore, the device also includes a detection module;

[0035] The detection module is used to obtain the keyword to be tested from the keyword table; detect whether the confidence of the keyword to be tested is greater than the first preset value; and if not, delete the keyword to be tested from the keyword table.

[0036] In a third aspect, one or more embodiments of the present specification provide a storage medium for storing computer-executable instructions, which, when executed, implement the steps of the Internet-based data collection method described in any one of the first aspects.

[0037] Compared with the prior art, this application can at least achieve the following technical effects:

[0038] (1) The present invention monitors the input of keywords and the access of web pages by users, collects records of the input of keywords and the access of web pages by users, and then monitors, calculates and counts the keywords. According to the correlation value between the keyword and the first web page, the confidence value of the keyword is corrected by the confidence, and then the keywords are screened, repeated or redundant keywords are deleted, and the corresponding link web pages with small correlation values ​​with the keywords are deleted. The present invention can ensure the quality of collected keywords and improve the accuracy of keywords.

[0039] (2) The present invention stores keywords and an information base formed by the keywords, and updates the information base according to the information of the first keyword, so that when the user operates next time, more accurate keywords and matching web pages are provided, thereby improving the user's search experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate one or more embodiments of this specification or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0041] Figure 1 A flow chart of an Internet-based data collection method provided for one or more embodiments of this specification;

[0042] Figure 2 A schematic diagram of the structure of an Internet-based data acquisition device provided for one or more embodiments of this specification. DETAILED DESCRIPTION

[0043] In order to enable those skilled in the art to better understand the technical solutions in one or more embodiments of this specification, the following will be combined with the drawings in one or more embodiments of this specification to clearly and completely describe the technical solutions in one or more embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on one or more embodiments of this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this document.

[0044] The present application embodiment provides a data collection method based on the Internet, such as Figure 1 As shown, the following steps are included:

[0045] Step 1: Collect keywords input by users and web page content data corresponding to the keywords.

[0046] In the embodiment of the present application, the user inputs keywords through the search engine to obtain the corresponding web pages. Based on the above operation process, the keywords input by the user and the web page content data corresponding to these keywords are obtained.

[0047] Step 2: Calculate the relevance between the webpage content data and the stored keywords in the preset keyword table.

[0048] In the embodiment of the present application, the confidence interval refers to the estimated interval of the population parameter constructed by the sample statistics. Therefore, the relevance between the webpage content data and the stored keywords in the preset keyword table is calculated to construct a sample with a determined confidence.

[0049] Step 3: Calculate the confidence of the keyword based on the relevance.

[0050] In the embodiment of the present application, the confidence level represents the relevance of these keywords to the content that the customer wants to view. The higher the confidence level, the greater the correlation between the two.

[0051] Step 4: If the confidence of the keyword is greater than a first preset value, the keyword is stored in the keyword table.

[0052] It can be seen that the present application establishes associations between newly input keywords and existing keywords based on confidence levels, so as to filter out repeated, redundant or even irrelevant new keywords, thereby ensuring the accuracy of the keywords in the keyword library.

[0053] In the embodiment of the present application, in addition to directly associating keywords based on word meanings, in order to further improve the accuracy of words, keywords can also be associated based on access probabilities. The specific process is as follows:

[0054] A1. Determine a first probability that a user uses a stored keyword to access a webpage corresponding to the keyword.

[0055] A2. Determine a second probability that a user uses a keyword to access a webpage corresponding to the keyword.

[0056] A3. Calculate the complementary confidence between the stored keyword and the keyword according to the first probability and the second probability.

[0057] A4. Add the confidence and the complementary confidence to obtain the total confidence of the other keywords and the first keyword.

[0058] A5. If the total confidence level is greater than the first preset value, store the keyword in the keyword table.

[0059] It should be noted that both the first probability and the second probability are obtained through statistical calculations. For example, count the number of times n1 that a user accesses webpage A with "potato" as the keyword, the number of times n2 that the user accesses webpage A with "potato tuber" as the keyword, and then count the total number of times S that the user accesses the webpage. Then n1 / S is the first probability, and n2 / S is the second probability.

[0060] In a real - world scenario, a user may focus on a certain field intensively within a period of time. Once having focused on it, the user may no longer pay attention. This will cause accumulation in the keyword library. To solve the above - mentioned problem, in the embodiment of the present application, obtain the keyword to be tested from the keyword table; detect whether the confidence level of the keyword to be tested is greater than the first preset value; if not, delete the keyword to be tested from the keyword table.

[0061] In the embodiment of the present application, when counting the confidence level, in order to ensure the accuracy of the data, it is necessary to eliminate some data with low relevance. Specifically:

[0062] The keyword table includes multiple stored keywords, and each stored keyword corresponds to a relevance

[0063] Select the target relevance greater than the preset relevance from multiple relevances;

[0064] Calculate the confidence level of the keyword according to the target relevance.

[0065] There are not many keywords that users use stably for a long time. But according to the existing rules, if a certain word has a large usage frequency within a period of time, this word will probably replace the keywords that users use stably for a long time from the word library. To solve the above - mentioned problem, in the embodiment of the present application, detect whether the confidence level of the stored keyword is greater than the second preset value; when the confidence level of the stored keyword is greater than the second preset value, add a trusted identifier and delete the retrieval record data corresponding to the stored keyword. By setting the trust identifier, these words can be called at any time, and at the same time, deleting the corresponding retrieval record data can save storage space.

[0066] In the embodiment of the present application, for words with a relatively low confidence level, detect whether the confidence level of the stored keyword is less than the third preset value. When the confidence level of the stored keyword is less than the third preset value, add a trusted identifier and delete the retrieval record data corresponding to the stored keyword.

[0067] The embodiment of the present application provides an Internet - based data acquisition device, including: a collection module 201, a data processing module 202, and a storage module 203;

[0068] The acquisition module 201 is used to acquire keywords input by users and web page content data corresponding to the keywords;

[0069] The data processing module 202 is used to calculate the relevance between the webpage content data and the stored keywords in the preset keyword table; and calculate the confidence of the keyword based on the relevance; and

[0070] The storage module 203 is configured to store the keyword in the keyword table if the confidence of the keyword is greater than a first preset value.

[0071] In the embodiment of the present application, the data processing module 202 is further used to determine a first probability that the user uses the stored keyword to access the webpage corresponding to the keyword; determine a second probability that the user uses the keyword to access the webpage corresponding to the keyword; calculate the complementary confidence of the stored keyword and the keyword based on the first probability and the second probability; add the confidence and the complementary confidence to obtain the total confidence of the other keywords and the first keyword; and

[0072] The storage module 203 is configured to store the keyword in the keyword table if the total confidence is greater than a first preset value.

[0073] In the embodiment of the present application, the device further includes a detection module;

[0074] The detection module is used to obtain the keyword to be tested from the keyword table; detect whether the confidence of the keyword to be tested is greater than the first preset value; and if not, delete the keyword to be tested from the keyword table.

[0075] An embodiment of the present application provides a storage medium for storing computer-executable instructions, and when the computer-executable instructions are executed, the steps of an Internet-based data collection method described in any one of the aforementioned embodiments are implemented.

[0076] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0077] In the 1930s, improvements to a technology could be clearly distinguished as hardware improvements (for example, improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the method flow). However, with the development of technology, many improvements to the method flow today can be regarded as direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that an improvement in a method flow cannot be implemented using a hardware entity module. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming it themselves, without having to ask a chip manufacturer to design and produce a dedicated integrated circuit chip. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages ​​and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.

[0078] The controller can be implemented in any appropriate manner, for example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (such as software or firmware) that can be executed by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that in addition to implementing the controller in a purely computer-readable program code manner, the controller can be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, this controller can be considered as a hardware component, and the devices included therein for implementing various functions can also be regarded as structures within the hardware component. Or even, the devices for implementing various functions can be regarded as both software modules for implementing the method and structures within the hardware component.

[0079] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0080] For the convenience of description, the above devices are described in terms of functions and are divided into various units. Of course, when implementing the embodiments of this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0081] It should be understood by those skilled in the art that one or more embodiments of this specification may be provided as a method, system or computer program product. Therefore, one or more embodiments of this specification may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0082] This specification is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of this specification. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0083] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0084] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0085] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0086] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0087] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0088] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0089] One or more embodiments of the present specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. One or more embodiments of the present specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0090] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0091] The above description is only an embodiment of this document and is not intended to limit this document. For those skilled in the art, this document may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this document should be included in the scope of the claims of this document.

Claims

1. A data collection method based on the Internet, characterized in that: include: Collecting keywords input by users and web page content data corresponding to the keywords; Calculating the relevance between the webpage content data and the stored keywords in the preset keyword table; Calculating the confidence of the keyword according to the relevance; as well as If the confidence of the keyword is greater than a first preset value, the keyword is stored in the keyword table.

2. The method according to claim 1, characterized in that The method further comprises: Determine a first probability that the user uses the stored keyword to access a webpage corresponding to the keyword; Determine a second probability that the user uses the keyword to access the webpage corresponding to the keyword; Calculating the complementary confidence between the stored keyword and the keyword according to the first probability and the second probability; Adding the confidence and the complementary confidence to obtain a total confidence of other keywords and the first keyword; and If the total confidence is greater than a first preset value, the keyword is stored in the keyword table.

3. The method according to claim 1, characterized in that: The method further comprises: Acquire the keywords to be tested from the keyword table; Detecting whether the confidence of the keyword to be tested is greater than the first preset value; If not, the keyword to be tested is deleted from the keyword table.

4. The method according to claim 1, characterized in that: The keyword table includes a plurality of stored keywords, each of which corresponds to a relevance; and Calculating the confidence of the keyword according to the relevance includes: Selecting a target correlation degree greater than a preset correlation degree from the plurality of correlation degrees; The confidence of the keyword is calculated according to the target relevance.

5. The method according to claim 1, characterized in that The method further comprises: Detecting whether the confidence of the stored keyword is greater than a second preset value; and When the confidence of the stored keyword is greater than a second preset value, a credible mark is added, and the search record data corresponding to the stored keyword is deleted.

6. The method according to claim 1, characterized in that The method further comprises: Detecting whether the confidence of the stored keyword is less than a third preset value; and When the confidence of the stored keyword is less than a third preset value, a credible mark is added, and the search record data corresponding to the stored keyword is deleted.

7. A data collection device based on the Internet, characterized in that: include: Acquisition module, data processing module and storage module; The acquisition module is used to acquire the keywords input by the user and the webpage content data corresponding to the keywords; The data processing module is used to calculate the relevance between the webpage content data and the stored keywords in the preset keyword table; and calculate the confidence of the keyword based on the relevance; as well as The storage module is used for storing the keyword in the keyword table if the confidence of the keyword is greater than a first preset value.

8. The device according to claim 7, characterized in that The data processing module is also used to determine a first probability that the user uses the stored keyword to access the webpage corresponding to the keyword; determine a second probability that the user uses the keyword to access the webpage corresponding to the keyword; calculate the complementary confidence of the stored keyword and the keyword based on the first probability and the second probability; add the confidence and the complementary confidence to obtain the total confidence of other keywords and the first keyword; as well as The storage module is used for storing the keyword in the keyword table if the total confidence is greater than a first preset value.

9. The device according to claim 7, characterized in that The device also includes a detection module; The detection module is used to obtain a keyword to be tested from the keyword table; and detect whether the confidence of the keyword to be tested is greater than the first preset value; And if not, deleting the keyword to be tested from the keyword table.

10. A storage medium for storing computer executable instructions, characterized in that: When the computer executable instructions are executed, the steps of an Internet-based data collection method according to any one of claims 1 to 6 are implemented.

Images