Data processing method and device based on acquisition frequency

Abstract

The present application provides a data processing method and device based on the frequency of collection, which relates to the field of data processing technology and solves the technical problem that the collection success rate of website data is low under the condition of high cost. The method includes: sending multiple collection requests to the website to be collected; if the feedback result corresponding to the first collection request in the multiple collection requests is a successful collection and the feedback result corresponding to the second collection request and the subsequent collection retry requests is a failed collection, then the retry interval between each successful request is determined according to the number of successful request sending of the second collection request and the collection retry request, and the number of successful request sending is determined as the number of request retries; if the number of request retries is greater than the preset number threshold and the retry interval is greater than the preset time threshold, the website to be collected is determined as the target website, and the collection request to be sent to the target website and the sending interval between the last collection request are extended.

Description

Technical Field

[0001] The present application relates to the technical field of data processing, and in particular to a data processing method and device based on acquisition frequency. Background Art

[0002] For the collection process of massive information from social media, news websites and other platforms, there is usually a need for managers to collect content from a certain website. For example, managers download social media and news data from multiple websites, and then extract relevant fields from the content to realize the collection process of massive information from social media, news websites and other platforms. Sometimes it is necessary to save web page related files locally, such as pictures, attachments, etc. However, managers often encounter collection failures during the process of website data collection, resulting in a low collection success rate at a high cost. Summary of the invention

[0003] The purpose of the present invention is to provide a data processing method and device based on collection frequency, so as to solve the technical problem that website data collection has a low collection success rate at a high cost.

[0004] In a first aspect, an embodiment of the present application provides a data processing method based on acquisition frequency, the method comprising:

[0005] Acquire a website to be collected, and send multiple collection requests to the website to be collected; wherein there is an interval time between the multiple collection requests;

[0006] If the feedback result corresponding to the first acquisition request among the multiple acquisition requests is acquisition success and the feedback result corresponding to the second acquisition request and subsequent acquisition retry requests is acquisition failure, then determine the retry interval between each number of successful request sending times according to the number of successful request sending times of the second acquisition request and the acquisition retry request, and determine the number of successful request sending times as the number of request retries;

[0007] If the number of request retries is greater than a preset number threshold and the retry interval is greater than a preset time threshold, the website to be collected is determined as a target website, the interval between the collection request to be sent to the target website and the last collection request is extended, and a first request retry number within a first preset time and a second request retry number within a second preset time are determined from the multiple request retries; wherein the second preset time is less than the first preset time;

[0008] If the first request retry number exceeds a first preset number and the second request retry number does not exceed a second preset number, the collection frequency of the collection request to be sent to the target website is reduced according to a first preset reduction ratio value; wherein the first preset number is less than the second preset number;

[0009] If the first request retry number exceeds the first preset number and the second request retry number exceeds the second preset number, the collection frequency of the collection requests to be sent to the target website will be reduced according to a second preset reduction ratio value; wherein the second preset reduction ratio value is greater than the first preset reduction ratio value.

[0010] In a possible implementation, after the step of determining the website to be collected as the target website if the number of request retries is greater than a preset number threshold and the retry interval is greater than a preset duration threshold, the step further includes:

[0011] Establishing multiple virtual containers in the cloud server for the websites to be collected, the virtual containers are used to communicate through designated IP addresses, each of the virtual containers corresponds to a designated IP address, and the designated IP addresses corresponding to the multiple virtual containers are all different;

[0012] In response to the collection feedback result corresponding to the collection request sent to the website to be collected through the first designated IP address among the multiple designated IP addresses being that the number of collection failures is greater than the preset number of failures, a collection request is sent to the website to be collected through the second designated IP address among the multiple designated IP addresses, so as to use the virtual container to replace the IP address for sending the collection request.

[0013] In a possible implementation, the designated organization's venue space includes a master control terminal of the designated organization and multiple in-organization terminals affiliated with the designated organization; after the step of determining the website to be collected as the target website if the number of request retries is greater than a preset number threshold and the retry interval is greater than a preset time threshold, the step further includes:

[0014] Calling at least one target in-organization terminal among the multiple in-organization terminals through the master control terminal, and controlling the target in-organization terminal to simulate access to the website to be collected;

[0015] In response to the target organization terminal successfully accessing the website to be collected, controlling the target organization terminal to read the website data of the website to be collected and sending the website data to the master control terminal, so as to retrieve the website data from the target organization terminal through the master control terminal;

[0016] Based on the website data in the master control terminal, a collection task for the website to be collected is executed.

[0017] In a possible implementation, in response to the target organization terminal successfully accessing the website to be collected, the step of controlling the target organization terminal to read the website data of the website to be collected and sending the website data to the master control terminal, so as to retrieve the website data from the target organization terminal through the master control terminal includes:

[0018] In response to the target organization's internal terminal successfully accessing the to-be-collected website, controlling the target organization's internal terminal to obtain the to-be-collected website's to-be-collected website's to-be-collected interface display content; wherein the to-be-collected interface display content includes the website data of the to-be-collected website;

[0019] The content to be displayed on the interface is processed to obtain a website image of the website to be collected, and the website image is sent to the master control terminal so that the master control terminal performs image content recognition on the website image to obtain the website data.

[0020] In a possible implementation, the step of calling at least one target intra-organization terminal among the multiple intra-organization terminals through the master control terminal includes:

[0021] Detecting the current usage status of the terminals in the multiple institutions through the master control terminal;

[0022] If there is a target in-organization terminal in the multiple in-organization terminals that is currently in an unused and non-operation standby state, calling the target in-organization terminal through the master control terminal;

[0023] If there is no target in-organization terminal in the multiple in-organization terminals whose current use state is in an unused and non-operation standby state, it is determined that the call to the multiple in-organization terminals is currently abandoned.

[0024] In a possible implementation, the step of controlling the terminal in the target organization to simulate access to the website to be collected includes:

[0025] The terminal in the target organization is remotely controlled to access the website to be collected through wireless communication to simulate the process of the terminal in the target organization accessing the website to be collected.

[0026] In a possible implementation, after the step of controlling the terminal in the target organization to read the website data of the website to be collected and sending the website data to the master control terminal in response to the terminal in the target organization successfully accessing the website to be collected, so as to retrieve the website data from the terminal in the target organization through the master control terminal, the method further includes:

[0027] Based on the process of the master control terminal controlling the multiple terminals within the organization to read the website data, the number of visits, visit frequency, visit success rate, and login data of each terminal within the organization to the website to be collected are recorded to obtain the visit history data of the multiple terminals within the organization to the website to be collected;

[0028] Determining a first in-organization terminal to be called preferentially from among the plurality of in-organization terminals according to the access history data;

[0029] The first in-organization terminal is controlled by the master control terminal to read the website data of the website to be collected according to the access history data corresponding to the first in-organization terminal.

[0030] In a second aspect, a data processing device based on acquisition frequency is provided, comprising:

[0031] An acquisition module, used to acquire a website to be collected, and send multiple acquisition requests to the website to be collected; wherein there is an interval between the multiple acquisition requests;

[0032] A determination module, configured to determine, if the feedback result corresponding to the first acquisition request among the multiple acquisition requests is acquisition success and the feedback results corresponding to the second acquisition request and subsequent acquisition retry requests are acquisition failures, determine the retry interval between each of the request sending success times according to the number of request sending successes of the second acquisition request and the acquisition retry request, and determine the number of request sending successes as the number of request retries;

[0033] An extension module, for determining the website to be collected as a target website if the number of request retries is greater than a preset number threshold and the retry interval is greater than a preset interval threshold, extending the interval between the collection request to be sent to the target website and the last collection request, and determining a first request retry number within a first preset time length and a second request retry number within a second preset time length from a plurality of request retries; wherein the second preset time length is less than the first preset time length;

[0034] A first reduction module, configured to reduce the frequency of the collection request to be sent to the target website according to a first preset reduction ratio value if the first request retry number exceeds a first preset number and the second request retry number does not exceed a second preset number; wherein the first preset number is less than the second preset number;

[0035] A second reduction module is used to reduce the collection frequency of the collection requests to be sent to the target website according to a second preset reduction ratio value if the number of retries of the first request exceeds the first preset number and the number of retries of the second request exceeds the second preset number; wherein the second preset reduction ratio value is greater than the first preset reduction ratio value.

[0036] In a third aspect, an embodiment of the present application provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program that can be executed on the processor, and when the processor executes the computer program, the method described in the first aspect is implemented.

[0037] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to execute the method described in the first aspect above.

[0038] The embodiments of the present application bring the following beneficial effects:

[0039] A data processing method and device based on collection frequency provided in an embodiment of the present application can obtain a website to be collected and send multiple collection requests to the website to be collected; wherein there is an interval time between the multiple collection requests; if the feedback result corresponding to the first collection request among the multiple collection requests is that the collection is successful and the feedback result corresponding to the second collection request and the subsequent collection retry requests is that the collection fails, then the retry interval between each successful request sending number is determined according to the number of successful request sending of the second collection request and the collection retry request, and the number of successful request sending is determined as the number of request retries; if the number of request retries is greater than a preset number threshold and the retry interval time is greater than a preset time threshold, the website to be collected is determined as a target website, and the collection request to be sent to the target website is compared with the sending of the previous collection request. The interval time is extended, and the first request retry number within the first preset time length and the second request retry number within the second preset time length are determined from the multiple request retry times; wherein the second preset time length is less than the first preset time length; if the first request retry number exceeds the first preset number and the second request retry number does not exceed the second preset number, the collection frequency of the collection request to be sent to the target website is reduced according to the first preset reduction ratio value; wherein the first preset number is less than the second preset number; if the first request retry number exceeds the first preset number and the second request retry number exceeds the second preset number, the collection frequency of the collection request to be sent to the target website is reduced according to the second preset reduction ratio value; wherein the second preset reduction ratio value is greater than the first preset reduction ratio value. In this scheme, when the collection request fails after multiple retries, the determination of the target website is not only based on the number of retries of failed collection, but also based on the length of the retry interval between these retries. The target website is determined only when the retry interval is long and the number of retries is large, so that the determination result of the target website is more accurate, avoiding the website that fails to collect after multiple retries due to a high retry density from being mistakenly determined as the target website. Then, when the target website is confirmed, the time interval of the collection request is also extended to improve the success rate of the collection, avoiding the failure of the retry collection due to a high retry density. Moreover, the frequency of the collection retry is also adjusted to different degrees based on the retry density previously requested: if the retry density previously requested is not dense enough, that is, the number of retries is small in a long time, then the frequency of the collection retry is reduced to a small extent; if the retry density previously requested is too dense, that is, the number of retries is large in a short time, then the frequency of the collection retry is reduced to a large extent.In this way, the collection frequency is adjusted to different degrees according to different retry intensities, which not only improves the success rate of collection, but also saves unnecessary collection request sending times and reduces the website data collection cost, thereby solving the technical problem of low collection success rate when the cost of website data collection is high.

[0040] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the specific implementation methods of the present application or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0042] Figure 1 A flow chart of a data processing method based on acquisition frequency provided in an embodiment of the present application;

[0043] Figure 2 An example of a remote control architecture through a wireless network in a data processing method based on acquisition frequency provided in an embodiment of the present application;

[0044] Figure 3 A schematic diagram of the structure of a data processing device based on acquisition frequency provided in an embodiment of the present application;

[0045] Figure 4 A schematic structural diagram of an electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution of the present application will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present application.

[0047] The terms "including" and "having" and any variations thereof mentioned in the embodiments of the present application are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device including a series of steps or units is not limited to the listed steps or units, but may optionally include other steps or units that are not listed, or may optionally include other steps or units that are inherent to these processes, methods, products or devices.

[0048] In order to collect and deeply mine massive amounts of information from social media, news websites and other platforms in real time, there is usually a need for managers to collect content from a website. At present, website data collection has a low collection success rate at a high cost. Based on this, an embodiment of the present application provides a data processing method and device based on collection frequency, which can solve the technical problem that website data collection has a low collection success rate at a high cost.

[0049] The embodiments of the present invention are further described below in conjunction with the accompanying drawings.

[0050] Figure 1 A flow chart of a data processing method based on acquisition frequency provided in an embodiment of the present application. Figure 1 As shown, the method includes:

[0051] Step S110, obtaining the website to be collected, and sending multiple collection requests to the website to be collected.

[0052] There is an interval between multiple collection requests. As an optional implementation, a scheduled task process may be performed, for example, a collection request is sent to the website to be collected every N hours.

[0053] Step S120, if the feedback result corresponding to the first collection request among multiple collection requests is collection success and the feedback result corresponding to the second collection request and subsequent collection retry requests is collection failure, then the retry interval between each successful request sending number is determined according to the number of successful request sending numbers of the second collection request and the collection retry request, and the number of successful request sending numbers is determined as the request retry number.

[0054] It should be noted that the number of successful request sending of the second collection request and the collection retry request only refers to the number of times the retry request is sent out. The successful sending of the request (successful request sending) does not mean that the feedback result corresponding to the collection request is also a successful collection, that is, successful request sending is not equivalent to successful collection.

[0055] In a possible implementation, the feedback result of sending the collection request for the first time is that the collection is successful, and the feedback result of sending the collection request for the second time is that the collection fails, or the feedback results of sending the collection request for the second time and multiple times thereafter are all that the collection fails. In this case, there is a corresponding number of retries after the second collection request fails, and there is also a time interval between each retry, that is, the retry interval duration.

[0056] As an optional implementation, the following formula can be constructed to implement this step: First, define the collection request feedback result: = "Collection successful", for k >1: = "Collection failed". Then, define whether the collection request is sent successfully: =1,ifth k The collection request was sent successfully; or, =0, otherwise. Then determine the number of request retries: if = "Collection successful" and for all k >1 both = "Collection failed", then request the number of retries R For all k >1.

[0057] Then, determine the number of successful requests: N Equal to the number of all successfully sent collection requests: N= Then determine the retry interval: Assume that all successful collection requests or collection retry requests have the same retry interval T , it can be defined as: T = f ( N ),in, f(N) can be an arbitrary function that can be N The value of determines the retry interval. For example, the simplest form can be T =c× N , where c is a constant representing the additional waiting time added after each successful transmission.

[0058] Let's combine these steps into a more concise formula:

[0059] Initialize variables: R =0, N = 0. Confirm the collection request feedback result: If it is the first collection request k = 1, its feedback result is defined as "successful acquisition"; if it is the second or subsequent acquisition request or retry k >1, their feedback result is defined as "collection failure". This describes an idealized scenario in which the first collection request always succeeds, while subsequent requests or retries always fail. Such conditional expressions help define the boundary conditions of the problem for subsequent logical processing and calculations. Then, determine the number of request retries. R . Determine the number of successful request sends: N= . Then determine the retry interval T = c × N .

[0060] To sum up, the whole process can be summarized in the following steps: Initial setup R =0, N = 0. If the first acquisition request is successful ( = "Collection successful") and all subsequent collection requests failed ( = "Collection failed", k >1), then R is the number of attempts starting from the second acquisition. Then calculate the number of successful sends in all acquisition requests ( N = ). Finally, use the number of successful sends N N To determine the retry interval length ( T = c × N ).

[0061] It should be noted that Indicates the feedback result corresponding to the first collection request; Indicates k Second-rate( k >1) Feedback results corresponding to the collection request or collection retry request; Indicates k Whether the collection request or collection retry request is sent successfully; T Indicates the preset retry interval duration; R Indicates the number of request retries; N Indicates the number of successful request sends. The formula used here is a general assumption based on the description provided, and may need to be adjusted according to specific circumstances in actual applications.

[0062] Step S130, if the number of request retries is greater than the preset number threshold and the retry interval is greater than the preset time threshold, the website to be collected is determined as the target website, the collection request to be sent to the target website and the sending interval between the last collection request are extended, and the first request retry number within the first preset time and the second request retry number within the second preset time are determined from multiple request retry times; wherein the second preset time is less than the first preset time.

[0063] As for the method of determining the target website, for example, the feedback result of sending the collection request for the first time is that the collection is successful, and the feedback results of sending the collection request for the second time and the X times thereafter are all that the collection fails, and the website in this case is determined as the target website. In addition, for the case of multiple requests, for example, the interval between each two requests, the browser parameters of the request, etc. can all correspond to the detection process.

[0064] As a possible implementation method, the corresponding collection frequency can be determined according to different situations of the website. For example, if the feedback result of sending the collection request for the first time is that the collection is successful, and the feedback results of sending the collection request for the second time and multiple times thereafter are all collection failures, then the threshold is set based on the number of retries, frequency, and interval after the second request failure. If the threshold of the number of retries, frequency, and interval setting is greater than the threshold, the target website is determined, and the interval between requests is extended, that is, no collection request is sent within this time period.

[0065] Step S140: if the first request retry number exceeds the first preset number and the second request retry number does not exceed the second preset number, the collection frequency of the collection request to be sent to the target website is reduced according to the first preset reduction ratio value.

[0066] The first preset number is less than the second preset number. For example, if the first request retry number exceeds three times (the first preset number) and the second request retry number does not exceed seven times (the second preset number), the collection frequency of the collection request to be sent to the target website is reduced according to the first preset reduction ratio value.

[0067] As an example of adjusting the collection frequency, if the number of requests exceeds a first preset number (such as three times) within a first preset time period (such as three hours), the collection frequency will be reduced by fifty percent. This reduction level is the first level.

[0068] Step S150: if the first request retry number exceeds the first preset number and the second request retry number exceeds the second preset number, the collection frequency of the collection request to be sent to the target website is reduced according to the second preset reduction ratio value.

[0069] The second preset reduction ratio value is greater than the first preset reduction ratio value. For example, if the number of retries of the first request exceeds three times (the first preset number) and the number of retries of the second request also exceeds seven times (the second preset number), the collection frequency of the collection request to be sent to the target website is reduced according to the second preset reduction ratio value.

[0070] As another example of adjusting the collection frequency, if the number of requests exceeds a second preset number (e.g., seven times) within a second preset time period (e.g., one hour), the collection frequency will be reduced by sixty percent (and the second preset reduction ratio is higher than the first preset reduction ratio of fifty percent). This reduction level is a higher-level second level.

[0071] In an embodiment of the present application, when the collection request fails after multiple retries, the target website is determined not only based on the number of retries of failed collection, but also based on the length of the retry interval between these retries. The target website is determined only when the retry interval is long and the number of retries is large, so that the determination result of the target website is more accurate, and it is avoided that the website that fails to collect after multiple retries due to a high retry density is not mistakenly determined as the target website. Then, when the target website is confirmed, the time interval of the collection request is also extended to improve the success rate of the collection, and to avoid the failure of the collection retry due to a high retry density. Moreover, the frequency of the collection retry is also adjusted to different degrees based on the retry density previously requested: if the retry density previously requested is not dense enough, that is, the number of retries is small in a long time, then the frequency of the collection retry is reduced to a small extent; if the retry density previously requested is too dense, that is, the number of retries is large in a short time, then the frequency of the collection retry is reduced to a large extent. In this way, the collection frequency is adjusted to different degrees according to different retry intensities, which not only improves the success rate of collection, but also saves unnecessary collection request times and reduces the cost of website data collection.

[0072] The above steps are described in detail below.

[0073] In some embodiments, in the above step S130, if the number of request retries is greater than a preset number threshold and the retry interval is greater than a preset time threshold, after determining the website to be collected as the target website, the method may further include the following steps:

[0074] For the websites to be collected, multiple virtual containers are established in the cloud server. The virtual containers are used to communicate through designated IP addresses. Each virtual container corresponds to a designated IP address, and the designated IP addresses corresponding to multiple virtual containers are all different.

[0075] In response to the fact that a collection feedback result corresponding to sending a collection request to a website to be collected through a first designated IP address among multiple designated IP addresses is that the number of collection failures is greater than a preset number of failures, a collection request is sent to the website to be collected through a second designated IP address among the multiple designated IP addresses, so as to replace the IP address for sending the collection request by using a virtual container.

[0076] As an optional implementation, if the target website causes the collection and retrieval to fail, a virtual container can be established in the cloud server and the IP address can be replaced using the virtual container, because the target website to be collected is likely to block the IP or server number, making it impossible to read and collect the website using the original IP.

[0077] In an embodiment of the present application, multiple virtual containers are established in a cloud server for websites to be collected, such as a target website. Each virtual container is used to communicate through a specified IP address. Each virtual container corresponds to a specified IP address. The multiple specified IP addresses corresponding to the multiple virtual containers are all different. In this way, the virtual container can be used to replace the IP address for collection, thereby avoiding the situation where the collection fails due to the target website blocking the original IP, thereby improving the success rate of website collection.

[0078] In some embodiments, the designated organization's venue space includes a master control terminal of the designated organization and multiple in-organization terminals affiliated to the designated organization; in the above step S130, if the number of request retries is greater than a preset number threshold and the retry interval is greater than a preset time threshold, after the process of determining the website to be collected as the target website, the method may further include the following steps:

[0079] At least one target in-organization terminal among multiple in-organization terminals is called through a master control terminal, and the target in-organization terminal is controlled to simulate access to a website to be collected; in response to the target in-organization terminal successfully accessing the website to be collected, the target in-organization terminal is controlled to read the website data of the website to be collected and send the website data to the master control terminal, so as to retrieve the website data from the target in-organization terminal through the master control terminal; and a collection task for the website to be collected is executed based on the website data in the master control terminal.

[0080] Exemplarily, the company's master control computer (the master control terminal of a designated organization) can call the in-house terminals (such as office computers) used for office work within the company. The company's master control computer controls certain in-house terminals to access the website to be collected. These in-house terminals can read and record the website information of the website to be collected in real time, and send the website information to the company's master control computer. The company's master control computer can then read and retrieve the collected website information, and realize the collection of website information of the website to be collected by manipulating the in-house terminals to simulate the process of the in-house terminals accessing the website, thereby further improving the collection success rate through the collection process of multiple different in-house terminals.

[0081] In some embodiments, in response to the target organization terminal successfully accessing the to-be-collected website, the process of controlling the target organization terminal to read the website data of the to-be-collected website and sending the website data to the master control terminal, so as to retrieve the website data from the target organization terminal through the master control terminal, may include the following steps:

[0082] In response to the target organization's terminal successfully accessing the website to be collected, controlling the target organization's terminal to obtain the to-be-collected website's to-be-collected website's to-be-collected interface display content; wherein the to-be-collected interface display content includes website data of the to-be-collected website;

[0083] The content displayed on the interface is processed to obtain a website image of the website to be collected, and the website image is sent to the master control terminal so that the master control terminal can perform image content recognition on the website image to obtain website data.

[0084] For example, the company's master control computer controls multiple internal terminals to read and record website screenshots of the internal terminal screens in real time, and sends the website screenshots to the company's master control computer. The company's master control computer reads, retrieves and collects website information by identifying the screenshots sent by the internal terminals. The above-mentioned website screenshot method improves the accuracy of the collected website information.

[0085] In some embodiments, the process of calling at least one target intra-organization terminal among the multiple intra-organization terminals through the master control terminal may include the following steps:

[0086] The current usage status of multiple in-organization terminals is detected through the master control terminal; if there is a target in-organization terminal in the multiple in-organization terminals that is currently in an unused and non-operation standby state, the target in-organization terminal is called through the master control terminal; if there is no target in-organization terminal in the multiple in-organization terminals that is currently in an unused and non-operation standby state, it is determined that the current call to the multiple in-organization terminals is abandoned.

[0087] As an example, before calling a terminal within the company, the company's main control computer first checks which terminal within the company is currently in standby and unused state. If a terminal within the company is currently in use, the call to the terminal within the company is abandoned to avoid affecting the normal office use of the terminal within the company.

[0088] In some embodiments, the process of controlling the terminal in the target organization to simulate access to the website to be collected may include the following steps: remotely controlling the terminal in the target organization to access the website to be collected via wireless communication to simulate the process of the terminal in the target organization accessing the website to be collected.

[0089] As a possible implementation, Figure 2 As shown, the company's main control computer group 101 can remotely control a terminal 102 within an organization to access the website to be collected through a wireless network, so as to simulate the process of the terminal within the organization accessing the website to be collected by remotely controlling the terminal within the organization, realize the collection of website information of the website to be collected, and improve the collection efficiency through wireless remote control.

[0090] In some embodiments, after the above step of controlling the terminal in the target organization to read the website data of the website to be collected and sending the website data to the master control terminal in response to the terminal in the target organization successfully accessing the website to be collected, so as to retrieve the website data from the terminal in the target organization through the master control terminal, the method may further include the following steps:

[0091] Based on the master control terminal, multiple terminals in the organization read the website data, record the number of visits, visit frequency, visit success rate, and login data of each terminal in the organization to the website to be collected, and obtain the visit history data of multiple terminals in the organization to the website to be collected;

[0092] According to the access history data, a first intra-institution terminal to be called preferentially is determined from multiple intra-institution terminals; and according to the access history data corresponding to the first intra-institution terminal, the first intra-institution terminal is controlled by the master control terminal to read the website data of the website to be collected.

[0093] As an example, the company's master control computer can summarize the number of logins to the collection website, the login frequency, the login success rate of each website, and other data of the terminal within each organization, so as to give priority to calling the terminal within a certain organization with a higher login success rate next time, thereby improving the control efficiency of the company's master control computer over the terminals within the organization.

[0094] Figure 3 A structural schematic diagram of a data processing device based on acquisition frequency is provided. Figure 3 As shown, the data processing device 300 based on the acquisition frequency includes:

[0095] The acquisition module 301 is used to acquire a website to be collected and send multiple collection requests to the website to be collected; wherein there is an interval between the multiple collection requests;

[0096] A first determining module 302 is configured to determine a retry interval between each number of successful request transmission times according to the number of successful request transmission times of the second acquisition request and the acquisition retry request, if the feedback result corresponding to the first acquisition request among the multiple acquisition requests is acquisition success and the feedback results corresponding to the second acquisition request and subsequent acquisition retry requests are acquisition failure, and determine the number of successful request transmission times as the number of request retries;

[0097] The extension module 303 is used to determine the website to be collected as the target website if the number of request retries is greater than a preset number threshold and the retry interval is greater than a preset time threshold, extend the interval between the collection request to be sent to the target website and the last collection request, and determine a first request retry number within a first preset time and a second request retry number within a second preset time from the multiple request retries; wherein the second preset time is less than the first preset time;

[0098] A first reduction module 304 is configured to reduce the frequency of the collection request to be sent to the target website according to a first preset reduction ratio value if the first request retry number exceeds a first preset number and the second request retry number does not exceed a second preset number; wherein the first preset number is less than the second preset number;

[0099] The second reduction module 305 is used to reduce the collection frequency of the collection requests to be sent to the target website according to a second preset reduction ratio value if the number of retries of the first request exceeds the first preset number and the number of retries of the second request exceeds the second preset number; wherein the second preset reduction ratio value is greater than the first preset reduction ratio value.

[0100] In some embodiments, the apparatus further comprises:

[0101] An establishment module, used for, after determining the website to be collected as a target website if the number of request retries is greater than a preset number threshold and the retry interval is greater than a preset duration threshold, establishing multiple virtual containers in a cloud server for the website to be collected, the virtual containers being used to communicate through a designated IP address, each of the virtual containers corresponding to a designated IP address, and the designated IP addresses corresponding to the multiple virtual containers being different;

[0102] A sending module is used to send a collection request to the website to be collected through a second designated IP address among the multiple designated IP addresses in response to a collection feedback result corresponding to the number of collection failures being greater than a preset number of failures when a collection request is sent to the website to be collected through a first designated IP address among the multiple designated IP addresses, so as to use the virtual container to replace the IP address for sending the collection request.

[0103] In some embodiments, the designated institution's space includes a master control terminal of the designated institution and a plurality of in-institution terminals affiliated with the designated institution; the device further includes:

[0104] A calling module, configured to call at least one target in-organization terminal among the multiple in-organization terminals through the master control terminal, and control the target in-organization terminal to simulate access to the in-organization website after the step of determining the in-organization website to be collected as the target website if the number of request retries is greater than a preset number threshold and the retry interval duration is greater than a preset duration threshold;

[0105] A first control module, configured to control the terminal in the target organization to read the website data of the website to be collected and send the website data to the master control terminal in response to the terminal in the target organization successfully accessing the website to be collected, so as to retrieve the website data from the terminal in the target organization through the master control terminal;

[0106] An execution module is used to execute a collection task for the website to be collected based on the website data in the master control terminal.

[0107] In some embodiments, the first control module is specifically configured to:

[0108] In response to the target organization's internal terminal successfully accessing the to-be-collected website, controlling the target organization's internal terminal to obtain the to-be-collected website's to-be-collected website's to-be-collected interface display content; wherein the to-be-collected interface display content includes the website data of the to-be-collected website;

[0109] The content to be displayed on the interface is processed to obtain a website image of the website to be collected, and the website image is sent to the master control terminal so that the master control terminal performs image content recognition on the website image to obtain the website data.

[0110] In some embodiments, the calling module is specifically used to:

[0111] Detecting the current usage status of the terminals in the multiple institutions through the master control terminal;

[0112] If there is a target in-organization terminal in the multiple in-organization terminals that is currently in an unused and non-operation standby state, calling the target in-organization terminal through the master control terminal;

[0113] If there is no target in-organization terminal in the multiple in-organization terminals whose current use state is in an unused and non-operation standby state, it is determined that the call to the multiple in-organization terminals is currently abandoned.

[0114] In some embodiments, the calling module is further configured to:

[0115] The terminal in the target organization is remotely controlled to access the website to be collected through wireless communication to simulate the process of the terminal in the target organization accessing the website to be collected.

[0116] In some embodiments, the apparatus further comprises:

[0117] a recording module, for, in response to the target in-organization terminal successfully accessing the to-be-collected website, controlling the target in-organization terminal to read the website data of the to-be-collected website and sending the website data to the master control terminal, so as to retrieve the website data from the target in-organization terminal through the master control terminal, and based on the process of the master control terminal controlling multiple in-organization terminals to read the website data, recording the number of visits, access frequency, access success rate, and login data of each in-organization terminal to the to-be-collected website, and obtaining the access history data of multiple in-organization terminals to the to-be-collected website;

[0118] A second determination module, configured to determine a first intra-institution terminal to be preferentially called from among the plurality of intra-institution terminals according to the access history data;

[0119] The second control module is used to control the terminal in the first organization to read the website data of the website to be collected through the master control terminal according to the access history data corresponding to the terminal in the first organization.

[0120] The data processing device based on acquisition frequency provided in the embodiment of the present application has the same technical features as the data processing method based on acquisition frequency provided in the above embodiment, so it can also solve the same technical problems and achieve the same technical effects.

[0121] An electronic device provided in an embodiment of the present application is Figure 4 As shown, the electronic device 400 includes a processor 402 and a memory 401, wherein the memory stores a computer program that can be run on the processor, and the processor implements the steps of the method provided in the above embodiment when executing the computer program.

[0122] See also Figure 4 The electronic device further includes: a bus 403 and a communication interface 404, a processor 402, a communication interface 404 and a memory 401 are connected via the bus 403; the processor 402 is used to execute an executable module stored in the memory 401, such as a computer program.

[0123] The memory 401 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk storage. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 404 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used.

[0124] The bus 403 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0125] Among them, the memory 401 is used to store programs, and the processor 402 executes the program after receiving the execution instruction. The method executed by the device defined by the process disclosed in any embodiment of the present application can be applied to the processor 402 or implemented by the processor 402.

[0126] The processor 402 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 402. The above processor 402 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware decoding processor to execute, or the hardware and software modules in the decoding processor are combined to execute. The software module may be located in a storage medium mature in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 401, and the processor 402 reads the information in the memory 401 and completes the steps of the above method in combination with its hardware.

[0127] Corresponding to the above-mentioned data processing method based on collection frequency, an embodiment of the present application also provides a computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to execute the steps of the above-mentioned data processing method based on collection frequency.

[0128] The data processing device based on the acquisition frequency provided in the embodiment of the present application can be specific hardware on the device or software or firmware installed on the device. The device provided in the embodiment of the present application, its implementation principle and the technical effect produced are the same as those in the aforementioned method embodiment. For the sake of brief description, the parts not mentioned in the device embodiment can refer to the corresponding contents in the aforementioned method embodiment. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, devices and units described above can all refer to the corresponding processes in the aforementioned method embodiment, and will not be repeated here.

[0129] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0130] For another example, the flowchart and block diagram in the accompanying drawings show the possible architecture, function and operation of the device, method and computer program product according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of the code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or the flowchart, and the combination of the boxes in the block diagram and / or the flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0131] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0132] In addition, each functional unit in the embodiments provided in the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0133] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the data processing method based on the acquisition frequency described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store program codes.

[0134] It should be noted that similar numbers and letters represent similar items in the following figures. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In addition, the terms "first", "second", "third", etc. are only used to distinguish the description and are not to be understood as indicating or implying relative importance.

[0135] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present application, which are used to illustrate the technical solution of the present application, rather than to limit it. The protection scope of the present application is not limited thereto. Although the present application is described in detail with reference to the aforementioned embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solution recorded in the aforementioned embodiments within the technical scope disclosed in the present application, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solution deviate from the scope of the technical solution of the embodiment of the present application. They should all be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A data processing method based on acquisition frequency, characterized in that: The method comprises: Acquire a website to be collected, and send multiple collection requests to the website to be collected; wherein there is an interval time between the multiple collection requests; If the feedback result corresponding to the first acquisition request among the multiple acquisition requests is acquisition success and the feedback result corresponding to the second acquisition request and subsequent acquisition retry requests is acquisition failure, then determine the retry interval between each number of successful request sending times according to the number of successful request sending times of the second acquisition request and the acquisition retry request, and determine the number of successful request sending times as the number of request retries; If the number of request retries is greater than a preset number threshold and the retry interval is greater than a preset time threshold, the website to be collected is determined as a target website, the interval between the collection request to be sent to the target website and the last collection request is extended, and a first request retry number within a first preset time and a second request retry number within a second preset time are determined from the multiple request retries; wherein the second preset time is less than the first preset time; If the first request retry number exceeds a first preset number and the second request retry number does not exceed a second preset number, the collection frequency of the collection request to be sent to the target website is reduced according to a first preset reduction ratio value; wherein the first preset number is less than the second preset number; If the first request retry number exceeds the first preset number and the second request retry number exceeds the second preset number, the collection frequency of the collection requests to be sent to the target website will be reduced according to a second preset reduction ratio value; wherein the second preset reduction ratio value is greater than the first preset reduction ratio value.

2. The method according to claim 1, characterized in that After the step of determining the website to be collected as the target website if the number of request retries is greater than a preset number threshold and the retry interval is greater than a preset duration threshold, the method further includes: Establishing multiple virtual containers in the cloud server for the websites to be collected, the virtual containers are used to communicate through designated IP addresses, each of the virtual containers corresponds to a designated IP address, and the designated IP addresses corresponding to the multiple virtual containers are all different; In response to the collection feedback result corresponding to the collection request sent to the website to be collected through the first designated IP address among the multiple designated IP addresses being that the number of collection failures is greater than the preset number of failures, a collection request is sent to the website to be collected through the second designated IP address among the multiple designated IP addresses, so as to use the virtual container to replace the IP address for sending the collection request.

3. The method according to claim 1, characterized in that The designated institution's location space includes a master control terminal of the designated institution and multiple in-institution terminals affiliated with the designated institution; after the step of determining the website to be collected as a target website if the number of request retries is greater than a preset number threshold and the retry interval is greater than a preset time threshold, the method further includes: Calling at least one target in-organization terminal among the multiple in-organization terminals through the master control terminal, and controlling the target in-organization terminal to simulate access to the website to be collected; In response to the target organization terminal successfully accessing the website to be collected, controlling the target organization terminal to read the website data of the website to be collected and sending the website data to the master control terminal, so as to retrieve the website data from the target organization terminal through the master control terminal; Based on the website data in the master control terminal, a collection task for the website to be collected is executed.

4. The method according to claim 3, characterized in that In response to the target organization terminal successfully accessing the to-be-collected website, the step of controlling the target organization terminal to read the website data of the to-be-collected website and sending the website data to the master control terminal, so as to retrieve the website data from the target organization terminal through the master control terminal, comprises: In response to the target organization's internal terminal successfully accessing the to-be-collected website, controlling the target organization's internal terminal to obtain the to-be-collected website's to-be-collected website's to-be-collected interface display content; wherein the to-be-collected interface display content includes the website data of the to-be-collected website; The content to be displayed on the interface is processed to obtain a website image of the website to be collected, and the website image is sent to the master control terminal so that the master control terminal performs image content recognition on the website image to obtain the website data.

5. The method according to claim 3, characterized in that: The step of calling at least one target intra-institution terminal among the multiple intra-institution terminals through the master control terminal comprises: Detecting the current usage status of the terminals in the multiple institutions through the master control terminal; If there is a target in-organization terminal in the multiple in-organization terminals that is currently in an unused and non-operation standby state, calling the target in-organization terminal through the master control terminal; If there is no target in-organization terminal in the multiple in-organization terminals whose current use state is in an unused and non-operation standby state, it is determined that the call to the multiple in-organization terminals is currently abandoned.

6. The method according to claim 3, characterized in that The step of controlling the terminal in the target organization to simulate access to the website to be collected includes: The terminal in the target organization is remotely controlled to access the website to be collected through wireless communication to simulate the process of the terminal in the target organization accessing the website to be collected.

7. The method according to claim 3, characterized in that After the step of controlling the terminal in the target organization to read the website data of the website to be collected and sending the website data to the master control terminal in response to the terminal in the target organization successfully accessing the website to be collected, so as to retrieve the website data from the terminal in the target organization through the master control terminal, the method further includes: Based on the process of the master control terminal controlling the multiple terminals within the organization to read the website data, the number of visits, visit frequency, visit success rate, and login data of each terminal within the organization to the website to be collected are recorded to obtain the visit history data of the multiple terminals within the organization to the website to be collected; Determining a first in-organization terminal to be called preferentially from among the plurality of in-organization terminals according to the access history data; The first in-organization terminal is controlled by the master control terminal to read the website data of the website to be collected according to the access history data corresponding to the first in-organization terminal.

8. A data processing device based on acquisition frequency, characterized in that: include: An acquisition module, used to acquire a website to be collected, and send multiple acquisition requests to the website to be collected; wherein there is an interval between the multiple acquisition requests; A determination module, configured to determine, if the feedback result corresponding to the first acquisition request among the multiple acquisition requests is acquisition success and the feedback results corresponding to the second acquisition request and subsequent acquisition retry requests are acquisition failures, determine the retry interval between each of the request sending success times according to the number of request sending successes of the second acquisition request and the acquisition retry request, and determine the number of request sending successes as the number of request retries; An extension module, for determining the website to be collected as a target website if the number of request retries is greater than a preset number threshold and the retry interval is greater than a preset interval threshold, extending the interval between the collection request to be sent to the target website and the last collection request, and determining a first request retry number within a first preset time length and a second request retry number within a second preset time length from a plurality of request retries; wherein the second preset time length is less than the first preset time length; A first reduction module, configured to reduce the frequency of the collection request to be sent to the target website according to a first preset reduction ratio value if the first request retry number exceeds a first preset number and the second request retry number does not exceed a second preset number; wherein the first preset number is less than the second preset number; A second reduction module is used to reduce the collection frequency of the collection requests to be sent to the target website according to a second preset reduction ratio value if the number of retries of the first request exceeds the first preset number and the number of retries of the second request exceeds the second preset number; wherein the second preset reduction ratio value is greater than the first preset reduction ratio value.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to execute the method according to any one of claims 1 to 7.

Images